Data encoding and decoding

By comparing and adjusting the cost of the motion vector predictor candidate list, the redundant candidate problem in VVC is solved, and the encoding efficiency and performance are improved.

CN119013968APending Publication Date: 2024-11-22CANON KK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380033410.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-06-09
Filing Date
2023-04-11
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

There is redundancy problem in the sorting and rearrangement of motion vector predictor candidate lists in existing video encoding standards VVC, which affects coding efficiency.

Method used

By comparing the cost of candidate predictors, use thresholds to determine whether to modify the list, remove or reorder redundant candidates to improve the diversity and coding efficiency of the list.

Benefits of technology

By optimizing the sorting of the motion vector predictor candidate list, redundant candidates are reduced, encoding efficiency is improved, bit rate is reduced, and encoding performance is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119013968A_ABST
    Figure CN119013968A_ABST
Patent Text Reader

Abstract

Improvements in processing of predictors are disclosed. A predictor list is obtained, the predictor list having at least two predictors. Using a first cost associated with a first predictor in the list and a second cost associated with a second predictor in the list, whether to modify the predictor list is determined based on a criterion. The predictor list is modified based on the determination. The criterion is threshold-based.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to encoding and decoding of predictors. Background Art

[0002] The Joint Video Experts Group (JVET), a collaborative team consisting of MPEG and VCEG of ITU-T Study Group 16, has released a new video coding standard called Versatile Video Coding (VVC). The goal of VVC is to provide a significant improvement in compression performance over the existing HEVC standard (i.e., typically twice as good as before). Key target applications and services include, but are not limited to, 360-degree and high dynamic range (HDR) video. Particular results were shown on ultra-high definition (UHD) video test material. Therefore, for the final standard, we can expect compression efficiency improvements far exceeding the targeted 50%.

[0003] Since the end of the VVC v1 standard, JVET has started the exploration phase by building the Exploration Tool (ECM). JVET collects additional tools and improves existing tools based on the VVC standard to achieve better coding efficiency.

[0004] Compared to HEVC, VVC has, among other modifications, a modified set of "merge modes" for motion vector prediction, which achieves better coding efficiency at the cost of greater complexity. Motion vector prediction is achieved by deriving a list of "motion vector predictor candidates", where the index of the selected candidate is signaled in the bitstream. The merge candidate list is generated for each coding unit (CU). But the CU can be split into smaller blocks for decoder-side Motion Vector Refinement (DMVR) or other methods.

[0005] The composition and order of the list can have a significant impact on coding efficiency, since an accurate motion vector predictor reduces the size of the residual or distortion of the block predictor, and having such a candidate at the top of the list reduces the number of bits required to signal the selected candidate. The present invention is directed to improving at least one of these aspects.

[0006] Modifications incorporated into VVC v1 and ECM mean that there can be up to 10 motion vector predictor candidates; this achieves diversity in the candidates, but the bit rate may increase if a candidate lower in the list is selected. The present invention broadly relates to improvements to the derivation and ordering of one or more "paired" motion vector predictor candidates in a list of motion vector predictor candidates. A "paired" motion vector predictor candidate is a candidate that is combined or averaged from two or more other candidates in a list of candidates. Summary of the invention

[0007] The present invention relates to improvements in the processing of lists of predictors (or predictor candidates), which achieve an improved ordering of predictors that provides coding efficiency at the expense of some increased complexity. For example, the method attempts to prevent redundant candidates with similar costs from being adjacent in a candidate list by comparing candidate costs to each other. In an embodiment, a balance is struck between the amount of reordering of candidates and the potential benefits of cost diversity in the resulting list to which the processing according to the invention is applied. The present invention may be particularly (but not exclusively) applied to predictors (or predictor candidates) for motion prediction of image portions in image or video coding. However, the present invention has broad applicability to any encoding when one data unit or data item can be predicted from another data unit or data item and a list of candidate predictors is generated for this purpose. For example, it is envisioned that, in addition to still and video image data, the present invention may also have applicability to at least point cloud data, mesh data and audio data.

[0008] According to an aspect of the present invention, a method for processing a predictor is provided, the method comprising: obtaining a list of predictors, the list of predictors having at least two predictors; determining whether to modify the list of predictors based on a criterion using a first cost associated with a first predictor in the list and a second cost associated with a second predictor in the list; modifying (or not modifying) the list of predictors based on the determination, wherein the criterion is based on a threshold. The first predictor and the second predictor may be adjacent predictors in the list (in other words, one follows the other). The predictor may be a candidate. For example, a motion information predictor candidate for an inter-mode such as a merge mode or a block predictor candidate in an intra-mode.

[0009] The determining may include adding the threshold to the first cost or the second cost, and the criterion is related to a comparison of a result with one of the first cost and the second cost without the addition of the threshold.

[0010] The criterion may be whether a result of adding the threshold to the first cost is less than the second cost.

[0011] The criterion may be whether a difference between the first cost and the second cost crosses the threshold.

[0012] Optionally, the difference is calculated as an absolute value and the criterion is whether the difference is smaller than the threshold value.

[0013] Optionally, modifying the list comprises removing the first candidate or the second candidate from the list of predictors.

[0014] Optionally, modifying the list comprises changing a position of one or more predictors in the list.

[0015] Optionally, changing the position of one or more candidates comprises moving the first predictor or the second predictor to a different position in the list.

[0016] Changing the ordering may include moving the first predictor or the second predictor to a position considered to be the end of the list.

[0017] Optionally, the first predictor and the second predictor are after the zeroth predictor at a position considered to be next to the beginning of the list. In some examples, the zeroth predictor may be excluded from the step of modifying the list of predictors (e.g., the zeroth predictor may be fixed at the position at the beginning of the list regardless of the reordering of other predictors in the list). In some examples, the method may further include a step for comparing a zeroth cost associated with the zeroth predictor to a zeroth threshold, wherein, in the event that the zeroth cost exceeds the zeroth threshold, the zeroth predictor is included in the step for modifying the list of predictors. The step for comparing the zeroth cost associated with the zeroth predictor to the zeroth threshold may be performed in the mode of the first set and not in the mode of the second set. In other words, including the zeroth predictor in the step for modifying the list of predictors may be mode dependent. In examples where the step for comparing the zeroth cost associated with the zeroth predictor to the zeroth threshold is performed, the zeroth threshold may be lower than the threshold used for the first predictor and the second predictor. In other examples, the thresholds may be the same.

[0018] In other examples, the first predictor may be at a position that is considered to be the beginning of the list. In such examples, the first cost may be set to zero. That is, regardless of the true computational cost associated with the first predictor, the method may include a step for overwriting that value to a zero cost. Alternatively, the first cost may be set to zero and not computed at all.

[0019] Optionally, the first cost and the second cost comprise template matching costs based on neighboring samples of the image portion to be encoded or decoded and neighboring samples of the image portion associated with the predictor.

[0020] Optionally, the first cost and the second cost comprise a difference between sample values ​​of two block predictors associated with the predictor.

[0021] Optionally, the first cost and the second cost comprise differences between samples associated with the predictor and at least one other predictor in the list.

[0022] Optionally, the first cost and the second cost are based on a subset of samples or neighboring samples of the predictor.

[0023] Optionally, the first cost and the second cost are based on samples corresponding to another image according to a different resolution.

[0024] Optionally, the values ​​of the samples used to calculate the first cost and the second cost are preprocessed.

[0025] Optionally, the first cost and the second cost are based on a coding rate.

[0026] Optionally, the first cost and the second cost are based on a distortion cost. The distortion cost may be any one of the following: Sum of Absolute Difference (SAD), Sum of Absolute Transform Difference (SATD), Sum of Squared Error (SSE), and Structural Similarity Index metric (SSIM).

[0027] The cost may be proportional to the number of samples in the unit being processed (image portion / prediction unit / data unit to be predicted). The cost may be proportional to the number of samples in a template (e.g., a template of neighboring samples). The cost may be based on samples used to determine the distortion cost and block size.

[0028] Optionally, the method further comprises deriving the threshold value.Alternatively or additionally, the method further comprises receiving information indicating the threshold value.

[0029] The threshold may be based on a quantization parameter (QP) value. According to the method of any one of claims 22 to 24, the threshold is based on a temporal distance or a picture order count (POC) distance between the current frame and one or more other frames. Alternatively or additionally, the threshold may be based on a temporal ID of a frame. Alternatively or additionally, the threshold may be based on a temporal direction of a reference frame. Alternatively or additionally, the threshold is based on whether bidirectional prediction is used for a block. The threshold may depend on one or more QP values ​​of available reference frames. The threshold may depend on a type of distortion used as at least one of the first cost and the second cost. The threshold may be based on a type of sample used to calculate the distortion as a measure of at least one of the first cost and the second cost. The threshold may be based on whether intra prediction or inter prediction is used to encode or decode the image portion to be predicted using the list of predictors. The threshold may be based on which one or more of a random access (RA) configuration, a low delay configuration, and an intra-only configuration is used for encoding or decoding.

[0030] The threshold may be based on a formula used for encoding and obtained by a decoder for calculating the threshold.

[0031] Additionally or alternatively, the threshold may be based on a shifted quantization parameter (QP) value. The shifted QP value may be used to determine a Lagrangian parameter λ as the threshold. The shifted quantization parameter may be used to obtain the Lagrangian parameter (e.g., Lambda) from a table, wherein an entry in the table associates a Lagrangian parameter value with a quantization parameter value (e.g., Lambda).

[0032] Additionally or alternatively, the threshold may be determined as a minimum cost difference that is a difference between consecutive predictors in the list that are not the first predictor and the second predictor.

[0033] Additionally or alternatively, the threshold may be determined using cost values ​​of all candidates or first N candidates in the list, where N is the maximum number of candidates allowed in the list.

[0034] Using the cost value may include calculating an average of differences in costs of consecutive candidates from among the candidates used to determine the threshold.

[0035] The threshold may be based on a Lagrangian parameter λ, and the criterion is whether the absolute difference between the first distortion cost and the second distortion cost is less than the Lagrangian parameter. For example, the threshold may be the Lagrangian parameter multiplied by the rate difference between the first predictor and the second predictor.

[0036] Optionally, at least one of the first distortion cost and the second distortion cost is multiplied by the threshold.

[0037] Optionally, the criterion is applied after a first sorting process. The first sorting process may be a cost-based sorting. The cost may be a template matching distortion cost.

[0038] Optionally, a further threshold is added to the result of the multiplication resulting from the difference between the first distortion cost and the second distortion cost.

[0039] Optionally, the determining is performed according to the criterion during a process of sorting the candidates according to their distortion costs.

[0040] Optionally, determining whether to modify the list is based on a minimum distortion difference and a threshold. The maximum number of reorderings may be limited to a predefined number. For example, the maximum number of reorderings may be a maximum allowed number of candidates in the list.

[0041] Optionally, the determination is based on a minimum distortion difference and is limited to a maximum number of reorderings.

[0042] Optionally, modifying the list comprises only removing candidates. Optionally, the number of candidates after removal must be greater than or equal to the maximum allowed number of candidates in the list. In an embodiment, when modifying the list by moving candidates, the maximum number of candidates allowed in the list is decremented.

[0043] Optionally, if the criterion is met, the second predictor is moved to a first position at which the difference in cost for subsequent predictors in the list is above the threshold. When the maximum number of possibilities for reordering is greater than the maximum number of candidates allowed in the final list, the modifying may include limiting the position to which the first predictor or the second predictor is moved to the end of the current list. Optionally, the processing of the predictor is applied to at least one of a normal merge mode, a template matching merge mode, a block matching (BM) merge mode, and an intra block copy (IBC) mode.

[0044] In an embodiment, the second predictor is immediately after the first predictor (in the list), and in the event that the criterion is determined not to be met, the second predictor is inserted after the subsequent predictor in the list. Optionally, in the event that the criterion is determined not to be met, the criterion is applied to the second cost and the individual costs of the previous candidates in the list, and wherein, in the event that the criterion is determined not to be met for each of the second cost and the previous candidate, the insertion of the second predictor after the subsequent predictor is not performed. The maximum possible number of reorderings can be set to the allowed number of candidates in the final list of processed candidates. Additionally, the processing of the predictor can be applied to at least one of a merge with MVD, an affine merge with MVD, and an MVD sign prediction of a motion vector residual.

[0045] The list of predictors may be a sub-list of predictors (combinable with another sub-list to form a final list, an intermediate list, or another sub-list), an intermediate list of predictors, or a final list.

[0046] In an embodiment, when template matching is used and one of at least two templates is available, the cost for all predictors (candidates) in the list can be calculated based on using the weighted cost of the available template as the cost for the unavailable template. For example, the weighting can be such that the cost is proportional to the height and width of the missing template. The template can be the left or top template.

[0047] In an embodiment, when template matching is used and one of at least two templates is available, the Lagrangian or Lambda is modified in proportion to the missing template.

[0048] In an embodiment, the motion vector (MV) threshold used for derivation is the same for all merge predictor (candidate) types. Such an embodiment can be advantageously applied to merge modes that apply different motion vector thresholds based on the type of predictor (candidate), such as conventional, template matching (TM), or block matching (BM) merge modes.

[0049] In one embodiment, when the method of the present invention as set forth in any of the above-mentioned aspects and embodiments is available (or enabled), ARMC processing is applied for all predictors (candidates) derived and ranked using ARMC.

[0050] In an embodiment, when the proposed method is available (or enabled), pairwise predictors (candidates) are added during or after the ranking process.

[0051] In another aspect of the invention, there is provided a method for processing a list of predictors, the method comprising: determining a minimum cost difference between two consecutive predictors (or predictor candidates) in the list. If the minimum difference is less than a threshold (Lambda), repositioning one of the candidates at a last position in the list. Optionally, the last position precedes a candidate in the list that is considered redundant. The cost may be a distortion. The cost difference may be an absolute cost difference.

[0052] According to another aspect of the invention, there is provided a method for processing a list of predictors, the method comprising: determining a minimum cost difference between two consecutive predictors (or predictor candidates), if the minimum difference is less than a threshold (Lambda), comparing the distortion of one of the (redundant) predictors with the distortion of a (further) subsequent predictor in the list, identifying the position of the first subsequent predictor having a distortion difference above Lambda, positioning the predictor at the identified position. The cost may be the distortion. The cost difference may be an absolute cost difference.

[0053] According to another aspect of the invention, there is provided a method for processing a list of predictors, the method comprising: for a candidate, determining a first, subsequent non-redundant predictor (or predictor candidate) in the list, after which the non-redundant predictor is inserted. The redundant candidate may be, for example, a candidate that has been determined to have a cost difference less than a threshold. The cost may be a distortion. The cost difference may be an absolute cost difference.

[0054] According to another aspect of the invention, there is provided a method for processing a list of predictors, the method comprising: for a predictor (or predictor candidate), determining a first, subsequent non-redundant predictor in the list, inserting the non-redundant candidate after the candidate unless the non-redundant candidate is determined to be redundant compared to all candidates before the candidate. The redundant candidate may be, for example, a candidate that has been determined to have a cost difference that is less than a threshold. The cost may be a distortion. The cost difference may be an absolute cost difference.

[0055] According to another aspect of the present invention, a method for processing a predictor is provided, the method comprising: obtaining a list of predictors, determining whether an absolute distortion cost difference between a first distortion cost of a first predictor and a second distortion cost of a second predictor is less than a Lambda value (e.g., a Lagrangian), wherein the second predictor immediately follows the first predictor in the list; and if the absolute distortion cost difference is less than the Lambda value, moving the position of the first predictor in the list. The second predictor may be moved to a position immediately following a last candidate in the list that is currently considered non-redundant and before any redundant candidates. For example, a redundant candidate may be a candidate that has been previously determined to have an absolute distortion difference with a previous candidate that is less than Lambda. The Lambda value may be derived based on a QP value or a shifted QP value. For example, the Lambda value may be obtained from a table having an entry indexed or associated with a shifted QP value.

[0056] According to another aspect of the present invention, a method for processing a list of predictors is provided, the method comprising: obtaining a list of predictors, the list of predictors having at least two predictors, determining whether to modify the list of predictors based on a comparison of a first cost associated with a first predictor in the list with a threshold, and modifying the list of predictors based on the determination. In some examples, the step of determining whether to modify the list of predictors is based on a comparison of a respective cost associated with each predictor in the list with a threshold. That is, each cost associated with each respective predictor in the list may be compared with a threshold. In some examples, the step of modifying the list comprises removing the candidate or candidates having a cost below the threshold from the list of predictors. Additionally or alternatively, the step of modifying the list may comprise moving the candidate or candidates having a cost below the threshold to a position in the list following a candidate having a cost greater than the threshold. In a further example, the step of modifying the list may comprise moving the candidate or candidates having a cost below the threshold to a position in the list that is considered to be the end of the list. In all examples, the step of modifying the list of predictors may be performed in the mode of the first set and not in the mode of the second set. That is, the reordering of predictors in the list can be pattern-dependent.

[0057] These further aspects may be usefully combined with any of the previous statements or embodiments already mentioned above.

[0058] The following optional features may be applied in combination with any of the above mentioned aspects and optional features. Optionally, the list of predictors is used to predict motion in an image portion. The predictors in the list may be any of intra block predictor candidates and inter motion information predictor candidates. Alternatively, the predictors in the predictor list may be used for any of: predicting a position in a point cloud, encoding a mesh, predicting audio data, predicting still image data, and predicting video data. The method may further comprise associating at least one syntax element with one or more predictors in the predictor list.

[0059] In an aspect according to the invention, there is provided a method for encoding image data in a bitstream, the method comprising processing a predictor list according to any aspect or embodiment described above.

[0060] In one aspect according to the invention, there is provided a method for decoding image data from a bitstream, the method comprising processing a predictor list according to any aspect or embodiment described above.

[0061] In an aspect according to the present invention, there is provided an apparatus for encoding image data in a bitstream, the apparatus being configured to process a predictor list according to any aspect or embodiment described above.

[0062] In an aspect according to the present invention, there is provided an apparatus for decoding image data from a bitstream, the apparatus being configured to process a predictor list according to any aspect or embodiment described above.

[0063] According to another aspect of the present invention, there is provided a (computer) program, which, when executed, causes a programmable device to perform a method according to any of the above aspects or embodiments. The program may be stored on a computer readable storage medium.

[0064] The program may be provided independently or may be on, carried by or in a carrier medium. The carrier medium may be non-transitory, such as a storage medium, in particular a computer-readable storage medium. The carrier medium may also be transient, such as a signal or other transmission medium. The signal may be transmitted via any suitable network, including the Internet. Other features of the invention are characterized by independent and dependent claims.

[0065] Any features in one aspect of the invention may be applied to other aspects of the invention in any appropriate combination. In particular, method aspects may be applied to apparatus aspects, and vice versa.

[0066] Furthermore, features implemented in hardware may be implemented in software and vice versa. Any references herein to software and hardware features should be interpreted accordingly.

[0067] Any apparatus features as described herein may also be provided as method features, and vice versa.As used herein, means-plus-function features may alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and associated memory.

[0068] It should also be understood that specific combinations of the various features described and defined in any aspect of the present invention may be independently implemented and / or provided and / or used. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Reference will now be made, by way of example, to the accompanying drawings, in which:

[0070] Figure 1 is a diagram for explaining the coding structure used in HEVC;

[0071] Figure 2 is a block diagram schematically illustrating a data communication system in which one or more embodiments of the present invention may be implemented;

[0072] Figure 3is a block diagram illustrating components of a processing device that may implement one or more embodiments of the present invention;

[0073] Figure 4 is a flow chart showing the steps of an encoding method according to an embodiment of the present invention;

[0074] Figure 5 is a flow chart showing the steps of a decoding method according to an embodiment of the present invention;

[0075] Figure 6 and 7 shows a labeling scheme for describing blocks located relative to a current block;

[0076] Figure 8 (a) and (b) illustrate affine (sub-block) patterns;

[0077] Fig. 9 (a), (b), (c), (d) illustrate geometric patterns;

[0078] Fig.10 Illustrate the first step of deriving a merge candidate list of a VVC;

[0079] Fig.11 Illustrate additional steps of deriving a merge candidate list for a VVC;

[0080] Fig.12 Illustrate the derivation of pairwise candidates;

[0081] Fig.13 Demonstrate the template matching method based on adjacent samples;

[0082] Fig.14 Example pair Fig.10 Modification of the first step of the merge candidate list derivation shown;

[0083] Fig.15 Example pair Fig.11 Modifications to the additional steps of deriving the merge candidate list shown;

[0084] Fig.16 Example pair Fig.12 Modification of the derivation of the pairs of candidates shown;

[0085] Fig.17 Cost determination of instantiated list candidates;

[0086] Fig.18 Illustrate a reordering process of a list of merge mode candidates;

[0087] Fig.19 illustrating pairwise candidate derivation during a reordering process of a list of merge mode candidates;

[0088] Fig. 20 Illustrate a ranking process of predictors based on distortion values;

[0089] Fig.21 An example of the sorting process of the distortion value-based predictor according to an embodiment of the present invention is provided;

[0090] Fig. 22 A reordering process of a distortion value based predictor according to an embodiment of the present invention is illustrated;

[0091] Fig.23 A reordering process of a distortion value based predictor according to an embodiment of the present invention is illustrated; Fig.24 A reordering process of a distortion value based predictor according to an embodiment of the present invention is illustrated; Fig.25 A reordering process of a distortion value based predictor according to an embodiment of the present invention is illustrated;

[0092] Fig.26 A reordering process of a distortion value based predictor according to an embodiment of the present invention is illustrated;

[0093] Fig. 27 A reordering process of a distortion value based predictor according to an embodiment of the present invention is illustrated;

[0094] Fig.28 A reordering process of a distortion value based predictor according to an embodiment of the present invention is illustrated;

[0095] Fig.29 A diagram illustrating a reordering process of a distortion value based predictor according to an embodiment of the present invention;

[0096] Fig.30 Illustrate ARMC reordering processing of sub-block merge list;

[0097] Fig.31 A reordering process of a distortion value based predictor according to an embodiment of the present invention is illustrated;

[0098] Fig.32 A reordering process of a distortion value based predictor according to an embodiment of the present invention is illustrated;

[0099] Fig.33 is a diagram showing a system including an encoder or a decoder and a communication network according to an embodiment of the present invention;

[0100] Fig.34 is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention;

[0101] Fig.35 is a diagram illustrating a network camera system;

[0102] Fig.36The figure illustrates an example of a smart phone. DETAILED DESCRIPTION

[0103] Figure 1 The invention relates to a coding structure used in the High Efficiency Video Coding (HEVC) video standard and the Diversified Video Coding (VVC) standard. A video sequence 1 consists of a series of digital images i. Each of these digital images is represented by one or more matrices. The matrix coefficients represent pixels.

[0104] The images 2 of the sequence may be partitioned into slices 3. In some cases, a slice may constitute the entire image. The slices are partitioned into non-overlapping coding tree units (CTUs). The coding tree unit (CTU) is the basic processing unit of the High Efficiency Video Coding (HEVC) video standard and conceptually corresponds in structure to the macroblock unit used in several previous video standards. The CTU is sometimes also referred to as the largest coding unit (LCU). The CTU has luma and chroma component parts, each of which is called a coding tree block (CTB). These different color components are not in the same Figure 1 Shown in.

[0105] For HEVC, a CTU is typically 64 pixels by 64 pixels in size, while for VVC it ​​may be 128 pixels by 128 pixels. Each CTU may be iteratively partitioned into smaller, variable-sized coding units (CUs) 5 using a quadtree decomposition.

[0106] The coding unit is the basic coding element and consists of two subunits called prediction units (PUs) and transform units (TUs). The maximum size of a PU or TU is equal to the CU size. A prediction unit corresponds to a partition of a CU for prediction of pixel values. Various different partitions of a CU into PUs are possible, as shown in 606, including a partition into 4 square PUs, and two different partitions into 2 rectangular PUs. A transform unit is a basic unit for spatial transformation using DCT. A CU can be partitioned into TUs based on a quadtree representation 607.

[0107] Each slice is embedded in a network abstraction layer (NAL) unit. In addition, the coding parameters of the video sequence are stored in a dedicated NAL unit called a parameter set. In HEVC and H.264 / AVC, two parameter set NAL units are used: first, the sequence parameter set (SPS) NAL unit, which collects all parameters that remain unchanged during the entire video sequence. Generally, it handles the encoding profile, the size of the video frame, and other parameters. Second, the picture parameter set (PPS) NAL unit, which includes parameters that can be changed from one image (or frame) of the sequence to other images (or frames). HEVC also includes a video parameter set (VPS) NAL unit, which contains parameters describing the overall structure of the bitstream. VPS is a type of parameter set defined in HEVC and is applied to all layers of the bitstream. A layer can contain multiple temporal sublayers, and all version 1 bitstreams are limited to a single layer. HEVC has certain hierarchical extensions for scalability and multi-view, and these extensions will allow multiple layers with a backward compatible version 1 base layer.

[0108] Other ways of splitting an image have been introduced in VVC including sub-pictures, which are independently coded groups of one or more slices.

[0109] Figure 2 The data communication system in which one or more embodiments of the present invention can be implemented is illustrated. The data communication system includes a transmission device (in this case, a server 201) which is operable to transmit data packets of a data stream to a receiving device (in this case, a client terminal 202) via a data communication network 200. The data communication network 200 can be a wide area network (WAN) or a local area network (LAN). Such a network can be, for example, a wireless network (Wifi / 802.11a or b or g), an Ethernet network, an Internet network, or a hybrid network consisting of several different networks. In a specific embodiment of the present invention, the data communication system can be a digital television broadcasting system, in which the server 201 sends the same data content to multiple clients.

[0110] The data stream 204 provided by the server 201 may consist of multimedia data representing video and audio data. In some embodiments of the present invention, the audio and video data streams may be captured by the server 201 using a microphone and a camera, respectively. In some embodiments, the data streams may be stored on the server 201 or received by the server 201 from other data providers, or generated at the server 201. The server 201 is provided with an encoder for encoding the video and audio streams, in particular for providing a compressed bit stream for transmission, which is a more compact representation of the data presented as input to the encoder.

[0111] In order to obtain a better ratio of quality of transmitted data to the amount of transmitted data, the video data may be compressed, for example, according to the HEVC format or the H.264 / AVC format or the VVC format.

[0112] The client 202 receives the transmitted bitstream and decodes the reconstructed bitstream to reproduce the video image on the display device and reproduce the audio data using the speaker.

[0113] Despite Figure 2 A streaming scenario is considered in the example of , but it will be appreciated that in some embodiments of the invention, data communication between the encoder and the decoder may be carried out using, for example, a media storage device (such as an optical disc).

[0114] In one or more embodiments of the present invention, a video image is transmitted along with data representing a compensating offset to be applied to reconstructed pixels of the image to provide filtered pixels in the final image.

[0115] Figure 3 A processing device 300 configured to implement at least one embodiment of the present invention is schematically illustrated. The processing device 300 may be a device such as a microcomputer, a workstation, or a light portable device. The device 300 includes a communication bus 313, which is connected to:

[0116] - a central processing unit 311 denoted as CPU, such as a microprocessor or the like;

[0117] - a read-only memory 306, denoted ROM, for storing a computer program implementing the invention;

[0118] - a random access memory 312, represented as a RAM, for storing executable codes of the method of the embodiment of the present invention, and registers suitable for recording variables and parameters required for implementing the method for encoding a digital image sequence and / or the method for decoding a bit stream according to the embodiment of the present invention; and

[0119] A communication interface 302 connected to a communication network 303, via which digital data to be processed are transmitted or received.

[0120] Optionally, the device 300 may further include the following components:

[0121] - a data storage component 304, such as a hard disk, for storing a computer program for implementing the method of one or more embodiments of the present invention and data used or generated during the implementation of one or more embodiments of the present invention;

[0122] a disk drive 305 for a disk 306, which is suitable for reading data from the disk 306 or writing data to said disk;

[0123] - A screen 309 for displaying data and / or serving as a graphical interface for interaction with a user by means of a keyboard 310 or any other pointing means.

[0124] Device 300 may be connected to various peripheral devices such as digital camera 320 or microphone 308 , each of which is connected to an input / output card (not shown) to provide multimedia data to device 300 .

[0125] The communication bus provides communication and interoperability between the various elements included in or connected to the device 300. The representation of a bus is not limiting, and in particular, the central processing unit is operable to communicate instructions to any element of the device 300 directly or via other elements of the device 300.

[0126] The disk 306 may be replaced by any information medium, such as a rewritable or non-rewritable compact disk (CD-ROM), a ZIP disk or a memory card, and, in general, by an information storage component that can be read by a microcomputer or a microprocessor, the disk 306 being integrated into the device or not, possibly removable and suitable for storing one or more programs whose execution enables the implementation of the method for encoding a digital image sequence and / or the method for decoding a bit stream according to the present invention.

[0127] The executable code may be stored in a read-only memory 306, on a hard disk 304 or on a removable digital medium such as, for example, a disk 306 as previously described, etc. According to a variant, the executable code of the program may be received via the interface 302 by means of a communication network 303 to be stored in one of the storage means of the device 300, such as a hard disk 304, etc., before being executed.

[0128] The central processing unit 311 is adapted to control and direct the execution of instructions or parts of software code of one or more programs according to the invention, the execution of instructions stored in one of the above-mentioned storage means. At power-on, one or more programs stored in non-volatile memory (e.g., on hard disk 304 or in read-only memory 306) are transferred to random access memory 312 (which then contains the executable code of one or more programs) and registers for storing variables and parameters necessary for implementing the invention.

[0129] In this embodiment, the device is a programmable device that implements the invention using software. Alternatively, however, the invention may be implemented in hardware (for example in the form of an application specific integrated circuit or ASIC).

[0130] Figure 4Block diagram illustrating an encoder according to at least one embodiment of the invention. The encoder is represented by connected modules, each module being adapted to implement, e.g. in the form of programming instructions executed by the CPU 311 of the apparatus 300, at least one corresponding step of a method for implementing at least one embodiment of encoding an image in a sequence of images according to one or more embodiments of the invention.

[0131] The encoder 400 receives as input a raw sequence 401 of digital images i0 to in. Each digital image is represented by a collection of samples (sometimes also called pixels) (hereinafter, they are referred to as pixels).

[0132] After implementing the encoding process, the encoder 400 outputs a bitstream 410. The bitstream 410 includes a plurality of coding units or slices, each slice including a slice header for transmitting an encoding value of an encoding parameter used for slice encoding, and a slice body including encoded video data.

[0133] Module 402 divides the input digital images i0 to in 401 into blocks of pixels. Blocks correspond to image parts and can have variable sizes (e.g., 4×4, 8×8, 16×16, 32×32, 64×64, 128×128 pixels, and several rectangular block sizes can also be considered). A coding mode is selected for each input block. Two families of coding modes are provided: coding modes based on spatial prediction coding (intra-frame prediction) and coding modes based on temporal prediction (inter-frame coding, merge, skip). Possible coding modes are tested.

[0134] Module 403 implements an intra prediction process in which a given block to be encoded is predicted by a predictor calculated from neighboring pixels of the block to be encoded. If intra coding is selected, an indication of the selected intra predictor and the difference between the given block and its predictor is encoded to provide a residual.

[0135] Temporal prediction is implemented by a motion estimation module 404 and a motion compensation module 405. First, a reference image from a reference image set 416 is selected, and a portion of the reference image (also referred to as a reference region or image portion) is selected by the motion estimation module 404, which is the region closest to a given block to be encoded (in terms of pixel value similarity). The motion compensation module 405 then uses the selected region to predict the block to be encoded. The difference between the selected reference region and the given block (also referred to as a residual block) is calculated by the motion compensation module 405. The selected reference region is indicated using a motion vector.

[0136] Thus, in both cases (spatial and temporal prediction), the residual is calculated by subtracting the predictor from the original block.

[0137] In the intra prediction implemented by module 403, the prediction direction is encoded. In the inter prediction implemented by modules 404, 405, 416, 418, 417, at least one motion vector or data identifying such a motion vector is encoded for temporal prediction.

[0138] If inter prediction is selected, information about the motion vector and the residual block is encoded. To further reduce the bit rate, the motion is assumed to be homogeneous and the motion vector is encoded by the difference relative to the motion vector predictor. The motion vector predictor in the set of motion information predictor candidates is obtained from the motion vector field 418 by the motion vector prediction and encoding module 417.

[0139] The encoder 400 further includes a selection module 406 for selecting a coding mode by applying a coding cost criterion such as a rate-distortion criterion, etc. To further reduce redundancy, a transform such as DCT is applied to the residual block by a transform module 407, and then the obtained transform data is quantized by a quantization module 408 and entropy encoded by an entropy encoding module 409. Finally, the encoded residual block of the current block being encoded is inserted into a bitstream 410.

[0140] The encoder 400 also performs decoding of the encoded images to generate reference images (e.g., reference images in reference image / picture 416) for motion estimation of subsequent images. This enables the encoder and decoder receiving the bitstream to have the same reference frame (using reconstructed images or image portions). The inverse quantization (dequantization) module 411 performs inverse quantization (dequantization) of the quantized data, followed by an inverse transform by the inverse transform module 412. The intra prediction module 413 uses the prediction information to determine which predictor to use for a given block, and the motion compensation module 414 actually adds the residual obtained by module 412 to the reference area obtained from the reference image set 416.

[0141] Post filtering is then applied by module 415 to filter the reconstructed pixel frame (image or image portion). In an embodiment of the present invention, a SAO loop filter is used, wherein a compensation offset is added to the pixel values ​​of the reconstructed pixels of the reconstructed image. It should be understood that post filtering is not always necessary. Furthermore, any other type of post filtering may be performed instead of or in addition to SAO loop filtering.

[0142] Figure 5 A block diagram of a decoder 60 according to an embodiment of the present invention is shown, which can be used to receive data from an encoder. The decoder is represented by connected modules, each module being suitable for implementing the corresponding steps of the method implemented by the decoder 60, for example in the form of programming instructions to be executed by the CPU 311 of the device 300.

[0143] The decoder 60 receives a bitstream 61 including coding units (e.g., data corresponding to a block or a decoding unit), each coding unit consisting of a header containing information related to the encoded parameters and a body containing the encoded video data. Figure 4 As illustrated, for a given block, the coded video data is entropy encoded on a predetermined number of bits and the index of the motion vector predictor is encoded. The received coded video data is entropy decoded by module 62. The residual data is then dequantized by module 63, after which an inverse transform is applied by module 64 to obtain pixel values.

[0144] Mode data indicating the encoding mode is also entropy decoded, and based on the mode, the encoding block (unit / set / group) of the image data is intra-type decoded or inter-type decoded.

[0145] In the case of intra mode, intra prediction module 65 determines the intra predictor based on the intra prediction mode specified in the bitstream.

[0146] If the mode is inter, motion prediction information is extracted from the bitstream to find (identify) the reference region used by the encoder. The motion prediction information includes a reference frame index and a motion vector residual. The motion vector predictor is added to the motion vector residual by the motion vector decoding module 70 to obtain the motion vector. Figures 6 to 10 The various motion predictor tools used in VVC are discussed in more detail.

[0147] The motion vector decoding module 70 applies motion vector decoding to each current block encoded by motion prediction. Once the index of the motion vector predictor for the current block has been obtained, the actual value of the motion vector associated with the current block can be decoded and used to apply motion compensation by module 66. The reference image portion indicated by the decoded motion vector is extracted from the reference image 68 to apply motion compensation 66. The motion vector field data 71 is updated with the decoded motion vector for prediction of subsequently decoded motion vectors.

[0148] Finally, a decoded block is obtained. Where appropriate, post-filtering is applied by a post-filtering module 67. The decoder 60 finally obtains and provides a decoded video signal 69.

[0149] Motion prediction (inter) mode

[0150] HEVC uses 3 different inter modes: inter mode (Advanced Motion Vector Prediction (AMVP)), "classic" merge mode (i.e. "non-affine merge mode" or also called "regular" merge mode) and "classic" merge skip mode (i.e. "non-affine merge skip" mode or also called "regular" merge skip mode). The main difference between these modes lies in the signaling of data in the bitstream. For motion vector coding, the current HEVC standard includes a competition-based scheme for motion vector prediction, which did not exist in earlier versions of the standard. This means that several candidates compete with the rate-distortion criterion on the encoder side to find the best motion vector predictor or best motion information for inter or merge mode (i.e. "classic / regular" merge mode or "classic / regular" merge skip mode), respectively. The index corresponding to the best predictor or best candidate for the motion information is then inserted into the bitstream together with a "residual" representing the difference between the predicted value and the actual value. The decoder can derive the same set of predictors or candidates and use the best one according to the decoded index. Using the residual, the decoder can then recreate the original value.

[0151] In the picture content extension of HEVC, a new coding tool called intra block copy (IBC) is signaled as any of these three inter modes, and the difference between IBC and the equivalent inter mode is determined by checking whether the reference frame is the current frame. This can be achieved, for example, by checking the reference index of list L0, and if it is the last frame in the list, it is inferred that this is an intra block copy. Another way is to compare the picture order counts of the current frame and the reference frame: if they are equal, this is an intra block copy.

[0152] The design of the derivation of predictors and candidates is important to achieve optimal coding efficiency without a disproportionate impact on complexity. In HEVC, two motion vector derivations are used: one for inter mode (Advanced Motion Vector Prediction (AMVP)) and one for merge mode (Merge Derivation Process (for Classic Merge Mode and Classic Merge Skip Mode)). The various motion predictor modes used in VVC are described below.

[0153] Figure 6 The marking scheme used herein to describe the blocks located between frames relative to the current block (i.e., the block currently being encoded / decoded) is shown ( Figure 6 ).

[0154] VVC Merge Mode

[0155] Compared to HEVC, several inter modes have been added in VVC. In particular, a new merge mode has been added to the normal merge mode of HEVC.

[0156] Affine mode (sub-block mode)

[0157] In HEVC, only the translational motion model is applied for motion compensated prediction (MCP). In the real world, there are many kinds of motions, such as zoom-in / zoom-out, rotation, perspective motion, and other irregular motions.

[0158] In JEM, a simplified affine transform motion compensated prediction is applied, and the general principles of the affine mode are described below based on an extract from the document JVET-G1001 presented at the JVET conference held in Torino, July 13-21, 2017. This entire document is incorporated herein by reference insofar as it describes other algorithms used in JEM.

[0159] like Figure 8 As shown in (a), the affine motion field of the block is described by two control point motion vectors.

[0160] Affine mode is a motion compensation mode like inter mode (AMVP, "classic" merge or "classic" merge skip). Its principle is to generate one motion information per pixel based on 2 or 3 neighboring motion information. In JEM, affine mode derives one motion information for each 4×4 block, such as Figure 8 (a) (each square is a 4×4 block, and Figure 8 The entire block in (a) is a 16×16 block, which is partitioned into 16 blocks of such 4×4 sized squares (each 4×4 square block has a motion vector associated with it). By enabling the affine mode with a flag, the affine mode can be used in AMVP mode and merge mode (i.e., the classic merge mode also referred to as "non-affine merge mode" and the classic merge skip mode also referred to as "non-affine merge skip mode").

[0161] In the VVC specification, affine mode is also called sub-block mode; these terms are used interchangeably in this specification.

[0162] The sub-block merge mode of VVC contains a sub-block based temporal merge candidate that inherits the motion vector field of the block in the previous frame pointed to by the spatial motion vector candidate. If the neighboring block has been coded using the inter-frame affine mode of sub-block merging, and then some affine candidates are derived as constructed before a certain zero Mv candidate, then the sub-block candidate is followed by the inherited affine motion candidate.

[0163] CIIP

[0164] In addition to the normal merge mode and the sub-block merge mode, the VVC standard also includes a combined inter merge / intra prediction (CIIP), also known as the Multi-Hypothesis Intra Inter (MHII) merge mode.

[0165] Combined Inter Merge / Intra Prediction (CIIP) merging can be considered as a combination of the regular merge mode and the intra mode and is referred to below as Fig.10 The block predictor for the current block (1001) of this mode is the average between the merged predictor block and the intra predictor block, such as Fig.10 As shown. The merge predictor block is obtained using exactly the same processing as the merge mode, so it is a dual predictor for a time block (1002) or two time blocks. Therefore, the merge index is signaled for this mode in the same way as the conventional merge mode. The intra predictor block is obtained based on the neighboring samples (1003) of the current block (1001). However, the number of available intra modes for the current block is limited compared to the intra block. In addition, there is no chroma intra predictor block signaled for the CIIP block. The chroma predictor is equal to the luminance predictor. As a result, 1, 2 or 3 bits are used to signal the intra predictor for the CIIP block.

[0166] The CIIP block predictor is obtained by merging a weighted average of the block predictor and the intra block predictor. The weights of the weighted average depend on the block size and / or the selected intra predictor block.

[0167] The obtained CIIP predictor is then added to the residual of the current block to obtain the reconstructed block. It should be noted that the CIIP mode is enabled only for non-skipped blocks. In fact, skipping using CIIP usually results in a loss of compression performance and an increase in encoder complexity. This is because the CIIP mode usually has a block residual that is opposite to other skip modes. Therefore, its signal notification for the skip mode increases the bit rate. When the current CU is skipped, CIIP is avoided. The result of this restriction is that the CIIP block cannot have a residual containing only 0 values, because it is impossible to encode a VVC block residual equal to 0. In fact, in VVC, for merge mode, the only way to signal that the block residual is equal to 0 is to use skip mode, because for merge mode, the CU CBF flag is inferred to be equal to true. And when the CBF flag is true, the block residual cannot be equal to 0.

[0168] In this manner, in this specification, CIIP should be interpreted as a mode that combines inter-frame and intra-frame prediction features, and not necessarily a label given to a specific mode.

[0169] CIIP uses the same motion vector candidate list as regular merge mode.

[0170] MMVD

[0171] The MMVD merge mode is a specific regular merge mode candidate derivation. It can be considered as an independent merge candidate list. For the current CU, the selected MMVD merge candidate is obtained by adding the offset value of one motion vector component (mvx or mvy) to the initial regular merge candidate. The offset value is added to the motion vector of the first list L0 or the motion vector of the second list L1, depending on the configuration of these reference frames (both are backward, both are forward, or forward and backward). The initial merge candidate is signaled using an index. The offset value is signaled using a distance index between 8 possible distances (1 / 4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels, 4 pixels, 8 pixels, 16 pixels, 32 pixels) and a direction index that gives the x or y axis and the offset sign.

[0172] In VVC, only the first two candidates of the regular merge list are used for MMVD derivation and signaled by a flag.

[0173] Geometric partitioning mode

[0174] The geometric (GEO) merge mode is a special bi-prediction mode. Fig. 9 exemplifies this particular block predictor generation. The block predictor contains one triangle (901 or 911) from the first block predictor and a second triangle (902 or 912) from the second block predictor. However, as Fig. 9 Several other possible splits of the block are possible, as described by (c) and (d). In this specification, geometric merging should be interpreted as a pattern of combining features of two inter non-square predictors, and not necessarily a label given to a specific pattern.

[0175] exist Fig. 9 In the example of (a), each partition (901 or 902) has a motion vector candidate as a unidirectional candidate. And for each partition, an index is signaled to obtain the corresponding motion vector candidate in the unidirectional candidate list at the decoder. And the first partition and the second partition cannot use the same candidate. The candidate list comes from a conventional merge candidate list, where for each candidate, one of the 2 components (L0 or L1) has been removed.

[0176] IBC

[0177] In VVC, the intra block copy (IBC) merge mode can also be enabled. IBC has an independent merge candidate derivation process.

[0178] Other motion information improvements

[0179] DMVR

[0180] Decoder-side motion vector derivation (DMVR) in VVC increases the accuracy of MVs in merge mode. For this method, decoder-side motion vector correction based on bilateral matching (BM) is applied. In this bi-prediction operation, the corrected MV is searched around the initial MV in reference picture list L0 and reference picture list L1. The BM method calculates the distortion between two candidate blocks in reference picture list L0 and list L1.

[0181] BDOF

[0182] VVC also integrates a bidirectional optical flow (BDOF) tool. BDOF (formerly known as BIO) is used to correct the dual prediction signal of a CU at the 4×4 sub-block level. BDOF is applied to a CU if the CU meets several conditions, in particular if the distances from two reference pictures to the current picture (i.e., the picture order count (POC) difference) are the same. As the name suggests, the BDOF mode is based on the concept of optical flow assuming that the motion of objects is smooth. For each 4×4 sub-block, the motion correction (v_x, v_y) is calculated by minimizing the difference between the L0 and L1 prediction samples. The motion correction is then used to adjust the dual prediction sample values ​​in the 4×4 sub-block.

[0183] PROF

[0184] Similarly, Prediction refinement with optical flow (PROF) is used for the affine mode.

[0185] AMVR and hpelIfIdx

[0186] VVC also includes adaptive motion vector resolution (AMVR). AMVR allows the motion vector difference of the CU to be encoded with different precisions. For example, for AMVP mode, consider a quarter brightness sample, half brightness sample, full brightness sample, or four brightness samples. The following table of the VVC specification gives the AMVR shift based on different syntax elements.

[0187]

[0188] AMVR may have an impact on the encoding of modes other than those that use motion vector difference encoding as a different merge mode. In fact, for some candidates, the parameter hpelIfIdx (which represents the index of the half-pixel precision luma interpolation filter) is propagated for some merge candidates. For example, for AMVP mode, hpelIfIdx is derived as follows:

[0189] hpelIfIdx=AmvrShift==3?1:0

[0190] Bi-prediction with CU-level weights (BCW)

[0191] In VVC, the bi-prediction mode with CU-level weights (BCW) is extended beyond simple averaging (as done in HEVC) to allow a weighted average of two prediction signals P0 and P1 according to the following equation.

[0192] P bi-pred =((8-w)*P0+w*P1+4)>>3

[0193] Five weights are allowed in the weighted average bi-prediction, where w∈{-2, 3, 4, 5, 10}.

[0194] For a non-merged CU, the weight index bcwIndex is signaled after the motion vector difference.

[0195] For a merged CU, the weight index is inferred from neighboring blocks based on the merge candidate index.

[0196] BCW is only used for CUs with 256 or more luma samples. In addition, for low-latency pictures, all 5 weights are used. For non-low-latency pictures, only 3 weights (w∈{3, 4, 5}) are used.

[0197] General merge list export

[0198] In VVC, Fig.10 and Fig.11 Derive the regular merge list. First, add the spatial candidates B1 (1002), A1 (1006), B0 (1010), A0 (1014) (such as Figure 7 and performing partial redundancy (1007) between the motion information of A1 and B1 to add A1 (1008), performing partial redundancy (1011) between the motion information of B0 and B1 to add B0 (1012), and performing partial redundancy (1015) between the motion information of A0 and A1 to add A0 (1016).

[0199] As merge candidates are added, the variable cnt is incremented (1015, 1009, 1013, 1017, 1023, 1027, 1115, 1108).

[0200] If the number of candidates in the list (cnt) is strictly less than 4 (1018), candidate B2 (1019) is added (1022) if it does not have the same motion information (1021) as A1 and B1.

[0201] Then add the temporal candidates. If the bottom right candidate (1024) is available (1025), add it (1026), otherwise if the center temporal candidate (1028) exists (1029), add the center temporal candidate (1026).

[0202] Then, the history-based (HMVP) is added (1101) if they do not have the same motion information as A1 and B1 (1103). In addition, the number of history-based candidates cannot exceed the maximum number of candidates in the merge candidate list minus 1 (1102). Therefore, in the merge candidate list, after the history-based candidate, at least one position is missing.

[0203] Then, if the number of candidates in the list is at least 2, a pair of candidates is constructed (1106) and added to the merge candidate list (1107).

[0204] Then, if there is an empty position in the merge candidate list (1109), a zero candidate is added (1110).

[0205] For spatial and history-based candidates, the parameters BCWidx and useAltHpelIf are set equal to the candidate's relevant parameters. For temporal and zero candidates, they are set equal to the default value of 0. These default values ​​essentially disable the method.

[0206] For paired candidates, BCCidx is set equal to 0, and hpelIfIdxp is set equal to the hpelIfIdxp of the first candidate (if the hpelIfIdxp of the first candidate is equal to the hpelIfIdxp of the second candidate, otherwise it is set to 0).

[0207] Paired Candidate Export

[0208] according to Fig.12 The algorithm constructs paired candidates (1106). As depicted, when 2 candidates are in the list (1201), hpelIfIdxp is derived as previously described (1204, 1202, 1203). The inter-frame direction (interDir) is then set equal to 0 (1205). For each list L0 and L1, if at least one reference frame is valid (different from -1) (1207), the parameter will be set. If both are valid (1208), the mv information of the candidate is derived (1209) and set equal to the reference frame of the first candidate, and the motion information is the average between the two motion vectors of the list, and the variable interDir is incremented. If only one of the candidates has motion information of the list (1210), the motion information of the paired candidates is set equal to the candidate (1212, 1211), and the inter-frame direction variable interDir is incremented.

[0209] ECM

[0210] Since the end of the VVC v1 standard, JVET has started the exploration phase by building the Exploration Software (ECM). The exploration phase collects additional tools and improves existing tools on top of the VVC standard to achieve better coding efficiency. Different additional tools compared to VVC are described in JVET-X2025.

[0211] ECM Merge Mode

[0212] Among all the tools added, some additional merge modes were added. Affine MMVD signals the offset of the merged affine candidate encoded by MVVD as the regular merge mode. Similarly, GEO MMVD was also added. CIIP PDPC is an extension of CIIP. And two template matching merge modes were added: regular template matching and GEO template matching.

[0213] Conventional template matching is based on template matching estimation, such as Fig.13 As depicted. At the decoder side, for the candidate corresponding to the relevant merge index and for the two lists (L0, L1) available, motion estimation is performed based on the neighboring samples of the current block (1301) and based on the neighboring samples of multiple corresponding block positions, the cost is calculated and the motion information that minimizes the cost is selected. The motion estimation is limited by the search range, and several restrictions on the search range are also used to reduce complexity.

[0214] In ECM, the regular template matching candidate list is based on the regular merge list, but with some additional steps and parameters added, which means that different merge candidate lists for the same block can be generated. In addition, only 4 candidates are available for the regular merge candidate list for template matching, compared to 10 candidates for the regular merge candidate list in ECM with the common test conditions defined by JVET.

[0215] General Merged List Export in ECM

[0216] In ECM, updated the general merge list export. Fig.14 and Fig.15 Shown based on Fig.10 and Fig.11 However, for clarity, the module of history-based candidates (1101) has been outlined in (1501).

[0217] In this Fig.15In , a new type of merge candidate is added: non-adjacent candidates (1540). These candidates come from blocks that are spatially located in the current frame rather than blocks in adjacent frames, because adjacent are spatial candidates. These candidates are selected based on distance and direction. For history-based candidates, a list of neighboring candidates can be added until the list reaches the maximum number of candidates minus 1, so that pairs of candidates can still be added.

[0218] Zero Candidate

[0219] If the list still has not reached the maximum number of candidates (Maxcand), a zero candidate is added to the list. Zero candidates are added based on possible reference frames or reference frame pairs. The following pseudo code gives the derivation of such candidates:

[0220]

[0221]

[0222] The pseudo code can be summarized as follows: For each reference frame index (unidirectional) or reference index pair (bi-prediction), add zero candidates. When all are added, only zero candidates with reference frame index 0 are added until the number of candidates reaches its maximum value. In this way, the merge list can include multiple zero candidates. In fact, it has been surprisingly found that this happens frequently in real video sequences, especially at the beginning of slices or frames and sequences.

[0223] In the latest modification of the derivation of merge candidates, the number of candidates in the list can be greater than the maximum number of candidates in the final list, Maxcand. However, this number of candidates in the initial list (MaxCandInitialList) is used for derivation. Therefore, zero candidates are added until the number of candidates is MaxCandInitialList, and not until it is Maxcand.

[0224] BM Merge Mode

[0225] BM merge is a merge mode dedicated to the adaptive decoder-side motion vector refinement method, which is an extension of the multi-pass DMVR of ECM. As described in JVET-X2025, this mode is equivalent to 2 merge modes that refine the MV in only one direction. Therefore, one merge mode is for L0 and one merge mode is for L1. Therefore, BM merge is enabled only when DMVR conditions can be enabled. For these two merge modes, only one list of merge candidates is derived, and all candidates comply with DMVR conditions.

[0226] Merge candidates for BM merge mode are derived from spatially neighboring coding blocks, TMVP, non-neighboring blocks, HMVP, pairwise candidates in a similar manner as for regular merge mode. The difference is that only those that meet the DMVR condition are added to the candidates. The merge index is encoded in a similar manner as for regular merge mode.

[0227] AMVP Merge Mode

[0228] The AVMP merge mode (also called dual predictor) is defined in JVET-X2025 as follows: It consists of an AMVP predictor in one direction and a merge predictor in the other direction. This mode can be enabled for a coding block when the selected merge predictor and AMVP predictor meet the DMVR condition, where there is at least one reference picture from the past and one reference picture from the future relative to the current picture, and the distances from these two reference pictures to the current picture are the same, and bilateral matching MV refinement is applied to the merged MV candidate and AMVP MVP as the starting point. Otherwise, if the template matching function is enabled, the template matching MV refinement is applied to the merged predictor or AMVP predictor with a higher template matching cost.

[0229] The AMVP part of the mode is signaled as regular unidirectional AMVP, ie, the reference index and MVD are signaled, and if template matching is used, the AMVP part has the derived MVP index, or the MVP index is signaled when template matching is disabled.

[0230] For the AMVP direction LX (where X can be 0 or 1), the merged part in the other direction (1-LX) is implicitly derived by minimizing the bilateral matching cost between the AMVP predictor and the merge predictor (i.e., for the pair of AMVP and merge motion vector). For each merge candidate in the merge candidate list with the motion vector in the other direction (1-LX), the bilateral matching cost is calculated using the merge candidate MV and the AMVP MV. The merge candidate with the minimum cost is selected. Bilateral matching refinement is applied to the coding block starting from the selected merge candidate MV and AMVP MV.

[0231] The third pass of 8x8 sub-PU BDOF refinement as multi-pass DMVR in multi-pass DMVR is enabled for AMVP merge mode coding blocks.

[0232] The mode is indicated by a flag, and if the mode is enabled, the AMVP direction LX is further indicated by a flag.

[0233] MVD Sign Prediction

[0234] The symbol prediction method is described in JVET-X0132. If the motion vector difference contains a non-zero component, the motion vector difference symbol prediction can be applied in the conventional inter-frame mode. In the current ECM version, symbol prediction is applied for AMVP, affine MVD and SMVD modes. The possible MVD symbol combinations are sorted according to the template matching cost, and the index corresponding to the true MVD symbol is derived, and the index is encoded with the context model. On the decoder side, the MVD symbol is derived as follows: 1 / parse the size of the MVD component; 2 / parse the MVD symbol prediction index of the context encoding; 3 / construct MV candidates by creating a combination between possible symbols and absolute MVD values, and add them to the MV predictor; 4 / derive the MVD symbol prediction cost for each derived MV based on the template matching cost and sorting; 5 / use the MVD symbol prediction index to select the true MVD symbol; 6 / add the true MVD to the MV predictor of the final MV.

[0235] TIMD

[0236] Intra prediction (fusion for template-based intra mode derivation (TIMD)) is described in JVET-X2025 as follows: For each intra prediction mode in MPM, the SATD between the prediction sample and the reconstructed sample of the template is calculated. The first two intra prediction modes with the smallest SATD are selected as TIMD modes. These two TIMD modes are fused with weights after applying PDPC processing, and such weighted intra prediction is used to encode the current CU. Position-dependent intra prediction combination (PDPC) is included in the derivation of TIMD mode.

[0237] Duplicate Check

[0238] exist Fig.14 and Fig.15 In, a duplication check is added for each candidate (1440, 1441, 1442, 1443, 1444, 1445 and 1530). However, repetition is also for non-adjacent candidates (1540) and history-based candidates (1501). The duplication check consists in comparing the motion information of the current candidate of index cnt with the motion information of the previous candidate. When the motion information is equal, it is considered to be a duplicate and the variable cnt is not incremented. Of course, the motion information means the inter-frame direction, reference frame index and motion vector for each list (L0, L1). Note that zero candidates corresponding to different reference frames are not considered to be duplicates.

[0239] MVTH

[0240] In ECM, a motion vector threshold is introduced for this duplicate check. This parameter changes the equality check in the following way: two motion vectors are considered equal if their absolute difference for each component is less than or equal to the motion vector threshold MvTh. For normal merge mode, MvTh is set equal to 1, which corresponds to the traditional duplicate check without a motion vector threshold.

[0241] As defined below, for the normal merge mode of template matching, MvTh is equal to a value that depends on the number of luma samples (nbSamples) in the current CU:

[0242] if(nbSamples<64)MvTh=1< <MV_FRACTIONAL=16

[0243] else if(nbSamples<256)MvTh=2< <MV_FRACTIONAL=32

[0244] else MvTh=4< <MV_FRACTIONAL=64

[0245] Where MV_FRACTIONAL corresponds to the inter-frame resolution of the codec. Therefore, in the current ECM, the 16th resolution is used, so MV_FRACTIONAL is equal to 4. << is a left shift operator. And where nbSamples = Height × Width (ie, the height and width of the current block).

[0246] For example, as described below, there is another threshold MvThBDMVRMvdThreshold used for GEO merge derivation and also for duplicate checking of non-adjacent candidates:

[0247]

[0248] ARMC

[0249] In ECM, in order to reduce the number of bits of the merge index, adaptive reordering of merge candidates with template matching (ARMC) is added. Fig.13 The candidates are reordered based on the cost of each candidate, using the template matching cost calculated in . In this method, only one cost is calculated for each candidate. This method is applied after the list is derived and is only applied to the first 5 candidates of the regular merge candidate list. It should be understood that the number 5 is selected to balance the complexity of the reordering process with the potential benefits, so that a larger number (e.g., all candidates) can be reordered.

[0250] Fig.18An example of this method is given for a conventional merge candidate list containing 10 candidates, such as in the CTC.

[0251] The method also applies to sub-block merging modes other than temporal candidates, and to regular TM mode for all 4 candidates.

[0252] In the proposal, the method is also extended to reorder and select candidates to be included in the final list of merge mode candidates. For example, in JVET-X0087, all possible non-adjacent candidates (1540) and history-based candidates (1501) are considered together with temporal non-adjacent candidates to obtain a list of candidates. The list of candidates is constructed without considering the maximum number of candidates. The list candidates are then reordered. Only the correct number of candidates from the list are added to the final list of merge candidates. The correct number of candidates corresponds to the first N candidates in the list. In this example, the correct number is the maximum number of candidates minus the number of spatial and temporal candidates already in the final list. In other words, non-adjacent candidates and history-based candidates are processed separately from adjacent spatial and temporal candidates. The processed list is used to supplement the adjacent spatial and temporal merge candidates already present in the merge candidate list to generate the final merge candidate list.

[0253] In JVET-X0091, ARMC is used to select a temporal candidate from 3 temporal candidates bi-dir, L0 or L1. The selected candidate is added to the merge candidate list.

[0254] In JVET-X0133, a merged temporal candidate is selected from several temporal candidates reordered using ARMC. In the same way, all possible neighboring candidates are subjected to ARMC, and up to 9 of these candidates can be added to the merge candidate list.

[0255] All of these proposed methods use classical ARMC to reorder the final list of merge candidates. JVET-X0087 reuses the costs computed during the reordering of non-adjacent and history-based candidates to avoid additional computational costs. JVET-X0133 applies systematic reordering to all candidates on the final list of merge candidates.

[0256] New ARMC

[0257] Since the first implementation of ARMC, the method has been added to several other modes. ARMC is additionally applied to regular and template matching merge modes, sub-block merge mode, and also to IBC, MMVD, affine MMVD, CIIP, CIIP with template matching, and BM merge modes. In addition, the principle of ARMC to re-order the list of candidates based on template matching cost is also applied to AMVP merge candidate derivation and for intra method TIMD to select the most likely predictor.

[0258] Furthermore, this principle is added for the signed residual prediction methods of the affine MVD, AMVP and SMVD methods.

[0259] For GEO merge mode and GEO with template matching and reference frame index prediction, there are additional tests for the use of this reordering.

[0260] Besides this almost systematic use of ARMC, there are several additional modifications to the derivation process for some candidates.

[0261] For example, ECM4.0 export includes cascaded ARMC processing for candidate export of conventional, TM and BM merge modes, such as Fig.19 as depicted in for conventional and TM merging modes.

[0262] Compared to the previous merge candidate derivation, first, 10 temporal positions are checked and added to the list of temporal candidates after non-repetitive checking. This temporal list can contain up to 9 candidates. Furthermore, since the MV threshold is always 1 in contrast to the motion threshold used for merge candidate derivation and does not depend on the merge mode, there is a specific threshold for this temporal list. Based on the list of up to the first 9 non-repetitive positions, the ARMC process is applied and the first temporal candidate is only added in the traditional list of merge candidates if it is non-repetitive compared to the previous candidate.

[0263] In the same way, non-adjacent spatial candidates are derived in 59 positions. A first non-repeating candidate list of up to 18 candidates is derived. But the motion threshold is different from the one used for temporal derivation and for the rest of the list and is independent of the regular or template merge mode. The motion threshold is set equal to the mv of the BDMVR. Up to 18 non-adjacent candidates are reordered and only the first 9 non-adjacent candidates are kept and added in the merge candidate list. Further candidates are then added unless the list already contains the maximum number of candidates. In addition, for the TM merge mode, the maximum number of candidates in the list MaxCandInitialList is higher than the maximum number of candidates Maxcand that the final list can contain.

[0264] Then, if Fig.19 As depicted, the ARMC process is applied to all candidates of the intermediate list including MaxCandInitialList. A maximum number of candidates Maxcand is set in the final list of candidates.

[0265] As for the template matching merge mode, similar modifications also apply to the BM merge mode.

[0266] ARMC template cost algorithm

[0267] Fig.17 The template cost calculation of the ARMC method is illustrated. The number of candidates considered in this process, NumMergeCandInList, is greater than or equal to the maximum number Maxcand (1712) that the list can contain.

[0268] For each candidate in the list (1701), if no cost was calculated during the first ARMC process for temporal and non-adjacent candidates (1702), the cost is set equal to 0 (1703). In an implementation, the uncalculated cost mergeList[i].cost associated with candidate "i" is set equal to the maximum value MAXVAL. If the upper template for the current block is available (1704), the distortion compared to the current block template is calculated (1705) and added to the current cost (1706). Then or otherwise, if the left template of the current block is available (1707), the distortion compared to the current block template is calculated (1708) and added to the current cost (1709). The cost of the current merge candidate mergeList[i].cost is then set equal to the calculated cost (1710), and the list is updated (1711). In this example, we consider the position of the current candidate i being set with respect to its cost compared to the costs of the other candidates. When all candidates have been processed, the number of candidates in the list, NumMergeCandInList, is set equal to the maximum number of possible candidates in the list, Maxcand.

[0269] Fig. 20 Show Fig.17 1710. First, the variable Shift is set equal to 0. Then, when Shift is less than Fig.17 When the current candidate i is found and the associated cost of the current candidate is lower than the cost of the previous candidate number i-1-shift (2002), the variable Shift is incremented (2003). When the loop ends, if the variable Shift is different from 0 (2007), the candidate number i is inserted at the position i-shift (2010).

[0270] Multiple Hypothesis Prediction (MHP)

[0271] Multiple hypothesis prediction (MHP) has also been added to ECM. With this approach, up to four motion compensated prediction signals for each block can be used (instead of two as in VVC). These individual prediction signals are superimposed to form an overall prediction signal. The motion parameters of the individual additional prediction hypotheses can be signaled explicitly by specifying the reference index, motion vector predictor index and motion vector difference, or implicitly by specifying the merge index. A separate multiple hypothesis merge flag distinguishes between these two signaling modes.

[0272] For spatial candidates, non-neighboring merge candidates, and history-based merge candidates, the multiple hypothesis parameter value "addHypNeighbours" is inherited from the candidate.

[0273] For temporal candidates, null candidates and paired candidates, the multiple hypothesis parameter values ​​"addHypNeighbours" are not kept (they are clear).

[0274] LIC

[0275] In ECM, local illumination compensation (LIC) is added. LIC is based on a linear model of illumination variation. The linear model is calculated using neighboring samples of the current block and neighboring samples of the previous block.

[0276] In ECM, LIC is enabled only for unidirectional prediction. LIC is signaled by means of a flag. For merge mode, the LIC flag is not transmitted but inherited from the merge candidate in the following manner.

[0277] For spatial candidates, non-adjacent merge candidates and history-based merge candidates, the value of the LIC flag is inherited.

[0278] For temporal candidates and zero candidates, the LIC flag is set equal to 0.

[0279] For paired candidates, such as Fig.16 The LIC flag is set as shown in the figure. Fig.12 , and modules 1620 and 1621 are added, and modules 1609, 1612, and 1611 are updated. The variable averageUsed is set equal to false (1620), if for the current list, the average for the paired candidates has been calculated, the LIC flag LICFlag[cnt] for the paired candidates is set equal to false, and the variable averageUsed is equal to true (1609). If only the candidate has motion information for the list (16121611), if averaging is not used, the LIC flag is updated. And it is set equal to the OR operation of its current value and the value of the candidate's LICflag.

[0280] And when the pair of candidates is Bidir (e.g., equal to 3), the LIC flag is equal to false.

[0281] However, as Fig.16 shown, the algorithm allows the LIC flag to be equal to a value different from true only when the two candidates have motion information of one list and each candidate has its own list. For example, candidate 0 has only the motion information of L0, while candidate 1 has only the motion information of L1. In this case, the LIC flag can be equal to a value other than 0, but since LIC is only used for unidirectional, it never occurs. Therefore, the LIC flag of the pair of candidates is always equal to false. Thus, when LIC may be needed, the pair of candidates cannot use LIC. This reduces the efficiency of the candidates and avoids the propagation of LIC for subsequent coding blocks, thereby reducing the coding efficiency.

[0282] In addition, the duplicate check in the ECM software results in some inefficiencies. As Fig.14 and Fig.15 depicted, each candidate is added to the list, and the duplicate check (1440, 1441, 1442, 1443, 1444, 1445, and 1530) only affects the increment of the variable cnt (1405, 1409, 1413, 1417, 1423, 1427, 1508). In addition, as Fig.16 described, the variable BCWidx is not initialized for the pair of candidates. Therefore, if the last candidate added to the list is a duplicate candidate, the value of BCWidx for the pair of candidates is the value of that previous duplicate candidate. This is not the case in VVC because these candidates are not added when a candidate is considered duplicate.

[0283] Embodiment

[0284] In one embodiment, when sorting the predictor list according to the cost value calculated for each predictor, the cost value includes a threshold and the cost calculated based on the decoded samples to obtain the costs to be compared. Therefore, the cost is calculated based on samples that are not the original block or based on the final decoded samples of the current block.

[0285] This embodiment lies in changing Fig. 20 module 2002 by "while((shift < i) AND (mergeList[i].cost + TH < mergeList[i - 1 - shift].cost))", where the threshold TH is added.

[0286] Fig.21 Illustrate an alternative embodiment; this figure is based on Fig. 20 . In this figure, the condition of the loop (2102) is the same as Fig. 20The same. When the Shift value has been determined (2102)(2103), and if the cost for the current candidate mergeList[i].cost plus the threshold TH is greater than or equal to the cost of the candidate at position i-1-shift (2108), then the candidate (2109) is not inserted at position i-shift (2110). Therefore, the current candidate maintains its position, even if its cost mergeList[i].cost is better.

[0287] Alternatively, and mathematically equivalently, in one embodiment, when the predictor list is sorted according to the cost values ​​calculated for each predictor, the difference or absolute difference between two costs is compared to a threshold. The costs include costs based on samples that are not original blocks.

[0288] This embodiment consists of Figure 23 to Figure 28 For example, in Fig.23 In the example, the absolute value between the costs is calculated (2305) to determine whether the candidate needs to be set to another position.

[0289] An advantage of these embodiments is that candidates may be re-ordered to improve diversity in the ordering of the candidate list, thereby increasing the likelihood that candidates earlier in the list will be selected, which results in improved coding efficiency.

[0290] In one embodiment, when the list of predictors is sorted according to the cost value calculated for each predictor, and some candidates are to be removed from the initial list of candidates, the cost value includes a threshold and a cost calculated based on the decoded samples to obtain the costs to be compared. Thus, the costs are calculated based on samples that are not the original block or based on the final decoded samples of the current block.

[0291] This embodiment can be modified by Fig.21 To obtain, to set the candidates (2108) that do not meet the conditions defined by the threshold value TH at the end of the list or to remove them completely from the list. In order to set the candidates at the end of the list, the Shift value is set equal to -Maxcand+1+i (2109), and the candidate mergeList[i] is inserted at the position i-shift (2110), thus at the position Maxcand-1.

[0292] Alternatively, and mathematically equivalently, in one embodiment, at least one candidate is removed from the list based on the cost value calculated for each predictor, comparing the difference or absolute difference between the two costs to a threshold. The costs include costs based on samples that are not original blocks.

[0293] An advantage of these embodiments is that candidates that do not provide enough diversity in terms of cost may be removed or moved to the end of the list since it is less likely to be selected, which optimizes the ordering of the list, which results in improved coding efficiency.

[0294] The predictor list can be intra or inter.

[0295] Some guidelines that can be applied

[0296] In one embodiment, at least one syntax element in the bitstream is associated with a predictor list, and the ordering affects the value of the syntax element. Thus, the syntax element value associated with the predictor list may be different when the ordering is applied and when the ordering is different from the ordering without considering the cost.

[0297] The advantage is that coding efficiency is improved, since the predictors are ordered according to some correlation between these predictors and the data representing the current block, which is generally more efficient than pre-ordering.

[0298] In one embodiment, each predictor of the list represents the value of a syntax element. For example, the method is applied to the signed MVD prediction method for motion vector residual coding of affine, conventional and SMVD inter-frame modes. For these modes, the reordered list is used by one or more syntax elements.

[0299] In one embodiment, some predictors of the list represent indexes. This embodiment corresponds to the merge index encoding of the MVD encoding of the merged MMVD and merged affine MMVD methods, or to the classic merge index encoding of the merge mode including IBC and GEO modes. In the same way, the index of the inter-frame predictor can also be considered as an inter-frame mode (AMVP), affine, IBC, SMVD. This embodiment can also be applied to the TIMD intra-frame mode in which the predictor list is reordered. In particular, the number of reordered candidates is higher than the maximum possible index value that the syntactic element can signal. This means that MaxCandInitialList is greater than MaxCand.

[0300] Possible cost calculation

[0301] The accuracy of the process of determining the relative "cost" of candidates at the decoder side is a determining factor in how effective the re-ordering process is. The following embodiments provide improvements to the cost determination process, which results in more accurate lists, lower computational cost complexity, or both.

[0302] It is important to note that the algorithm used to perform the reordering can be changed to prioritize the accuracy or complexity (speed) of the reordering process.

[0303] The following embodiments are intended to produce an accurate indication of relative costs while minimizing the overall complexity of the operation.

[0304] In one example, the re-ranking is based on a cost including a metric between samples. A sample associated with each candidate is compared to another sample to produce a relative metric of cost.

[0305] For example, the cost of the candidate can be calculated based on the neighboring samples of the predictor block and the neighboring samples of the current block. Such samples are easily available at the decoder side.

[0306] The cost may be calculated between samples of the 2 block predictors corresponding to the candidate in the list. For example, when the candidate is a bi-prediction candidate, the cost may be the distortion between the two block predictors. Bi-prediction candidates are discussed in more detail below.

[0307] The cost can also be calculated by comparing with another predictor candidate. For example, one other predictor candidate can be the most likely candidate or predictor. The cost of a candidate is calculated by comparing its samples with the samples of the most likely candidate.

[0308] The cost may be calculated for a subset of neighboring samples or a subset of the samples of the predictor. For example, if there are multiple neighboring samples that can be used to determine the cost, these samples are sampled to reduce the complexity of the calculation.

[0309] The cost may be calculated based on samples corresponding to an image according to another resolution. A high similarity to an image from a higher resolution is a good indication of a low cost (ie a good predictor).

[0310] The values ​​of the samples used to calculate the cost may be pre-processed. Assuming only relative cost values ​​are needed (i.e., only order is important), pre-processing the values ​​means simpler calculations and is less likely to significantly affect the efficacy of the re-ordering process. Based on the pre-processing, the calculated costs improve the re-ordering process.

[0311] In the above examples, the cost may be a measure of distortion such as the sum of absolute differences (SAD), the sum of absolute transformed differences (SATD), the sum of squared errors (SSE), or the structural similarity index measure (SSIM).

[0312] Alternatively, the cost is a measure of distortion, and a weight can be applied to the distortion. The rate or estimated rate can also be considered. Or a threshold.

[0313] The cost can also be a weighted cost, where the weights differ depending on the type of predictor or candidate or the initial position of the candidate in the list.

[0314] In an embodiment, the cost is proportional to the number of samples.

[0315] For example, the number of samples is the number of samples for the current block, and the distortion (cost based on the sample values) is divided by that number.

[0316] In an alternative embodiment, the number of samples is the number of samples used in the calculation of the distortion or the number of comparisons of the distortion. And the distortion (cost based on the sample values) is divided by that number.

[0317] In one embodiment, the cost based on the sample values is divided by the number of comparisons for the distortion, and the result is multiplied by the number of samples which is the block.

[0318] For example, for a template that matches the upper row and the left column of a block (for the row and column, only one sample), the cost associated with candidate I, mergeList[i].cost, is set to be equal to mergeList[i].cost = ((height × width) × mergeList[i].cost) / (height + width).

[0319] The advantage of this embodiment is that the cost is proportional to the block size. Thus, the parameters for the criterion can be adapted to all block sizes. More precisely, this provides a more efficient use of the Lambda value as defined in the following embodiments.

[0320] Additional criterion

[0321] In one embodiment, the threshold is the Lagrangian parameter Lambda (λ) used in the rate-distortion criterion J = D + λR. The rate-distortion criterion is used on the encoder side to determine the best coding possibility for a block, or more generally, to select the best coding possibility from a set of coding possibilities.

[0322] This embodiment provides a significant improvement in coding efficiency. If we consider two candidates or predictors cand1 (candidate 1) and cand2 (candidate 2), which have their respective distortions D1 and D2. And if D1 > D2 but their RD costs J1 and J2 are such that J1 < J2, we can consider that these candidates are too close or too redundant in the RD sense, because the rate of signaling cand2 is higher than its distortion benefit. This can be written as:

[0323] J1 < J2

[0324] D1 + λR1 < D2 + λR2

[0325] D1 - D2 < λ(R2 - R1)

[0326] (D1 - D2) / (R2 - R1) < λ

[0327] Since many of the indexes in the video codec use unary max codes, we can assume that R2 - R1 = 1. Therefore, D1 - D2 < λ. Thus, if |D1 - D2| < λ or D1 < D2 + λ, we can consider cand1 and cand2 to be redundant.

[0328] Thus, in this embodiment, a candidate is considered redundant when the absolute value between the distortion of the current candidate and the distortion of another candidate is less than Lambda.

[0329] This embodiment is illustrated in Figures 21 to 28 For example, in Fig.23 the minimum (min) absolute distortion difference (2305) is compared with the Lambda value (2308).

[0330] This formula is particularly efficient and is also particularly efficient when the rate difference between a candidate and its previous candidate is not 1.

[0331] The previous embodiments do not consider the rate difference between two candidates / predictors to determine whether they are redundant. Therefore, to improve coding efficiency, this rate difference is calculated to determine whether a candidate is redundant compared to another candidate. This embodiment is useful when the syntax element associated with the candidate / predictor list is encoded with something other than a unary code. For example, the merged motion difference of a regular MMVD or an affine MMVD is encoded with a Golomb code. In this case, the number of bin or bit differences between one index value and its subsequent index value can be 0, 1, 2, or more than 2.

[0332] In an additional embodiment, the actual rate is considered. Thus, the actual rate when using CABAC is taken into account. In this case, this embodiment can also be applied when the index is encoded with a unary max code.

[0333] This embodiment can be implemented by varying the Lambda value by taking advantage of λ(R2 - R1) in Figure 23 to Figure 28 or by changing the cost for the candidate by adding Lambda multiplied by the rate. Similarly, for Figure 20 to Figure 22 the criterion can be D2 + λ×R2 < D1 + λ×R1 or D2 + λ(R2 - R1) < D1, where D2 is mergeList[i].cost. (In Fig. 20 in the loop (2002) the criterion looks for a candidate better than the current candidate. But the criterion for determining whether a candidate is redundant is D1 < D2 + λ(R2 - R1). For example, this embodiment consists of changing Fig. 20 module 2002 by the following formula:

[0334] “while((shift <i)AND(mergeList[i].cost+TH*mergeList[i].rate<mergeList[i-1-shift].cost+TH*mergeList[i-1-shift].rate))”

[0335] In one embodiment, when the candidates are reordered after the first ordering, such as Figure 23 to Figure 28 As described in , the criteria (2305) for determining that a candidate is redundant includes multiplication of a distortion threshold, for example, D2-Th.D1<0, or alternatively Th.D2-D1<0 or Th2.D2-Th1.D1<0.

[0336] The advantage of this embodiment is that the coding efficiency is improved. In particular, the criterion can adjust the redundancy of two candidates based on the ratio of distortion. Due to this criterion, two candidates with small distortion are not considered redundant. On the contrary, compared with the previous criterion, two candidates with high distortion will be considered redundant more frequently.

[0337] In one embodiment, the criterion includes at least one threshold balancing the 2 distortions, and adding a threshold.For example, the criterion as defined previously becomes: |D2-Th.D1|-λ<0 or alternatively |Th2.D2-Th1.D1|-λ<0.

[0338] An advantage of this embodiment is coding efficiency improvement for some cases where a penalty for distortion of some candidates gives a coding efficiency improvement. For example, the penalty depends on the distance of the current block and the type of candidate.

[0339] Lambda Setup

[0340] In the encoder implementation, the lambda value depends on several parameters that affect the trade-off between distortion and rate and how the distortion is calculated and how the rate is estimated.

[0341] In one embodiment, Lambda depends on one or more than one parameter and the value of Lambda is calculated at the decoder side. The advantage is that no additional rate is required to transmit the Lambda value.

[0342] In one embodiment, the Lambda value (or threshold) is derived or obtained at the encoder side and transmitted at the decoder side (in the bitstream).

[0343] An advantage of this embodiment is that an encoder implementation can set a Lambda value that is more corresponding to the implementation or sequence type.

[0344] In an embodiment, the Lambda used at the decoder side or the transmitted Lambda depends on the QP value used for the current slice or picture. In another embodiment, when QP is set at the block level, the Lambda value is adjusted at the block level based on the block QP value. The QP value is the most important parameter affecting the quality of the reconstructed blocks with block residuals and therefore affecting the rate-distortion tradeoff.

[0345] In an embodiment, the Lambda used at the decoder side or the transmitted Lambda depends on the picture order count distance or the temporal distance between the current frame and its reference frame. As some modes, as a regular merge mode, the POC distances for one candidate may be different from each other. Therefore, when several POC distances are possible, it is preferred to consider the minimum POC or the minimum temporal distance between the current frame and its reference frame. The temporal distance between the current frame and its reference frame or the temporal distance to the closest reference frame affects the quality of the block predictor. Therefore, it affects the quality of the reconstructed block with and without block residuals.

[0346] In an embodiment, the Lambda used at the decoder side or the transmitted Lambda depends on the temporal ID. In many video coding standards, frames have a temporal ID indicating the depth used for temporal diversity. Therefore, this temporal ID can affect the quality tradeoff required for the frame, so the Lambda can also depend on this parameter.

[0347] In an embodiment, the Lambda used at the decoder side or the transmitted Lambda depends on the time direction. The time direction means that the reference frame is from the past or from the future or from both directions. It is known that on average the quality of the inter-frame predictor is better when the block predictor comes from both directions. Therefore, it has an impact on the rate-distortion tradeoff. As some modes, as a regular merge mode, one candidate time direction can be different from each other, it is preferred that the Lambda depends on a large number of time directions, therefore, from one time direction or from two time directions.

[0348] In an embodiment, the Lambda used at the decoder side or the transmitted Lambda depends on uni-prediction or bi-prediction. It is known that when the block predictor comes from two block predictors, on average the quality of the predictor is better even if the two reference blocks are from the same temporal direction. Therefore, it has an impact on the rate-distortion tradeoff. As in some modes such as normal merge mode, one candidate may be bi-predicted and the other candidate uni-predicted, the preferred Lambda depends on whether bi-prediction can be used.

[0349] In one embodiment, the Lambda used at the decoder side or the transmitted Lambda depends on the QP value of the reference frame of the current slice. In an embodiment, to avoid complex Lambda settings for some modes, the Lambda depends on the QP value of the closest reference frame in each or both directions or the minimum QP value of each reference frame direction. When combined with the previous embodiment, the Lambda can depend on the difference between the QP values ​​of the selected QP for the reference frame. The QP of the reference affects the quality of the reference and therefore affects the QP of the current frame, thus affecting the rate-distortion tradeoff.

[0350] In an embodiment, the Lambda used at the decoder side or the transmitted Lambda depends on the distortion metric used. As is known, Lambda depends on the distortion used at the encoder side (SAD, SSE, etc.). Therefore, for the decoder side, Lambda also depends on the type of distortion.

[0351] In an embodiment, the Lambda used at the decoder side or the transmitted Lambda depends on the type of samples used to calculate the distortion. In fact, as mentioned before, criteria can be applied for different reordering or removal algorithms based on different types of samples. This can also be considered as setting the Lambda value. For example, since the template matching uses fewer samples than the current block, the distortion is calculated for fewer samples and therefore can be taken into account.

[0352] As mentioned previously, an alternative to this embodiment is to adjust the distortion according to the number of samples.

[0353] In one embodiment, the Lambda used or transmitted at the decoder side depends on the type of prediction. At the encoder side, the Lambda value may depend on the type of intra / inter prediction, and the Lambda used at the decoder side also depends on this characteristic.

[0354] In an embodiment, the Lambda used at the decoder side or the transmitted Lambda depends on the GOP configuration type. For example, depending on the configuration type such as random access configuration RA or low delay configuration or intra-only configuration, the Lambda can be shifted or be a Lambda based on another QP other than the QP of the current frame.

[0355] In an embodiment, the Lambda used at the decoder side or the transmitted Lambda is based on the Lambda formula used at the encoder to select among the best candidates / predictors in a list, etc., and the Lambda is used at the decoder side. The formula used may be signaled to the decoder, for example, in the bitstream by an index indicating which formula to use to calculate the Lambda from a predetermined list of available formulas.

[0356] Note that in other embodiments, the threshold or Lambda value may depend on a combination of any of the above factors, which may be optimally introduced to provide an optimal balance of coding efficiency and complexity.

[0357] One possible implementation of setting the correct Lambda is to derive it using the traditional formula (which usually depends on the QP value), and shift the QP value according to the parameters as defined previously. In a specific example that gives significant benefits, a Lambda formula used to select among the best merge candidates is used. When the current frame is in random access configuration, the Lambda formula is set with QP-4. Where QP is the QP for the current slice / frame. And when the frame is in low delay configuration, the QP value used to obtain the Lambda value at the decoder is the current QP.

[0358] Of course, Lambda can be stored in a predefined table instead of using a formula. In this case, the individual values ​​of the table are related to the QP value.

[0359] Threshold (TH) is not Lambda

[0360] As described for the previous embodiments, the Lambda value is theoretically a good value for the criterion, but depends on several parameters and is encoder implementation dependent. Therefore, in other embodiments, other values ​​may be used as the threshold.

[0361] In an embodiment, the threshold value is Fig.25 The minimum values ​​depicted are described below under the heading "Algorithm".

[0362] In an embodiment, the threshold is calculated based on the cost values ​​of all candidates (NumMergeCandInList) or from the first Maxcand candidate.

[0363] In an embodiment, the threshold is the average of the differences between the distortions of each candidate compared to its previous candidate. Figures 21 to 28 All relevant algorithms are valid and the lambda value or TH value is replaced by the average value.

[0364] An advantage of these embodiments is that the threshold value does not depend on several parameters or need to be transmitted.

[0365] Possible Algorithms

[0366] The algorithm to reorder the list based on the defined criteria or to remove candidates based on the criteria may be specific since the costs are calculated at the decoder side from samples that are not samples of the current block and furthermore each mode has its specificity.

[0367] In one embodiment, in Fig. 20 The criteria of the present invention are applied during the sorting process, which includes changing the criteria of the loop. In another example of implementation, Fig.21 and Fig. 22 This embodiment is also involved.

[0368] The advantage of this embodiment is the low impact on complexity, since only one cycle is needed to apply the proposed criterion.

[0369] Fig.23 An algorithm is illustrated which is applied after a first sort based on cost. Thus, at the beginning of the algorithm, all candidates of the list are sorted according to their cost / distortion. For example, after the ARMC method. The list of candidates contains NumMergeCandInList candidates, and NumMergeCandInList may be greater than or equal to the maximum number of candidates Maxcand used in the final list.

[0370] In the figure, the number of reorderings MaxNumReorder is limited to the number of candidates in the list NumMergeCandInList minus one (2301)(2302). For example, as defined in embodiment (2302), the algorithm is applied when the number of possible reorderings is greater than one and the minimum difference between a candidate and its subsequent candidate is greater than or equal to the Lambda (threshold) value. The minimum cost difference between candidate i and its subsequent candidate i+1 is determined (2304)(2305)(2306), and the variable CandToMove is set equal to the subsequent candidate i+1. The candidate is considered redundant compared to the predecessor candidate. If the minimum cost is lower than Lambda (threshold) (2308), the candidate is moved (2310) from its position to the position MaxNumReorder-1 (2309). Thus, the candidate is set at the farthest position, but before the previous candidate that is considered redundant by the algorithm. MaxNumReorder is then decremented.

[0371] The algorithm can be summarized as:

[0372] -Determine the minimum cost difference between 2 consecutive candidates.

[0373] - If that minimum difference is less than Lambda (or other threshold), then set that candidate at the furthest position in the list (and specifically before previous candidates that were considered redundant by the algorithm)

[0374] Fig.29An example of the reordering obtained by the algorithm is given. In this example, Lambda (i.e., threshold) is set equal to 16. At the beginning of the first step "Step 1", a list of 6 merge candidates is sorted according to their cost (mergeList[i].cost) from the lowest cost 2264 to the highest cost 2750. The absolute cost difference between each candidate and its predecessor candidate is then calculated (2305), and the minimum value is identified (2306). Fig.29 In this example, in step 1, the minimum absolute cost difference is 5 (min=5), and the relevant candidate CandToMove is candidate index i=4. The cost 5 is less than or equal to the Lambda value of 16 (2308). Therefore, in "step 2", candidate index 4 is moved to the end of the list and MaxNumReorder is decremented. Therefore, this candidate is not considered for the second step of the loop. In step 2, the list considered contains 5 candidates and the minimum absolute cost difference is 8 (corresponding to the merge candidate index i=1). For step 3, the candidate is inserted at position 4 in the merge candidate list, and the candidate is not considered for step 3. During "step 3", the minimum absolute cost difference is 18, which is greater than the Lambda value of 16 (2308), so the algorithm stops as defined in (2302). At the end of step 3, the merge list candidates are sorted not by their cost value (from smallest to largest) but by an algorithm that provides diversity improvement by moving candidates that are likely to be redundant (due to their similarity in cost to other candidates) toward the end of the list. More specifically, redundant candidates are moved to positions following the candidates that have been considered non-redundant so far.

[0375] and Fig.21 One advantage of this embodiment is that the redundancy is sorted based on the most redundant candidates. Another advantage is that the reordering is first by removing redundant candidates at the end according to the redundancy setting, and then the less redundant candidates are in the final set. In addition, the number of removals / reorderings is also limited. Therefore, compared with the embodiment of Fig.21 This embodiment produces an improvement in coding efficiency compared to the embodiment of . However, this requires a second reordering process that introduces more complexity.

[0376] In another embodiment, MaxNumReorder is not set equal to the number of candidates, but is set to Maxcand or alternatively to a smaller value. In fact, when the list is significantly large (NumMergeCandInList) compared to the number of candidates with indices (Maxcand), the proposed reordering sets inefficient candidates. Therefore, it is preferred to limit the possible reordering to a small number of values. For example, with the merged MMVD of ARMC, the MVD position list contains 96 candidates, and the maximum index of the MVD signaled is only 12. Therefore, in this case, it is preferred to use MaxNumReorder set equal to 12.

[0377] Fig.25 The examples shown in the embodiment Fig.23 In this algorithm, in contrast to the previous embodiment, there is no comparison with Lambda / Threshold (2502) and the minimum cost difference. The most redundant candidate CandToMove is always inserted at the end of the list. In order to obtain coding efficiency, Fig.23 Compared to the embodiment of , the maximum number of possible reorderings MaxNumReorder is limited by the variable LimitReorder (2502). Therefore, in this case, only LimitReorder reordering can be applied. Note that this embodiment is similar to Fig.23 , where Lambda / Threshold is set equal to the maximum value that is always larger than the possible absolute difference between the 2 distortions.

[0378] The advantage of this embodiment is that the Lambda value does not need to be derived or transmitted on the decoder side, so it is applicable to all encoder selection algorithms. However, the coding efficiency is reduced. However, for some modes, this restriction is effective.

[0379] exist Fig.23 In an alternative embodiment of the embodiment, the algorithm only removes candidates. This can be achieved by modifying the criteria of the loop (2302). In this case, the loop stops when MaxNumReorder is greater than or equal to Maxcand instead of 1. In addition, for better coding efficiency, the minimum distortion difference (2304)(2305)(2306)(2307) is determined only in the first part of the list. For example, the candidate that will (currently) be in the final list (Maxcand). This can be achieved by changing the criteria of 2304 and limiting i to Maxcand.

[0380] As for Fig.23 Compared with the re-ranking and removal described above, simply removing candidates is inefficient on average unless it affects the maximum number of candidates Maxcand in the final list.

[0381] In one embodiment, candidates that are considered redundant are removed from the list and the maximum size of the final list is reduced accordingly.

[0382] Fig.24 To illustrate this embodiment, in the figure similar to Fig.23 , each time a candidate is placed at the end of the list (2410), the maximum number of candidates Maxcand in the final list is reduced (2412). This is particularly efficient when the relevant syntax element is not encoded using a unary max code or when the Maxcand value is small. However, this creates parsing problems because the decoder needs to fully reconstruct to parse the relevant syntax element.

[0383] Fig.26 Example Fig.23 An alternative algorithm to the algorithm of Fig.23 In contrast, the position where the redundant candidate is inserted is not always MaxNumReorder-1 (2309), but the first position (2613) (2614) (2615) where the candidate becomes non-redundant. This is achieved by identifying a first candidate (2614) among the candidates after the current redundant candidate (2613) (CandToMove) for which the distortion difference between the redundant candidate (CandToMove) and the candidate is greater than Lambda (threshold). The insertion position isertPos for the redundant candidate is set equal to the position of the second candidate i (2615).

[0384] The algorithm can be summarized as:

[0385] - Determine the minimum cost difference between 2 consecutive candidates;

[0386] - If the minimum difference is less than Lambda, the consecutive candidates are considered redundant;

[0387] - comparing the distortion of the redundant candidate with the distortion of subsequent candidates in the list;

[0388] - identifying the position of the first subsequent candidate having a distortion difference above Lambda;

[0389] - Placing a redundant candidate at the identified position.

[0390] In an additional embodiment, when the value of the maximum number of possible reorderings MaxNumReorder is greater than the maximum number of candidates in the final list Maxcand (2612), isertPos is maintained equal to MaxNumReorder-1.

[0391] and Fig.23This embodiment improves the coding efficiency compared to the embodiment of . In fact, the distortion cost is not the real distortion compared to the original block, and the first reordering is usually correct, so this restriction on the position change of candidates considered to be redundant is usually correct.

[0392] This embodiment is particularly efficient for normal merge mode, template matching merge mode and BM merge mode, and when enabled for IBC merge mode.

[0393] Please note that the algorithm of the figure can be adjusted. For example, an additional step can check whether the number of merge candidates NumMergeCandInList in the list is greater than 2 to apply the algorithm. To avoid applying the algorithm when it is certain that the candidate is not reordered. In fact, the algorithm can change the order of the candidates only when the list contains more than 2 candidates. In addition, the number of candidates considered in the algorithm can be the number of candidates that have been reordered by ARMC, because other candidates do not have relevant distortions. In another possible implementation, step 2602 can be changed to a "for" loop (instead of a "while" loop) from the maximum number of reordered candidates NumMergeCandInList to 2, and step 2611 can be removed. When less than the minimum min, the algorithm stops. All algorithms described can be based on a table that associates the initial candidate position with its reordered position and another table containing the relevant costs of each candidate. In this case, in order to avoid complex exchanges, the exchange of the final reordered candidate list can be performed only at the end of the ARMC algorithm and the diversity reordering algorithm.

[0394] Fig. 27 Another possible implementation of the reordering algorithm after the first reordering based on distortion in an embodiment is shown. Fig.23 In contrast, the algorithm is not based on the minimum distortion difference, but is dedicated to identifying non-redundant candidates.

[0395] For each candidate cand (2702) of the list (2701), the algorithm looks for the first subsequent candidate cand2 (candidate 2) (2704) which is not redundant in the sense of the criteria defined in the present invention (2705). If no candidate is non-redundant, the next candidate is considered and the criteria are applied to determine whether the candidate is redundant or non-redundant. Otherwise, the first non-redundant candidate (2706) is inserted (2707) after the candidate (isertPos=cand+1). Of course, insertion means removing the non-redundant candidate from its original position.

[0396] The algorithm can be summarized as:

[0397] - for a candidate, determine the first and subsequent non-redundant candidates in the list;

[0398] - Insert the non-redundant candidate after this candidate.

[0399] This embodiment is particularly efficient for lists where candidates are almost all redundant. Thus, for example, this embodiment is efficient for MMVD merges, affine or conventional, where MVD residuals are close in position. In the same way and for the same reasons, this algorithm is particularly efficient for MVD sign prediction.

[0400] Fig.28 Example Fig. 20 In this figure, when a candidate is considered non-redundant (2805), it is also tested whether the candidate is non-redundant compared to all previous candidates (2808) of the current candidate (2809).

[0401] The algorithm can be summarized as:

[0402] - for a candidate, determine the first and subsequent non-redundant candidates in the list;

[0403] - Insert the non-redundant candidate after this candidate unless the non-redundant candidate is redundant compared to all previous candidates of this candidate.

[0404] As with the previous embodiments, this embodiment is particularly efficient for lists where the candidates are almost all redundant. Thus, for example, this embodiment is efficient for MMVD merges, affine or conventional, where the MVD residuals are close in position. In the same way and for the same reasons, the algorithm is particularly efficient for MVD sign prediction. Due to this further restriction of the reordering, a better coding efficiency is obtained compared to the previous embodiments. Even though Fig.26 The algorithm gives better results than this embodiment for the conventional, TM, and BM merge modes, and the algorithm also gives significant gains for these modes.

[0405] In an alternative embodiment, SizeCandList is not set equal to the number of candidates NumMergeCandInList (2701)(2801), but is set to Maxcand or alternatively to a smaller value. In fact, when the list is significantly large compared to the number of candidates with indices, the proposed reordering will set inefficient candidates. Therefore, it is preferred to limit the possible reordering to a small number of values. For example, with a merged MMVD using ARMC, the list of MVD positions for one candidate is 96, and the index of the signaled MVD is only 12. Therefore, in this case, it is preferred to use MaxNumReorder set equal to 12.

[0406] Note that this limit (2701)(2801) of Maxcand may be optional if the difference between NumMergeCandInList and Maxcand is not high.

[0407] Other embodiments

[0408] The following features may be usefully combined with the above-described embodiments to form further embodiments of the invention.

[0409] In one embodiment, when the list contains only 2 candidates, the proposed method is not applied. This is equivalent to setting Th equal to 0. For example, when using Fig. 20 In the embodiment of , this results in improved coding efficiency. In fact, when there are only 2 candidates in the candidate list, it is preferred to set the candidate with the best distortion as the first candidate.

[0410] For other embodiments where reordering is applied after the first ordering, e.g. Fig.23 , when there are only 2 candidates, the criterion does not change anything.

[0411] In an embodiment, when only one template for the current block is available, the costs for all candidates of the list are set proportionally relative to the costs of the available templates.

[0412] For example, when the unavailable template is the left template, the cost for the current block is cost=CostUp+(CostUp×height) / width.

[0413] Where height and width correspond to the height and width of the block, because here we consider the upper template to be the upper row on the upper side of the block, and the left template to be the left row on the left side of the block. But longer or shorter templates can be considered with adjustments to the formula. Multiplication and division can be replaced by shift operations to reduce complexity.

[0414] The advantage of this embodiment is that the coding efficiency is improved. In fact, for some rate-distortion trade-offs, the Lambda value is fixed. Therefore, Lambda is adapted to this rate-distortion trade-off. With a distortion corresponding to an equal number of samples for all blocks of the same size, a coding improvement is observed.

[0415] Alternatively, in this case, Lambda can be modified instead of modifying the individual distortion costs. For the same example, when the left template is not available, Lambda or threshold becomes Lambda = Lambda / (1+(height) / width). The calculation can be modified to use only shifts without division to reduce complexity.

[0416] An advantage of this embodiment is reduced complexity compared to the previous embodiment, since in this case the new Lambda is calculated once instead of changing the individual distortion costs.

[0417] In the current ECM, the motion vector thresholds for normal, TM and BM merge modes are different for each type of candidate. This creates a kind of diversity in the candidate subset.

[0418] In one embodiment, when applying the proposed invention, the MV thresholds used for derivation are the same for all merge candidate types. For example, the motion vector thresholds for temporal and non-neighboring candidates are the same as those for all other types of candidates.

[0419] This improves the coding efficiency because the diversity created by the proposed method of the invention is more efficient than using different motion vector thresholds.

[0420] In the same way, Fig.19 The merger type based ARMC depicted in gives some benefits because there is a diversity control based on the merger type.

[0421] In one embodiment, when the method of the present invention is enabled, ARMC processing is applied to all candidates derived and sorted using ARMC. Fig.19 In comparison, up to 9 temporal candidates and up to 22 (or 18, depending on the implementation) non-adjacent candidates are added to the list of other merge candidates (B1, A1, etc.). In order to use at least the same number of ARMC comparisons as for the ARMC process for each merge type, the initial list of candidates is not limited to the defined MaxCandForFirstList, but to MaxCandForFirstList + the number of temporal candidates and the number of non-adjacent candidates. However, this number can be limited to the maximum number that the buffer of merge candidates can contain. For example, it can be fixed to 30.

[0422] One advantage of this embodiment is that the coding efficiency is increased, since the diversity is preferably controlled due to the method proposed in the present invention.In addition, this embodiment reduces the complexity, since ARMC is not applied to repeated candidates in the temporal list or the non-contiguous list.

[0423] In one embodiment, the number of temporal candidates and / or the number of non-adjacent candidates is reduced. The advantage is that the complexity of the cost calculation of all these candidates is reduced.

[0424] Coding efficiency improvements have been shown when pairwise candidates are added after the ARMC process. However, in competition with merge-based ARMC processes, the gains are significantly reduced due to the significant increase in the number of possible candidates.

[0425] In one embodiment, when the proposed method is enabled, pairs of candidates are added during or after the ranking process. Optionally, the pairs of candidates in this case are constructed based on the first candidate given by the ranking of the ARMC and one or more other candidates in the list. Optionally, these added candidates are re-ranked compared to their costs and the costs of other candidates in the list due to the criteria proposed in the present invention, or a second re-ranking process is applied after this first re-ranking process.

[0426] The advantage is improved coding efficiency. In fact, paired candidates are sometimes candidates that are too close to existing candidates in the list. Therefore, when ARMC is used to sort based on cost, these too redundant candidates are at the top of the list without creating enough diversity. When an embodiment of the present invention is applied to the list, the list is more diverse, and paired candidates are sorted differently, or removed when paired candidates are too redundant. Therefore, improved coding efficiency can be obtained.

[0427] In an additional embodiment, when a pair of candidates is added after a re-ordering process of other conventional candidates, the pair of candidates is in a separate list. In this case, a table contains the order of the various candidates and another table contains the associated costs. These tables are set up during the ARMC process and the diversity re-ordering process. The final list is then set based on the candidates from both lists according to the determined order. The advantage is that the complexity is reduced because the maximum size of the buffer can be reduced.

[0428] In one embodiment, the number of pairwise candidates is limited to a fixed value, for example, 4.

[0429] An advantage of this embodiment is that it limits the maximum number of candidates to which ARMC processing is applied.

[0430] In one embodiment, the maximum number of paired candidates that can be added is defined as the number of target candidates in the final list or a fixed value. The maximum number of paired candidates is considered to limit the maximum number of candidates in the initial list (before ARMC processing). Therefore, in this case, NumMergeCandInList is set to be equal to NumMergeCandInList minus the maximum number of paired candidates that can be added. An offset (positive or negative) can also be added. It should be noted that it is impossible to know the true number of paired candidates that can be added when building paired candidates after the first reordering.

[0431] The advantages of this embodiment are reduced complexity and limitation because the maximum number of candidates with ARMC cost calculation is limited.

[0432] In one embodiment, pairwise candidates are not added for some modes. For example, pairwise candidates are not added to the TM merge list.

[0433] The advantage is also reduced complexity.

[0434] Predictive Coding Applications

[0435] The above embodiments have been described as implementations for predicting image portions (prediction units) in the context of video or image encoding and decoding. However, it is envisioned that the above methods may be applied to any compression technique that may use a predictor or predictor candidate. For example, in point cloud encoding, the predictor may be for a point in the cloud or for an element of a grid. Alternatively, the predictor may be for an audio data portion that predicts the audio data to be encoded or decoded from another audio data.

[0436] Unless explicitly stated otherwise, all described embodiments may be combined. In fact, many combinations are synergistic and can produce efficiency improvements that are greater than the sum of their parts.

[0437] Re-rank first candidate

[0438] For some specific predictor lists, ARMC is not applied for one or more than one first candidate. As mentioned before, for example, ARMC is not applied for the sub-block temporal predictor (STMVP) of the sub-block merge list. The diversity criteria and related algorithms can be adapted to these specific cases.

[0439] In addition, ARMC is applied to several modes, which are then competed together. ARMC sets the most likely predictor at the beginning of each list. This most likely predictor can be considered the so-called "zeroth" predictor, because it is usually desirable to exclude this predictor from reordering, so that the zeroth predictor is the first (as Cand0) and after any reordering process (as NewCand0, see Fig.30 ) are both at the beginning of the list. However, setting similar candidates at the beginning of each list is not desirable in terms of coding efficiency because there is not enough diversity between prediction modes.

[0440] In one embodiment, reordering based on the diversity criterion is applied to candidates with costs. For the example of STMVP candidates that are not reordered and costed, the candidate is maintained at the beginning of the list. In the same way, using the diversity criterion, ARMC (such as Fig.31 The candidate NewCand1 obtained after (as shown) is maintained at the second position of the list.

[0441] An alternative embodiment is to calculate the cost of STMVP candidates and apply the diversity algorithm to all sub-block merging candidates. In this case, the calculation of STMVP candidates can be conditioned on the threshold / Lambda value of the diversity criterion or on the activation of the diversity algorithm. If the threshold / Lambda value is equal to zero, the cost of STMVP is not calculated.

[0442] In one embodiment, the cost of the candidate at the beginning of the unevaluated list is set equal to 0, and the diversity algorithm uses this candidate as the zeroth candidate of the list. In the example of the STMVP candidate, the distortion cost is not calculated for the STMVP, but the associated value is set equal to 0, and the diversity algorithm is applied to all sub-block candidates in the list including this candidate. Of course, the diversity algorithm does not change the position of the STMVP candidate, but can change Fig.31 The position of the "first" candidate (although it is in the position after the zeroth candidate) NewCand1 (new candidate 1). In this embodiment, when the first candidate is considered to be reordered, it means that its cost is close to 0.

[0443] The advantage of this embodiment is that the coding efficiency is improved compared to the previous embodiment. In this embodiment, the first candidate is considered to be too redundant compared to its own cost and the threshold of the diversity criterion. Compared to the previous embodiment, the candidate can be moved to another position.

[0444] An alternative to this embodiment is to calculate the cost of the STMVP candidate and set it at the beginning of the list, regardless of the calculated cost. Therefore, the ARMC algorithm does not consider this cost for reordering. The diversity criterion can then be applied to all candidates in the list.

[0445] Compared with the previous embodiment, the diversity criterion can correctly evaluate the diversity between the zeroth candidate (ie, the STMVP candidate) and its successor NewCand1. Note that in the diversity algorithm, the candidate at the second position of the list is not always the same, depending on each cycle.

[0446] An advantage of this embodiment is therefore that the coding efficiency is improved, since the criterion is calculated from correct data.

[0447] Consider the zeroth candidate in the re-ranking

[0448] In one embodiment, the diversity algorithm may reorder all candidates in the list or all candidates with a cost. For the candidate at the beginning of the list, since there is no previous candidate and in order to consider the candidate in the diversity algorithm, its cost is compared to a threshold. If we consider the threshold to be a Lambda value, and if we consider D0 to be a distortion of NewCand0 (new candidate 0), the diversity criterion becomes: D0 < λ.

[0449] Fig.31 Example based on Fig.26 Only in this figure, step 3103 is changed compared to step 2603. Therefore, the minimum cost is initialized by the cost of the first candidate in the list, and the variable CandToMove representing the index of the candidate that can be moved is initialized by the value 0.

[0450] An advantage of this embodiment is that the coding efficiency is improved, since candidates at the beginning of the list are also considered for the reordering.

[0451] In an additional embodiment, the possibility to reorder the zeroth candidate is limited to some modes. For example, the possibility to reorder the zeroth candidate is limited to BM merge mode or TM merge mode. In fact, TM and BM are more efficient when they are not enabled for each other. This means that these two modes as well as the regular merge mode produce similar block predictors. There are differences in the candidate derivation to get a diversity between the candidate sets for these modes. But the proposed method gives better results because diversity is generated between the sets. Compared to Fig.31 In contrast, step 3103 is applied only for the relevant modes, otherwise the minimum value is set equal to MAX.

[0452] An advantage of this embodiment is that the coding efficiency is improved because of the diversity created among the candidate sets.

[0453] In one embodiment, for the zeroth candidate in the list, the threshold is different than the threshold used in the diversity algorithm for the other candidates.

[0454] In this case, for the zeroth candidate in the list, the criterion is D0<λ1, and for the other candidates, the criterion is |D i -D (i+1) |<λ2. Fig.31 In the example, step 3108 is suitable for changing the Lambda value when CandToMove is equal to 0 or other situations and step 3114.

[0455] The advantage is a better setting of the parameters, since the probability of re-ranking the zeroth candidate is different from re-ranking the other predictors.

[0456] In one embodiment, the threshold for the zeroth candidate is less than the threshold of the diversity criterion.

[0457] In fact, the movement or removal of the zeroth candidate is restricted compared to other candidates.

[0458] All previously described embodiments related to threshold values ​​or Lambda values ​​(eg, methods of setting or calculating threshold values ​​or Lambda values) may be applied to the above-described embodiments.

[0459] In one embodiment, according to the criterion D i <λ, move or remove candidates based on their cost values.

[0460] In one embodiment, all candidates with a distortion less than Lambda are removed from the set of predictors. For example, when the maximum number of predictors in the current list (MaxNumReorder) is greater than the maximum number of predictors that the final list can contain (Maxcand), all predictors with a cost less than Lambda are removed from the list until the MaxNumReorder is set equal to Maxcand.

[0461] In one embodiment, all candidates with distortion less than Lambda are moved to another position where the previous candidate had a cost distortion higher than the Lambda value.

[0462] Fig.32 This embodiment is illustrated. For each candidate (3201), it is tested whether its cost is less than Lambda (3202). If this is the case, the candidate (3203) is moved to the first position (3205) where the candidate cost is greater than or equal to Lambda, and the current candidate (3203) is inserted in the next position (3207). Of course, the algorithm is only applied if there is at least one candidate with a cost value greater than or equal to Lambda.

[0463] In an alternative embodiment, candidates with lower cost than Lambda are moved to the end of the list.

[0464] The advantage is that all candidates are considered according to the criterion compared to the previous embodiment, and those candidates with a cost less than Lambda are penalized more. This embodiment is more efficient when the number of predictors in the predictor list is high.

[0465] In an additional embodiment, the proposed moves or removals are applied only for some modes. For example, the possibility to reorder the first candidate is limited to BM merge mode or TM merge mode. In fact, as mentioned before, TM and BM are more efficient when the other is not enabled.

[0466] An advantage of this embodiment is that the coding efficiency is improved due to the diversity created among the candidate sets.

[0467] Implementation of the invention

[0468] Fig.33 A system 191, 195 according to an embodiment of the present invention is shown, which includes at least one of the encoder 150 or the decoder 100 and a communication network 199. According to an embodiment, the system 195 is used to process and provide content (e.g., video and audio content for display / output or streaming video / audio content) to a user, who accesses the decoder 100, for example, through a user terminal including the decoder 100 or a user interface of a user terminal that can communicate with the decoder 100. Such a user terminal can be a computer, a mobile phone, a tablet computer, or any other type of device capable of providing / displaying (provided / streamed) content to a user. The system 195 obtains / receives the bitstream 101 (in the form of a continuous stream or signal (e.g., when displaying / outputting earlier video / audio)) via the communication network 199. According to an embodiment, the system 191 is used to process content and store processed content, such as video and audio content processed for display / output / streaming at a later time. The system 191 obtains / receives content including an original image sequence 151, which is received and processed by the encoder 150 (including filtering using a deblocking filter according to the present invention), and the encoder 150 generates a bitstream 101 to be transmitted to the decoder 100 via the communication network 191. The bitstream 101 is then transmitted to the decoder 100 in a variety of ways, for example, it can be pre-generated by the encoder 150 and stored as data in a storage device in the communication network 199 (for example, on a server or cloud storage device) until the user requests the content (i.e., bitstream data) from the storage device, at which time the data is transmitted / streamed from the storage device to the decoder 100. The system 191 may also include a content providing device for providing / streaming to the user (for example, by transmitting data of a user interface to be displayed on a user terminal) content information (for example, the title of the content and other meta / storage location data for identifying, selecting and requesting the content) of the content stored in the storage device, and for receiving and processing user requests for content so that the requested content can be transmitted / streamed from the storage device to the user terminal. Alternatively, the encoder 150 generates the bitstream 101 and transmits / streams it directly to the decoder 100 when the user requests the content. The decoder 100 then receives the bitstream 101 (or signal) and filters it using the deblocking filter according to the present invention to obtain / generate a video signal 109 and / or an audio signal, which is then used by the user terminal to provide the requested content to the user.

[0469] Any step of the method / processing according to the present invention or the function described herein may be implemented in hardware, software, firmware or any combination thereof. If implemented in software, the step / function may be stored as one or more instructions or codes or programs or computer-readable media on one or more hardware-based processing units or sent via one or more hardware-based processing units, and executed by one or more hardware-based processing units, such as a programmable computing machine, which may be a PC ("personal computer"), a DSP ("digital signal processor"), a circuit, a circuit system, a processor and memory, a general-purpose microprocessor or central processing unit, a microcontroller, an ASIC ("application-specific integrated circuit"), a field programmable logic array (FPGA) or other equivalent integrated or discrete logic circuit system. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures or any other structures suitable for implementing the technology described herein.

[0470] Embodiments of the present invention may also be implemented by various devices or apparatuses, including wireless handsets, integrated circuits (ICs), or JC collections (e.g., chipsets). Various components, modules, or units are described herein to illustrate functional aspects of the apparatus / device configured to perform these embodiments, but do not necessarily need to be implemented by different hardware units. Instead, the various modules / units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units, which include one or more processors in conjunction with appropriate software / firmware.

[0471] Embodiments of the present invention can be implemented by a computer of a system or device that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium to perform one or more modules / units / functions in the above-mentioned embodiments and / or includes one or more processing units or circuits for performing one or more functions in the above-mentioned embodiments, and can be implemented by a method performed by a computer of a system or device, for example, reading out and executing computer executable instructions from a storage medium to perform one or more functions in the above-mentioned embodiments and / or controlling one or more processing units or circuits to perform one or more functions in the above-mentioned embodiments. The computer may include a network of separate computers or separate processing units to read out and execute computer executable instructions. Computer executable instructions may be provided to a computer from a computer-readable medium such as a communication medium, for example, via a network or a tangible storage medium. The communication medium may be a signal / bit stream / carrier. A tangible storage medium is a “non-transitory computer-readable storage medium” which may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage device of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD) TM ), one or more of a flash memory device, a memory card, etc. At least some steps / functions may also be implemented in hardware by a machine or dedicated components such as an FPGA (“field programmable gate array”) or an ASIC (“application-specific integrated circuit”).

[0472] Fig.343600 is a schematic block diagram of a computing device 3600 for implementing one or more embodiments of the present invention. The computing device 3600 may be a device such as a microcomputer, a workstation, or a light portable device. The computing device 3600 includes a communication bus connected to the following: - a central processing unit (CPU) 3601, such as a microprocessor; - a random access memory (RAM) 3602 for storing executable code of the method of the embodiment of the present invention and registers suitable for recording variables and parameters required to implement the method for encoding or decoding at least a part of an image according to the embodiment of the present invention, and its storage capacity can be expanded, for example, by an optional RAM connected to an expansion port; - a read-only memory (ROM) 3603 for storing a computer program for implementing an embodiment of the present invention; - a network interface (NET) 3604, which is usually connected to a communication network through which the digital data to be processed is transmitted or received, and the network interface (NET) 3604 may be a single network interface, or composed of a group of different network interfaces (for example, wired and wireless interfaces, or different kinds of wired or wireless interfaces), which are connected to the CPU when running. 3601, data packets are written to the network interface for transmission or read from the network interface for reception; - a user interface (UI) 3605, which can be used to receive input from a user or display information to a user; - a hard disk (HD) 3606, which can be set as a mass storage device; - an input / output module (IO) 3607, which can be used to receive / send data from / to an external device (such as a video source or a display, etc.). The executable code can be stored in ROM 3603, on HD3606, or on a removable digital medium such as a disk. According to a variant, the executable code of the program can be received via NET 3604 by means of a communication network to be stored in one of the storage components (such as HD 3606, etc.) of the computing device 3600 before being executed. CPU 3601 is suitable for controlling and directing the execution of instructions or parts of the software code of one or more programs according to an embodiment of the present invention, which instructions are stored in one of the aforementioned storage components. For example, after power-up, CPU 3601 is capable of executing those instructions relating to a software application from main RAM memory 3602 after loading instructions from program ROM 3603 or HD 3606. Such a software application, when executed by CPU 3601, causes the steps of a method according to the invention to be performed.

[0473] It should also be understood that according to other embodiments of the present invention, a decoder according to the above embodiments is provided in a user terminal such as a computer, a mobile phone (cellular phone), a tablet or any other type of device (e.g., a display device) capable of providing / displaying content to a user. According to yet another embodiment, an encoder according to the above embodiments is provided in an image capture device, which also includes a camera, a video camera or a webcam (e.g., a closed-circuit television or video surveillance camera) for capturing and providing content for encoding by the encoder. See below Fig.35 and Fig.36 Two such examples are provided.

[0474] Fig.35 is a diagram illustrating a network camera system 3700 including a network camera 3702 and a client device 202 .

[0475] The network camera 3702 includes an imaging unit 3706 , an encoding section 3708 , a communication unit 3710 , and a control unit 3712 .

[0476] The network camera 3702 and the client device 202 are connected to each other via the network 200 to be able to communicate with each other.

[0477] The camera unit 3706 includes a lens and an image sensor (eg, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS)), and captures an image of a subject and generates image data based on the image. The image may be a still image or a video image.

[0478] The encoding section 3708 encodes the image data by using the encoding method described above or a combination of the encoding methods described above.

[0479] The communication unit 3710 of the network camera 3702 transmits the encoded image data encoded by the encoding section 3708 to the client device 202 .

[0480] Furthermore, the communication unit 3710 receives commands from the client device 202. The commands include commands for setting parameters for encoding by the encoding section 3708.

[0481] The control unit 3712 controls other units in the network camera 3702 according to the command received by the communication unit 3710 .

[0482] The client device 202 includes a communication unit 3714 , a decoding unit 3716 , and a control unit 3718 .

[0483] The communication unit 3714 of the client device 202 transmits a command to the network camera 3702 .

[0484] Furthermore, the communication unit 3714 of the client device 202 receives the encoded image data from the web camera 3702 .

[0485] The decoding unit 3716 decodes the encoded image data by using the decoding method described above or a combination of the decoding methods described above.

[0486] The control unit 3718 of the client device 202 controls other units in the client device 202 according to a user operation or command received by the communication unit 3714 .

[0487] The control unit 3718 of the client device 202 controls the display device 2120 to display the image decoded by the decoding section 3716 .

[0488] The control unit 3718 of the client device 202 also controls the display device 2120 to display a GUI (Graphical User Interface) for specifying values ​​of parameters of the network camera 3702 (including parameters for encoding by the encoding section 3708 ).

[0489] The control unit 3718 of the client device 202 also controls other units in the client device 202 according to user operation input to the GUI displayed by the display device 2120 .

[0490] The control unit 3718 of the client device 202 controls the communication unit 3714 of the client device 202 according to the user operation input to the GUI displayed by the display device 2120 to transmit a command for specifying the value of the parameter of the network camera 3702 to the network camera 3702 .

[0491] Fig.36 The diagram illustrates an example of a smart phone 3800 .

[0492] The smartphone 3800 includes a communication unit 3802 , a decoding / encoding section 3804 , a control unit 3806 , and a display unit 3808 .

[0493] The communication unit 3802 receives the encoded image data via the network 200 .

[0494] The decoding / encoding section 3804 decodes the encoded image data received by the communication unit 3802 .

[0495] The decoding / encoding section 3804 decodes / encodes the encoded image data by using the decoding method described above.

[0496] The control unit 3806 controls other units in the smartphone 3800 according to user operations or commands received by the communication unit 3802 .

[0497] For example, the control unit 3806 controls the display unit 3808 to display the image decoded by the decoding / encoding section 3804. The smartphone 3800 may also include a sensor 3812 and an image recording device 3810. In this way, the smartphone 3800 may record an image and encode the image (using the method described above).

[0498] The smartphone 3800 may then decode the encoded image (using the method described above) and display it via the display unit 3808 , or transmit it to another device via the communication unit 3802 and the network 200 .

[0499] Replacements and modifications

[0500] Although the present invention has been described with reference to the embodiments, it should be understood that the present invention is not limited to the disclosed embodiments. It will be appreciated by those skilled in the art that various changes and modifications may be made without departing from the scope of the present invention as defined in the appended claims. All features disclosed in this specification (including any appended claims, abstracts and drawings) and / or all steps of any disclosed method or process may be combined in any combination, except for at least some mutually exclusive combinations of such features and / or steps. Unless otherwise expressly stated, each feature disclosed in this specification (including any appended claims, abstracts and drawings) may be replaced by alternative features for the same, equivalent or similar purposes. Therefore, unless otherwise expressly stated, each feature disclosed is only an example of a general series of equivalent or similar features.

[0501] It should also be understood that any results of the above-described comparison, determination, evaluation, selection, execution, proceeding, or consideration (e.g., selections made during an encoding or filtering process) may be indicated in data in the bitstream (e.g., a flag or data indicating the result) or may be determined / inferred from data in the bitstream, such that the indicated or determined / inferred result may be used for processing rather than actually being compared, determined, evaluated, selected, executed, performed, or considered, for example, during a decoding process.

[0502] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

[0503] Reference signs appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims.

Claims

1. A method for processing a predictor, the method comprising: obtaining a list of predictors, the list of predictors having at least two predictors; determining whether to modify the list of predictors based on a criterion using a first cost associated with a first predictor in the list and a second cost associated with a second predictor in the list; as well as modifying the list of predictors based on the determination, Therein, the criterion is based on a threshold value.

2. The method according to claim 1, wherein: The determining includes adding the threshold to the first cost or the second cost, and the criterion is related to a comparison of a result with one of the first cost and the second cost without adding the threshold.

3. The method according to claim 2, wherein: The criterion is whether a result of adding the threshold to the first cost is less than the second cost.

4. The method according to claim 1, wherein: The criterion is whether the difference between the first cost and the second cost crosses the threshold.

5. The method according to claim 4, wherein: The difference is calculated as an absolute value, and the criterion is whether the difference is smaller than the threshold value.

6. The method according to any one of claims 1 to 5, wherein: Modifying the list includes removing the first candidate or the second candidate from the list of the predictor.

7. The method according to any one of claims 1 to 6, wherein: Modifying the list includes changing the position of one or more predictors in the list.

8. The method according to claim 7, wherein: Changing the position of one or more candidates comprises moving the first predictor or the second predictor to a different position in the list.

9. The method according to claim 7, wherein: Changing the ordering includes moving the first predictor or the second predictor to a position considered to be the end of the list.

10. A method according to any one of the preceding claims, wherein: The first predictor and the second predictor follow the zeroth predictor at a position considered to follow the beginning of the list.

11. The method according to claim 10, wherein: The zeroth predictor is excluded from the step of modifying the list of predictors.

12. The method of claim 10, further comprising the step of comparing a zeroth cost associated with the zeroth predictor to a zeroth threshold, wherein: In case the zeroth cost exceeds the zeroth threshold, the zeroth predictor is included in the step for modifying the list of predictors.

13. The method according to claim 12, wherein: The step of comparing the zeroth cost associated with the zeroth predictor to a zeroth threshold is performed in the modes of the first set but not in the modes of the second set.

14. The method according to claim 12 or 13, wherein: The zeroth threshold is lower than the thresholds used for the first predictor and the second predictor.

15. The method according to any one of claims 1 to 9, wherein: The first predictor is at a position considered to be the beginning of the list, and wherein the first cost is set to zero.

16. The method according to any one of claims 1 to 14, wherein: The first cost and the second cost comprise template matching costs based on neighboring samples of the image portion to be encoded or decoded and neighboring samples of the image portion associated with the predictor.

17. The method according to any one of claims 1 to 14, wherein: The first cost and the second cost comprise a difference between sample values ​​of two block predictors associated with the predictor.

18. The method according to any one of claims 1 to 14, wherein: The first cost and the second cost comprise differences between samples associated with the predictor and at least one other predictor in the list.

19. The method according to any one of claims 1 to 14, wherein: The first cost and the second cost are based on a subset of samples or neighboring samples of the predictor.

20. The method according to any one of claims 1 to 14, wherein: The first cost and the second cost are based on samples corresponding to another image according to a different resolution.

21. A method according to any one of the preceding claims, wherein: The values ​​of the samples used to calculate the first cost and the second cost are pre-processed.

22. The method according to any one of claims 16 to 21, wherein: The first cost and the second cost are based on a coding rate.

23. The method according to any one of claims 16 to 21, wherein: The first cost and the second cost are based on distortion cost.

24. The method according to claim 23, wherein: The distortion cost is any one of the following: Sum of Absolute Differences (SAD), Sum of Absolute Transform Differences (SATD), Sum of Squared Errors (SSE), and Structural Similarity Index (SSIM).

25. A method according to any one of the preceding claims, wherein: The cost is proportional to the number of samples in the cell being processed.

26. The method according to claim 23 or 24, wherein: The cost is proportional to the number of samples in the template.

27. The method according to claim 23 or 24, wherein: The cost is based on the samples used to determine the distortion and block size.

28. The method according to any one of the preceding claims, further comprising: The threshold value is derived.

29. The method according to any one of claims 1 to 27, further comprising: Information indicative of the threshold is received.

30. The method according to claim 28 or 29, wherein: The threshold is based on the quantization parameter value, ie, QP value.

31. The method according to claim 28 or 29, wherein: The threshold is based on the temporal distance or the picture order count distance, ie, POC distance, between the current frame and one or more other frames.

32. The method according to claim 28 or 29, wherein: The threshold is based on the temporal ID of the frame.

33. The method according to claim 28 or 29, wherein: The threshold is based on the temporal direction of the reference frame.

34. The method according to claim 28 or 29, wherein: The threshold is based on whether bi-directional prediction is used.

35. The method according to claim 28 or 29, wherein: The threshold depends on one or more QP values ​​of available reference frames.

36. The method according to claim 28 or 29, wherein: The threshold value depends on a type of distortion used as at least one of the first cost and the second cost.

37. The method according to claim 28 or 29, wherein: The threshold is based on a type of sample used to calculate distortion as a measure of at least one of the first cost and the second cost.

38. The method according to claim 28 or 29, wherein: The threshold is based on whether intra prediction or inter prediction is used for encoding or decoding the portion of the image to be predicted using the list of predictors.

39. The method according to claim 28 or 29, wherein: The threshold is based on which of the random access configurations, ie, one or more of the RA configuration, the low delay configuration, and the intra-only configuration, is used for encoding or decoding.

40. The method according to claim 28 or 29, wherein: The threshold is based on a formula used for encoding and is obtained by the decoder for calculating the threshold.

41. The method according to claim 28 or 29, wherein: The threshold is based on the shifted quantization parameter value, ie the shifted QP value.

42. The method of claim 40, wherein: The shifted QP value is used to determine the Lagrangian parameter λ as the threshold.

43. A method according to any one of claims 41 to 52, wherein: The shifted quantization parameter is used to obtain the Lagrangian parameter from a table, wherein entries in the table associate Lagrangian parameter values ​​with quantization parameter values.

44. The method according to any one of claims 1 to 28, wherein: The threshold is determined as a minimum cost difference that is a difference between consecutive predictors in the list that are not the first predictor and the second predictor.

45. The method according to any one of claims 1 to 28, wherein: The threshold is determined using cost values ​​of all candidates or the first N candidates in the list, where N is the maximum number of candidates allowed in the list.

46. ​​The method of claim 45, wherein: Using the cost value includes calculating an average of differences in costs of consecutive candidates from among the candidates used to determine the threshold.

47. A method according to any one of claims 23 to 43, wherein: The threshold is based on a Lagrangian parameter λ, and the criterion is whether the absolute difference between the first distortion cost and the second distortion cost is smaller than the Lagrangian parameter.

48. The method of claim 47, wherein: The threshold is the Lagrangian parameter multiplied by a rate difference between the first predictor and the second predictor.

49. A method according to any one of claims 23 to 43, wherein: At least one of the first distortion cost and the second distortion cost is multiplied by the threshold.

50. The method of claim 37, wherein: The criteria are applied after the first sorting process.

51. The method of claim 37 or 50, wherein: A further threshold is added to the result of the difference between the first distortion cost and the second distortion cost resulting from the multiplication.

52. A method according to any one of the preceding claims, wherein: The determination is made according to the criterion during a process of ranking the candidates according to their distortion costs.

53. The method of claim 52, wherein: Determining whether to modify the list is based on a minimum distortion difference and a threshold.

54. The method of claim 53, wherein: The maximum number of reorders is limited to a predefined number.

55. The method of claim 54, wherein: The maximum number of reorderings is the maximum allowed number of candidates in the list.

56. The method according to any one of claims 1 to 52, wherein: The determination is based on the minimum distortion difference and is limited to a maximum number of reorderings.

57. The method according to any one of claims 52 to 56, wherein: Modifying the list merely involves removing candidates.

58. The method of claim 57, wherein: The number of candidates after removal must be greater than or equal to the maximum allowed number of candidates in the list.

59. A method according to the preceding claim, wherein: When the list is modified by moving a candidate, the maximum number of candidates allowed in the list is decremented.

60. The method of claims 53 to 55, wherein: In the event that the criterion is met, the second predictor is moved to a first position where the difference in cost for subsequent predictors in the list is above the threshold.

61. The method of claim 60, wherein: When the maximum number of possibilities having re-ordering is greater than the maximum number of candidates allowed in the final list, the modifying includes limiting the position where the first predictor or the second predictor is moved to the end of the current list.

62. The method according to claims 60 and 61, wherein: The processing of the predictor is applied to at least one of a normal merge mode, a template matching merge mode, a bilateral matching merge mode (BM merge mode), and an intra block copy mode (IBC mode).

63. The method according to any one of claims 1 to 52, wherein: The second predictor immediately follows the first predictor, and wherein, if it is determined that the criterion is not met, the second predictor is inserted after a subsequent predictor in the list.

64. The method of claim 63, wherein: In the event that it is determined that the criterion is not met, the criterion is applied to the second cost and the individual costs of the previous candidates in the list, and wherein, in the event that it is determined that the criterion is not met for each of the second cost and the previous candidate, the second predictor is not inserted after the subsequent predictor.

65. The method according to any one of claims 52 to 64, wherein: The maximum possible number of reorderings is set to the allowed number of candidates in the final list of processed candidates.

66. The method of claim 64 or 65, wherein: The processing of the predictor is applied to at least one of a merge with MVD, an affine merge with MVD, and an MVD sign prediction of a motion vector residual.

67. A method according to any one of the preceding claims, wherein: The list of predictors is used to predict motion in a picture portion.

68. The method of claim 67, wherein: The predictor in the list is any one of an intra block predictor candidate and an inter motion information predictor candidate.

69. The method according to any one of claims 1 to 66, wherein: The predictors in the list are used for any of: predicting positions in a point cloud, encoding a mesh, predicting audio data, predicting still image data, and predicting video data.

70. The method of claim 1, further comprising: At least one syntax element is associated with one or more predictors in the list of predictors.

71. A method for processing a predictor, the method comprising: obtaining a list of predictors, the list of predictors having at least two predictors; determining whether to modify the list of predictors based on a comparison of a first cost associated with a first predictor in the list to a threshold; as well as The list of predictors is modified based on the determination.

72. The method of claim 71, wherein: The step of determining whether to modify the list of predictors is based on a comparison of a respective cost associated with each predictor in the list with a threshold value.

73. The method of claim 71 or 72, wherein: Modifying the list comprises removing the or each candidate from the predictor's list having a cost below the threshold.

74. The method of claim 71 or 72, wherein: Modifying the list comprises moving the or each candidate in the list having a cost below the threshold to a position in the list following a candidate having a cost above the threshold.

75. The method of claim 71 or 72, wherein: Modifying the list comprises moving the or each candidate in the list having a cost below the threshold to a position considered to be at the end of the list.

76. The method according to any one of claims 71 to 75, wherein: The step of modifying the list of predictors is performed in the mode of the first set but not in the mode of the second set.

77. A method for encoding image data in a bitstream, comprising processing a list of predictors according to the method of any one of claims 1 to 76.

78. A method for decoding image data from a bitstream, comprising processing a list of predictors according to the method of any one of claims 1 to 76.

79. A device for encoding image data in a bitstream, the device being configured to perform the method of claim 77.

80. An apparatus for decoding image data from a bitstream, the apparatus being configured to perform the method of claim 78.

81. A program which, when executed, causes a programmable device to carry out a method according to any one of claims 1 to 76.

82. A computer-readable storage medium having stored thereon the program according to claim 81.