Video Encoding and Decoding

Abstract

An improvement is disclosed for deriving and reordering motion vector prediction candidates for predicting motion in an image portion. A list of motion vector prediction candidates for predicting motion in an image portion is generated, a first set of motion vector prediction candidates is added to the list, and if the number of the first set of motion vector prediction candidates is less than a maximum number of candidates, a second set of motion vector prediction candidates is added to the list such that the total number of candidates is equal to the maximum number of candidates. The list of candidates is reordered, and the second set of candidates is excluded from the reordering.

Description

[Technical field]

[0001] The present invention relates to video encoding and decoding. [Background technology]

[0002] The Joint Video Experts Team (JVET), a joint team formed by MPEG and ITU-T Study Group 16 VCEG, has introduced a new video coding standard called Versatile Video Coding (VVC). The goal of VVC is to achieve significant compression improvements over the existing HEVC standard (i.e., typically twice as fast). Primary target applications and services include, but are not limited to, 360-degree video and high dynamic range (HDR) video. In particular, the test material has demonstrated its effectiveness in ultra-high definition (UHD) video. Thus, we can expect compression efficiency improvements well beyond the 50% targeted in the final standard.

[0003] After completing the standardization of VVCv1, JVET established the Exploration Software (ECM) and started the exploration phase. ECM aims to improve coding efficiency by gathering additional tools and improvements of existing tools on top of the VVC standard.

[0004] Compared to HEVC, VVC has a modified set of "merge modes" for motion vector prediction that achieves higher coding efficiency at the cost of more complexity. Motion vector prediction is enabled by deriving a list of "motion vector prediction candidates", and the index of the selected candidate is signaled in the bitstream. A merge candidate list is generated per coding unit (CU). However, CUs may be split into smaller blocks for decoder-side motion vector refinement (DMVR) or other methods.

[0005] The composition and order of this list can have a significant impact on coding efficiency, since accurate motion vector prediction reduces the size of the residual or distortion of the block prediction, and having such candidates at the top of the list reduces the number of bits required to signal the selected candidate. The present invention aims to improve on at least one of these aspects.

[0006] The modifications incorporated in VVCv1 and ECM result in a maximum of 10 motion vector prediction candidates. This allows for candidate diversity, but may increase bitrate if a candidate lower in the list is selected. The present invention broadly relates to an improved ordering of candidates in a list of motion vector prediction candidates, especially when the calculated cost of the candidate is not representative of how likely it is to be selected. Reordering the list can lead to improved coding efficiency, and an efficient method of reordering can lead to reduced complexity. Various embodiments of the present invention achieve one or both of these advantages. Summary of the Invention

[0007] According to an aspect of the present invention, there is provided a method for generating a list of motion vector prediction candidates for predicting motion in an image portion, comprising: adding a first set of motion vector prediction candidates to said list; if the number of said first set of motion vector prediction candidates is less than a maximum number of candidates, adding a second set of motion vector prediction candidates to said list such that a total number of candidates is equal to said maximum number of candidates; and reordering said list of candidates, wherein said second set of candidates is excluded from said reordering. Optionally, said first set does not include duplicates.

[0008] According to another aspect of the present invention, there is provided a method for generating a list of motion vector prediction candidates for predicting motion in an image portion, comprising: adding a first set of motion vector prediction candidates to the list, determining duplicate candidates in the list, if the number of the first set of motion vector prediction candidates is less than a maximum number of candidates, adding a second set of motion vector prediction candidates to the list such that a total number of candidates is equal to the maximum number of candidates, and reordering the list of candidates, wherein the duplicate candidates are excluded from the reordering. Optionally, the second set of candidates is also excluded from the reordering.

[0009] The following statements apply to both aspects:

[0010] Optionally, said second set of motion vector predictions are zero candidates and said first set of candidates are non-zero candidates.

[0011] The first set of candidates may be candidates derived from at least one previously decoded or encoded motion information. Optionally, the first candidate set of candidates are candidates derived from true samples including one or more of: temporal candidates, spatial candidates, historical candidates, previously used candidates, or candidates derived from other candidates.

[0012] The reordering may be based on calculated relative costs of the candidates, optionally where at least one candidate is derived from at least one spatially or temporally matched template, where templates inside a bounded region are available and templates outside the bounded region are unavailable, and where a non-zero cost is calculated for a candidate if at least one template is unavailable.

[0013] According to yet another aspect of the present invention, there is provided a method for generating a list of motion vector prediction candidates for predicting motion in an image portion, the method comprising: adding a plurality of motion vector prediction candidates to a list; calculating a cost associated with at least one candidate in the list; the at least one candidate being derived from at least one spatially or temporally consistent template, where templates inside a bounded area for the image portion are available and templates outside the bounded area are unavailable; if at least one template is unavailable, calculating a non-zero cost for a candidate; and reordering the list in dependence on the calculated costs.

[0014] The cost may be determined based on the size of the template or templates from which the candidates are derived.

[0015] The cost may be determined based on the number of samples of the template that are available and the number of samples of the template that are unavailable, for example, the cost may be determined based on the ratio of available samples to unavailable samples.

[0016] Optionally, the division used in determining the cost is approximated by a shift operation.

[0017] Optionally, a cost value for the unavailable template is set equal to the cost value of the associated available template.

[0018] Optionally, if no template is available, the cost is set to a maximum value. For example, candidates without an associated available template may be reordered in the list immediately above the added motion vector prediction candidates, so that the total number of candidates is equal to the maximum number of candidates when the number of added motion vector prediction candidates is less than the maximum number of candidates. Optionally, the maximum value is less than the cost assigned to the added motion vector prediction candidates, so that the total number of candidates is equal to the maximum number of candidates when the number of added motion vector prediction candidates is less than the maximum number of candidates.

[0019] In one embodiment, the cost of the candidates is assigned a maximum value.

[0020] Optionally, at least one candidate is a bidirectional candidate having two spatially matched templates, and a cost of the available template is used to replace a cost of the unavailable template when determining a cost of the bidirectional candidate.

[0021] Optionally, at least one candidate is a bidirectional candidate having two spatially matched templates, and the cost of said bidirectional candidate is calculated using only said available templates.

[0022] If one template is partially available, the cost may be calculated for the available sample or the cost of the unavailable template relative to the cost of the associated available template.

[0023] The method may further include applying a weight to the calculated costs of candidates for which no template is available.

[0024] According to yet another aspect of the present invention, there is provided a method for generating a list of motion vector prediction candidates for predicting motion in an image portion, comprising: adding a plurality of motion vector prediction candidates to a list, at least one candidate in the list being derived from at least one spatially or temporally matched template, templates inside a delimited area for the image portion being available and templates outside the delimited area being unavailable, and reordering the list unless at least one template is unavailable. Optionally, the reordering is performed unless all templates are unavailable.

[0025] According to another aspect of the present invention, there is provided a method for generating a list of motion vector prediction candidates for predicting motion in an image portion, the method comprising: determining availability of at least one template related to a template of the image portion; and deriving the motion vector prediction candidates according to the availability of the template.

[0026] Optionally, if at least one template is unavailable, the method comprises adding at least one further temporal candidate to the list.

[0027] Optionally, the number of temporal candidates remains the same regardless of the availability of templates.

[0028] Optionally, the method further comprises decreasing the maximum number of candidates.

[0029] Optionally, the method further comprises decreasing the motion vector threshold before adding said further temporal candidates.

[0030] The cost of a candidate in the list may be calculated based on a comparison measure of at least one sample associated with the candidate with at least one other sample. The cost of a candidate may be calculated based on a difference between a neighbouring sample of a predicted block and said neighbouring sample of a current block. Optionally, the cost of a candidate is calculated by calculating a difference between two block predictions. Optionally, the cost of a candidate is calculated by calculating a difference with another candidate in the list. Optionally, said other candidate is the most likely candidate. Optionally, the cost is based on a sub-sampling of neighbours or samples of the prediction. Optionally, the cost is based on samples corresponding to images of other resolutions. Optionally, values ​​of samples used to calculate the cost are pre-processed.

[0031] Optionally, the cost corresponds to a distortion, which may be SAD, SATD, SSE or SSIM.

[0032] Optionally, the cost comprises a weight, the weight differing between motion vector prediction candidates. The method includes deriving a variable corresponding to a number of motion vector prediction candidates in the first set, and reordering the list is performed in dependence on the variable.

[0033] According to the above aspects and embodiments, optionally, the variable identifies a first candidate from the second set of motion vector prediction candidates. Each motion vector prediction candidate in the second set may be associated with a variable, and the sorting is performed depending on the variable. Optionally, the method includes setting a non-sorted motion vector prediction candidate to the end of the list.

[0034] The method may include performing a second reordering process on the unreordered motion vector prediction candidates.

[0035] The method may include performing a second reordering operation on the unreordered motion vector prediction candidates if the first set does not include more than one candidate.

[0036] The method may include performing a second reordering operation on the non-reordered motion vector prediction candidates depending on the coding mode.

[0037] Optionally, the second sorting is not applied to the sub-block merging mode.

[0038] Optionally, the method further comprises performing a second reordering operation on the non-reordered motion vector prediction candidates if a mode has a number of candidates that exceeds a threshold.

[0039] According to another aspect of the present invention, there is provided a method of encoding image data into a bitstream, the method comprising generating a list of motion vector prediction candidates according to any of the aspects and embodiments described above.

[0040] According to another aspect of the present invention, there is provided a method of decoding image data from a bitstream, the method comprising generating a list of motion vector prediction candidates according to any of the aspects and embodiments described above.

[0041] According to another aspect of the present invention, there is provided an apparatus for encoding image data into a bitstream, the apparatus being configured to perform a method for generating a list of motion vector prediction candidates according to any of the aspects and embodiments described above.

[0042] According to another aspect of the present invention, there is provided an apparatus for decoding image data from a bitstream, the apparatus being configured to perform a method for generating a list of motion vector prediction candidates according to any of the aspects and embodiments described above.

[0043] According to another aspect of the present invention there is provided a computer program which, when executed, performs the method according to any of the above aspects and embodiments. The computer program may be stored on a computer readable carrier medium, which may be transitory or non-transitory.

[0044] Further aspects of the invention relate to corresponding encoding methods, encoding devices, decoding devices and computer programs operable to perform the inventive decoding and / or encoding methods.

[0045] The program may be provided by itself or on, by or in a carrier medium. The carrier medium may be non-transitory, such as a storage medium, in particular a computer-readable storage medium. The carrier medium may also be transitory, such as a signal or other transmission medium. The signal may be transmitted via any suitable network, including the Internet. Further features of the invention are characterized by the independent and dependent claims.

[0046] Features in one aspect of the invention may be applied to other aspects of the invention in any appropriate combination, in particular method aspects may be applied to apparatus aspects and vice versa.

[0047] Furthermore, functionality implemented in hardware may be implemented in software and vice versa, and references herein to software and hardware features should be construed accordingly.

[0048] Apparatus features as described herein may also be provided as method features, and vice versa. As used herein, means-plus-function features may alternatively be expressed in terms of corresponding structure, such as a suitably programmed processor and associated memory.

[0049] It is also to be understood that particular combinations of the various features described and defined in any aspect of the present invention can be implemented and / or provided and / or used independently.

[0050] Further aspects of the invention are provided by the independent and dependent claims. [Brief description of the drawings]

[0051] Reference will now be made, by way of example only, to the accompanying drawings, in which:

[0052] [Figure 1] FIG. 1 is a diagram for explaining the coding structure used in HEVC. [Diagram 2] 1 is a block diagram that illustrates generally a data communications system in which one or more embodiments of the present invention may be implemented; [Diagram 3] 1 is a block diagram illustrating components of a processing device in which one or more embodiments of the present invention may be implemented. [Figure 4] 4 is a flow chart illustrating steps of an encoding method according to an embodiment of the present invention. [Diagram 5] 4 is a flow chart illustrating steps of a decoding method according to an embodiment of the present invention. [Figure 6] FIG. 2 illustrates a labeling scheme used to describe multiple blocks relative to a current block. [Figure 7] FIG. 2 illustrates a labeling scheme used to describe multiple blocks relative to a current block. [Figure 8] 13(a) and (b) are diagrams illustrating affine (sub-block) mode. [Figure 9] 1A, 1B, 1C, and 1D are diagrams illustrating geometric modes. [Figure 10-1] FIG. 1 illustrates the first step of VVC merge candidate list derivation. [Figure 10-2] FIG. 1 illustrates the first step of VVC merge candidate list derivation. [Figure 11] FIG. 13 illustrates a further step in deriving a merge candidate list for VVC. [Figure 12] FIG. 13 is a diagram illustrating the derivation of pair candidates. [Figure 13] FIG. 1 illustrates a template matching method based on neighboring samples. [Figure 14-1] FIG. 11 is a modification of the first step of deriving the merge candidate list shown in FIG. [Figure 14-2] FIG. 11 is a modification of the first step of deriving the merge candidate list shown in FIG. [Figure 15] FIG. 12 is a modification of the further steps of the merge candidate list derivation shown in FIG. [Figure 16] FIG. 13 is a diagram in which the derivation of pair candidates shown in FIG. 12 is modified. [Figure 17] FIG. 13 illustrates cost determination for list candidates. [Figure 18] FIG. 13 is a diagram illustrating a process of rearranging the merge mode candidate list. [Figure 19] 13A and 13B are diagrams illustrating derivation of pair candidates in a sorting process of a merge mode candidate list. [Figure 20] FIG. 2 illustrates the derivation of a merge candidate list of the present invention. [Figure 21] FIG. 13 is a diagram showing a process of rearranging a merge mode candidate list according to the present invention. [Figure 22] FIG. 13 shows three examples of templates for out-of-area candidates. [Figure 23] FIG. 13 shows three examples of templates for a current block outside the region. [Figure 24] FIG. 13 is a diagram showing an example of one template outside the region for bi-prediction. [Diagram 25] FIG. 1 illustrates a system including an encoder or decoder and a communication network according to an embodiment of the present invention. [Figure 26] 1 is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention. [Figure 27]FIG. 1 illustrates a network camera system. [Figure 28] FIG. 1 illustrates a smartphone. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0053] Figure 1 relates to the coding structure used in the High Efficiency Video Coding (HEVC) video and Versatile Video Coding (VVC) standards. A video sequence 1 is composed of a succession of digital images i, each such digital image being represented by one or more matrices, whose coefficients represent pixels.

[0054] An image 2 of a sequence may be divided into multiple slices 3. A slice may constitute the entire image. These slices are divided into non-overlapping coding tree units (CTUs). A coding tree unit (CTU) is the basic processing unit of the High Efficiency Video Coding (HEVC) video standard and is a structure that conceptually corresponds to the macroblock unit used in some earlier video standards. A CTU is sometimes called a maximal coding unit (LCU). A CTU has luma and chroma component parts, each of which is called a coding tree block (CTB). These different color components are not shown in Figure 1.

[0055] A CTU is typically 64 pixels by 64 pixels in HEVC, but 128 pixels by 128 pixels in VVC. Each CTU can be iteratively divided into smaller variable-sized coding units (CUs) 5 using a quad-tree decomposition.

[0056] A coding unit is a basic coding element and consists of two types of subunits called prediction units (PUs) and transform units (TUs). The maximum size of a PU or TU is equal to the CU size. A prediction unit corresponds to a partition of a CU for predicting pixel values. Various different divisions of a CU into PUs are possible, such as a division into four square PUs and two different divisions into two rectangular PUs, as shown in 606. A transform unit is an element unit that undergoes spatial transformation using DCT. A CU can be divided into TUs based on a quadtree representation 607.

[0057] Each slice is embedded in one network abstraction layer (NAL) unit. Furthermore, the coding parameters of a video sequence are stored in a dedicated NAL unit called a parameter set. HEVC and H.264 / AVC employ two types of parameter set NAL units. First, the sequence parameter set (SPS) NAL unit collects all parameters that do not change throughout the video sequence. It typically deals with the coding profile, the size of the video frames, and other parameters. Second, the picture parameter set (PPS) NAL unit contains parameters that may change from one picture (or frame) of the sequence to another. HEVC also includes the video parameter set (VPS) NAL unit, which contains parameters that describe the overall structure of the bitstream. VPS is a type of parameter set defined in HEVC that applies to all layers of the bitstream. A layer may contain multiple temporal sublayers, while all version 1 bitstreams are limited to one layer. HEVC has specific layer extensions for scalability and multiview, which allow for a backward-compatible version 1 base layer and multiple layers.

[0058] VVC introduces another way of dividing an image: sub-pictures, which are independently coded groups of one or more slices.

[0059] 2 illustrates a data communication system in which one or more embodiments of the present invention may be implemented. The data communication system includes a transmitting device (in this case a server 201) operable to transmit data packets of a data stream to a receiving device (in this case a client terminal 202) via a data communication network 200. The data communication network 200 may be a wide area network (WAN) or a local area network (LAN). Such a network may be, for example, a wireless network (Wifi / 802.11a or b or g), an Ethernet network, an Internet network, or a mixed network including several different networks. In a particular embodiment of the present invention, the data communication system may be a digital television broadcasting system in which the server 201 transmits the same data content to several clients.

[0060] The data streams 204 provided by the server 201 may consist of multimedia data representing video and audio data. The audio and video data streams may, in some embodiments of the invention, be captured by the server 201 using a microphone and a camera, respectively. In some embodiments, the data streams are stored in the server 201, received by the server 201 from other data providers, or generated at the server 201. The server 201 comprises, inter alia, an encoder for encoding the video and audio streams to provide a compressed bitstream for transmission, which is a more compact representation of the data presented as input to the encoder.

[0061] In order to obtain a better ratio of the quality of the transmitted data to the amount of transmitted data, the compression of the video data may for example be according to the HEVC format or the H.264 / AVC format or the VVC format.

[0062] Client 202 receives the transmitted bitstream, decodes the reconstructed bitstream, and plays the video images on a display device and the audio data on a loudspeaker.

[0063] Although the example of FIG. 2 considers a streaming scenario, it will be appreciated that in some embodiments of the invention, data communication between the encoder and the decoder may be performed using a media storage device, such as an optical disc.

[0064] In one or more embodiments of the invention, a video image is transmitted along with data representative of correction offsets that are applied to reconstructed pixels of the image to provide filtered pixels in a final image.

[0065] 3 shows a schematic diagram of a processing device 300 configured to implement at least one embodiment of the present invention. The processing device 300 may be a device such as a microcomputer, a workstation or a lightweight handheld device. The device 300 comprises a communication bus 313 connected to: - a central processing unit 311 (denoted as CPU), such as a microprocessor; - a read-only memory 306 (denoted ROM) for storing a computer program for implementing the invention; a random access memory 312 (denoted RAM) for storing the executable code of the method according to an embodiment of the invention and registers suitable for recording variables and parameters necessary for implementing the method for encoding a sequence of digital images according to an embodiment of the invention and / or the method for decoding a bitstream according to an embodiment of the invention; and a communications interface 302 connected to a communications network 303 over which the digital data to be processed is sent and received;

[0066] Optionally, device 300 may also include the following components: - data storage means 304, such as a hard disk, for storing computer programs for implementing the methods of one or more embodiments of the invention, and data used or generated during the implementation of one or more embodiments of the invention; a disk drive 305 for a disk 306. The disk drive is adapted to read data from or write data to the disk 306; A screen 309 for displaying data and / or acting as a graphical interface with the user, via a keyboard 310 or other pointing means.

[0067] The device 300 can be connected to a variety of peripherals, such as a digital camera 320 and a microphone 308 , each connected to an I / O card (not shown), which provide multimedia data to the device 300 .

[0068] The communication bus provides communication and interoperability between the various elements included in or connected to the device 300. The representation of the bus is not limiting, and in particular the central processing unit is operable to communicate instructions to any element of the device 300 directly or by other elements of the device 300.

[0069] The disk 306 may be replaced by any information medium, such as, for example, a compact disk (CD-ROM), rewritable or not, a ZIP disk or a memory card, and generally speaking, by any information storage means readable by a microcomputer or microprocessor, whether integrated in the device or not, possibly removable, and suitable for storing one or more programs, the execution of which can implement the method of encoding a sequence of digital images and / or the method of decoding a bitstream according to the present invention.

[0070] The executable code may, as mentioned above, either be stored in the read-only memory 306, on the hard disk 304 or on a removable digital medium such as for example the disk 306. According to a variant, the executable code of the program may be received by the communications network 303 via the interface 302, to be stored in one of the storage means of the device 300, for example the hard disk 304, before being executed.

[0071] The central processing unit 311 is adapted to control and direct the execution of instructions or parts of the software code of a program or program according to the invention, instructions stored in one of the storage means mentioned above. On power-up, the program or programs stored in a non-volatile memory (for example the hard disk 304 or the read-only memory 306) are transferred to a random access memory 312, which contains the executable code of the program or programs, as well as registers for storing variables and parameters necessary for the implementation of the invention.

[0072] In this embodiment, the device is a programmable device that uses software to implement the invention, but the invention could alternatively be implemented in hardware (e.g., in the form of an application specific integrated circuit (ASIC)).

[0073] 4 is a block diagram of an encoder according to at least one embodiment of the present invention, the encoder being represented by a number of connected modules, each adapted to implement, for example in the form of programming instructions executed by the CPU 311 of the device 300, at least one corresponding step of a method for implementing at least one embodiment of encoding images of an image sequence according to one or more embodiments of the present invention.

[0074] The encoder 400 receives as input an original sequence of digital images i0 to in401. Each digital image is represented by a collection of samples, also called pixels (hereafter also referred to as pixels).

[0075] A bitstream 410 is output by the encoder 400 after the encoding process is performed. The bitstream 410 includes multiple coding units or slices, each of which includes a slice header for transmitting coded values ​​of coding parameters used to code the slice, and a slice body that includes coded video data.

[0076] An input digital image i0 to 401 is divided by a module 402 into blocks of pixels. The blocks correspond to image portions and may be of variable size (for example 4x4, 8x8, 16x16, 32x32, 64x64, 128x128 pixels, and several rectangular block sizes can also be considered). A coding mode is selected for each input block. Two families of coding modes are provided: coding modes based on spatial predictive coding (intra prediction) and coding modes based on temporal predictive coding (inter coding, merge, SKIP). Possible coding modes are tested.

[0077] Module 403 performs an intra prediction process in which a given block to be coded is predicted by a prediction calculated from pixels in the neighborhood of the block to be coded. The indication of the selected intra prediction and the difference between a given block and its prediction are coded to provide a residual when intra coding is selected.

[0078] Temporal prediction is performed by a motion estimation module 404 and a motion compensation module 405. First, a reference image is selected from a set of reference images 416, and the motion estimation module 404 selects a part of the reference image (also called a reference region or image portion), which is the region closest to a given block to be coded (closest in terms of pixel value similarity). The motion compensation module 405 then uses the selected region to predict the block to be coded. The difference between the selected reference region and the given block (also called a residual block) is calculated by the motion compensation module 405. The selected reference region is indicated by a motion vector.

[0079] Therefore, in both cases (spatial and temporal prediction), the residual is calculated by subtracting the predicted value from the original block.

[0080] In the intra prediction implemented by module 403, the prediction direction is coded. In the inter prediction implemented by modules 404, 405, 416, 418, 417, at least one motion vector or data for identifying such a motion vector is coded for the temporal prediction.

[0081] If inter prediction is selected, the motion vector and information related to the residual block are coded. To further reduce the bit rate, assuming the motion is homogeneous, the motion vector is coded by the difference to the motion vector prediction. The motion vector prediction and coding module 417 obtains the motion vector prediction from a set of motion information prediction candidates from the motion vector field 418.

[0082] The encoder 400 further comprises a selection module 406 for selecting a coding mode applying a coding cost criterion, such as a rate-distortion criterion. To further reduce redundancy, a transform (such as a DCT) is applied to the residual block by a transform module 407, and the resulting transformed data is quantized by a quantization module 408 and entropy coded by an entropy coding module 409. Finally, the coded residual block of the current block being coded is inserted into a bitstream 410.

[0083] The encoder 400 also performs decoding of the encoded image to generate reference images (e.g., those of the reference image / picture 416) for motion estimation of subsequent images. This allows the encoder and decoder receiving the bitstream to have the same reference frame (reconstructed image or image portion is used). The inverse quantization ("dequantization") module 411 performs inverse quantization ("dequantization") of the quantized data, followed by inverse transformation by the inverse transformation module 412. The intra prediction module 413 uses the prediction information to decide which prediction to use for a given block, and the motion compensation module 414 actually adds the residual obtained by the module 412 to the reference area obtained from the set of reference images 416.

[0084] Post-filtering is then applied by module 415 to filter the reconstructed frame of pixels (image or image portion). In an embodiment of the invention, an SAO loop filter is used in which a compensation offset is added to the pixel values ​​of the reconstructed pixels of the reconstructed image. It is understood that post-filtering does not always have to be performed. Also, other types of post-filtering can be performed in addition to or instead of SAO loop filtering.

[0085] 5 is a block diagram of a decoder 60 that may be used to receive data from an encoder according to an embodiment of the present invention. The decoder is represented by a number of connected modules, each adapted to implement a corresponding step of a method implemented by the decoder 60, for example in the form of programming instructions executed by the CPU 311 of the device 300.

[0086] The decoder 60 receives a bitstream 61 containing coding units (e.g. data corresponding to blocks or coding units), each including a header containing information about coding parameters and a body containing the coded video data. As explained with respect to Fig. 4, the coded video data is entropy coded, and an index of a motion vector prediction is coded with a given number of bits for a given block. The received coded video data is entropy decoded by module 62. The residual data is inverse quantized by module 63 before an inverse transform is applied by module 64 to obtain pixel values.

[0087] Mode data indicating the encoding mode is also entropy decoded, and intra-type or inter-type decoding is performed on the block (unit / set / group) of encoded image data based on the mode.

[0088] For intra modes, intra prediction is determined by intra prediction module 65 based on the intra prediction mode specified in the bitstream.

[0089] If the mode is inter, motion prediction information is extracted from the bitstream to find (identify) the reference region to be used by the encoder. The motion prediction information includes a reference frame index and a motion vector residual. The motion vector prediction is added to the motion vector residual by the motion vector decoding module 70 to obtain a motion vector. The various motion prediction tools used in VVC are described in more detail below with reference to Figures 6 to 10.

[0090] A motion vector decoding module 70 applies motion vector decoding to each current block that has been coded with motion prediction. Once the index of the motion vector prediction for the current block is obtained, the actual value of the motion vector associated with the current block can be decoded and used to apply motion compensation by module 66. The reference image portion indicated by the decoded motion vector is extracted from reference image 68 for applying motion compensation 66. Motion vector field data 71 is updated with the decoded motion vector for use in predicting subsequent decoded motion vectors.

[0091] Finally, a decoded block is obtained. If necessary, post-filtering is applied by a post-filtering module 67. A decoded video signal 69 is finally obtained and provided by the decoder 60.

[0092] Multiple motion estimation (inter) modes HEVC uses three different inter modes: Inter mode (Advanced Motion Vector Prediction: AMVP), "classical" merge mode (also called "non-affine merge mode" or "normal" merge mode), and "classical" merge Skip mode (also called "non-affine merge skip" mode or "normal" merge skip mode). The main difference between these modes is the data signal in the bitstream. Regarding motion vector coding, the current HEVC standard includes a contention-based scheme for motion vector prediction. This means that for each inter or merge mode (i.e., "classical / normal" merge mode or "classical / normal" merge skip mode), multiple candidates compete with a rate-distortion criterion on the encoder side to find the best motion vector prediction or the best motion information. Then, an index corresponding to the best candidate for the best prediction or motion information is inserted into the bitstream along with a "residual" that represents the difference between the predicted value and the actual value. The decoder can derive the same set of predictions or candidates and uses the best one according to the decoded index. Using the residual, the decoder can recreate the original value.

[0093] In the Screen Content Extension of HEVC, a new coding tool called Intra Block Copy (IBC) is signaled as one of these three inter modes, and the difference between IBC and its equivalent inter modes is that it checks if the reference frame is the current one. This can be implemented, for example, by checking the reference index of list L0 and inferring that if this is the last frame in that list, this is an intra block copy. Another way is to compare the picture order counts of the current and reference frames.

[0094] The design of prediction and candidate derivation is important to achieve the best coding efficiency without disproportionately impacting the complexity. In HEVC, two motion vector derivations are used: one for inter modes (Advanced Motion Vector Prediction (AMVP)) and one for merge modes (merge derivation processes - classical merge mode and classical merge skip mode). In the following, the different motion prediction modes used in VVC are described.

[0095] FIG. 6 illustrates the labeling scheme used herein to describe blocks located relative to the current block (ie, the block currently being coded / decoded) between frames (FIG. 6).

[0096] VVC Merge Mode VVC adds some inter modes compared to HEVC, in particular adding new merge modes to the usual merge modes of HEVC.

[0097] Affine mode (sub-block mode) In HEVC, only the translational motion model is applied for motion compensated prediction (MCP), but in the real world there are many kinds of motion, including zoom, rotation, perspective, and other irregular motions.

[0098] In JEM, a simplified affine transform motion compensation prediction is applied and the general principles of the affine mode are explained below based on an excerpt from document JVET-G1001 presented at the JVET Conference held in Turin on 13-21 July 2017. This document is incorporated herein in its entirety by reference insofar as other algorithms used in JEM are described.

[0099] As shown in FIG. 8(a), the affine motion field of a block is described by two control point motion vectors.

[0100] Affine mode is a motion compensation mode like inter modes (AMVP, "classical" merge, or "classical" merge skip). Its principle is to generate one motion information per pixel according to two or three neighboring motion information. In JEM, affine mode derives one motion information for each 4x4 block as depicted in Fig. 8(a) (each square is a 4x4 block, and the whole block in Fig. 8(a) is a 16x16 block divided into 16 blocks of 4x4 size squares - each 4x4 square block has a motion vector associated with it). Affine mode is available in AMVP and merge modes (i.e., classical merge mode, also called "non-affine merge mode", and classical merge skip mode, also called "non-affine merge skip mode") by enabling affine mode with a flag.

[0101] In the VVC specification, the affine mode is also called the sub-block mode.

[0102] The subblock merging mode of VVC includes a subblock-based temporal merging candidate that inherits the motion vector field of the block of the previous frame pointed to by the spatial motion vector candidate. This subblock candidate follows the inherited affine motion candidate if the neighboring block is coded in the inter-affine mode of subblock merging, and some constructed affine candidates are derived before some zero Mv candidates.

[0103] CIIP In addition to the normal and sub-block merge modes, the VVC standard also includes a Combined Inter-Merge / Intra Prediction (CIIP), also known as the Multi-Hypothesis Intra-Inter (MHII) merge mode.

[0104] Combined Inter Merge / Intra Prediction (CIIP) merge can be considered as a combination of the normal merge mode and the intra mode, and is described below with reference to FIG. 10 (partially illustrated in FIG. 10-1 and FIG. 10-2). The block prediction of the current block (1001) in this mode is the average between the merge prediction block and the intra prediction block, as depicted in FIG. 10. The merge prediction block is obtained by exactly the same process as in the merge mode, and is therefore bi-predictive of a temporal block (1002) or two temporal blocks. Therefore, the merge index is signaled for this mode in the same way as in the normal merge mode. The intra prediction block is obtained based on the neighboring samples (1003) of the current block (1001). However, the amount of intra modes available for the current block is limited compared to intra blocks. Furthermore, there is no chroma intra prediction block signaled for CIIP blocks. Chroma prediction is equal to luma prediction. As a result, 1, 2, or 3 bits are used for the intra prediction signal of a CIIP block.

[0105] The CIIP block prediction is obtained by a weighted average of the merge block prediction and the intra block prediction, where the weighting of the weighted average depends on the selected block size and / or intra prediction block.

[0106] Then, the obtained CIIP prediction is added to the residual of the current block to obtain the reconstructed block. It should be noted that CIIP mode is only valid for non-skip blocks. In fact, using CIIP skip usually leads to a worse compression performance and an increased encoder complexity. This is because CIIP modes often have the opposite block residual to other skip modes. As a result, signaling skip mode increases the bitrate. -CIIP is avoided if the current CU is skip. In fact, in VVC, the only way to signal a block residual equal to 0 for merge mode is to use skip mode, because the CU_CBF flag is inferred to be true for merge mode. And when this CBF flag is true, the block residual cannot be 0.

[0107] Thus, in this specification, CIIP should be interpreted as a mode that combines features of inter prediction and intra prediction, and not necessarily as a label given to one particular mode.

[0108] CIIP uses the same motion vector candidate list as the normal merge mode.

[0109] MMVD The MMVD merge mode is the derivation of a certain regular merge mode candidate. It can be seen as an independent merge candidate list. The MMVD merge candidate selected for the current CU is obtained by adding an offset value to one motion vector component (mvx or mvy) of the first regular merge candidate. The offset value is added to the motion vector of the first list L0 or to the motion vector of the second list L1 depending on the configuration of these reference frames (both backward, both forward, or forward and backward). The first merge candidate is indicated by an index. The offset value is indicated by a distance index between eight possible distances (1 / 4-pel, 1 / 2-pel, 1-pel, 2-pel, 4-pel, 8-pel, 16-pel, 32-pel) and a direction index giving the x- or y-axis and the sign of the offset.

[0110] In VVC, only the first two candidates in the regular merge list are used to derive the MMVD and are signaled with one flag.

[0111] Geometric division mode Geometric (GEO) merge mode is a particular bi-prediction mode. Figure 9 illustrates the generation of this particular block prediction. The block prediction includes one triangle from the first block prediction (901 or 911) and a second triangle from the second block prediction (902 or 912). However, several other possible divisions of the block are possible, as depicted in Figures 9(c) and 9(d). Geometric merge should be interpreted herein as a mode that combines features of two inter non-square predictions, and not necessarily as a label given to one particular mode.

[0112] In the example of Fig. 9(a), each partition (901 or 902) has a motion vector candidate that is a unidirectional candidate. And for each partition, an index is signaled to get the corresponding motion vector candidate in the list of unidirectional candidates at the decoder. And the first and second cannot be the same candidate. This list of candidates comes from the list of regular merge candidates, and for each candidate, one of the two components (L0 or L1) is removed.

[0113] IBC VVC also allows the enabling of intra-block copy (IBC) merge mode, which has an independent merge candidate derivation process.

[0114] Other movement information improvements DMVR Decoder-side motion vector derivation (DMVR) in VVC improves the accuracy of MV in merge mode. In this method, bilateral matching (BM) based decoder-side motion vector refinement is applied. In this bi-prediction operation, a refined MV is searched around the initial MV in reference picture list L0 and reference picture list L1. The BM method calculates the distortion between two candidate blocks in reference picture list L0 and list L1.

[0115] BDOF VVC also integrates a bidirectional optical flow (BDOF) tool. BDOF, previously called BIO, is used to refine the bidirectional prediction signal of a CU at the 4x4 subblock level. BDOF is applied when a CU satisfies some conditions, in particular when the distances (i.e., POC (Picture Order Count) difference) from two reference pictures to the current picture are the same. As the name suggests, the BDOF mode is based on the concept of optical flow, which assumes that object motion is smooth. For each 4x4 subblock, a motion refinement (v_x,v_y) is calculated by minimizing the difference between the L0 predicted sample and the L1 predicted sample. This motion refinement is then used to adjust the two predicted sample values ​​of the 4x4 subblock.

[0116] PROF Similarly, prediction refinement by optical flow (PROF) is used for the affine modes.

[0117] AMVR and hpelIfIdx VVC also includes Adaptive Motion Vector Resolution (AMVR). AMVR allows the motion vector difference of a CU to be coded with different precision. For example, in AMVP mode, 1 / 4-luma samples, 1 / 2-luma samples, integer-luma samples, or 4-luma samples are considered. The following table from the VVC specification shows the AMVR shifts based on different syntax elements: TIFF2025510514000002.tif45153

[0118] AMVR may affect coding modes other than those using motion vector differential coding as different merging modes. Indeed, for some merging candidates, a parameter hpelIfIdx is propagated, which represents the index of the luma interpolation filter with half-pel precision. For example, in AMVP mode, hpelIfIdx is derived as follows: hpelIfIdx=AmvrShift==3?1:0

[0119] Bi-prediction with CU-level weights (BCW) In VVC, the Bi-prediction with CU-level weighting (BCW) mode is extended to allow not only simple averaging (as performed in HEVC) but also weighted averaging of the two prediction signals P0 and P1 according to the following formula: P bi-pred =((8-w)*P0+w*P1+4)≫3

[0120] Weighted average biprediction allows 5 weights, w∈{-2, 3, 4, 5, 10}.

[0121] For non-merged CUs, the weight index bcwIndex is signaled after the motion vector difference.

[0122] For a merged CU, the weight index is inferred from neighboring blocks based on the merge candidate indexes.

[0123] BCW is only used for CUs with 256 or more luma samples. In addition, for low-latency pictures, all five weights are used. For non-low-latency pictures, only three weights (w∈{3,4,5}) are used.

[0124] Normal merge list derivation In VVC, a normal merge list is derived as shown in Figures 10 and 11. First, if spatial candidates B1 (1002), A1 (1006), B0 (1010), and A0 (1014) (shown in Figure 7) exist, they are added. Then, partial redundancy is performed by adding A1 (1008) between the motion information of A1 and B1 (1007), adding B0 (1012) between the motion information of B0 and B1 (1011), and adding A0 (1016) between the motion information of A0 and A1 (1015).

[0125] When a merge candidate is added, the variable cnt is incremented (1015, 1009, 1013, 1017, 1023, 1027, 1115, 1108).

[0126] If the number of candidates in the list (cnt) is strictly less than four (1018), candidate B2 (1019) is added (1022) if it does not have the same motion information as A1 and B1 (1021).

[0127] Next, temporal candidates are added: if the bottom right candidate (1024) exists (1025), it is added (1026), otherwise the middle temporal candidate (1028) exists (1026), it is added (1029).

[0128] Next, a history-based (HMVP) is added (1101), but only if it has the same motion information as A1, B1 (1103). Furthermore, the number of history-based candidates must not exceed the maximum number of candidates in the merge candidate list minus one (1102). Therefore, there is at least one position missing in the merge candidate list after a history-based candidate.

[0129] Next, if the number of candidates in the list is at least two, we build pair candidates (1106) and add them to the merge candidate list (1107).

[0130] Next, if there are any empty positions in the merge candidate list (1109), a zero candidate is added (1110).

[0131] For spatial and history-based candidates, the parameters BCWidx and useAltHpelIf are set equal to the relevant parameters of the candidate. For spatial and zero candidates, they are set equal to the default value of 0. These default values ​​essentially disable the method.

[0132] For paired candidates, BCldx is set to 0, and hpelIfIdxp is set to the hpelIfIdxp of the first candidate if it is equal to the hpelIfIdxp of the second candidate, otherwise it is set to 0.

[0133] Pair candidates The pair candidates are constructed (1106) according to the algorithm of FIG. 12. As depicted, when there are two candidates in the list (1201), hpelIfIdxp is derived as described above (1204, 1202, 1203). Then, the inter direction (interDir) is set equal to 0 (1205). For each list, L0 and L1, if at least one reference frame is valid (different from -1) (1207), the parameter is set. If both are valid (1208), the mv information of this candidate is derived (1209) and set equal to the reference frame of the first candidate, the motion information is the average between the two motion vectors of this list, and the variable interDir is incremented. If only one of the candidates has motion information for this list (1210), the motion information of the pair candidate is set equal to this candidate (1212, 1211) and the inter direction variable interDir is incremented.

[0134] ECM After the standardization of VVCv1 was completed, JVET launched the Exploration Software (ECM) and started the exploration phase. ECM aims to improve the efficiency of encoding by adding additional tools and improving existing tools on top of the VVC standard. Various additional tools compared to VVC are described in JVET-X2025.

[0135] ECM Merge Mode Among all the tools added, some merging modes have been added. Affine MMVD signal offset is for merging affine candidates as MVVD encoding in normal merging mode. Similarly, GEO_MMVD has also been added. CIIP_PDPC is an extension of CIIP. Also, two template matching merging modes have been added: normal template matching and GEO template matching.

[0136] Normal template matching is performed based on template matching estimation as shown in Figure 13. At the decoder side, for the candidate corresponding to the associated merge index and both available lists (L0, L1), motion estimation based on neighboring samples of the current block (1301) and motion estimation based on neighboring samples of corresponding multiple block positions is performed, the cost is calculated, and the motion information that minimizes the cost is selected. The motion estimation is limited by a search range, and some restrictions on this search range are also used to reduce the complexity.

[0137] In ECM, the normal template matching candidate list is based on the normal merge list, but some additional steps and parameters are added, which may result in different merge candidate lists for the same block. Moreover, the normal merge candidate list for template matching is only available with 4 candidates, while the normal merge candidate list for ECM with common test conditions defined in JVET is 10 candidates.

[0138] Derivation of normal merge lists in ECM In ECM, the normal merge list derivation has been updated. Figure 14 (divided into Figure 14-1 and Figure 14-2) and Figure 15 show the updates based on Figures 10 and 11, respectively. However, for clarity, the modules of the history-based candidates (1101) are collapsed into (1501).

[0139] In this Fig. 15, a new type of merge candidate is added: non-adjacent candidates (1540). These candidates come from blocks that are spatially located in the current frame, but not from adjacent blocks. They are selected by distance and direction. On a history basis, the list of adjacent candidates can be added until it reaches the maximum number of candidates minus one.

[0140] Zero Candidate If the list has not yet reached the maximum number of candidates (Maxcand), zero candidates are added to the list. The zero candidates are added according to the possible reference frames or pairs of reference frames. The following pseudocode gives the derivation of such candidates: int iNumRefIdx=slice.isInterB()?min(MaxRefL0,MaxRefL1):MaxRefL0; int r=0 int refcnt=0; while (nbCand <Maxcand) { if (slice.isInterB()) { addZero(L0(Mv(0,0),Refix(r)),L1(Mv(0,0),Refix(r))); } else { addZero(Mv(0,0),Refix(r)); } nbCand++; if (refcnt==iNumRefIdx-1) { r=0; } else { ++r; ++refcnt; } }

[0141] The pseudocode can be summarized as follows: For each reference frame index (unidirectional) or pair of reference indices (bi-predictive), a zero candidate is added. When all are added, only zero candidates with reference frame index 0 are added until the number of candidates reaches a maximum value. In this way, the merge list can contain multiple zero candidates. In fact, it has been surprisingly found that such phenomena occur frequently in real video sequences, especially at the beginning of slices, frames, and sequences.

[0142] In a recent modification of the derivation of merge candidates, the number of candidates in the list can be greater than the maximum number of candidates in the final list, Maxcand, but this initial list of candidates, MaxCandInitialList, is used in the derivation, resulting in zero candidates being added up to MaxCandInitialList, but not up to Maxcand.

[0143] BM merge mode BM merge is a merge mode dedicated to the adaptive decoder-side motion vector refinement method that extends the multi-pass DMVR of ECM. As described in JVET-X2025, this mode corresponds to two merge modes, which refine the MVs in only one direction: one merge mode for L0 and one merge mode for L1. In other words, BM merge is enabled only if the DMVR condition is valid. For these two merge modes, only one list of merge candidates is derived, and all candidates respect the DMVR condition.

[0144] Merge candidates for BM merge mode are derived from spatially adjacent coded blocks, TMVP, non-adjacent blocks, HMVP, and pairwise candidates, similar to the normal merge mode. The difference is that only those that satisfy the DMVR condition are added to the candidates. The merge index is coded in the same way as the normal merge mode.

[0145] AMVP merge mode AVMP merge mode, also known as bidirectional prediction, is defined in JVET-X2025 as follows: It consists of AMVP prediction in one direction and merge prediction in the other direction. This mode is valid for a coding block if the selected merge prediction and AMVP prediction satisfy the DMVR condition, there is at least one reference picture from the past and one reference picture from the future relative to the current picture, and the distances from the two reference pictures to the current picture are the same, then bilateral matching MV refinement is applied to the merge MV candidate and the AMVP MVP as starting points. Otherwise, if the template matching function is enabled, template matching MV refinement is applied to the merge prediction or AMVP prediction with the higher template matching cost.

[0146] The AMVP part of the mode is signaled as normal unidirectional AMVP, i.e. the reference index and MVD are signaled, the derived MVP index if template matching is used, or the MVP index if template matching is disabled.

[0147] For an AMVP direction LX (X is 0 or 1), the merge part of the other direction (1-LX) is implicitly derived by minimizing the bilateral matching cost between the AMVP prediction and the merge prediction, i.e., for a pair of AMVP and merge motion vectors. For all merge candidates in the merge candidate list with other direction (1-LX) motion vectors, the bilateral matching cost is calculated using the merge candidate MV and the AMVP MV. The merge candidate with the smallest cost is selected. Starting from the selected merge candidate MV and the AMVP MV, bilateral matching refinement is applied to the coding block.

[0148] The third pass of the multi-pass DMVR, which is the BDOF refinement of the 8x8 sub-PUs of the multi-pass DMVR, is valid for the AMVP-merge mode coding block.

[0149] The mode is indicated by a flag, and if valid, the AMVP direction LX is further indicated by a flag.

[0150] MVD Code Prediction The code prediction method is described in JVET-X0132. Motion vector differential code prediction can also be applied in normal inter modes if the motion vector differential contains non-zero components. In the current ECM version, it is applied to AMVP, affine MVD, and SMVD modes. The possible MVD code combinations are sorted according to the template matching cost, and the index corresponding to the true MVD code is derived and coded in the context model. On the decoding side, the MVD code is derived as follows: 1 / Parse the magnitude of the MVD components. 2 / Parse the context-coded MVD code prediction index. 3 / Create MV candidates by the combination of possible codes and MVD absolute values, and add them to the MV predictor. 4 / Derive the MVD code prediction cost for each MV based on the template matching cost and sorting. 5 / Select the true MVD code using the MVD code prediction index. 6 / Add the true MVD to the MV predictor for the final MV.

[0151] TIM D Regarding intra prediction, template-based intra mode derivation (TIMD) fusion, JVET-X2025 describes it as follows: For each intra prediction mode of MPM, the SATD between the template prediction sample and the reconstructed sample is calculated. First, the two intra prediction modes with the smallest SATD are selected as the TIMD mode. These two TIMD modes are fused with weights after applying PDPC processing, and such weighted intra prediction is used to code the current CU. Position-dependent intra prediction fusion (PDPC) is included in the derivation of TIMD modes.

[0152] Duplicate Check In Figures 14 and 15, a duplicate check for each candidate is added (1440, 1441, 1442, 1443, 1444, 1445, 1530). However, duplicates are also present for non-adjacent candidates (1540) and history-based candidates (1501). This involves comparing the motion information of the current candidate with index cnt with the motion information of other previous candidates. If this motion information is equal, it is considered a duplicate and the variable cnt is not incremented. Of course, by motion information we mean the mutual orientation, the reference frame index and the motion vector for each list (L0,L1). Note that zero candidates corresponding to different reference frames are not considered as duplicates.

[0153] MVTH In ECM, a motion vector threshold has been added for this overlap check. This parameter modifies the equality check by considering two motion vectors equal if the absolute value of their difference is less than or equal to the motion vector threshold MvTh for each component. In normal merge mode, MvTh is set to 1, which corresponds to the traditional overlap check without a motion vector threshold.

[0154] MvTh is equal to a value that depends on the number of luma samples of the current CU, nbSamples, in template matching normal merge mode defined as follows: if (nbSamples<64) MvTh=1< <MV_FRACTIONAL=16 else if (nbSamples<256) MvTh=2< <MV_FRACTIONAL=32 else MvTh=4< <MV_FRACTIONAL=64

[0155] Here MV_FRACTIONAL corresponds to the inter resolution of the codec, i.e. currently the 16th resolution is used in the ECM, so MV_FRACTIONAL is equal to 4. << is the left shift operator. Also, nbSamples=HeightxWidth (height and width of the current block).

[0156] For example, there is another threshold called MvThBDMVRMvdThreshold, which is used for GEO merge derivation as well as for overlap checks of non-adjacent candidates, which will be described later. if (nbSamples<64) MvThBDMVRMvdThreshold=(1<<MV_FRACTIONAL)> >2=4 else if (nbSamples<256) MvThBDMVRMvdThreshold=(1<<MV_FRACTIONAL)> >1=8 else MvThBDMVRMvdThreshold=(1<<MV_FRACTIONAL)> >0=16

[0157] AMRC In ECM, Adaptive Reordering of Merge Candidates Using Template Matching (AMRC) was added to reduce the number of bits in the merge index. Candidates are reordered based on the cost of each candidate according to the template matching cost calculated as in Figure 13. In this method, only one cost is calculated per candidate. This method is applied only to the first five candidates in the regular merge candidate list after this list is derived. It should be understood that the number five was chosen to balance the complexity of the reordering process with the potential benefits, and a larger number (e.g., all candidates) could be reordered.

[0158] FIG. 18 is an example of this method for a regular merge candidate list containing 10 candidates, such as CTC.

[0159] This method is also applied to the sub-block merging mode except for the spatial candidates, and to the regular TM mode for all four candidates.

[0160] In one proposal, this method was also extended to sort and select candidates to be included in the final merge mode candidate list. For example, in JVET-X0087, all possible non-adjacent candidates (1540) and history-based candidates (1501) are considered together with the temporally non-adjacent candidates to create a candidate list. This candidate list is created without considering the maximum number of candidates. This candidate list is then sorted. Only the correct number of candidates from this list are added to the final merge candidate list. The correct number of candidates corresponds to the first N candidates in the list. In this example, the correct number is the maximum number of candidates minus the number of spatial and temporal candidates already in the final list. In other words, the non-adjacent and history-based candidates are processed separately from the adjacent spatial and temporal candidates. The processed list is used to supplement the adjacent spatial and temporal merge candidates already present in the merge candidate list to generate the final merge candidate list.

[0161] In JVET-X0091, ARMC is used to select one temporal candidate from three temporal candidates: bi-dir, L0, and L1. The selected candidate is added to the merge candidate list.

[0162] In JVET-X0133, merge candidates are selected from multiple temporal candidates sorted using ARMC. Similarly, all neighbor candidates are subjected to ARMC and up to 9 candidates are added to the merge candidate list.

[0163] All these proposed methods use classical ARMC sorting to sort the final list of merge candidates. JVET-X0087 reuses the costs calculated during sorting of non-adjacent and history-based candidates to avoid additional computational costs. JVET-X0133 applies a systematic sorting to all candidates on the final merge candidate list.

[0164] New AMRC Since the initial implementation of ARMC, the method has been added to several other modes: ARMC applies to IBC, MMVD, affine MMVD, CIIP, CIIP with template matching, and BM merge modes, in addition to normal, template matching, and subblock merge modes. Furthermore, the ARMC principle of sorting candidate lists based on template matching cost is also applied to deriving AMVP merge candidates, as well as the intra-method TIMD for selecting the most probable prediction.

[0165] We also add rationale for the sign residual prediction methods of affine MVD, AMVP, and SMVD, and provide additional testing on the use of this reordering for GEO merge mode and template matching, and for reference frame index prediction.

[0166] In addition to this mostly systematic use of ARMC, for some candidates some additional modifications were made to the derivation process.

[0167] For example, the derivation of ECM4.0 includes a cascaded AMRC process for the derivation candidates of normal mode, TM mode, and BM merge mode, as shown in Figure 19 for normal mode and TM merge mode.

[0168] The first 10 temporal locations are checked against the previous merge candidate derivation and, after overlap check, are added to a temporal candidate list. This temporal list can contain up to 9 candidates. Furthermore, there is a specific threshold for this temporal list since the MV threshold is always 1 and is independent of the merge mode compared to the motion threshold used for merge candidate derivation. Based on the list of up to 9 first non-overlapping locations, the ARMC process is applied and only the first temporal candidate is added to the conventional merge candidate list if it is non-overlapping compared to the previous candidate.

[0169] In a similar manner, non-adjacent spatial candidates are derived from among the 59 positions. The initial list of non-overlapping candidates amounts to 18 candidates. However, the motion threshold is different from that used for the temporal derivation and the remaining list, and does not depend on the normal or template merge modes. It is set equal to mv in BDMVR. The maximum 18 non-adjacent candidates are sorted and only the first 9 non-adjacent candidates are kept and added to the merge candidate list. After that, other candidates are added unless the list already contains the maximum number of candidates. Furthermore, in TM merge mode, the maximum number of candidates in the list, MaxCandInitialList, is greater than the maximum number of candidates that the final list can contain, Maxcand.

[0170] Then, the ARMC process is applied to all candidates in the intermediate list, including MaxCandInitialList, as shown in Figure 19. The maximum number of candidates, Maxcand, is set in the final list of candidates.

[0171] Similar modifications apply to the BM merge mode as well as the template matching merge mode.

[0172] ARMC template cost algorithm Figure 17 shows the template cost calculation of the ARMC method. The number of candidates considered in this process, NumMergeCandInList, is greater than or equal to the maximum number Maxcand that the list can contain (1712).

[0173] For each candidate in the list (1701), if the cost was not calculated during the first ARMC process for temporal and non-adjacent candidates (1702), the cost is set equal to 0 (1703). In the implementation, the not calculated cost mergeList[i].cost associated with candidate "i" is set equal to the maximum value MAXVAL. If the head template of the current block is available (1704), the distortion compared to the current block template is calculated (1705) and added to the current cost (1706). Next, or if not, if the left template of the current block is available (1707), the distortion compared to the current block template is calculated (1708) and added to the current cost (1709). Then, the cost of the current merge candidate, mergeList[i].cost, is set equal to the calculated cost (1710) and the list is updated (1711). In this example, we consider the current candidate i to be set to its position in terms of cost compared to the costs of the other candidates. Once all candidates have been advanced, the number of candidates in the list, NumMergeCandInList, is set equal to the maximum number of possible candidates in the list, Maxcand.

[0174] Multiple Hypothesis Prediction (MHP) ECM also adds multiple hypothesis prediction (MHP), which allows up to four motion compensated prediction signals per block (versus two in VVC). These individual prediction signals are superimposed to form an overall prediction signal. The motion parameters for each additional prediction hypothesis can be signaled explicitly by specifying a reference index, a motion vector prediction index, and a motion vector difference, or implicitly by specifying a merge index. A separate multiple hypothesis merge flag distinguishes between these two signaling modes.

[0175] For spatial, non-adjacent, and history-based merging candidates, the multiple hypothesis parameter value "addHypNeighbours" is inherited from the candidate.

[0176] In the case of temporal, zero and paired candidates, the multiple hypothesis parameter value "addHypNeighbours" is not retained (it is explicit).

[0177] LIC In ECM, Local Illumination Compensation (LIC) has been added, which is based on a linear model for illumination changes, which is calculated thanks to the neighboring samples of the current block and the previous block.

[0178] In ECM, LIC is only valid for unidirectional prediction. LIC is signaled by a flag. In merge mode, the LIC flag is not sent, but instead the LIC flag is inherited from the merge candidate in the following way:

[0179] For spatial candidates, non-adjacent merge candidates, and history-based merge candidates, the value of the LIC flag is inherited.

[0180] For temporal candidates and zero candidates, the LIC flag is set to 0.

[0181] For the pair candidates, the value of the LIC flag is set as shown in FIG. 16. This figure is based on FIG. 12, with the addition of modules 1620 and 1621, and the updates of modules 1609, 1612, 1611. The variable average is set to false (1620), and if the average of the pair was calculated for the current list, the LIC flag of the pair LICFlag[cnt] is set to false and the variable averageUsed is set to true (1609). If the candidate only has list motion information (1612-1611), and the average was not used, the LIC flag is updated and set equal to the OR operation of the current value and the value of the LIC flag of the candidate.

[0182] Also, if the pair candidate is Bidir (eg, 3), the LIC flag becomes false.

[0183] However, in the algorithm of FIG. 16, LICflag can be different from true only if two candidates have motion information of one list, and each candidate has its own list. For example, candidate 0 has motion information of only L0, and candidate 1 has motion information of only L1. In this case, the LIC flag can be non-zero, but this does not happen because LIC is only unidirectional. Therefore, the LIC flag of the pair is always false. As a result, the pair candidate cannot use LIC when it is potentially needed. This reduces the efficiency of the candidate and avoids the propagation of LIC to the next coding block, resulting in a reduction in coding efficiency.

[0184] Furthermore, the duplicate check in the ECM software introduces some inefficiencies. As depicted in Figures 14 and 15, each candidate is added to a list and the duplicate check (1440, 1441, 1442, 1443, 1444, 1445, 1530) only affects the increment of the variable cnt (1405, 1409, 1413, 1417, 1423, 1427, 1508). Also, as explained in Figure 16, the variable BCWidx is not initialized for pair candidates. As a result, if the last candidate added to the list is a duplicate candidate, the value of the pair candidate BCWidx is the value of the previous duplicate candidate. This does not happen in VVC, since if a candidate is determined to be a duplicate, the candidate is not added.

[0185] Embodiment As mentioned above, reordering the motion vector prediction candidates in the list generally means that the most likely predictions are placed at the top of the list, which results in fewer bits required for encoding. However, it is pointed out that this does not always happen. The following examples aim to improve the reordering process by defining specific situations or categories of candidates for which no reordering process is used, reduced, and / or secondary reordering process is performed.

[0186] In the following embodiments, the list of candidates or predictions is sorted according to the calculated costs. For example, as described above, some lists of merge candidates or motion vector predictions are sorted according to template cost, similar to intra predictions.

[0187] In the following examples, the list of candidates or predictions can be a list of predictions for deriving intra or inter blocks, or other than predictions. For example, in the above description, the MVD code prediction method sorts the list of possible MVD code indexes. This disclosure focuses primarily on deriving merge candidates.

[0188] Exclude categories of candidates from the sorting process In one example, the list of candidates or predictions is sorted according to the calculated costs. The list derived before sorting can be obtained from previously decoded or coded motion information, for example, from a spatial location, a temporal location, or a spatial / temporal non-adjacent location, or from a list of previously decoded candidates, or from candidates derived from other candidates, or from other decoded or estimated samples. Such candidates are selected based on the likelihood that the corresponding samples are correlated with the sample being coded / decoded.

[0189] The list of candidates will always be less than the maximum number of candidates (Maxcand) that the final list can contain. The maximum number of candidates in the list corresponds to the maximum index value that the decoder can decode. However, the candidate list before reordering may contain more candidates than this maximum number, and the reordering algorithm based on the cost values, as described in Figure 17, may remove additional candidates in this list. Therefore, the maximum number of candidates during derivation may be greater than the final maximum number of candidates after reordering.

[0190] If the candidate list does not reach the maximum number of candidates, some candidates are added.

[0191] The coding efficiency improves when the candidates added to reach the maximum number of candidates are not considered during the reordering process (i.e., they are excluded from the reordering process). This is mainly because if these candidates are duplicates and the reordering process sets them to an earlier position (earlier than the selected / best candidate), multiple consecutive merge indices will be used to signal the same candidate, thus increasing the merge index rate or avoiding the selection of the best candidate. In addition, it is also because these candidates are on average less efficient since they are not correlated with the current block, especially when the list of candidates is large.

[0192] Including these as part of the sorting process is unlikely to provide any benefit and would add unnecessary complexity (especially for large lists).

[0193] In a particularly advantageous case, the zero candidates added to the list are not considered in the sorting process, the advantage of which is reduced complexity since the number of cost calculations is limited.

[0194] An additional advantage of this embodiment is that coding efficiency is improved because zero candidates are often inefficient and occupy the positions of useful candidates when they are sorted, and if they are duplicate zero candidates, several consecutive indices at the beginning or middle of the list represent the exact same candidate, and signaling of other candidates after these candidates will have a higher rate than necessary.

[0195] This scenario has been found to be likely at the beginning of a video sequence.

[0196] The reordering process can be applied only in some coding modes. In a particularly advantageous embodiment, the reordering process is applied for one mode of the full list. The advantage is an increase in coding efficiency. Indeed, if a zero candidate is interesting in terms of coding efficiency, there exists at least one mode in which it can be represented with the least number of bits.

[0197] An example of such a mode where the reordering is not done on the complete list is the sub-block merge mode, so in the normal merge, TM merge, BM merge and IBC merge modes, no reordering is applied to the candidates to fill the list as zero candidates.

[0198] In another example, the modes in which the reordering process is applied to the complete list are those in which the number of candidates is below a threshold (i.e., the number of candidates is small) and / or the list often contains zero candidates, but the number of zero candidates is small (e.g., below a threshold). In other words, for modes in which the number of candidates is above a threshold and / or the number of zero candidates is above a threshold, the reordering process is not performed on the candidates set to fill the list as zero candidates.

[0199] Determining the cost function that defines the sorting process The accuracy of the process of determining the relative "cost" of the candidates at the decoder determines how effective the reordering process is. The following examples provide refinements to the cost determination process that result in more accurate lists, lower cost calculation complexity, or both.

[0200] It is important to note that the algorithm that performs this sorting can be modified to trade off accuracy versus complexity (speed) of the sorting process.

[0201] The following examples are intended to pinpoint relative costs while minimizing the overall complexity of the computations.

[0202] In one example, the sorting is based on cost, which includes a measure between samples: the sample associated with each candidate is compared to another sample and a relative cost is calculated.

[0203] For example, the cost of a candidate can be calculated based on the neighboring samples of its predicted block and the neighboring samples of the current block, which are easily available at the decoder side.

[0204] The cost can be calculated between the samples of the two block predictions corresponding to the candidates in the list. For example, if the candidate is a bi-prediction candidate, the cost can be the distortion between the two block predictions. Bi-prediction candidates are described in more detail below.

[0205] Costs can also be calculated relative to other candidates. For example, another candidate can be the most likely candidate or prediction. The cost of a candidate is calculated based on the samples of that candidate and the samples of the most likely candidate.

[0206] The cost can be calculated on a subset of the neighborhood samples or on a subset of the prediction samples, e.g., if there are multiple adjacent samples that can be used to determine the cost, these samples are sampled to reduce computational complexity.

[0207] The cost can be calculated based on samples corresponding to images at another resolution: high similarity with the higher resolution image is a good sign of low cost (i.e. good prediction).

[0208] The sample values ​​used in the cost calculation can be pre-processed. Given that only the relative cost values ​​are needed (i.e., only the order matters), pre-processing the values ​​means the calculation will be simpler and is unlikely to have a significant impact on the effectiveness of the sorting process. Pre-processing will improve the sorting process by calculating the costs.

[0209] In the above example, the cost could be a measure of distortion such as the sum of absolute differences (SAD), the sum of absolute transformed differences (SATD), the sum of squared errors (SSE), or the structural similarity index measure (SSIM).

[0210] Alternatively, the cost could be a measure of distortion, with a weight applied to this distortion. It could also be a rate or an estimated rate. Or it could be a threshold.

[0211] It is also possible to have weighted costs, with different weights depending on the type of prediction or candidate, or on the initial position in the list of candidates.

[0212] Deciding which candidates to sort (signaling) If some candidates in the list are not reordered, both the encoder and the decoder need to decide which candidates are excluded from reordering (or for which the reordering process is performed). One way to achieve this is to derive, by additional signaling, a variable corresponding to the number of candidates to be reordered.

[0213] In one example, before any candidates added to fill the candidate list are added to the candidate list, a variable numMaxNonZeroCand is set equal to the current number of candidates. The sorting is then limited to this variable value, and not to the maximum number the list can contain. This variable is the difference between the current number of candidates and Maxcand.

[0214] In one embodiment, the variable is coded in the sequence, picture set, picture, and slice level headers. This variable can be set equal to the average of numMaxNonZeroCand for better coding efficiency. For that coding, this variable is preferably the difference between the current number of candidates and Maxcand. This may require fewer bits to code, since the difference (i.e., the number of candidates excluded from reordering) is more likely to be lower than the current number of candidates (i.e., the number of candidates to which reordering is applied).

[0215] If the maximum number of candidates in the list is equal to or greater than the maximum number of candidates the final list can contain, this does not limit the computational cost.

[0216] A variation of this is to specify a variable to identify the first candidate added to fill the list, and all candidates after this variable are unsorted.

[0217] This value is used to avoid sorting zero candidates, or at least to avoid cost calculation of zero candidates.

[0218] Figure 20 shows this embodiment for zero candidates. This figure is based on Figure 15. In this figure, numMaxNonZeroCand is set equal to the current number of candidates before zero candidates are added (2001). Furthermore, in this figure, zero candidates are added up to the maximum number of candidates in the initial list MaxCandInitialList instead of Maxcand (2009)(2010). In fact, the initial candidate list can be larger than the final candidate list of the last ECM version.

[0219] This value numMaxNonZeroCand is then used to avoid computing costs for these candidates (2102), as shown in FIG. 21. FIG. 21 is based on FIG.

[0220] Another implementation is to change the value of NumMergeCandInList by the variable numMaxNonZeroCand in module 2101 or 1701. This is an implementation issue and depends on the initialization of the variables.

[0221] In another example, each candidate added to fill the candidate list (e.g., the zero candidate) is identified by a variable. During the sorting process of all the candidates in the list, this variable is used to apply or not apply a cost calculation. In this way, the same sorting process is applied, but the candidates that are not sorted are ignored because they do not have an associated cost.

[0222] Handling Unsorted Candidates In an advantageous embodiment of these different embodiments, the candidates used to fill the candidate list (e.g., as the zero candidate) are set to the end of the list. In one embodiment, they are ordered according to the order in which they were derived in the initial list. This property results in coding efficiency as explained above. For example, in FIG. 21, if the current number of candidates is better than the value numMaxNonZeroCand (2113) corresponding to the zero candidate, the cost remains equal to the maximum value MAXVAL. As a result, during the candidate list update process 2110, the zero candidate is placed at the end of the list.

[0223] To provide further coding efficiency, a cost is calculated for the candidates that the candidate uses to fill the candidate list (e.g., the zero candidate) and a separate second sorting process is performed on them and they are set to the end of the list. This example improves coding efficiency, especially when the number of such candidates is large.

[0224] In one example, if the list does not contain more than one candidate, excluding the candidates set to fill the list (e.g., zero candidates), then no sorting process is applied. This is shown in module (2114) of FIG.

[0225] Duplicate candidates In one example, predictors that are duplicated in the list are not sorted during the sorting process. One implementation of this is to change the derivation of the candidates. For example, zero candidates are added with a duplicate check (either when all are added or during addition), and the variable numNonDuplicateCand is set equal to the current number of candidates in the list. Then, zero candidates are added to the list without a duplicate check. During sorting, candidates with high indices are not sorted and are placed at the end of the list. For example, in cost calculations, signaling, handling of non-sorted candidates, etc.

[0226] The advantage of this is that it improves coding efficiency by not using multiple consecutive indices at the beginning of the list, which is particularly inefficient when the cost of the zero candidate is small. Such scenarios are surprisingly common, and this change has a large impact on coding efficiency, especially for short video sequences.

[0227] Depending on the encoding mode, the reordering process can be changed dynamically, for example, except in the unreorder duplicate candidates only mode, the candidates used to fill the list are not reordered within the list.

[0228] Template matching cost calculation The following examples solve some problems related to the availability of templates for cost computation. These examples are mainly specific to template matching methods. The availability of a template means that the template is outside the reference frame or outside a bounded region. A bounded region can be defined to reduce memory accesses for such methods.

[0229] Figure 22 shows three cases where the prediction template is unavailable. In this figure, 2201, 2202, 2203 represent the boundaries of a reference frame or a bounded region. In the first case, the left template (2204) is unavailable, but the top template is available. In the second case, the top template of the prediction block is outside the reference frame or restricted region (2202), and the left template is available (2207).

[0230] If a template is missing, its associated cost defaults to 0, so that these candidates have a higher chance of being selected compared to other candidates.

[0231] In the third case, both templates (2208, 2209) are unavailable. Candidates related to these cases should have block predictions outside the reference frame, and as a result, they will generally generate worse block predictions compared to blocks coming from the true reference frame. Indeed, if a sample is outside the current frame, the missing sample is replaced by a sample from the border or column of the reference frame.

[0232] To improve this problem of calculating the cost of a non-existent template, if one template of the prediction block is not present, the cost that cannot be calculated is replaced by a value based on the cost value of the associated available template. The advantage of this example is an increase in coding efficiency. In fact, the cost corresponds to the closest value to the value obtained when both templates are available. Compared to existing methods, the candidate in this case is not set to an earlier position since its associated cost has been calculated on fewer samples than the other candidate.

[0233] Alternatively, the cost can be calculated based on the size of the template, e.g., proportional to the size of the template. The advantage of this is that the cost of unavailable templates will be statistically closest to the actual cost, making the comparison with the costs of other candidates fairer. This method may also be computationally simpler.

[0234] This proportionality may depend on the number of samples in the available template and the number of samples in the unavailable template. For example, in case 1 of Figure 22, the cost of the unavailable left template (CAO4), costLeft, is set equal to the cost of the up template (2205), costUP, divided by the width of the block multiplied by the height: costLeft=(costUp / width)xheight So the cost of the current candidate is: cost=costUp+(costUp / width)xheight

[0235] Figure 22 shows the specific cases 1 and 2, but there may also be cases where only parts of the template are missing, in which case we can compute partial costs over all available samples.

[0236] In such cases, the proportionality depends on the number of available and unavailable samples. For example, given the number of available samples, numAvailableS, the number of unavailable samples, numNonAvailableS, and the cost of available samples, costAvailableS, the cost of a candidate is: cost=costAvailableS+(costAvailableS / numAvailableS)xnumNonAvailableS

[0237] This can be applied to determine the cost of one template or all templates.The advantage compared to the previous embodiment is that it allows a better estimation of the actual cost of the candidates.

[0238] To reduce complexity, the proportionality is calculated with shift operations, i.e. the shift operation closest to the actual calculation can be used, e.g. the denominator is approximated to the nearest power of 2. The advantage of this is that it simplifies the implementation, especially in hardware, by not using a division (shift operations are less complex than division).

[0239] In the alternative, the cost value of the unavailable template is set equal to the cost value of the available template. For example, for case 1 in Figure 22, the cost is: cost=costUp+costUp

[0240] this is, cost=costUp<<1 where << is a shift operation. This is mathematically cost=costUpx2 is equivalent to

[0241] Since most of the blocks are square, the impact on coding efficiency is minor, but it has the advantage of allowing further simplification.

[0242] It has been pointed out that for unavailable templates, the cost may be underestimated. To remedy this, a weight w can be applied to the formula that penalizes the estimated cost of a candidate. In this case, the basic formula for calculating the cost is: cost=costUp+wx(costUp / width)xheight

[0243] where w is greater than 1 to penalize the estimated costs.

[0244] The advantage of this embodiment is improved coding efficiency. Indeed, if a template is missing, the associated block prediction is likely to be less relevant. This weighting may result in a high cost, placing the candidate at the bottom of the final list.

[0245] In the above examples, we considered top and left templates with one line of samples like ECM. However, longer templates or more templates can be considered. An example would be a top right template or a bottom left template. Similarly, further line or row templates can be considered to calculate the cost. Also, bi-prediction can be considered for each of the above examples.

[0246] The third case in Figure 22 shows when there is no template for the current prediction (2208, 2209). In this case, any cost can be calculated.

[0247] If there is no template available for prediction, the value is set equal to the maximum value, which ensures that the relevant candidate is not set to the top of the list.

[0248] The advantage of this is that the coding efficiency is improved since these candidates are not at the beginning of the list compared to when the cost is set to 0. In reality, these candidates are less efficient since they contain estimated samples rather than true samples.

[0249] In a variant, the maximum is less than the maximum set on the candidates used to fill the list (e.g., zero candidates), and in fact these candidates are more efficient than such candidates that are added to satisfy a requirement to have a certain number of candidates.

[0250] In another variation, candidates for which no template is available are placed at the end of the list, before any candidates added to satisfy the requirement of having a certain number of candidates (e.g., the zero candidate). In practice, this can be achieved by setting their cost values ​​equal to the maximum and highest values ​​of the zero candidate, as described above.

[0251] In another example, if there is no template, the cost of the candidate is set equal to the maximum value so that the candidate is at the end of the list. The advantage of this is reduced complexity, since there is no need to calculate the cost for the available templates and no cost adaptation is required. Therefore, the coding efficiency is lower, but the complexity is lower, compared to the conventional embodiment with one available template.

[0252] Advantageously, the maximum is inferior to the maximums set on the candidates used to fill the list (e.g., the zero candidate), and in fact these candidates are more efficient than such candidates.

[0253] In that case, when there are multiple candidates, these candidates will keep the same order as they were in the initial list, i.e. if candidate 1 was before candidate 2 in the initial list, then candidate 1 will be before candidate 2 in the sorted list.

[0254] If one candidate is missing only one template and other candidates are missing more, the candidate with only one missing template will be set before the candidate with all the missing templates.

[0255] Sorting restrictions based on the presence or absence of templates Figure 23 shows three cases when no template is available for the current block (case 6) or when the template is partially available (cases 4 and 5). These cases are surprisingly common for low bitrates, small sequences or small slices.

[0256] In one example, if at least one template for the current block is not available, the list is not reordered, which can be achieved by not calculating the cost for any candidate.

[0257] The advantage is reduced complexity and small impact on coding efficiency.

[0258] Increase the number of temporal candidates to improve the case where at least one template is not available. For example, in FIG. 19, the number of temporal candidates is not 1, but a larger value. It can be, for example, all temporal candidates. In the current ECM implementation example, this results in 9 temporal candidates. Note that in this case, in the first ARMC process, there is no AMRC for temporal candidates and non-adjacent candidates.

[0259] Furthermore, the number of temporal positions used to derive the temporal candidates can be increased so that additionally more diverse temporal candidates can be derived.

[0260] Alternatively, the number of temporal candidates is the same as the number of templates available.

[0261] The advantage of this embodiment is that when no template is available (case 6), the coding efficiency is improved because there are no spatial candidates, history-based templates, or non-adjacent candidates. Also, when there is only one temporal candidate, no pair candidate can be derived. Therefore, adding more temporal candidates improves the coding efficiency. This is especially efficient for low bit rates, small sequences, or small slices such as cases 4, 5, and 6.

[0262] In one alternative or addition, the maximum number of candidates for those blocks is reduced. If only temporal considerations are taken into account, the maximum number of candidates is, for example, 1. In that case, there is no cost associated with the current block.

[0263] Additionally, when none are available temporally because temporal candidates are prohibited, or for the first inter frame of a GOP, some modes can be avoided such as all inter modes without motion vector residuals as merge modes.

[0264] The advantage of this is that the coding efficiency of the associated blocks is increased.

[0265] In a further example, if no reordering is possible, the motion vector threshold used is modified: since there are fewer possible spatial candidates, the motion vector threshold is lowered (if possible) to get more candidates in the list. This allows additional candidates to be added to the list without the possibility of them being considered duplicates and removed (replacing zero candidates).

[0266] This improves coding efficiency since the list contains more candidates and fewer less useful zero candidates.

[0267] Bidirectional candidate If the candidates are bidirectional, there are two possibilities to calculate the cost associated with the templates. The first possibility is to calculate the cost of each template independently for the first and second lists and then sum these costs. If unidirectional prediction is possible in the candidate set, the cost is divided by two.

[0268] The second possibility is to apply bidirectional computation to obtain only one template each for the top and left templates, as in this example, and to compute the costs of these bidirectional templates with the template of the current block. This second possibility is particularly efficient when the BCW (CU-level weighted bidirectional prediction) method is available.

[0269] FIG. 24 illustrates bidirectional prediction when the left template of the L0 block predictor is unavailable.

[0270] In one embodiment, the cost of each template is calculated independently for bidirectionally predicted candidates, and if a template is not available for one direction but is available for a second direction, the cost of the available template is used to replace the cost of the unavailable template. For example, in FIG. 24, if the cost costLeftL0 cannot be calculated, the final cost is set equal to: cost=costLeftL1<<1+costTopL0+costTopL1 Here, costLeftL1 is the cost of the left template in L1 (2406), costTopL0 is the cost of the top template in L0, and costTopL1 is the cost of the top template in L1. Also, << is the left shift operator. That is, costLeftL1<<1 means that costLeftL1 is multiplied by 2.

[0271] For reference, if all templates are available, the cost is calculated using the following formula: cost=costLeftL0+costLeftL1+costTopL0+costTopL1

[0272] The advantage is improved coding efficiency.

[0273] In a related example, if one template is partially available, the cost is calculated for the available samples, and the cost of the unavailable template corresponds to the cost of the associated available template, the advantage of which is that it is possible to calculate a cost that is closest to the actual cost.

[0274] In a similar manner as above, the penalty may be applied to a cost in lieu of the current cost or to the final cost.

[0275] It is understood that many more templates, rows and columns for the templates are possible.

[0276] In a related but different example, if the candidates are bidirectional and one template is unavailable in one direction but available in a second direction, the cost is calculated on the available templates without deriving the bidirectional prediction. For example, in FIG. 24, if the left template of L0 is unavailable, only the unidirectional prediction of the left template is considered to calculate the cost of the left template.

[0277] The advantage is improved coding efficiency.

[0278] Similar to above, if one template is partially available, bidirectional prediction is calculated for samples available in both directions, and if not available in the second direction, unidirectional prediction is considered for samples available in one direction.

[0279] The advantage is that it is possible to calculate a cost that is closest to the actual cost.

[0280] In a similar manner as above, the penalty may be applied to a cost in lieu of the current cost or to the final cost.

[0281] It is also possible to consider more templates, rows and columns as templates.

[0282] All of these embodiments may be combined unless expressly stated otherwise, and in fact many combinations may be synergistic, resulting in efficiency gains greater than the sum of their parts.

[0283] Implementation of the Invention Fig. 25 shows a system 191, 195 including at least one of the encoder 150 or the decoder 100 and a communication network 199 according to an embodiment of the present invention. According to an embodiment, the system 195 is for processing and providing content (e.g. video and audio content for displaying / outputting or streaming the video and audio content) to a user having access to the decoder 100, for example via a user interface of a user terminal constituting the decoder 100 or capable of communicating with the decoder 100. Such a user terminal may be a computer, a mobile phone, a tablet or any other type of device capable of providing / displaying (provided / streamed) content to a user. The system 195 obtains / receives the bitstream 101 (in the form of a continuous stream or signal - for example while a previous video / audio is being displayed / output) via the communication network 199. According to an embodiment, the system 191 is for processing content and storing the processed content, for example the processed video and audio content for later displaying / outputting / streaming. The system 191 obtains / receives content including an original image sequence 151, which is received and processed (including filtered by a deblocking filter according to the invention) by an encoder 150, which generates a bitstream 101 to be communicated to the decoder 100 via a communication network 191. The bitstream 101 is then communicated to the decoder 100 in several ways, for example it may be pre-generated by the encoder 150 and stored as data in a storage device (e.g. on a server or cloud storage) in the communication network 199 until a user requests the content (i.e. bitstream data) from the storage device, at which point the data is communicated / streamed from the storage device to the decoder 100.The system 191 may also comprise a content providing device for providing / streaming to the user (e.g., by communicating data for a user interface to be displayed on the user terminal) content information (e.g., title of the content and other meta / storage location data for identifying, selecting and requesting the content) of the content stored in the storage device and for receiving and processing user requests for content so that the requested content is delivered / streamed from the storage device to the user terminal. Alternatively, the encoder 150 generates the bitstream 101 and communicates / streams it directly to the decoder 100 when the user requests the content. The decoder 100 then receives the bitstream 101 (or signal) and performs filtering with a deblocking filter according to the present invention to obtain / generate a video signal 109 and / or an audio signal, which is used by the user terminal to provide the requested content to the user.

[0284] Any step of the method / process according to the present invention or function described herein may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the step / function is stored or transmitted as one or more instructions, codes, programs, or computer readable media and executed by one or more hardware-based processing units, such as a PC ("personal computer"), a DSP ("digital signal processor"), a programmable computing machine, which may be a circuit, processor and memory, a general-purpose microprocessor or central processing unit, a microcontroller, an ASIC ("application specific integrated circuit"), a field programmable logic array (FPGA), or other equivalent integrated or discrete logic circuit. Thus, the term "processor" as used herein may refer to any of the aforementioned structures, or other structures suitable for implementing the techniques described herein.

[0285] The embodiments of the present invention may also be implemented by a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described herein to illustrate functional aspects of a device / apparatus configured to perform the embodiments, but need not necessarily be implemented by different hardware units. Rather, the various modules / units may be combined into a codec hardware unit or provided by a collection of interoperating hardware units including one or more processors in combination with appropriate software / firmware.

[0286] The embodiments of the present invention may be realized by a computer of a system or apparatus that reads and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium to perform the modules / units / functions of one or more of the above-mentioned embodiments, and / or includes one or more processing units or circuits for performing one or more functions of the above-mentioned embodiments, and may also be realized by a method implemented by the computer of the system or apparatus, e.g., by reading and executing computer executable instructions from a storage medium to perform one or more functions of the above-mentioned embodiments, and / or by controlling one or more processing units or circuits for performing one or more functions of the above-mentioned embodiments. The computer may include a separate computer or a network of separate processing units for reading and executing the computer executable instructions. The computer executable instructions may be provided to the computer from a computer readable medium, e.g., a communication medium via a network or a tangible storage medium. The communication medium may be a signal / bit stream / carrier wave. The tangible storage medium is a "non-transitory computer-readable storage medium" and may include, for example, one or more of a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), a Blu-ray disk (BD)™), a flash memory device, a memory card, etc. At least some of the steps / functions may also be implemented in hardware by machines or dedicated components such as an FPGA ("field programmable gate array") or an ASIC ("application specific integrated circuit").

[0287] Fig. 26 is a schematic block diagram of a computing device 3600 for implementing one or more embodiments of the present invention. The computing device 3600 can be a device such as a microcomputer, a workstation, or a lightweight portable device. The computing device 3600 comprises a communication bus connected to: - a central processing unit (CPU) 3601, such as a microprocessor; - a random access memory (RAM) 3602 for storing executable codes of the method according to the embodiments of the present invention and registers for recording variables and parameters required for implementing the method of encoding or decoding at least a part of an image according to the embodiments of the present invention, the memory capacity of which can be expanded, for example, by an optional RAM connected to an expansion port; - a read only memory (ROM) 3603 for storing computer programs for implementing the embodiments of the present invention; - a network interface (NET) 3604, typically connected to a communication network through which digital data to be processed is transmitted and received. The network interface (NET) 3604 can be a single network interface or can be composed of a collection of different network interfaces (for example, a wired interface and a wireless interface, or different types of wired or wireless interfaces). Data packets are written to the network interface for transmission or read from the network interface for reception under the control of a software application executed by the CPU 3601. The executable code can be stored either in the ROM 3603, in the HD 3606 or in a removable digital medium, such as a disk. According to a variant, the executable code of the program can be received by the communication network, via the NET 3604, to be stored in one of the storage means of the communication device 3600, such as the HD 3606, before being executed. The CPU 3601 is adapted to control and direct the execution of the program or parts of the program's instructions or software code according to an embodiment of the invention, these instructions being stored in one of the aforementioned storage means.After power-up, CPU 3601 can execute instructions from main RAM memory 3602 associated with software applications after those instructions have been loaded, for example, from program ROM 3603 or HD 3606. Such software applications, when executed by CPU 3601, cause the steps of the method according to the present invention to be performed.

[0288] It is also understood that according to another embodiment of the invention, the decoder according to the aforementioned embodiment is provided in a user terminal such as a computer, a mobile phone (cellular telephone), a table or any other type of device (e.g. a display device) capable of providing / displaying content to a user. According to yet another embodiment, the encoder according to the aforementioned embodiment is provided in an image capture device, also comprising a camera, a video camera or a network camera (e.g. a closed circuit television or a video surveillance camera) that captures and provides content for the encoder to encode. Two such examples are given below with reference to Figures 37 and 38.

[0289] FIG. 27 is a diagram showing a network camera system 3700 including a network camera 3702 and a client device 202 .

[0290] The network camera 3702 includes an imaging unit 3706 , an encoding unit 3708 , a communication unit 3710 , and a control unit 3712 .

[0291] The network camera 3702 and the client device 202 are connected to each other via the network 200 so that they can communicate with each other.

[0292] The imaging unit 3706 includes a lens and an image sensor (e.g., a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS)) to capture an image of an object and generate image data based on the image. The image may be a still image or a moving image.

[0293] The encoding unit 3708 encodes the image data using the encoding method described above.

[0294] The communication unit 3710 of the network camera 3702 transmits the encoded image data encoded by the encoding unit 3708 to the client device 202 .

[0295] Furthermore, the communication unit 3710 receives commands from the client device 202. The commands include commands for setting parameters for encoding in the encoding unit 3708.

[0296] The control unit 3712 controls other units within the network camera 3702 according to commands received by the communication unit 3712 .

[0297] The client device 202 includes a communication unit 3714 , a decryption unit 3716 , and a control unit 3718 .

[0298] The communication unit 3714 of the client device 202 transmits a command to the network camera 3702 .

[0299] Furthermore, the communication unit 3714 of the client device 202 receives the encoded image data from the network camera 3712 .

[0300] The decoder 3716 decodes the encoded image data using any of the decoding methods described above, or a combination of the decoding methods described above.

[0301] The control unit 3718 of the client device 202 controls other units within the client device 202 in response to user operations or commands received by the communication unit 3714 .

[0302] The control unit 3718 of the client device 202 controls the display device 2120 to display the image decoded by the decoding unit 3716 .

[0303] In addition, the control unit 3718 of the client device 202 controls the display device 2120 to display a GUI (graphical user interface) for specifying parameter values ​​of the network camera 3702, and also displays the encoding parameters of the encoding unit 3708.

[0304] Furthermore, the control unit 3718 of the client device 202 controls other units within the client device 202 in response to a user's operation input to the GUI displayed by the display device 2120 .

[0305] The control unit 3718 of the client device 202 controls the communication unit 3714 of the client device 202 to send a command specifying a parameter value for the network camera 3702 to the network camera 3702 in response to a user's operational input to the GUI displayed by the display device 2120.

[0306] FIG. 28 is a diagram showing a smartphone 3800.

[0307] The smartphone 3800 includes a communication unit 3802, a decoding unit 3804, a control unit 3806, and a display unit 3808.

[0308] The communication unit 3802 receives the encoded image data via the network 200 .

[0309] The decoding unit 3804 decodes the encoded image data received by the communication unit 3802 .

[0310] The decoding / encoding unit 3804 decodes / encodes the encoded image data using the decoding method described above.

[0311] The control unit 3806 controls other units within the smartphone 3800 in response to a user's operation or a command received by the communication unit 3806 .

[0312] For example, the control unit 3806 controls the display unit 3808 to display the image decoded by the decoding unit 3804. The smartphone 3800 may also include a sensor 3812 and an image recording device 3810. In this manner, the smartphone 3800 can record an image and encode the image (using the methods described above).

[0313] The smartphone 3800 can then decode the encoded image (as described above) and display it via the display device 3808 or transmit the encoded image to other devices via the communication device 3802 and the network 200.

[0314] Alternatives and fixes Although the present invention has been described with reference to the embodiments, it should be understood that the present invention is not limited to the disclosed embodiments. Those skilled in the art will understand that various changes and modifications can be made without departing from the scope of the present invention as defined in the appended claims. All features disclosed in this specification (including the appended claims, abstract and drawings) and / or all steps of any method or process so disclosed can be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. Each feature disclosed in this specification (including the appended claims, abstract and drawings) can be replaced with an alternative feature serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is an example of a generic series of equivalent or similar features.

[0315] It will also be understood that the result of the above-mentioned comparison, determination, evaluation, selection, execution, performing, or consideration, e.g., a selection made during an encoding or filtering process, may be indicated or determinable / inferable in data in the bitstream, e.g., a flag or data indicating the result, and the indicated or determined / inferred result may be used in processing, e.g., during a decoding process, instead of actually performing the comparison, determination, evaluation, selection, execution, performing, or consideration.

[0316] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

[0317] Any reference numerals appearing in the claims are for illustrative purposes only and shall not be construed as limiting the scope of the claims.

Claims

1. 1. A method for generating a list of candidate motion vector predictors for predicting motion in an image portion, comprising: adding a first set of motion vector prediction candidates to the list; adding a second set of motion vector prediction candidates to the list if the number of the first set of motion vector prediction candidates is less than a maximum number of candidates, such that a total number of candidates is equal to the maximum number of candidates; Sorting said list of candidates; Including, The second set of candidates is excluded from the permutation. method.

2. The first set does not include duplicates. The method of claim 1.

3. 1. A method for generating a list of candidate motion vector predictors for predicting motion in an image portion, comprising: adding a first set of motion vector prediction candidates to the list; determining duplicate candidates in the list; adding a second set of motion vector prediction candidates to the list if the number of the first set of motion vector prediction candidates is less than a maximum number of candidates, such that a total number of candidates is equal to the maximum number of candidates; Sorting said list of candidates; Including, The duplicate candidates are excluded from the sorting. method.

4. The second set of candidates is also excluded from the sorting. The method according to claim 3.

5. The second set of motion vector predictions are zero candidates and the first set of candidates are not zero candidates.

5. The method according to any one of claims 1 to 4.

6. The first set of candidates are candidates derived from at least one previously decoded or encoded motion information.

6. The method according to any one of claims 1 to 5.

7. The first candidate set of candidates are candidates derived from true samples corresponding to one or more of: temporal candidates, spatial candidates, historical candidates, previously used candidates, or candidates derived from other candidates. The method according to claim 6.

8. The sorting is based on the calculated relative costs of the candidates.

8. The method according to any one of claims 1 to 7.

9. At least one candidate is derived from at least one spatially or temporally matched template; Templates inside the delimited area are available and templates outside the delimited area are unavailable; If at least one template is unavailable, calculate a non-zero cost for the candidate. The method according to claim 8.

10. 1. A method for generating a list of candidate motion vector predictors for predicting motion in an image portion, comprising: adding a plurality of motion vector prediction candidates to a list; calculating a cost associated with at least one candidate in the list; the at least one candidate is derived from at least one spatially or temporally consistent template; a template inside a delimited area for the image portion is available and a template outside the delimited area is unavailable; if at least one template is unavailable, calculating a non-zero cost for the candidate; Sorting the list in dependence on the calculated costs; A method comprising:

11. The cost is determined based on the size of the template or templates from which the candidates are derived.

11. The method according to claim 9 or 10.

12. The cost is determined based on the number of samples of the available template and the number of samples of the unavailable template.

12. The method according to any one of claims 9 to 11.

13. The cost is determined based on the ratio of available samples to unavailable samples. The method of claim 12.

14. The division used in determining the cost is approximated by a shift operation.

14. The method according to any one of claims 11 to 13.

15. A cost value for the unavailable template is set equal to the cost value of the associated available template.

11. The method according to any one of claims 9 to 10.

16. If no template is available, the cost is set to the maximum value.

24. The method according to any one of claims 9 to 23.

17. If the number of added motion vector prediction candidates is less than a maximum number of candidates, candidates without an associated available template are reordered in the list immediately above the added motion vector prediction candidates so that the total number of candidates is equal to the maximum number of candidates.

17. The method of claim 16.

18. The maximum value is less than a cost assigned to the added motion vector prediction candidates, so that if the number of added motion vector prediction candidates is less than the maximum number of candidates, the total number of candidates is equal to the maximum number of candidates.

18. The method of claim 16 or 17.

19. The cost of the candidate is assigned the maximum value.

19. The method according to any one of claims 9 to 18.

20. At least one candidate is a bidirectional candidate having two spatially matched templates; The cost of the available templates is used to replace the cost of the unavailable templates when determining the cost of the bidirectional candidates.

20. The method according to any one of claims 9 to 19.

21. At least one candidate is a bidirectional candidate having two spatially matched templates; The cost of the bidirectional candidate is calculated using only the available templates.

20. The method according to any one of claims 9 to 19.

22. If a template is partially available, the cost is calculated for the available sample or the unavailable template cost corresponding to the associated available template cost.

22. The method of claim 20 or 21.

23. and applying a weight to the calculated cost of the candidate for which a template is not available.

23. The method according to any one of claims 9 to 22.

24. 1. A method for generating a list of candidate motion vector predictors for predicting motion in an image portion, comprising: adding a plurality of motion vector prediction candidates to a list; At least one candidate in the list is derived from at least one spatially or temporally matched template; a template inside a delimited area for the image portion is available and a template outside the delimited area is unavailable; reordering said list unless at least one template is unavailable; A method comprising:

25. The sorting is performed unless all templates are unavailable.

25. The method of claim 24.

26. 1. A method for generating a list of candidate motion vector predictors for predicting motion in an image portion, comprising: determining availability of at least one template associated with the template of the image portion; deriving the motion vector prediction candidates according to template availability; A method comprising:

27. If at least one template is unavailable, the method includes adding at least one further temporal candidate to the list.

27. The method of claim 26.

28. The number of temporal candidates is the same regardless of template availability.

28. The method of claim 26 or 27.

29. Including reducing the maximum number of candidates 29. The method of any one of claims 26 to 28.

30. Reducing the motion vector threshold 30. The method of any one of claims 26 to 29.

31. A cost of a candidate in the list is calculated based on a comparison measure of at least one sample associated with the candidate and at least one other sample.

31. The method according to any one of claims 8 to 30.

32. The cost of the candidate is calculated based on the difference between the adjacent samples of the prediction block and the adjacent samples of the current block.

32. The method of claim 31 .

33. The cost of a candidate is calculated by computing the difference between the two block predictions.

32. The method of claim 31 .

34. The cost of a candidate is calculated by calculating its difference from other candidates in the list.

32. The method of claim 31 .

35. The other candidate is the most likely candidate.

35. The method of claim 34.

36. The cost is based on a sub-sampling of the neighborhood or samples of the prediction.

32. The method of claim 31 .

37. The cost is based on samples corresponding to images at other resolutions.

32. The method of claim 31 .

38. The sample values ​​used to calculate the cost are pre-processed.

38. The method of any one of claims 31 to 37.

39. The cost corresponds to the distortion 39. The method of any one of claims 31 to 38.

40. The distortion is SAD, SATD, SSE or SSIM.

40. The method of claim 39.

41. The cost includes a weight 41. The method of any one of claims 8 to 40.

42. The weights differ among the motion vector prediction candidates.

42. The method of claim 41.

43. deriving a variable corresponding to a number of motion vector prediction candidates in the first set; Sorting the list is performed depending on the variables.

43. The method of any one of claims 1 to 42.

44. The variable identifies a first candidate from the second set of motion vector prediction candidates.

44. The method of claim 43.

45. Each motion vector prediction candidate in the first set is associated with a variable; The sorting is performed depending on the variables. The method according to claim 1 or 2.

46. and setting the non-reordered motion vector prediction candidates to the end of the list.

46. ​​The method of any one of claims 1 to 45.

47. performing a second reordering process on the unreordered motion vector prediction candidates.

47. The method of claim 46.

48. performing a second reordering process on the unreordered motion vector prediction candidates if the first set does not include more than one candidate.

47. The method of claim 46.

49. and performing a second reordering process on the non-reordered motion vector prediction candidates depending on the coding mode.

47. The method of claim 46.

50. The second sorting is not applied to the sub-block merging mode.

50. The method of claim 49.

51. performing a second reordering process on the unreordered motion vector prediction candidates if the mode has a number of candidates that exceeds a threshold.

47. The method of claim 46.

52. A method of encoding image data into a bitstream, the method comprising generating a list of motion vector prediction candidates according to any one of claims 1 to 51.

53. A method of decoding image data from a bitstream, comprising generating a list of motion vector prediction candidates according to any one of claims 1 to 51.

54. 52. An apparatus for encoding image data into a bitstream, the apparatus being configured to perform a method for generating a list of motion vector prediction candidates according to any one of claims 1 to 51.

55. 52. An apparatus for decoding image data from a bitstream, the apparatus being configured to perform a method for generating a list of motion vector prediction candidates according to any one of claims 1 to 51.

56. A computer program which, when executed, performs the method of any one of claims 1 to 51.

57. 57. A computer readable carrier medium having stored thereon a computer program according to claim 56.

Images