HMVC for Affine and SBTMVP Motion Vector Prediction Modes

Abstract

An apparatus for encoding or decoding blocks of a current image encodes sub-blocks of a first block of the current image, the sub-blocks being encoded or decoded based on a motion vector determined according to motion information data associated with the first block. In a second step, a set of second motion information data is determined as a function of the first motion information data. These second motion information data are added to a list of motion information data used to determine motion information data of further blocks of the current image to be encoded or decoded.

Description

[Technical Field]

[0001] Technical field At least one of these embodiments relates in general to a method or apparatus for video encoding or decoding, and more specifically to a method or apparatus for encoding and decoding blocks of a video image based on motion vectors determined from pre-encoded or decoded blocks. [Background technology]

[0002] background To achieve high compression efficiency, image and video coding schemes typically utilize the spatial and temporal redundancy of video content using prediction and transformation, including motion vector prediction. Generally, intra- or inter-frame prediction is used to utilize intra-frame or inter-frame correlation, and then the difference between the original and predicted images—often referred to as prediction error or prediction residual—is transformed, quantized, and entropy-encoded. To reconstruct the video, the compressed data is decoded by the reverse process corresponding to entropy coding, quantization, transformation, and prediction.

[0003] Recent advancements in high-compression techniques include the use of motion models based on affine modeling and / or subblock-based temporal motion vector predictors (SbTMVP). In particular, these models are used for motion compensation in the encoding and decoding of video images. Generally, affine modeling is a model that uses at least two parameters, for example, two control-point motion vectors (CPMVs) representing the motion at each corner of an image block, to derive a motion field for the entire image block, allowing for the simulation of, for example, rotation and similarity transformations (zoom). The motion field is generally discretized within a set of motion vectors associated with subblocks of the block.

[0004] Recent developments in this field include the use of history-based motion vector prediction (HMVP) methods, where HMVP candidates are defined as motion information for pre-encoded blocks. During the encoding / decoding process, a table containing multiple HMVP candidates is maintained. Whenever there are inter-encoded blocks that are not based on subblocks, the associated motion information is added to the last entry in the table as a new HMVP candidate. These candidates can then be used to encode / decode further blocks, particularly adjacent blocks. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, since only inter-encoded blocks that are not based on subblocks contribute to the HMVP list, if a block to be encoded or decoded is surrounded by inter-encoded blocks that are based on subblocks, this block cannot benefit from the motion vectors of the previously encoded / decoded blocks. There is a lack of solutions to this problem. [Means for solving the problem]

[0006] overview The shortcomings and deficiencies of the prior art are addressed by general embodiments described herein that concern storing motion information relating to inter-encoded blocks based on subblocks for encoding and decoding further blocks. A method is provided according to a first embodiment. The method includes the step of decoding a subblock of a first block of a current image. The subblock is decoded based on a motion vector determined according to first motion information data associated with the first block. The method further includes determining a set of second motion information data as a function of the first motion information data and adding the set of second motion information data to a list of motion information data. Items of the list may be used to determine the motion information data of a second block of the current image.

[0007] A second method is provided according to another embodiment. This method includes the step of encoding a subblock of a first block of the current image. The subblock is encoded based on a motion vector determined according to first motion information data associated with the first block. This method further includes determining a set of second motion information data as a function of the first motion information data, and adding the set of second motion information data to a list of motion information data. Items of the list may be used to determine the motion information data of a second block of the current image.

[0008] The apparatus is provided according to another embodiment. The apparatus includes a processor. The processor may be configured to encode a current block of images of a video or decode a bitstream by performing one of the methods described above.

[0009] In another general aspect of at least one embodiment, a device is provided comprising an apparatus according to any of the decoding embodiments, and at least one of the following: (i) an antenna configured to receive a signal containing the decoded blocks of an image; (ii) a band limiter configured to limit the received signal to a frequency band containing the decoded blocks of an image; and (iii) a display configured to display an output representing the decoded blocks of an image.

[0010] A non-temporary computer-readable medium is provided that includes data content generated according to either the encoding embodiment or the modified form described above, according to another general aspect of at least one embodiment.

[0011] A signal is provided that includes video data generated according to either the above-described encoding embodiment or the modified form, in accordance with another general aspect of at least one embodiment.

[0012] According to another general aspect of at least one embodiment, the bitstream is formatted to include data content generated according to either the encoding embodiment or the modified form described above.

[0013] According to another general aspect of at least one embodiment, there is provided a computer program product including instructions that, when executed by a computer, cause the computer to execute any of the above-described decoding embodiments or variants.

[0014] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] Shows an encoder. [Figure 2] Shows a block diagram of a video decoder. [Figure 3] Shows a block diagram of an example of a system in which various aspects and embodiments are implemented. [Figure 4] Shows a coding tree and a coding tree unit (CTU) structure used to represent a compressed image, for example, in accordance with the HEVC video compression standard. [Figure 5A] Shows an affine motion vector field based on a 4×4 sub-block of two control points. [Figure 5B] Shows an affine motion vector field based on a 4×4 sub-block of three control points. [Figure 7] Schematically shows a method 70 for encoding / decoding a block of a current image. DETAILED DESCRIPTION OF THE INVENTION

[0016] DETAILED DESCRIPTION The general aspects described herein pertain to the field of video compression. These aspects are aimed at improving compression efficiency compared to existing video compression systems.

[0017] This application describes various embodiments, including tools, features, embodiments, models, approaches, etc. Many of these embodiments are described specifically, and often in a way that seems restrictive, at least to illustrate individual features. However, this is for illustrative purposes only and does not limit the uses or scope of these embodiments. In practice, all of the different embodiments can be combined and interchangeable to provide further embodiments. Furthermore, these embodiments can be combined and interchangeable with embodiments described in previous applications.

[0018] The embodiments described and discussed in this application can be implemented in many different forms. Figures 1, 2, and 3 below illustrate some embodiments, but other embodiments are also conceivable, and the discussion relating to Figures 1, 2, and 3 does not limit the scope of implementation forms. At least one embodiment generally relates to video encoding and decoding, and at least one other embodiment generally relates to the transmission of a generated or encoded bitstream. These and other embodiments can be implemented as methods, apparatus, computer-readable storage media storing instructions for encoding or decoding video data according to any of the above-described methods, and / or computer-readable storage media storing a bitstream generated according to any of the above-described methods.

[0019] In this application, the terms “reconstructed” and “decoded” may be used interchangeably, the terms “pixel” and “sample” may be used interchangeably, and the terms “image,” “picture,” and “frame” may be used interchangeably. Typically, though not always, the term “reconstructed” is used on the encoder side, while the term “decoded” is used on the decoder side.

[0020] Various methods are described herein, each method comprising one or more steps or actions to implement the method described above. Unless a specific sequence of steps or actions is required for the method to function correctly, the order and / or use of any particular steps and / or actions may be modified or combined.

[0021] The modules of the video encoder 100 and decoder 200 shown in Figures 1 and 2, such as motion compensation modules 170 and 275, can be modified using the various methods and other embodiments described herein. Furthermore, these embodiments are not limited to VVC or HEVC and can be applied to other standards and recommendations, whether existing or to be developed in the future, and to any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the embodiments described herein can be used individually or in combination.

[0022] Figure 1 shows the encoder 100. While various configurations of this encoder 100 are possible, for the sake of brevity, the encoder 100 will be described below without describing all possible configurations.

[0023] Before encoding, the video sequence undergoes pre-processing 101, which may involve applying a color conversion to the input color image (e.g., from RGB4:4:4 to YCbCr4:2:0) or performing a remapping of the input image components to make the signal distribution more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and added to the bitstream.

[0024] In encoder 100, the image is encoded by encoder elements as described later. The encoded image is divided (102) and processed, for example, into CU units. Each unit is encoded using either intra or intermode. If a unit is encoded in intra mode, intra prediction (160) is performed. In intermode, motion estimation (175) and correction (170) are performed. The encoder decides (105) whether to use intra mode or intermode to encode the unit and indicates the intra / inter decision, for example, by a prediction mode flag. For example, the prediction residual is calculated by subtracting the prediction block from the original image block (110).

[0025] The predicted residual is then transformed (125) and quantized (130). The quantized transformed coefficients are entropy encoded (145) along with the motion vector and other syntactic elements to output the bitstream. The encoder can skip the transformation and apply quantization directly to the untransformed residual signal. The encoder can bypass both transformation and quantization, i.e., the residual is encoded directly without applying any transformation or quantization process.

[0026] The encoder decodes the encoded blocks to provide a reference for further predictions. The quantization transformation coefficients are dequantized (140) and inversely transformed (150) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (155) to reconstruct the image blocks. An in-loop filter (165) is applied to the reconstructed image to reduce encoding artifacts, for example, by performing deblocking / SAO (sample adaptive offset) filtering. The filtered image is stored in a reference image buffer (180).

[0027] Figure 2 shows a block diagram of the video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements as described later. The video decoder 200 generally performs the decoding pass and the encoding pass alternately, as described in Figure 1. The encoder 100 also generally performs video decoding as part of the encoding of video data.

[0028] In particular, the decoder input includes a video bitstream that can be generated by the video encoder 100. The bitstream is first entropy-decoded (230) to obtain transformation coefficients, motion vectors, and other encoded information. The image segmentation information indicates how the image will be segmented. The decoder can therefore segment the image according to the decoded image segmentation information (235). The transformation coefficients are dequantized (240) and inversely transformed (250) to decode the prediction residuals. The image blocks are reconstructed by combining the decoded prediction residuals and prediction blocks (255). The prediction blocks can be obtained (270) from intra-predictions (260) or motion-compensated predictions (i.e., inter-predictions) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference image buffer (280).

[0029] The decoded image may undergo further post-decoded processing (285), such as inverse color transformation (e.g., conversion from YCbCr4:2:0 to RGB4:4:4) or inverse remapping, which performs the inverse transformation of the remapping process performed in pre-encoding processing (101). The post-decoded processing can use metadata derived in pre-encoding processing and signaled to the bitstream.

[0030] Figure 3 shows a block diagram of an example of a system in which various embodiments and forms are implemented. System 1000 may be implemented as a device including various elements described below and configured to perform one or more of the embodiments described herein. Multiple examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected consumer electronics, and servers. The elements of System 1000 may be implemented individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of System 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, System 1000 is coupled communicably to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, System 1000 is configured to implement one or more embodiments described herein.

[0031] System 1000 includes, for example, at least one processor 1010 configured to execute instructions loaded therein to perform various embodiments described herein. Processor 1010 may include embedded memory, input / output interfaces and various other circuits known in the Art. System 1000 includes at least one memory 1020 (e.g., volatile memory devices and / or non-volatile memory devices). System 1000 includes a storage device 1040 which may include, but is not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drives and / or optical disk drives, as well as non-volatile memory and / or volatile memory. Storage device 1040 may, in non-limiting examples, include built-in storage devices, mounted storage devices (including removable and non-removable storage devices) and / or network-accessible storage devices.

[0032] System 1000 includes, for example, an encoder / decoder module 1030 configured to process data and provide encoded or decoded video, the encoder / decoder module 1030 may include its processor and memory. The encoder / decoder module 1030 represents a module that may be included in the device to perform encoding and / or decoding functions. As is well known, the device may include one or both of the encoding and decoding modules. The encoder / decoder module 1030 may also be implemented as a separate element of System 1000 or incorporated into the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0033] To perform the various embodiments described herein, program code to be loaded into the processor 1010 or encoder / decoder 1030 can be stored in the storage device 1040 and then loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more processors 1010, memory 1020, storage device 1040 and encoder / decoder modules 1030 can store one or more items while performing the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables and equations, formulas, operations and intermediate or final results from the processing of operational logic.

[0034] In some embodiments, instructions are stored using memory within the processor 1010 and / or encoder / decoder module 1030, and working memory is provided for processing required during encoding or decoding. However, in other embodiments, external memory of the processing unit (for example, the processing unit may be either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory is used to store, for example, the television operating system. In at least one embodiment, high-speed external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations in MPEG-2 (MPEG refers to the Video Experts Group, also known as ISO / IEC 13818, 13818-1 also known as H.222, and 13818-2 also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (JVET, i.e., the Joint Video Experts Team, a new standard under development for multi-purpose video coding).

[0035] Inputs to the elements of system 1000 can be provided through various input devices, as shown in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section for receiving RF signals wirelessly transmitted by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-resolution multimedia interface (HDMI) input terminal. Other examples not shown in Figure 10 include composite video.

[0036] In various embodiments, the input device of block 1130 has associated input processing elements known in the Art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) downconverting the selected signal, (iii) again band-limiting it to a narrower frequency band to select a signal frequency band which may be referred to as a channel in certain embodiments, (iv) demodulating the downconverted band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a stream of desired data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs these various functions, for example, downconverting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to the baseband) or the baseband. In one embodiment of the set-top box, the RF unit and associated input processing elements receive an RF signal transmitted via a wired (e.g., cable) medium, perform frequency selection by filtering, downconvert, and filter again to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Addition of elements may include inserting elements between existing elements, such as inserting amplifiers and analog / digital converters. In various embodiments, the RF unit includes an antenna.

[0037] Furthermore, the USB and / or HDMI terminals may include interface processors that connect the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various forms of input processing, such as Reed-Solomon error correction, can be implemented as needed, for example, within separate input processing ICs or within processor 1010. Similarly, forms of USB or HDMI interface processing can be implemented as needed, within separate interface ICs or within processor 1010. Demodulation, error correction, and demultiplexed streams are provided to various processing elements, such as processor 1010 and encoder / decoder 1030, which operate in combination with memory and storage elements to process the data stream as needed for display to an output device.

[0038] The various elements of system 1000 may be housed within an integrated enclosure, where the various elements are interconnected and can transmit data between them using an internal bus known in the art, such as an inter-IC (I2C) bus, wiring, and printed circuit board.

[0039] System 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0040] In various embodiments, data is streamed to or provided to system 1000 via a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 1060 and a communication interface 1050 adapted for Wi-Fi communication. In these embodiments, the communication channel 1060 is typically connected to an access point or router that provides access to an external network, including the Internet, to enable streaming applications and other over-the-top communications. Other embodiments provide streaming data to system 1000 using a set-top box that distributes data via an HDMI connection on input block 1130. Yet another embodiment also provides streaming data to system 1000 using an RF connection on input block 1130. As described above, various embodiments provide data non-streaming. Furthermore, various embodiments use wireless networks other than Wi-Fi, e.g., cellular networks or Bluetooth networks.

[0041] System 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripheral devices 1120. In various embodiments, the display 1100 includes, for example, one or more touchscreen displays, organic light-emitting diode (OLED) displays, curved displays, and / or foldable displays. The display 1100 may be for televisions, tablets, laptops, mobile phones, or other devices. The display 1100 may also be integrated with other elements (e.g., a mobile phone) or separate (e.g., an external monitor for a laptop). In various examples of the multiple embodiments, the other peripheral devices 1120 include one or more standalone digital video discs (or digital multi-purpose discs) (both abbreviated as DVRs), disc players, stereo systems, and / or lighting systems. Various embodiments utilize one or more peripheral devices 1120 that provide functions based on the output of System 1000. For example, a disc player performs the function of playing back the output of System 1000.

[0042] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speaker 1110, or other peripheral devices 1120 using signaling protocols, such as AV.Link, consumer electronics control (CEC), or other communication protocols that enable inter-device control, with or without user intervention. Output devices may be coupled to the system 1000 via dedicated connectors through their respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to the system 1000 via communication interface 1050 using communication channel 1060. The display 1100 and speaker 1110 may be integrated into a single unit with other elements of the system 1000 in an electronic device such as a television. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0043] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may instead be separate from one or more other elements. In various embodiments where the display 1100 and speaker 1110 are external elements, the output signal may be provided via a dedicated output connection, such as an HDMI port, a USB port, or a COMP port.

[0044] The embodiments can be implemented by computer software implemented by the processor 1010 or hardware, or by a combination of hardware and software. In non-limiting examples, the embodiments described above may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment, and in non-limiting examples, may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor memory devices, fixed memory, and removable memory. The processor 1010 may be of any type suitable for the technical environment, and in non-limiting examples, may include one or more microprocessors, multipurpose computers, special-purpose computers, and processors based on multicore architectures.

[0045] Figure 4 shows, for example, the coding tree and coding tree unit (CTU) structure used to represent a compressed image using the HEVC video compression standard. In video compression standards such as HEVC, motion-compensated temporal prediction is used to utilize the redundancy present between a series of images in a video. A motion vector is associated with each prediction unit (PU) contained in a CTU. A set of CTUs in an image is represented as a coding tree of the compression region. This is a quadtree partition of CTUs, where each leaf is called a coding unit (CU). Each CU is given multiple intra or interpretation parameters (prediction information). To do this, a CU is spatially divided into one or more prediction units (PUs), and some prediction information is assigned to each PU. An intra or inter-coding mode is assigned to the CU level. One motion vector is assigned to each PU in HEVC. This motion vector is used for the motion-compensated temporal prediction of the PU under consideration. Therefore, in video compression standards such as HEVC, the motion model that links prediction blocks to their reference blocks includes translation.

[0046] Figures 5A and 5B show affine motion vector fields based on 4x4 subblocks for two and three control points, respectively. Recent revised video compression standards, such as the Joint Search Model (JEM) and the VVC (Variable-Purpose Video Coding) test model later developed by the JVET (Joint Video Search Team) group, support several improved motion models to enhance temporal prediction. For this purpose, a PU can be spatially divided into subPUs, and a more robust model can be used to assign a dedicated motion vector to each subPU. A single CU cannot be further divided into PUs or TUs; some motion data is directly assigned to each CU. In this new codec design, a CU can be divided into subCUs, and one motion vector can be computed for each subCU. One of the new motion models introduced in JEM is the affine model, which uses the affine model to represent motion vectors within a CU.

[0047] The affine motion field for the two control points of FIG. 5A, also called a four-parameter affine model, includes the component values of the motion vectors for each position (x, y) within the block frame under consideration, according to Equation [1].

Number

[0048] A model having three control points as shown in FIG. 5B, called a six-parameter affine motion model, can also be used to represent a motion field based on a sub-block of a given coding unit. The motion field in the case of the six-parameter affine model is calculated as in Equation [2], where (v 0x , v 0y ) is vector 51B, (v 1x , v 1y ) is vector 52B, and (v 2x , v 2y ) is vector 53B.

Number

[0049] In practice, to maintain a reasonable level of complexity, the same motion vector is calculated for the pixels of the 4x4 subblocks (sub-CUs) of the CU under consideration, as shown in Figures 5A and 5B. An affine motion vector is calculated from the control point motion vector at the center position of each subblock. Thus, the obtained motion vector is expressed with an accuracy of 1 / 16 pixels. As a result, the prediction unit (PU) of the coding unit in affine mode is constructed through motion-compensated predictions using the motion vectors of each subblock. Affine motion compensation can be used in two ways in the VVC test model (VTM): affine AMVP and affine merge. These are described below.

[0050] In VTM, CUs larger than 8x8 can be predicted in affine AMVP mode. This is signaled via a flag in a bitstream encoded at the CU level. Generating the affine motion field between these CUs involves the decoder determining the control point motion vector (CPMV) obtained by adding the control point motion vector prediction (CPMVP) to the motion vector difference. The CPMVP is a pair of motion vectors (for a 4-parameter affine model specified by two control point motion vectors) or a triplet (for a 6-parameter affine model specified by three control point motion vectors) used as the predictor of the current CU's CPMV. The current CU's CPMVP can be inherited from an affine adjacent CU (similar to affine merge mode). The spatial position from which the inherited CPMVP is obtained is selected from a given ordered list of candidate positions. The inherited CPMVP is considered valid if its reference image is equal to the reference image of the current CU.

[0051] In affine merge mode, the CU level flag indicates whether the merge mode CU uses affine motion compensation. If so, in JEM, the first available adjacent CU encoded in affine mode is selected from a given ordered list of candidate positions. Once the first adjacent CU in affine mode is obtained, the CPMVP of the current CU can be inherited from that affine adjacent CU. Three motion vectors from the top-left, top-right, and bottom-left corners of the adjacent affine CU.

number

number

[0052] For a CU in 4-parameter affine mode, the two CPMVs of the current CU are derived as follows:

number

[0053] For a 6-parameter affine mode control unit (CU), the motion vectors of the three control points of the current CU are derived as follows.

number

[0054] Current CU control point motion vector

number

number

[0055] Since VTM-3.0, SbTMVP (also known as ATMVP, a subblock-based time-motion vector predictor) candidates are part of the affine merge as the first subblock candidates evaluated in the rate-distortion optimization process in affine merge list mode. SbTMVP predicts the motion vectors of 8x8 subblocks in the current CU as a regular TMVP predictor by collecting motion information of corresponding subblocks in consecutive images.

[0056] Figure 6 shows an example of list updating using the HMVP method. This also leads to the history-based motion vector prediction (HMVP) tool used in VTM-3.0 and later. "History-based" means maintaining a list created from multiple motion information (motion vectors, associated reference frames, BCW subscripts, ...) used to encode / decode CUs preceding the current CU. Whenever a non-affine and non-triangular interCU, which is a block not based on subblocks, is encoded / decoded, the associated motion information is inserted at the end of the list as a new HMVP candidate. As shown in Figure 6, when inserting a new motion candidate into the table, a constrained FIFO rule is used, and a redundancy check is first applied to see if the same HMVP already exists in the table. If found, the same HMVP is removed from the table, and all subsequent HMVP candidates are moved forward, i.e., their subscripts are reduced by 1.

[0057] HMVP candidates can be used in the merge candidate list construction process. Several of the most recent HMVP candidates in the table are sequentially examined and inserted after the TMVP candidates in the candidate list. The HMVP candidates are then pruned to obtain spatial or temporal merge candidates (i.e., ATMVPs) that exclude subblock movement candidates.

[0058] According to current principles, motion information for an interCU encoded in subblock mode, i.e., a block whose subblock is decoded based on a motion vector determined according to the motion information data associated with that block, is added to a list of motion information data, such as an HMVP list. The items in the list are used to determine the motion information data for further blocks of the same current image. Since affine and SbTMVP modes use motion compensation based on subblocks, multiple motion vectors are stored for each CU encoded in subblock mode. The position within the CU, i.e., the subblock from which the motion vector stored in the list is selected, may affect the performance of the BD rate.

[0059] Currently, motion information derived from CUs encoded in subblock or triangular mode is not considered in the creation and updating of the HMVP list. According to the current principle, zero, one, or more motion information data determined according to interCUs encoded in subblock mode are inserted into the HMVP list as steady intermotion information.

[0060] The current principle can be applied to all subblock coding modes. The subblock coding modes are not limited to the three modes of VTM-5.0, namely (i) affine AMVP, (ii) affine merge, and (iii) SbTMVP. However, they can be extended to all already known subblock modes such as planar motion vector, regression, triangle, FRUC, interwoven affine, etc., and all new subblock modes that may be proposed in the future. In HMVP list saving, it is possible to consider only one of these subblock modes, a combination of modes (e.g., both affine modes), or a combination of all subblock modes.

[0061] Figure 7 schematically illustrates method 70 for encoding / decoding a block of the current image. In step 71, a first block is encoded / decoded according to a subblock mode. The motion vector used for encoding / decoding the subblock of the first block is determined according to the motion information contained in the CU of the first block, for example, as shown in Figures 5A and 5B. The motion information data includes different data used for motion prediction, such as motion vectors, associated reference frames, BCW subscripts, etc.

[0062] In accordance with current principles, in step 72, a second set of motion information data is determined as a function of the first set of motion information data. The number of determined motion information data in the set may be zero, one, or more. The size of the set is chosen based on the efficiency of its selection for the encoding / decoding process. The size is parameterized and shared by the encoder and decoder. The relevant parameters may be encoded in the header of the video stream.

[0063] In one embodiment, second motion information is generated from first motion information, i.e., data of motion information associated with the first block. The motion vector of the second motion information is determined as a function of a subset of motion vectors determined for the subblocks of the first block. A subblock that provides that motion vector (i.e., a function of motion vectors) for that combination is selected according to the subblock's position within the first block. For example, a unique subblock is selected, e.g., a subblock centered in the first block (e.g., in the case of a 4x4 subblock partition, subblocks [1,1] or [1,2]), and its function is the identity function. Thus, the motion vector of the determined second motion information data is the motion vector of the selected subblock. In another example, the motion vector of the second motion information data is the average of two, three, or four motion vectors of subblocks selected to distribute their positions within the first block, e.g., a subblock with two centers aligned or three subblocks, i.e., two at the top corners and one at the bottom center or four at each corner of the first block. The function does not have to be an averaging function. The selection of the function and position of the selected subblocks is guided by the efficiency of the selection for the encoding / decoding process. This selection is parameterized and shared by the encoder and decoder. The relevant parameters may be encoded in the header of the video stream.

[0064] In another embodiment, the second motion information data includes a motion vector determined as a function of the motion vector of the first motion information data. For example, for a first block encoded / decoded according to a four-parameter affine model, the motion vector of the second motion information data may be a weighted average of two control point vectors. Zero, one, or more motion information data can be determined by different functions and / or different parameters, depending on the embodiment.

[0065] According to one variant applicable to the two embodiments described above, the selected function of the motion vector and / or position of the selected subblock depends on the size of the first block and / or the size of the subblock. For example, if the size of the first block is smaller than a given size (e.g., 128×128, 64×64, 32×32, or 16×16), the function is identification, and the selected subblock is selected at the center of the first block. However, if the first block is larger than a given size, the function is the weighted average of the two vectors of the subblock at the corners of the first block (weights depending on the size of the first block).

[0066] In one variant, the number of determined second motion information data depends on the size of the first block and / or the size of the subblocks. For example, if the size of the first block is smaller than a given size, no second motion data is determined. In another example, the subblocks of the first block can be divided into areas of fixed size (e.g., 64×64, 32×32, or 16×16) or areas of adapted size, for example, areas obtained by dividing the subblock CU into 2, 4, or 8 parts. For these areas, one second motion information data is determined according to one of the embodiments and variants described above. For example, if the area is of fixed size 32×32, only one second motion information data is determined for the 16×32 subblock CU, and four second motion information data are determined for the 64×64 subblock CU. In another example, if the size of the region is adaptive and the subblock CU is divided into four regions, there will always be four motion information data to be stored, one in each 8x16 subblock in the case of a 16x32 subblock CU, and one in each 32x32 subblock in the case of a 64x64 subblock CU.

[0067] The number of determined second motion information data points and the choice of different functions for determining them are guided by the efficiency of that choice for the encoding / decoding process. This choice is parameterized and shared by the encoder and decoder. The relevant parameters may be encoded within the video stream header. For example, an effective choice might be to determine one motion information data point for each subblock with an area of ​​256 square pixels or more, along with the motion vector of the centrally placed subblock.

[0068] In step 73 of method 70 in Figure 7, the determined set of motion information data is added to a list of motion information data, for example, the HMVP list of the VCC test model. This list is defined as determining motion information data for further blocks of the current image that are encoded / decoded using its items.

[0069] Various implementations include decoding. As used in this application, “decoding” may include, for example, all or part of the processing performed on the received encoded sequence to produce a final output suitable for display. In various embodiments, such processing includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processing also includes, or alternatively, processing performed by the decoder in the various implementations described in this application, such as managing motion information data lists to send to the feed motion compensation prediction modules 170 and 275 in Figures 1 and 2.

[0070] As further examples, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" refers specifically to a subset of operations or to a broader decoding process in general will become clear from the context of the specific description and will be well understood by those skilled in the art.

[0071] Various implementations include encoding. As with the above discussion concerning "decoding," "encoding" as used in this application may include, for example, all or part of the processing performed on the input video sequence to generate an encoded bitstream. In various embodiments, such processing includes one or more processes typically performed by an encoder, such as splitting, differential encoding, transformation, quantization, and entropy encoding. In various embodiments, such processing also includes, or alternatively, processing performed by the encoder in the various implementations described in this application, such as managing motion information data lists to send to the motion compensation prediction modules 170 and 275 in Figures 1 and 2.

[0072] As further examples, in one embodiment, "encoding" refers only to entropy coding; in another embodiment, "encoding" refers only to differential coding; and in yet another embodiment, "encoding" refers to a combination of differential coding and entropy coding. Whether the phrase "encoding process" refers specifically to a subset of operations or more broadly to encoding methods will become clear from the context of the specific description and will be well understood by those skilled in the art.

[0073] Please note that the syntactic elements used in this specification are descriptive terms. Therefore, these terms do not preclude the use of other syntactic element names.

[0074] When a drawing is presented as a flow chart, it should be understood that it also shows a block diagram of the corresponding device. Similarly, when a drawing is presented as a block diagram, it should be understood that it also shows a flow chart of the corresponding method / process.

[0075] The implementations and embodiments described herein may be implemented in methods or processes, apparatus, software programs, data streams, or signals. Even when discussed only as a single implementation (e.g., discussed only as a method), the implementations of the features discussed may be implemented in other forms (e.g., apparatus or programs). Apparatus may be implemented, for example, in appropriate hardware, software, and firmware. The method may be implemented in a processor, which generally refers to an apparatus, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as computers, mobile phones, personal digital assistants ("PDAs"), and other devices that enable information communication between end users.

[0076] References to "one embodiment" or "one implementation" or "one implementation" mean, as with other variations, that the specific features, structures, characteristics, etc. described in relation to that embodiment are included in at least one embodiment. Therefore, the appearance of the phrases "in one embodiment" or "in one embodiment" or "in one implementation" does not necessarily refer to the same embodiment in all instances, along with other variations that appear in various places throughout this application.

[0077] Furthermore, this application may refer to the "determination" of various information items. Determination of information may include, for example, one or more of the following: estimation of information, calculation of information, prediction of information, or retrieval of information from memory.

[0078] Furthermore, this application may refer to “access” to various information items. Access to information may include, for example, one or more of the following: receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0079] Furthermore, this application may refer to the "reception" of various information items. Reception is intended to be as broad as "access." Reception of information may include, for example, one or more of the following: access to information or retrieval of information (e.g., from memory). Moreover, "reception" is typically performed in some way during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0080] It should be noted that the use of any of the following " / ", "and / or", or "at least one" in the cases of "A / B", "A and / or B", or "at least one of A and B" is intended to include the selection of only the first enumerated option (A), only the second enumerated option (B), or both options (A and B). As further examples, in the cases of "A, B and / or C" and "at least one of A, B and C", such phrases are intended to include the selection of only the first enumerated option (A), only the second enumerated option (B), only the third enumerated option (C), or only the first and second enumerated options (A and B), or only the first and third enumerated options (A and C), or only the second and third enumerated options (B and C), or the selection of all three options (A, B, and C). This can be extended to all enumerated items, as will be apparent to those skilled in the art in this and related fields.

[0081] Furthermore, the term "signaling" as used herein refers, in particular, to instruct a corresponding decoder in some way. For example, in certain embodiments, an encoder signals one particular parameter of several parameters for determining second information data from a block encoded in subblock mode. Thus, in one embodiment, the same parameter is used on both the encoder and decoder sides. Therefore, for example, the encoder can transmit a particular parameter to the decoder (explicit signaling) so that the decoder can use the same particular parameter. Conversely, if the decoder already has that particular parameter along with others, it can simply recognize and select the particular parameter using signaling (implicit signaling) without transmitting it. Bit saving is achieved in various embodiments by avoiding the transmission of the actual function. It should be recognized that signaling can be implemented in various ways. For example, in various embodiments, information is signaled to the corresponding decoder using one or more syntactic elements, flags, etc. The above concerns the verb form of the term "signaling," but the term "signal" can also be used as a noun in this specification.

[0082] As will become apparent to those skilled in the art, multiple implementations can generate various signals formatted to hold, for example, information that can be stored or transmitted. This information may include, for example, instructions for performing a method or data generated by one of the above-described implementations. For example, a signal can be formatted to hold the bitstream of the above-described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, the steps of encoding a data stream and modulating a carrier by the encoded data stream. The information held by the signal may be, for example, analog or digital information. The signal can be transmitted over various different wired or wireless links known. The signal can be stored in a processor-readable medium.

Claims

1. - decoding a sub-block of a first block of the current image, said sub-block being decoded based on a motion vector determined according to first motion information data associated with said first block; determining a second set of motion information data as a function of said first motion information data; adding said second set of motion information data to a list of motion information data, the entries of said list being used to determine motion information data of a second block of said current image; A method comprising:

2. 1. An apparatus for decoding a block of a current image, comprising: - decoding a sub-block of a first block of the current image, said sub-block being decoded based on a motion vector determined according to first motion information data associated with said first block; determining a second set of motion information data as a function of said first motion information data; adding said second set of motion information data to a list of motion information data, the entries of said list being used to determine motion information data of a second block of said current image; 12. An apparatus comprising: a processor configured to:

3. - encoding a sub-block of a first block of the current image, said sub-block being encoded on the basis of a motion vector determined according to first motion information data associated with said first block; determining a second set of motion information data as a function of said first motion information data; adding said second set of motion information data to a list of motion information data, the entries of said list being used to determine motion information data of a second block of said current image; A method comprising:

4. 1. An apparatus for encoding a block of a current image, comprising: - encoding a sub-block of a first block of the current image, said sub-block being encoded on the basis of a motion vector determined according to motion information data associated with said first block; determining a second set of motion information data as a function of said first motion information data; adding said second set of motion information data to a list of motion information data, the entries of said list being used to determine motion information data of a second block of said current image; 12. An apparatus comprising: a processor configured to:

5. 5. The method of claim 1 or 3 or the apparatus of claim 2 or 4, wherein the second motion information data comprises a motion vector determined as a function of a motion vector of a selected sub-block of the first block, the selected sub-block being at a given position within the first block.

6. 5. The method of claim 1 or 3 or the apparatus of claim 2 or 4, wherein the second motion information data comprises motion vectors determined as a function of motion vectors comprised in the first motion information data.

7. The method or apparatus according to claim 5 or 6, wherein the function of the motion vector depends on the size of the first block or the size of the sub-block.

8. 8. The method of claim 1, 3 or 5 to 7 or the apparatus of claim 2 or 4 to 7, wherein the second set of motion information data comprises a number of items, the number of items depending on the size of the first block or the size of the sub-block.

9. - a device according to any one of claims 1 or 5 to 8, at least one of (i) an antenna configured to receive a signal including the decoded block of the image, (ii) a band limiter configured to limit the received signal to a band of frequencies including the decoded block of the image, and (iii) a display configured to display an output representing the decoded block of the image; Equipment including.

10. A non-transitory computer readable medium containing data content generated according to the method of any one of claims 3 or 5 to 8 or by the apparatus of any one of claims 4 to 8 for playback using a processor.

11. A computer program product comprising instructions that, when the program is run by a computer, cause the computer to carry out the method of any one of claims 3 or 5 to 8.

12. A signal comprising video data generated according to the method of any one of claims 3 or 5 to 8 or by the apparatus of any one of claims 4 to 8 for playback using a processor.

Images