Precision refinement for motion compensation with optical flow

Abstract

To provide systems and methods for refining motion-compensated predictions in block-based video coding.SOLUTION: A video coding method comprises: obtaining an initial predicted sample value, based on motion-compensated prediction, for each of sample positions in a current block or subblock of samples; determining a motion vector refinement; determining, at each of the sample positions, a spatial gradient of sample values; determining a sample difference value, based on a scalar product of the spatial gradient and the motion vector refinement, for each of the sample positions; and modifying the initial predicted sample value based on the sample difference value.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application claiming the benefit under 35 U.S.C. §119(e) from U.S. Provisional Patent Application No. 62 / 864,825, entitled “Precision Refinement for Motion Compensation with Optical Flow,” filed June 21, 2019, which is incorporated by reference in its entirety herein. [Background technology]

[0002] Video coding systems are widely used to compress digital video signals to reduce the storage needs and / or transmission bandwidth of such signals. Among various types of video coding systems, such as block-based, wavelet-based, and object-based systems, block-based hybrid video coding systems are the most widely used and deployed today. Examples of block-based video coding systems include international video coding standards such as MPEG-1 / 2 / 4 Part 2, H.264 / MPEG-4 Part 10 AVC, VC-1, and High Efficiency Video Coding (HEVC), which was developed by ITU-T / SG16 / Q.6 / VCEG and the JCT-VC (Joint Collaboration Team on Video Coding) of ISO / IEC / MPEG.

[0003] In October 2017, a joint Call for Proposals (CfP) for video compression with capabilities beyond HEVC was issued by ITU-T and ISO / IEC. In April 2018, 22 CfP responses for the standard dynamic range category were accepted and evaluated at the 10th JVET meeting, demonstrating an improvement in compression rate over HEVC of approximately 40%. Based on such evaluation results, the Joint Video Experts Team (JVET) launched a new project to develop a new generation of video coding standard called Versatile Video Coding (VVC). In the same month, a reference software code base called VVC Test Model (VTM) was established to demonstrate a reference implementation of the VVC standard. In the initial VTM-1.0, most coding modules, including intra prediction, inter prediction, transform / inverse transform, quantization / inverse quantization, as well as in-loop filters, follow the existing HEVC design, except that a multi-type tree-based block partition structure is used in VTM. Meanwhile, another reference software base called Benchmark Set (BMS) was also generated to facilitate the evaluation of new coding tools. In the BMS code base, a list of coding tools inherited from the Joint Exploration Model (JEM), which offers higher coding efficiency and moderate implementation complexity, is included on top of VTM and is used as a benchmark when evaluating similar coding technologies in the process of VVC standardization. Specifically, there are nine JEM coding tools integrated into BMS-1.0, including 65 angular intra prediction directions, modified coefficient coding, advanced multiple transform (AMT) + 4 × 4 non-separable quadratic transform (NSST), affine motion model, generalized adaptive loop filter (GALF), advanced temporal motion vector prediction (ATMVP), adaptive motion vector precision, decoder-side motion vector refinement (DMVR), and linear model (LM) chroma mode. Summary of the Invention

[0004] The embodiments described herein include methods used in video encoding and decoding (collectively "coding"). A system and method for refining motion-compensated prediction in block-based video coding are described. A video coding method according to some embodiments includes: generating an initial predicted sample value for at least a first sample position in a current block of samples using motion-compensated prediction; determining a refinement of a motion vector associated with at least the first sample position; determining a spatial gradient of the sample value at the first sample position; determining a sample difference value by calculating a scalar product of the spatial gradient and the refinement of the motion vector; and adding the sample difference value to the initial predicted sample value to generate a refined sample value.

[0005] In some embodiments, the method is performed by a decoder and determining the motion vector refinement includes decoding the motion vector refinement from a bitstream.

[0006] Some embodiments further include decoding refinement precision information from the bitstream, and determining the sample difference value includes scaling the scalar product by an amount indicated by the precision information. Scaling the scalar product may include bit-shifting the scalar product by the amount indicated by the precision information.

[0007] In some embodiments, the motion compensated prediction is performed using at least one motion vector having an initial precision, and the refinement precision information includes a precision difference value representing the difference between the initial precision and the refinement precision.

[0008] In some embodiments, scaling the scalar product includes right-shifting the scalar product by a number of bits equal to the sum of the precision difference value and the initial precision.

[0009] In some embodiments, the method is performed by an encoder, and determining a motion vector refinement includes selecting a motion vector refinement to substantially minimize a prediction error for the input video block, and the method further includes encoding the motion vector refinement in the bitstream.

[0010] In some embodiments, the refinement of the motion vector is signaled in the bitstream as an index. In some embodiments, the index may identify one of a plurality of motion vector refinements from the group consisting of (0,-1), (1,0), (0,1), and (-1,0). In some other embodiments, the index may identify one of a plurality of motion vector refinements from the group consisting of (0,-1), (1,0), (0,1), (-1,0), (-1,-1), (1,-1), (1,1), and (-1,1).

[0011] In some embodiments, the refinement of the motion vector is associated with all sample positions within the current block (or current sub-block), while in some other embodiments, the refinement of the motion vector is determined on a sample-by-sample basis and may be different for different samples.

[0012] Some embodiments include at least one processor configured to perform any of the methods described herein. In some such embodiments, a computer-readable medium (e.g., a non-transitory medium) is provided that stores instructions that operate to perform any of the methods described herein.

[0013] Some embodiments include a computer-readable medium (eg, a non-transitory medium) that stores video encoded using one or more of the methods disclosed herein.

[0014] An encoder or decoder system may include a processor and a non-transitory computer-readable medium storing instructions for performing the methods described herein.

[0015] One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods described above. The present embodiments also provide a computer-readable storage medium (e.g., a non-transitory medium) having stored thereon a bitstream generated according to the methods described above. The present embodiments also provide a method and apparatus for transmitting a bitstream generated according to the methods described above. The present embodiments also provide a computer program product including instructions for performing any of the methods described. [Brief description of the drawings]

[0016] [Figure 1A] FIG. 1 is a system diagram illustrating an example communication system in which one or more disclosed embodiments may be implemented. [Figure 1B] 1B is a system diagram illustrating an example of a wireless transmit / receive unit (WTRU) that may be used within the communications system illustrated in FIG. 1A, according to one embodiment. [Figure 1C] FIG. 1 is a functional block diagram of a system used in some embodiments described herein. [Figure 2A] FIG. 1 is a functional block diagram of a block-based video encoder such as the encoder used in VVC. [Figure 2B] FIG. 2 is a functional block diagram of a block-based video decoder such as the decoder used in VVC. [Figure 3A-3B] 1 shows motion compensated prediction. [Figure 4A] A four-parameter affine motion model is presented. [Figure 4B] 1 illustrates sub-block level motion derivation for affine blocks. [Diagram 5] A 6-parameter affine mode is shown, where V0, V1 and V2 are the motion vectors of the control points, and (MVx,MVy) is the motion vector of the sub-block centered at position (x,y). [Figure 6]The motion vector VSB of the subblock and the pixel refinement Δv(i,j) are indicated. [Figure 7] 1 illustrates the use of adjacent positions to signal additional motion vectors. [Figure 8] 4 is a flowchart illustrating an encoding and decoding method performed in some embodiments, including coding functions that can be performed in both the encoder and decoder. [Figure 9] FIG. 1 illustrates an example of a communication system.

[0017] Example Devices and Networks for Implementing the Embodiments 1A illustrates an example communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multi-access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access schemes, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tailed Unique Word DFT-Spread OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multi-Carrier (FBMC), etc.

[0018] 1A, the communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104 / 113, a CN 106 / 115, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, although it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, mobile phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearables, head mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronics devices, device networks operating with commercial and / or industrial wireless, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be referred to interchangeably as a UE.

[0019] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106 / 115, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode B, a home Node B, a home eNode B, a gNB, an NR NodeB, a site controller, an access point (AP), a wireless router, etc. Although the base stations 114a, 114b are each shown as a single element, it will be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0020] The base station 114a may be part of the RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage to a particular geographic area that may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one for each sector of the cell. In one embodiment, the base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0021] The base stations 114a, 114b may communicate with one or more WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, microwave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0022] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base stations 114a and the WTRUs 102a, 102b, 102c of the RAN 104 / 113 may implement a radio technology, such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115 / 116 / 117 using Wideband CDMA (WCDMA). WCDMA may include communications protocols, such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0023] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0024] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR radio access, which may establish the air interface 116 using New Radio (NR).

[0025] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement LTE radio access and NR radio access together, for example, using a dual connectivity (DC) principle. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to and from multiple types of base stations (e.g., eNBs and gNBs).

[0026] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a wireless technology such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Global Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), GSM EDGE (GERAN), or the like.

[0027] The base station 114b of FIG. 1A may be, for example, a wireless router, a Home Node B, a Home eNode B, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a local area, such as an office, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, etc. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology, such as IEEE 802.11, to establish a wireless local area network (WLAN). In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology, such as IEEE 802.15, to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or femtocell. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not need to access the Internet 110 via the CN 106 / 115.

[0028] The RAN 104 / 113 may communicate with the CN 106 / 115, which may be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more WTRUs 102a, 102b, 102c, 102d. The data may have various Quality of Service (QoS) requirements, such as different throughput requirements, delay requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 / 115 may provide call control, charging services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform high level security functions, such as user authentication. Although not shown in FIG. 1A, it will be understood that the RAN 104 / 113 and / or the CN 106 / 115 may communicate directly or indirectly with other RANs that use the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which may utilize NR radio technology, the CN 106 / 115 may also communicate with another RAN (not shown) that uses GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0029] The CN 106 / 115 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) of the TCP / IP Internet protocol suite. The networks 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may use the same RAT as the RANs 104 / 113 or a different RAT.

[0030] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with a base station 114a, which may employ a cellular-based wireless technology, and with a base station 114b, which may employ an IEEE 802 wireless technology.

[0031] Figure 1B is a system diagram illustrating an example WTRU 102. As shown in Figure 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any subcombination of the foregoing elements while remaining consistent with an embodiment.

[0032] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC) thereof, a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other function that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although FIG. 1B depicts the processor 118 and the transceiver 120 as separate components, it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0033] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It will be appreciated that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0034] 1B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0035] The transceiver 120 may be configured to modulate signals transmitted by the transmit / receive element 122 and demodulate signals received by the transmit / receive element 122. As mentioned above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as, for example, NR and IEEE 802.11.

[0036] The processor 118 of the WTRU 102 may be coupled to and may receive user input data from the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from and store data in any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include random access memory (RAM), read only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, or the like. In other embodiments, the processor 118 may access information from and store data in memory that is not physically located on the WTRU 102, such as a server or home computer (not shown).

[0037] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control the power to other components within the WTRU 102. The power source 134 may be any suitable device for providing power to the WTRU 102. For example, the power source 134 may include one or more dry batteries (e.g., nickel cadmium (NiCd), nickel zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0038] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or instead of information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface 116 and / or may determine its location based on the timing of signals received from two or more nearby base stations. It will be understood that the WTRU 102 may obtain location information by way of any suitable location determination method while remaining consistent with an embodiment.

[0039] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functions, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, and / or the like. The peripherals 138 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, a direction sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0040] The WTRU 102 may include a full-duplex radio for transmitting and receiving some or all of the signals (e.g., associated with a particular subframe) on both the UL (e.g., for transmission) and downlink (e.g., for reception) in parallel and / or simultaneously. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., via a separate processor (not shown) or via the processor 118). In one embodiment, the WRTU 102 may include a half-duplex radio for transmitting and receiving some or all of the signals (e.g., associated with a particular subframe on either the UL (e.g., for transmission) or downlink (e.g., for reception).

[0041] Although the WTRU is depicted in FIGS. 1A-1B as a wireless terminal, in certain representative embodiments it is contemplated that such a terminal may use a wired communications interface (e.g., temporarily or permanently) with the communications network.

[0042] In an exemplary embodiment, the other network 112 may be a WLAN.

[0043] 1A-1B and in view of the corresponding description, one or more or all of the functions described herein may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more or all of the functions described herein. For example, the emulation devices may be used to test other devices and / or to simulate network and / or WTRU functions.

[0044] The emulation device may be designed to implement one or more tests of other devices in a lab environment and / or an operator network environment. For example, one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices in the communication network. One or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for testing purposes and / or may perform testing using over-the-air wireless communication.

[0045] The one or more emulation devices may perform one or more functions, inclusive, while not implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in test scenarios in a test lab and / or in an undeployed (e.g., test) wired and / or wireless communication network to implement testing of one or more components. The one or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (which may include, for example, one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0046] An exemplary system. Some embodiments are implemented using a system such as the system of FIG. 1C. FIG. 1C illustrates a block diagram of an example system in which various aspects and embodiments are implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television sets, personal video recording systems, connected appliances, and servers. The elements of system 1000, either alone or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or separate components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or separate components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, the system 1000 is configured to implement one or more of the aspects described herein.

[0047] The system 1000 includes at least one processor 1010 configured to execute loaded instructions, for example, to implement various aspects described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits, as known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. The storage device 1040 may include, by way of non-limiting example, an internal storage device, an attached storage device (including removable and non-removable storage devices), and / or a network-accessible storage device.

[0048] The system 1000 includes an encoder / decoder module 1030 configured to process data to provide, for example, encoded or decoded video, which may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included in a device that performs encoding and / or decoding functions. As is known, a device may include one or both of an encoding and decoding module. Furthermore, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated within the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0049] Program code loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and computational logic.

[0050] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, an external non-volatile flash memory is used to store, for example, the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also referred to as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by JVET, i.e. the Joint Video Experts Team).

[0051] Inputs to the elements of system 1000 may be provided through various input devices as shown in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, an RF signal transmitted over the air by a broadcast station, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high definition multimedia interface (HDMI) input terminal. Other examples not shown in FIG. 1C include composite video.

[0052] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulating the down-converted, band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements that perform these functions, e.g., a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include, for example, a tuner that performs various of these functions, including down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to baseband) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and filtering again to a desired frequency band an RF signal transmitted over a wired (e.g., cable) medium. In various embodiments, the order of the above (and other) elements is rearranged, some of these elements are removed, and / or other elements that perform similar or different functions are added. Adding elements can include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0053] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or within the processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented in a separate interface IC or within the processor 1010, as desired. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements including, for example, the processor 1010 and an encoder / decoder 1030 operating in combination with memory and storage elements to process the data stream as desired for display on an output device.

[0054] The various elements of the system 1000 may be provided within a unitary housing in which the various elements are interconnected and can transmit data between them using a suitable connection arrangement 1140, for example, an internal bus as is well known in the art, including an Inter-IC (I2C) bus, hardwired, and printed circuit boards.

[0055] The system 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented in a wired and / or wireless medium, for example.

[0056] Data is streamed or otherwise provided to the system 1000 in various embodiments using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received via a communication channel 1060 and a communication interface 1050 adapted for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, enabling streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 1000 using a set-top box that delivers data via an HDMI connection in the input block 1130. Still other embodiments provide streamed data to the system 1000 using an RF connection in the input block 1130. As noted above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, e.g., a cellular network or a Bluetooth network.

[0057] The system 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be for a television, a tablet, a laptop, a mobile phone, or other device. The display 1100 can also be integrated with other components (e.g., as in a smartphone) or can be separate (e.g., an external monitor for a laptop). The other peripheral devices 1120 include, in various example embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (both terms DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functionality based on the output of the system 1000. For example, a disc player performs the function of playing the output of the system 1000.

[0058] In various embodiments, control signals are communicated between the system 1000 and the display 1100, speaker 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that allow inter-device control with or without user intervention. The output devices may be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices may be connected to the system 1000 using a communication channel 1060 via the communication interface 1050. The display 1100 and speaker 1110 may be integrated into a single unit with other components of the system 1000 in an electronic device such as, for example, a television. In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0059] The display 1100 and speakers 1110 may alternatively be separate from one or more of the other components, for example, if the RF section of the input 1130 is part of a separate set-top box. In various embodiments in which the display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0060] The embodiments may be executed by the processor 1010 or by computer software implemented by hardware or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memories, and removable memories. The processor 1010 may be of any type suitable for the technical environment and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0061] Block-based video coding. Like HEVC, VVC is built on a block-based hybrid video coding framework. Figure 2A provides a block diagram of a block-based hybrid video encoding system 200. Although variations of this encoder 200 are contemplated, the encoder 200 is described below for clarity without describing all possible variations.

[0062] Before being encoded, the video sequence may undergo a pre-encoding process (204), such as applying a color transformation to the input color image (e.g., from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input image components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata can be attached to the bitstream associated with the pre-processing.

[0063] An input video signal 202 containing an image to be coded is divided (206) and processed block by block. Some blocks may be referred to as coding units (CUs). Different CUs may have different sizes. In VTM-1.0, a CU can be up to 128x128 pixels. However, unlike HEVC, which divides blocks based only on quad-trees, in VTM-1.0, coding tree units (CTUs) are divided into CUs to adapt to different local characteristics based on quad / binary / ternary trees. Additionally, the concept of multiple partition unit types in HEVC is removed, and the separation of CUs, prediction units (PUs), and transform units (TUs) no longer exists in VVC-1.0; instead, each CU is always used as the basic unit for both prediction and transformation without additional partitions. In the multi-type tree structure, the CTU is first divided by a quad-tree structure. Then, each quad-tree leaf node can be further divided by binary and ternary tree structures. There are five splitting types: 4-way split, 2-way split vertically, 2-way split horizontally, 3-way split vertically, and 3-way split horizontally.

[0064] The encoder of FIG. 2A may perform spatial prediction (208) and / or temporal prediction (210). Spatial prediction (or "intra prediction") predicts a current video block using pixels from samples (called reference samples) of already coded neighboring blocks in the same video image / slice. Spatial prediction reduces spatial redundancy inherent in video signals. Temporal prediction (also called "inter prediction" or "motion compensated prediction") predicts a current video block using pixels reconstructed from already coded video images. Temporal prediction reduces temporal redundancy inherent in video signals. The temporal prediction signal for a given CU may be signaled by one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal reference. Also, if multiple reference images are supported, a reference image index may be additionally transmitted and used to identify which reference image in the reference image store (212) the temporal prediction signal comes from.

[0065] A mode decision block (214) at the encoder selects the best prediction mode, for example, based on a rate-distortion optimization method. This selection can be made after spatial and / or temporal prediction is performed. The intra / inter decision may be indicated, for example, by a prediction mode flag. The prediction block is subtracted from the current video block (216) to generate a prediction residual. The prediction residual is decorrelated using a transform (218) and quantization (220). (In some blocks, the encoder may bypass both the transform and quantization, in which case the residual may be directly coded without applying a transform or quantization process.) The quantized residual coefficients are inverse quantized (222) and inverse transformed (224) to form a reconstructed residual, which is then fed back to the prediction block (226) to form a reconstructed signal for the CU. Further in-loop filtering, such as deblocking / SAO (sample adaptive offset) filtering, may be applied to the reconstructed CU (228) to reduce coding artifacts before it is placed into a reference picture store (212) and used to code future video blocks. To form the output video bitstream 230, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy coding unit (108) for further compression and packing to form the bitstream.

[0066] 2B provides a block diagram of a block-based video decoder 250. In the decoder 250, the bitstream is decoded by decoder elements as described below. The video decoder 250 generally performs a decoding pass that is the inverse of the encoding pass as described in FIG. 2A. The encoder 200 also generally performs video decoding as part of the encoding of the video data.

[0067] In particular, the decoder input includes a video bitstream 252, which may be generated by the video encoder 200. The video bitstream 252 is first unpacked and entropy decoded in an entropy decoding unit 254 to obtain transform coefficients, motion vectors, and other coding information. The image partition information indicates how the image is partitioned. Thus, the decoder may partition the image according to the decoded image partition information (256). The coding mode and prediction information are sent to either a spatial prediction unit 258 (if intra-coded) or a temporal prediction unit 260 (if inter-coded) to form a prediction block. The residual transform coefficients are sent to an inverse quantization unit 262 and an inverse transform unit 264 to reconstruct the residual block. The prediction block and the residual block are then added together at 266 to generate a reconstructed block. The reconstructed block may further pass through in-loop filtering 268 before being stored in a reference image store 270, as it is used to predict future video blocks.

[0068] The decoded image 272 may further undergo post-decoding processing (274), such as an inverse color conversion (e.g., YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that reverses the remapping process performed in the pre-encoding process (204). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream. The decoded and processed video may be sent to a display device 276. The display device 276 may be a separate device from the decoder 250, or the decoder 250 and the display device 276 may be components of the same device.

[0069] Various methods and other aspects described in this disclosure may be used to modify modules of video encoder 200 or decoder 250. Additionally, the systems and methods disclosed herein are not limited to VVC or HEVC, but may be applied, for example, to other standards and recommendations, whether existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, aspects described in this disclosure may be used individually or in combination.

[0070] Inter prediction. 3A and 3B are diagrams illustrating examples of motion prediction of video blocks (e.g., using inter prediction module 210 or 260). FIG. 3B, illustrating an example of block-level motion within an image, is a diagram illustrating an example decoded image buffer including reference pictures "Ref pic 0", "Ref pic 1", and "Ref pic2". Blocks B0, B1, and B2 of the current image may be predicted from blocks of reference pictures "Ref pic 0", "Ref pic 1", and "Ref pic2", respectively. Motion prediction may predict the current video block using video blocks from neighboring video frames. Motion prediction may exploit temporal correlation and / or remove temporal redundancy inherent in video signals. For example, in H.264 / AVC and HEVC, temporal prediction may be performed on video blocks of various sizes (e.g., for the luma component, the temporal prediction block size may vary from 16×16 to 4×4 in H.264 / AVC and from 64×64 to 4×4 in HEVC). According to the motion vector of (mvx, mvy), temporal prediction can be performed as provided in the following equation: P(x,y)=ref(x-mvx,y-mvy) where ref(x,y) may be a pixel value at a position (x,y) in a reference picture, and P(x,y) may be a predicted block. The video coding system may support inter prediction with fractional pixel accuracy. If the motion vector (mvx,mvy) has fractional pixel values, one or more interpolation filters may be applied to obtain pixel values ​​at fractional pixel positions. The block-based video coding system may use multi-hypothesis prediction to improve temporal prediction, e.g., a prediction signal may be formed by combining multiple prediction signals from different reference pictures. For example, H.264 / AVC and / or HEVC may use bi-prediction, which may combine two prediction signals. The bi-prediction may combine two prediction signals, each from a reference picture, to form a prediction as in the following equation:

number

[0071] Affine mode. In HEVC, only translational motion models are applied for motion compensated prediction. In the real world, there are various types of motion, such as zoom in / zoom out, rotation, perspective motion, and other irregular motions. In VTM-2.0, affine motion compensated prediction is applied. Affine motion models can be either 4-parameter or 6-parameter. The first flag of each inter coded block is signaled to indicate whether translational or affine motion model is applied for inter prediction. In case of affine motion model, a second flag is sent to indicate whether 4-parameter or 6-parameter model is used.

[0072] A four-parameter affine motion model can have the following parameters: two parameters for horizontal and vertical translational motion, one parameter for zoom motion in both directions, and one parameter for rotational motion in both directions. A single zoom parameter is used because the horizontal zoom parameter is equal to the vertical zoom parameter. A single rotation parameter is used because the horizontal rotation parameter is equal to the vertical rotation parameter. A four-parameter affine motion model is coded in the VTM using two motion vectors at two control point locations defined at the top-left and top-right corners of the current block. As shown in Figure 4A, the affine motion field of a block is coded using two control point motion vectors (V 0 ,V 1 Based on the motion of the control points, the motion field of the affine coded block (v x ,v y ) is written as follows:

number

[0073] In equation (1), (v 0x ,v 0y ) is the motion vector of the control point in the upper left corner, as shown in FIG. 4A, and (v 1x ,v 1y ) is the motion vector of the control point in the upper right corner, and w is the width of the block. In VTM-2.0, the motion field of an affine-coded block is derived at the 4 × 4 subblock level, with (v x ,v y ) is derived and applied to the corresponding 4×4 sub-block. (Note that a set of samples that is referred to as a block in some contexts may be referred to as a sub-block in other contexts. For clarity, various terms may be used in the contexts.)

[0074] These four parameters of the four-parameter affine model can be estimated iteratively. The MV pair at step k is

number

number

[0075] In equation (2), at step k, (a,b) are the delta translation parameters, and (c,d) are the delta zoom and rotation parameters. The delta MVs at a control point can be derived using its coordinates as shown in equations (3) and (4). For example, (0,0), (w,0) are the coordinates of the top-left and top-right control points, respectively.

number

number

[0076] Based on the optical flow equation, the relationship between the change in luminance and the spatial gradient and temporal motion is formulated as follows:

number

[0077] In equation (2),

number

[0078] Since all samples in the block satisfy equation (6), the parameter set (a,b,c,d) can be solved using the least squares error method. The two control points at step (k+1)

number

[0079] An affine motion model with six parameters may have two parameters for horizontal and vertical translational motion, one parameter for zoom motion and one parameter for horizontal rotational motion, and one parameter for zoom motion and one parameter for vertical rotational motion. The six-parameter affine motion model is coded using three MVs at three control points. As shown in the example of Figure 5, the three control points of a six-parameter affine coded block are defined at the top left corner, top right corner, and bottom left corner of the block. The motion at the top left control point is related to translational motion, the motion at the top right control point is related to horizontal rotational and zoom motion, and the motion at the bottom left control point is related to rotational and vertical zoom motion. For the six-parameter affine motion model, the horizontal rotational and zoom motion may not be the same as the vertical motion. Each sub-block (v x ,v y ) can be derived using the three MVs at the control points as follows:

number

[0080] In equation (7), (v 2x ,v 2y ) is the motion vector of the bottom-left control point, (x,y) is the center position of the sub-block, and w and h are the width and height of the block.

[0081] The six parameters of the six-parameter affine model can be estimated in a similar manner as used in the four-parameter model. Equation (2) can be modified to equation (8) as follows:

number

[0082] In equation (8), at step k, (a,b) are the delta translation parameters, (c,d) are the horizontal delta zoom and rotation parameters, and (e,f) are the vertical delta zoom and rotation parameters. Equation (6) is modified accordingly to obtain equation (9). I' k (i,j)-I(i,j)=(g x (i,j)*i)*c+g x (i,j)*j)*d+g y (i,j)*i)*e+(g y (i,j)*j)*f+g x (i,j)*a+g y (i,j)*b (9)

[0083] The parameter set (a,b,c,d,e,f) can be solved using the least squares method by considering all samples in the block.

number

number

number

number

number

[0084] Refining predictions with affine mode optical flow (PROF) To achieve finer granularity of motion compensation, a method is proposed to refine the sub-block-based affine motion compensated prediction by optical flow, as described in Jiancong (Daniel) Luo, Yuwen He, “CE2-related: Prediction refinement with optical flow for affine mode”, JVET-N0236, March 2019, Geneva, Switzerland. After the sub-block-based affine motion compensation is performed, each luma prediction sample is refined by adding the difference derived by the optical flow equation. The proposed PROF is described as including the following steps:

[0085] In the first step, sub-block based affine motion compensation is performed to generate the sub-block prediction I(i,j).

[0086] In the second step, the spatial gradient of the subblock prediction g x (i,j) and g y (i,j) is computed at each sample location using a 3-tap filter [-1,0,1]. g x (i,j) = I(i+1,j) - I(i-1,j) g y (i,j)=I(i,j+1)-I(i,j-1) The subblock prediction is extended by one pixel on each side for gradient computation. To reduce memory bandwidth and complexity, the pixels on the extended boundary are copied from the nearest integer pixel location of the reference image. Thus, additional interpolation of the padding area is avoided.

[0087] In the third step, a refinement of the luminance prediction is calculated by the optical flow equation. ΔI(i,j)=g x (i,j)*Δv x (i,j)+g y (i,j)*Δv y (i,j) (12) Here, Δv(i,j) is the difference between the pixel MV calculated for sample position (i,j), denoted by bv(i,j)y, and the sub-block MV of the sub-block to which pixel (i,j) belongs, as shown in FIG. 6. Since the parameters of the affine model and the pixel position relative to the center of the subblock do not change for each subblock, Δv(i,j) can be calculated for the first subblock and reused for other subblocks within the same block. Let x and y be the horizontal and vertical offsets from the pixel position to the center of the subblock, Δv(x,y) can be derived by the following equation:

number

number

number

[0088] In the fourth step, the refinement of the luma prediction is added to the sub-block prediction I(i,j). The final prediction I′ may be generated using the following equation: I'(i,j)=I(i,j)+ΔI(i,j) (13)

[0089] PROBLEM ADDRESSED IN SOME EMBODIMENTS Current motion compensation processes are usually limited by the precision of the motion vector. For example, the motion vector indicated at the decoder side is used for motion compensated prediction at the sample level, subblock level, or block level, which may determine integer reference sample positions and interpolation filters at fractional positions. The precision of the associated motion vector is a factor of the precision of the motion compensated prediction at each sample. Using four additional bits for the fractional part of the motion vector, 1 / 16 PEL precision may be achieved. However, this precision limitation creates potential problems. One problem is that if the motion compensated prediction is already accurate enough, the extra bits are a waste of signaling overhead. Another problem is that in some cases, the number of additional bits provided may still not be sufficient, and even higher precision may be desirable. To provide a more efficient representation of precision, a more flexible and accurate method may be beneficial to improve the motion compensated precision.

[0090] VTM-5.0 prevents the use of arbitrary precision because if a particular granularity of precision is required, the corresponding interpolation filters are predefined at that precision or higher. For example, if a decoder achieves 1 / 32 PEL precision, which is finer than the 1 / 16 PEL precision of VTM-5.0, the interpolation filters for 1 / 32 PEL precision or higher (e.g., 1 / 64 PEL) can be predefined interpolation filters (e.g., filters defined in the video standard specification).

[0091] Overview of some embodiments. This disclosure describes systems and methods for improving prediction accuracy of a motion compensation (MC) prediction process using optical flow in a flexible manner for use with precision refinement. In some embodiments, after motion compensation is performed, the prediction at each sample is refined by adding a difference value derived by the optical flow equation. Such refinement may be referred to as motion compensation precision refinement by optical flow (MCPROF). The optical flow may be signaled as a refinement of the motion vector at the block level (which may be at the prediction unit level, such as the CU level or the sub-CU level). Since the difference value derived by the optical flow equation may represent different precisions, a more refined precision may be achieved. Some embodiments described herein can achieve pixel-level granularity without significantly increasing the complexity and can keep the worst-case memory access bandwidth the same as normal block-level motion compensation. Various embodiments described herein may be applied to any sub-block-based inter prediction mode and / or block-based inter prediction mode. The embodiments described herein may be applied to both single prediction and bi-prediction, and they may be applied to both inter merge and non-merge modes. One potential advantage of some embodiments is that they provide refinement of accuracy without requiring additional interpolation filters.

[0092] Refining Motion Compensation Accuracy Using Optical Flow (MCPROF) In order to achieve finer accuracy of motion compensation, some embodiments employ a method for refining the motion compensated prediction using optical flow. After the motion compensation process is performed, the luma or / and chroma prediction at each sample is refined by adding the difference derived by the optical flow equation. An example of an encoding method using MCPROF is as follows:

[0093] In the exemplary method, a motion compensation process is used to generate a prediction I(i,j) at each sample position (i,j) after motion estimation in inter non-merge mode. The motion compensation process may be performed using existing inter prediction processes (including single prediction, bi-prediction, and affine prediction). There may be one or more motion compensation processes performed in this step (e.g., multiple motion vector candidates are available). In the case of multiple motion compensation processes, one of the multiple motion compensation processes may be selected according to a predefined criterion (e.g., the motion compensation process with the minimum rate-distortion cost), as is done in VTM5.0.

[0094] One or selected motion compensation process is evaluated to determine whether to use precision refinement. For example, if the existing precision (e.g., 1 / 4 PEL) provided after the MC process is sufficiently accurate, a decision may be made not to proceed with precision refinement. Such a decision may be made, for example, when the residual value is close to 0 (e.g., less than a predetermined magnitude threshold) at the current motion compensation precision.

[0095] However, in some circumstances, it is determined to proceed with precision refinement. In this case, a precision difference N may be determined that is used to convey the degree of refinement. For example, a determination that a substantially optimal (or accurate) residual value is about 9 / 16 PEL may be made through a motion compensation process. To correspond to the optimal precision, a motion compensation precision of 1 / 16 PEL may be desired. However, if the current motion compensation process uses a precision of 1 / 4 PEL (e.g., provided by an existing predefined interpolation filter), the precision difference N between the current precision and the desired precision is 2.

[0096] In some embodiments, the refinement of the motion compensated prediction is determined using optical flow as follows: Let the existing motion vector(s) be MCP(i,j), the uncompressed original input sample value be O(i,j), and the horizontal spatial gradient of MCP(i,j) be g x (i), and the vertical spatial gradient of MCP(i,j) is g y(i,j) is the motion compensated prediction. x ,Δmv y ) is used for optical flow. Motion vector refinement (Δmv x ,Δmv y ) may be selected to substantially satisfy Equation 14. O(i,j)=MCP(i,j)+g x (i,j)*Δmv x +g y (i,j)*Δmv y (14)

[0097] In some embodiments, the refinement of the motion vector (Δmv x ,Δmv y ) is estimated by the least squares method as in equation (15).

number

[0098] If it is decided to use finer precision (e.g., in terms of bit depth), the refinement value of the motion vector Δmv(i,j) (i.e., (Δmv x (i,j),Δmv y (i,j))) may be signaled in the bitstream. In some embodiments, the associated precision difference N may also be signaled in the bitstream. To save signaling overhead, N may be signaled at different levels, such as slice / picture level, CTU level, or CU (or other block) level. Similarly, refinement values ​​of motion vectors may be signaled at different levels, such as slice / picture level, CTU level, CU (or other block) level, or sample level. If refinement values ​​of motion vectors are not signaled at the sample level, refinement values ​​of motion vectors in a block or sub-block may be the same for each sample in that block or sub-block.

[0099] In some embodiments, additional motion vector refinement values ​​may be signaled in the form of specific neighboring sample positions. As shown in FIG. 7, neighboring positions may be used to indicate the motion vector refinement value, which may be represented by one of the four nearest neighboring positions or the eight nearest neighboring positions (e.g., one pixel distance) or even further neighboring positions (e.g., more than one pixel distance). In the example shown in FIG. 7, if a neighboring sample position (i,j-1) is signaled, it has the effect of signaling a motion vector refinement value Δmv(i,j)=(0,-1), where Δmv x (i,j)=0, Δmv y If the adjacent sample position (i-1,j-1) is signaled, it has the effect of signaling a refinement value of the motion vector Δmv(i,j)=(-1,-1), where Δmv x (i,j)=-1, Δmv y (i,j)=-1.

[0100] In some embodiments, the refinement value of the motion vector may be signaled in the form of an index value. For example, if four nearest neighboring positions are used, the corresponding top, bottom, left, and right neighboring positions are indexed as 0, 1, 2, 3 as an example. The indexes may be binarized with a variable length codeword. For example, if eight nearest neighboring positions are allowed, the indexes of the four nearest neighboring positions may be coded using a shorter codeword than the indexes of the other four neighboring positions in the two diagonal directions.

[0101] In some embodiments, the associated precision difference N may be an integer value equal to the bit depth difference between the current precision and the desired precision. For example, if the current precision of a motion compensated prediction process is ¼ PEL and the required precision is 1 / 16 PEL, the signaled precision may be N=2. In some embodiments, the encoder signals a flag indicating whether MCPROF is being used.

[0102] Prediction using precision refinement with optical flow. The precision difference N and the refinement value of the motion vector can be used by the encoder (e.g., in module 210) or the decoder (e.g., in module 260) in generating a prediction for a block or subblock. If MCPROF is used (e.g., if an enablement flag is signaled in the bitstream), the signaled values ​​of Δmv(i,j) and N are obtained. The spatial gradient g x (i,j) and g y (i,j) is calculated for each sample position (i,j). The spatial gradient determination may be performed using a 3-tap filter as described above, or using other techniques.

[0103] An initial motion compensated prediction I(i,j) is generated for the current block using, for example, uni-prediction, bi-prediction, and / or affine prediction. The precision refinement is calculated using a scalar product of the motion vector refinement and the spatial gradient according to Equation 16. ΔI(i,j)=(gx(i,j)*Δmv x (i,j)+g y (i,j)*Δmv y (i,j))≫(N+existing MC precision) (16) where Δmv(i,j) is the signaled motion vector refinement value (e.g., received from the encoder), N is the signaled bit-depth difference between the current precision and the desired precision, and g(i,j) is the spatial gradient calculated as above.

[0104] The motion compensated prediction at each sample is refined by adding a change in intensity (e.g., luma or chroma). The final prediction I′ may be generated according to the following equation: I'(i,j)=I(i,j)+ΔI(i,j) (17)

[0105] In the above exemplary embodiment, four adjacent positions or eight adjacent positions are considered, and each selected adjacent position is used to indicate the direction of improvement in accuracy, without relying on waiting for the motion compensation process of the adjacent samples to be completed.

[0106] An example method is shown in Figure 8. In a video encoder, an initial predicted sample value is obtained for at least a first sample position in a current block of samples based on a motion compensated prediction (802). A motion vector refinement is determined for at least the first sample position (804). The motion vector refinement may be encoded into a bitstream (806), e.g., for storage or transmission.

[0107] A spatial gradient of the sample value is determined at the first sample location (808). A sample difference value is determined based on a scalar product of the spatial gradient and a refinement of the motion vector (810). In some embodiments, determining the sample difference value may include scaling (e.g., bit shifting) the sample difference value, and precision information indicating the amount of scaling may be encoded into the bitstream. The initial predicted sample value is modified based on the sample difference value (812), e.g., adding the sample difference value to the initial predicted sample value to generate a refined sample value.

[0108] In some embodiments, determining a motion vector refinement (804) may include selecting a motion vector refinement to substantially minimize a prediction error for the input video block. In embodiments in which the motion vector refinement is selected on a sample-by-sample basis, the prediction error may be based on a difference (e.g., absolute difference or squared difference) between the refined sample value and a corresponding sample value of the input video block. In embodiments in which the motion vector refinement is selected on a block-by-block (or sub-block) basis, the prediction error may be based on a sum of differences (e.g., sum of absolute difference or squared difference) between the refined sample value and a corresponding input video block sample value across multiple sample positions within the block (or sub-block).

[0109] In some embodiments, the encoder may use the refined sample values ​​generated at 812 in determining a prediction residual, and may also encode the prediction residual into the bitstream.

[0110] In a method performed by a video decoder, the decoder obtains an initial predicted sample value for at least a first sample position in a current block of samples based on the motion compensated prediction (814). A motion vector refinement is determined for at least the first sample position (816), e.g., by decoding the motion vector refinement from a bitstream. A spatial gradient of the sample value is determined at the first sample position (818). A sample difference value is determined by computing a scalar product of the spatial gradient and the motion vector refinement (820). In some embodiments, determining the sample difference value may further include scaling the sample difference value based on precision information decoded from the bitstream. The initial predicted sample value is modified based on the sample difference value (822). For example, the sample difference value may be added to the initial predicted sample value to generate a refined sample value.

[0111] In some embodiments, the decoder may further decode a prediction residual from the bitstream, which may be used in determining a reconstructed sample value at the first sample position. The reconstructed sample value may be displayed or communicated to another display device for display.

[0112] 1 is an exemplary communication system. FIG. 9 illustrates an example of a communication system. The communication system 900 may include an encoder 902, a communication network 904, and a decoder 906. The encoder 902 may communicate with the network 904 via a connection 908, which may be a wired or wireless connection. The encoder 902 may be similar to the block-based video encoder of FIG. 2A. The encoder 902 may include a single-layer or multi-layer codec. The decoder 906 may communicate with the network 904 via a connection 910, which may be a wired or wireless connection. The decoder 906 may be similar to the block-based video decoder of FIG. 2B. The decoder 906 may include a single-layer or multi-layer codec.

[0113] The encoder 902 and / or decoder 906 may be incorporated into a wide variety of wired communication devices and / or wireless transmit / receive units (WTRUs), such as, but not limited to, digital televisions, wireless broadcast systems, network elements / terminals, servers such as content or web servers (e.g., HyperText Transfer Protocol (HTTP) servers), personal digital assistants (PDAs), laptop or desktop computers, tablet computers, digital cameras, digital recording devices, video game devices, video game consoles, cellular or satellite wireless telephones, digital media players, and / or wireless transmit / receive units (WTRUs).

[0114] The communication network 904 may be any suitable type of communication network. For example, the communication network 904 may be a multi-access system that provides content, such as, for example, voice, data, video, messaging, broadcasts, etc., to multiple wireless users. The communication network 904 may enable multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication network 904 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), etc. The communication network 904 may include multiple connected communication networks. The communication network 904 may include the Internet and / or one or more private commercial networks, such as, for example, cellular networks, WiFi hotspots, Internet Service Provider (ISP) networks, etc.

[0115] Additional embodiments. A block-based video coding method according to some embodiments includes, at a decoder, generating a motion-compensated prediction of sample values ​​in a current block of samples; decoding from the bitstream a precision difference value and a motion vector refinement for the current block; for each predicted sample in the current block, determining a spatial gradient at the sample; computing a scalar product of the spatial gradient and the motion vector refinement; scaling the scalar product by an amount indicated by the precision difference value to generate a sample difference value; and adding the sample difference value to the predicted sample value to generate a refined sample value.

[0116] In some embodiments, the motion compensated predictions of the sample values ​​in the current block are generated uni-predictively.

[0117] In some embodiments, the motion compensated predictions of the sample values ​​in the current block are generated using bi-prediction.

[0118] In some embodiments, the refinement of the motion vector is signaled in the bitstream as an index. The index may identify one of a plurality of motion vector refinements of the form (i,j), where i and j are integers. The index may identify one of a plurality of motion vector refinements from the group consisting of (0,-1), (1,0), (0,1) and (-1,0). The index may identify one of a plurality of motion vector refinements from the group consisting of (0,-1), (1,0), (0,1), (-1,0), (-1,-1), (1,-1), (1,1) and (-1,1).

[0119] In some embodiments, scaling the scalar product includes bit-shifting the scalar product.

[0120] In some embodiments, the precision difference value is N, and scaling the scalar product includes right-shifting the scalar product by a number of bits equal to the sum of the signaled precision difference value N and the existing MC precision.

[0121] A block-based video coding method according to some embodiments includes, at an encoder, generating, for an input video block, a motion compensated prediction of sample values ​​in a current block of samples; selecting a precision difference value; determining a respective spatial gradient at the samples; determining a motion vector refinement for the current block, where the motion vector refinement is selected to substantially minimize an error between (i) a scalar product of the spatial gradient and the motion vector refinement and (ii) a difference between the input video block and the motion compensated prediction; and signaling in a bitstream the precision difference value and the motion vector refinement for the current block.

[0122] In some embodiments, the motion compensated predictions of the sample values ​​in the current block are generated uni-predictively.

[0123] In some embodiments, the motion compensated predictions of the sample values ​​in the current block are generated using bi-prediction.

[0124] In some embodiments, the refinement of the motion vector is signaled in the bitstream as an index. The index may identify one of a plurality of motion vector refinements of the form (I,j), where i and j are integers. The index may identify one of a plurality of motion vector refinements from the group consisting of (0,-1), (1,0), (0,1) and (-1,0). The index may identify one of a plurality of motion vector refinements from the group consisting of (0,-1), (1,0), (0,1), (-1,0), (-1,-1), (1,-1), (1,1) and (-1,1).

[0125] In some embodiments, the motion vector refinement is selected to substantially minimize (i) the sum of the scalar product of the spatial gradient and the motion vector refinement, and (ii) the sum of the squared differences between the difference between the input video block and the motion compensated prediction.

[0126] Some embodiments comprise a processor and a non-transitory computer-readable medium operative to perform any of the functions described herein.

[0127] This disclosure describes a wide variety of aspects, including tools, features, embodiments, models, approaches, and the like. Many of these aspects are specifically described and may be described in a manner that sounds restrictive to at least indicate their individual characteristics. However, this is for clarity of description and does not limit the disclosure or scope of these aspects. In fact, all of the different aspects can be combined and interchanged to provide further aspects. Additionally, these aspects can also be combined and interchanged with aspects described in previous applications.

[0128] The aspects described and contemplated in this application can be implemented in many different formats. While some embodiments are specifically illustrated, other embodiments are contemplated, and discussion of a particular embodiment is not intended to limit the breadth of implementation. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0129] In this disclosure, the terms "reconstructed" and "decoded" may be used synonymously, the terms "pixel" and "sample" may be used synonymously, and the terms "image," "picture," and "frame" may be used synonymously.

[0130] Various methods are described herein, each of which includes one or more steps or acts for achieving the described method. Insofar as a particular order of steps or acts is not necessary for the correct operation of the method, the order and / or use of particular steps and / or acts may be modified or combined. Furthermore, terms such as "first", "second", etc. may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decode" and "second decode". The use of such terms does not imply a modified order of operations, unless specifically required. Thus, in this example, the first decode need not be performed before the second decode, but may occur, for example, before, during, or during an overlapping period with the second decode.

[0131] For example, various numerical values ​​may be used in this disclosure, and the specific values ​​are for illustrative purposes and the described aspects are not limited to these specific values.

[0132] The embodiments described herein may be performed by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The processor may be of any type appropriate to the technical environment, and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0133] Various implementations involve decoding. "Decoding" as used herein may, for example, encompass all or part of the processes performed on a received encoded sequence to generate a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes may also or alternatively include processes performed by the decoder of various implementations described herein, such as extracting an image from a tiled (packed) image, determining an upsample filter to use, and then upsampling the image, and flipping the image back to the intended orientation.

[0134] As a further example, in one embodiment, "decoding" refers to only entropy decoding, in another embodiment, "decoding" refers to only differential decoding, and in another embodiment, "decoding" refers to a combination of entropy and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process in general will be clear based on the context of a particular description and will be well understood by one of ordinary skill in the art.

[0135] Various implementations involve encoding. Similar to the above discussion regarding "decoding," "encoding" as used herein can encompass all or part of the processes performed on an input video sequence to generate, for example, an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transforming, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by an encoder in the various implementations described herein.

[0136] As a further example, in one embodiment, "encoding" refers to only entropy encoding, in another embodiment, "encoding" refers to only differential encoding, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to the broader encoding process in general will be clear based on the context of the particular description and will be well understood by one of ordinary skill in the art.

[0137] Where a diagram is presented as a flow diagram, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, where a diagram is presented as a block diagram, it should be understood that it also provides a flow diagram of the corresponding method / process.

[0138] Various embodiments refer to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is usually considered, often given computational complexity constraints. Rate-distortion optimization is usually formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. There are various approaches to solving the rate-distortion optimization problem. For example, the approach may be based on an extensive test of all encoding options, including all modes or coding parameter values ​​considered, with a full evaluation of the coding cost and associated distortion of the reconstructed signal after coding and decoding. In particular, faster approaches may be used to reduce the coding complexity by calculating approximate distortion based on a prediction or prediction residual signal rather than a reconstructed signal. These two approaches may also be used in combination, such as by using approximate distortion for only some of the possible encoding options and full distortion for others. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a full evaluation of both the coding cost and the associated distortion.

[0139] The implementations and aspects described herein may be implemented, for example, in a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only a single implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). The apparatus may be implemented, for example, in appropriate hardware, software, and firmware. The methods may be implemented, for example, in a processor, which refers generally to processing devices including, for example, computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0140] References to "one embodiment" or "one implementation" or "one implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in one embodiment" or "in one implementation," as well as any other variations, in various places throughout this application are not necessarily all referring to the same embodiment.

[0141] Additionally, this disclosure may refer to "determining" various portions of information. Determining the information may include, for example, one or more of evaluating the information, calculating the information, predicting the information, or retrieving the information from a memory.

[0142] Additionally, the application may refer to "accessing" various portions of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or evaluating information.

[0143] Additionally, the application may refer to "receiving" various portions of information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Additionally, "receiving" typically involves in some manner, for example, an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or evaluating information.

[0144] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to cover the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to cover the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of only the third listed option (C), or the selection of only the first and second listed options (A and B), or the selection of only the first and third listed options (A and C), or the selection of only the second and third listed options (B and C), or the selection of all three options (A and B and C). This may be extended to any number of items listed.

[0145] Also, as used herein, the word "signaling" refers to, among other things, instructing a corresponding decoder to do something. For example, in a particular embodiment, an encoder signals a particular one of a plurality of parameters for refinement. In this way, in an embodiment, the same parameters are used at both the encoder and decoder sides. Thus, for example, an encoder can transmit a particular parameter to a decoder (explicit signaling), so that the decoder can use the same particular parameter. Conversely, if the decoder already has a particular parameter as well as other parameters, signaling can be used without transmission (implicit signaling) to allow the decoder to easily recognize and select the particular parameter. By avoiding the transmission of any actual function, bit savings are realized in various embodiments. It should be appreciated that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used in various embodiments to signal information to a corresponding decoder. Although the above is directed to the verb form of the word "signaling", the word "signaling" can also be used herein as a noun.

[0146] The implementations can generate various signals formatted to carry information that can be stored or transmitted, for example. The information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signals can be formatted to carry a bit stream of the described embodiments. Such signals can be formatted, for example, as electromagnetic waves (e.g., using a radio frequency portion of the spectrum) or as baseband signals. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signals carry can be, for example, analog or digital information. The signals can be transmitted over a variety of different wired or wireless links, as is known. The signals can be stored on a processor-readable medium.

[0147] Several embodiments are described. The features of these embodiments can be provided alone or in any combination across various claim categories and types. In addition, the embodiments can include one or more of the following features, devices, or aspects, alone or in any combination across various claim categories and types. A bitstream or signal including syntax conveying information generated according to any of the described embodiments. Creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments. · A method, process, apparatus, instruction storage medium, data storage medium, or signal according to any of the described embodiments. A TV, set-top box, mobile phone, tablet or other electronic device performing the encoding or decoding method according to any of the described embodiments. A TV, set-top box, mobile phone, tablet, or other electronic device that performs the decoding method according to any of the described embodiments and displays the resulting image (e.g., using a monitor, screen, or other type of display). A TV, set-top box, mobile phone, tablet, or other electronic device that selects a channel (e.g., using a tuner) to receive a signal containing an encoded image and performs decoding according to any of the described embodiments. A TV, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing an encoded image wirelessly (e.g., using an antenna) and performs decoding according to any of the described embodiments.

[0148] It should be noted that various hardware elements of one or more of the described embodiments are referred to as "modules" that perform (i.e., perform, implement, etc.) various functions described herein with respect to the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) that is deemed suitable by one of ordinary skill in the relevant art for a given implementation. It should be noted that each described module may also include executable instructions to perform one or more functions described as being performed by the respective module, which may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., or may be stored in any suitable non-transitory computer-readable medium(s), commonly referred to as RAM, ROM, etc.

[0149] Although the features and elements are described above in certain combinations, one skilled in the art will understand that each feature or element may be used alone or in any combination with the other features and elements. Furthermore, the methods described herein may be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. 1. A video coding method, comprising: obtaining an initial predicted sample value based on the motion compensated prediction for at least a first sample position in a current block of samples; determining a refinement of a motion vector associated with at least the first sample position; determining a spatial gradient of sample values ​​at the first sample location; determining a sample difference value based on a scalar product of the spatial gradient and a refinement of the motion vector; and modifying the initial predicted sample value based on the sample difference value.

2. The method of claim 1 , wherein determining the motion vector refinement is performed by a decoder, the decoder comprising decoding the motion vector refinement from a bitstream.

3. 3. The method of claim 2, further comprising decoding refinement precision information from the bitstream, and wherein determining the sample difference value comprises scaling the scalar product by an amount indicated by the precision information.

4. The method of claim 3 , wherein scaling the scalar product comprises bit-shifting the scalar product by an amount indicated by the precision information.

5. 5. The method of claim 3, wherein the motion compensated prediction is performed using at least one motion vector having an initial precision, and the refinement precision information comprises a precision difference value representing a difference between the initial precision and the refinement precision.

6. The method of any one of claims 3 to 5, wherein scaling the scalar product comprises right-shifting the scalar product by a number of bits equal to the sum of the precision difference value and the initial precision.

7. 2. The method of claim 1, wherein determining a motion vector refinement comprises selecting a motion vector refinement to substantially minimize a prediction error for an input video block, and the method further comprises encoding the motion vector refinement in a bitstream.

8. The method of any one of claims 2 to 7, wherein the refinement of the motion vector is signaled in the bitstream as an index.

9. 9. The method of claim 8, wherein the index identifies one of a number of motion vector refinements from the group consisting of (0,-1), (1,0), (0,1), and (-1,0).

10. 9. The method of claim 8, wherein the index identifies one of a number of motion vector refinements from the group consisting of: (0,-1), (1,0), (0,1), (-1,0), (-1,-1), (1,-1), (1,1), and (-1,1).

11. A method according to any one of claims 1 to 10, wherein the refinement of the motion vector is associated with all sample positions in the current block.

12. An apparatus comprising a processor, the processor comprising at least obtaining an initial predicted sample value based on the motion compensated prediction for at least a first sample position in a current block of samples; determining a refinement of a motion vector associated with at least the first sample position; determining a spatial gradient of sample values ​​at the first sample location; determining a sample difference value based on a scalar product of the spatial gradient and a refinement of the motion vector; and modifying the initial predicted sample value based on the sample difference value.

13. The apparatus of claim 12 , wherein the apparatus includes a decoder, and wherein determining a refinement of a motion vector includes decoding the refinement of the motion vector from a bitstream.

14. 14. The apparatus of claim 13, wherein the processor is further configured to decode refinement precision information from the bitstream, and wherein determining the sample difference value comprises scaling the scalar product by an amount indicated by the precision information.

15. 13. The apparatus of claim 12, wherein the apparatus comprises an encoder, and determining a refinement of the motion vector comprises selecting a refinement of the motion vector to substantially minimize an error between the refined sample values ​​and corresponding sample values ​​of an input video block, and the method further comprises encoding the refinement of the motion vector in a bitstream.

Images