Precision refinement with motion compensation of optical flow

Abstract

Systems and methods for refining motion compensated prediction in block-based video coding are described. In an example embodiment, motion compensated prediction is used to generate predicted sample values in a current block of samples. A motion vector refinement and a precision difference value for the current block are signaled in a bitstream. For each sample in the current block, a spatial gradient is computed at that sample, and a dot product between the spatial gradient and the motion vector refinement is computed. The dot product is scaled (e.g., bit shifted) by an amount indicated by the precision difference value to produce a sample difference value, and the sample difference value is added to the predicted sample value to produce a refined sample value.

Description

[0001] Cross-reference to related applications

[0002] This application is a non-provisional application of the following application and claims the benefit of it under 35 USC §119(e): U.S. Provisional Patent Application No. 62 / 864,825, filed June 21, 2019, entitled “Precision Refinement for Motion Compensation with Optical Flow,” which is incorporated herein by reference in its entirety. Background Technology

[0003] Video decoding systems are widely used to compress digital video signals to reduce the storage requirements and / or transmission bandwidth of such signals. Among various types of video decoding systems, such as block-based, wavelet-based, and object-based systems, block-based hybrid video decoding systems are currently the most widely used and deployed. Examples of block-based video decoding systems include international video decoding standards such as MPEG-1 / 2 / 4 Part 2, H.264 / MPEG-4 Part 10AVC, VC-1, and High Efficiency Video Decoding (HEVC), which was developed by ITU-T / SG16 / Q.6 / VCEG and ISO / IEC / MPEG's JCT-VC (Joint Collaboration Team for Video Decoding).

[0004] In October 2017, ITU-T and ISO / IEC issued a joint proposal (CfP) on video compression capabilities exceeding HEVC. In April 2018, at the 10th JVET meeting, 22 CfP responses in the standard dynamic range category were received and evaluated, demonstrating compression efficiency gains exceeding HEVC by approximately 40%. Based on these evaluation results, the Joint Video Experts Group (JVET) launched a new project to develop a next-generation video decoding standard called Universal Video Decoding (VVC). In the same month, a reference software codebase called the VVC Test Model (VTM) was established to demonstrate a reference implementation of the VVC standard. For the initial VTM-1.0, most decoding modules, including intra-frame prediction, inter-frame prediction, transform / inverse transform, quantization / dequantization, and in-loop filters, follow the existing HEVC design, except for the use of a multi-type tree-based block partitioning structure in the VTM. Simultaneously, another reference software library called the Baseline Set (BMS) was generated to facilitate the evaluation of the new decoding tools. The BMS codebase includes a list of decoding tools inherited from the Joint Exploratory Model (JEM), offering higher coding efficiency and moderate implementation complexity. This list is included on top of VTM and used as a benchmark when evaluating similar decoding techniques during the VVC standardization process. Specifically, nine JEM decoding tools are integrated in BMS-1.0, including 65-angle intra-frame prediction direction, modified coefficient decoding, Advanced Multiple Transform (AMT) + 4×4 Inseparable Quadratic Transform (NSST), Affine Motion Model, Generalized Adaptive Loop Filter (GALF), Advanced Temporal Motion Vector Prediction (ATMVP), Adaptive Motion Vector Precision, Decoder-Side Motion Vector Refinement (DMVR), and Linear Model (LM) chroma mode. Summary of the Invention

[0005] The embodiments described herein include methods used in video encoding and decoding (collectively, "decoding"). Systems and methods for refining motion-compensated predictions in block-based video coding are described. A video decoding method according to some embodiments includes: generating initial predicted sample values ​​for at least a first sample location in a current sample block using motion-compensated predictions; determining a motion vector refinement associated with at least the first sample location; determining a spatial gradient of the sample values ​​at the first sample location; determining a sample difference by computing a scalar product of the spatial gradient and the motion vector refinement; and adding the sample difference to the initial predicted sample values ​​to produce refined sample values.

[0006] In some embodiments, the method is performed by a decoder, and determining the motion vector refinement includes decoding the motion vector refinement from the bitstream.

[0007] Some embodiments further include decoding refinement precision information from the bitstream, wherein determining the sample difference includes scaling the scalar product by an amount indicated by the precision information. The scaling of the scalar product may include shifting the scalar product bits by an amount indicated by the precision information.

[0008] In some embodiments, the motion compensation prediction is performed using at least one motion vector with initial accuracy, and the refinement accuracy information includes an accuracy difference value representing the difference between the initial accuracy and the refinement accuracy.

[0009] In some embodiments, scaling the scalar product includes shifting the scalar product to the right by a number of bits equal to the sum of the precision difference and the initial precision.

[0010] In some embodiments, the method is performed by an encoder, the determination of the motion vector refinement includes: selecting the motion vector refinement to substantially minimize the prediction error relative to the input video block, and the method further includes: encoding the motion vector refinement in a bitstream.

[0011] In some embodiments, the motion vector thinning is transmitted as an index in the bitstream using a signal. In some embodiments, the index may identify one of a plurality of motion vector thinnings from a group consisting of (0,-1), (1,0), (0,1), and (-1,0). In some other embodiments, the index may identify one of a plurality of motion vector thinnings from a group consisting of (0,-1), (1,0), (0,1), (-1,0), (-1,-1), (1,-1), (1,1), and (-1,1).

[0012] In some embodiments, the motion vector refinement is associated with the positions of all samples in the current block (or the current sub-block). In some other embodiments, the motion vector refinement is determined on a sample-by-sample basis and may be different for different samples.

[0013] Some embodiments include at least one processor configured to perform any of the methods described herein. In some such embodiments, a computer-readable medium (e.g., a non-transitory medium) is provided to store instructions that operate to perform any of the methods described herein.

[0014] Some embodiments include a computer-readable medium (e.g., a non-transitory medium) for storing video encoded using one or more of the methods disclosed herein.

[0015] An encoder or decoder system may include a processor and a non-transitory computer-readable medium storing instructions for performing the methods described herein.

[0016] One or more embodiments of the present invention also provide a computer-readable storage medium storing instructions for encoding or decoding video data according to any of the methods described above. This embodiment also provides a computer-readable storage medium (e.g., a non-transitory medium) storing a bitstream generated according to the methods described above. This embodiment also provides a method and apparatus for transmitting a bitstream generated according to the methods described above. This embodiment also provides a computer program product including instructions for performing any of the described methods. Attached Figure Description

[0017] Figure 1A This is a system diagram illustrating an example communication system in which one or more of the disclosed embodiments may be implemented.

[0018] Figure 1B It is shown that, according to one embodiment, it is possible to Figure 1A The diagram shows a system diagram of an example wireless transmit / receive unit (WTRU) used in a communication system.

[0019] Figure 1C This is a functional block diagram of the system used in some of the embodiments described herein.

[0020] Figure 2A This is a functional block diagram of a block-based video encoder (e.g., an encoder for VVC).

[0021] Figure 2B This is a functional block diagram of a block-based video decoder (e.g., a decoder used for VVC).

[0022] Figures 3A-3B The motion compensation prediction is shown.

[0023] Figure 4A A four-parameter affine motion model is shown.

[0024] Figure 4B The sub-block level motion derivation of the affine block is shown.

[0025] Figure 5 The six-parameter affine pattern is shown: V0, V1, and V2 are the motion vectors of the control points; (MVx, MVy) is the motion vector of the sub-block centered at position (x, y).

[0026] Figure 6 Showing the sub-block motion vector V SB And pixel thinning Δv(i,j).

[0027] Figure 7 This demonstrates using adjacent locations to send additional motion vectors via signals.

[0028] Figure 8 This is a flowchart illustrating encoding and decoding methods performed in some embodiments, which includes decoding functions that can be performed at both the encoder and the decoder.

[0029] Figure 9 This is a diagram illustrating an example communication system.

[0030] Example devices and networks used for implementing the embodiments

[0031] Figure 1A This is an illustration of an example communication system 100 in which one or more of the disclosed embodiments may be implemented. The communication system 100 may be a multi-access system that provides content such as voice, data, video, messaging, broadcasting, etc., to multiple wireless users. The communication system 100 enables multiple wireless users to access this content through the sharing of system resources (including wireless bandwidth). For example, the communication system 100 may use one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero Tail Unique Word DFT-Extended OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multicarrier (FBMC), etc.

[0032] like Figure 1AAs shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, Internet 110, and other networks 112. However, it should be understood that the disclosed embodiments cover any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a wireless environment. As an example, any of WTRUs 102a, 102b, 102c, and 102d may be referred to as a “station” and / or “STA”, and may be configured to transmit and / or receive wireless signals. They may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscriber-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain environments), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, and so on. Any of WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0033] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a and 114b may be configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks (e.g., CN 106 / 115, Internet 110, and / or Network 112). As an example, base stations 114a and 114b may be base transceiver stations (BTS), node B, e-node B, home node B, home e-node B, gNB, NR node B, site controller, access point (AP), wireless router, etc. Although each of base stations 114a and 114b is described as a single element, it should be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0034] Base station 114a may be part of RAN 104 / 113, and may also include other base station and / or network elements (not shown), such as base station controller (BSC), radio network controller (RNC), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. Cells can provide coverage of radio services to specific geographic areas that may be relatively fixed or change over time. Cells may also be divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, for example, one transceiver for each sector of the cell. In one embodiment, base station 114a may use multiple-input multiple-output (MIMO) technology and may use multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0035] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116, which can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Air interface 116 can be established using any suitable radio access technology (RAT).

[0036] More specifically, as described above, the communication system 100 can be a multi-access system and can use one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN 104 / 113 can implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can use Wideband CDMA (WCDMA) to establish air interfaces 115 / 116 / 117. WCDMA can include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA can include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0037] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement radio technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may use Long Term Evolution (LTE) and / or LTE-A Advanced (LTE-A) and / or LTE-A Pro to establish air interface 116.

[0038] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement radio technologies such as NR radio access, which can use New Radio (NR) to establish air interface 116.

[0039] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement various radio access technologies. For example, base station 114a and WTRUs 102a, 102b, 102c can, for example, use the dual connectivity (DC) principle to implement LTE radio access and NR radio access. Therefore, the air interface used by WTRUs 102a, 102b, 102c can be characterized by various types of radio access technologies and / or transmissions sent to / from various types of base stations (e.g., eNB and gNB).

[0040] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate GSM Evolution (EDGE), and GSM EDGE (GERAN).

[0041] Figure 1ABase station 114b can be, for example, a wireless router, a home node B, a home e-node B, or an access point, and can use any suitable RAT to facilitate wireless connectivity in localized areas such as commercial areas, homes, vehicles, campuses, industrial facilities, air corridors (e.g., for use by drones), and roads. In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d can use cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish picocells or femtocells. Figure 1A As shown, base station 114b can have a direct connection to the Internet 110. Therefore, base station 114b does not need to access the Internet 110 via CN 106 / 115.

[0042] RAN 104 / 113 can communicate with CN 106 / 115, which can be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRU 102a, 102b, 102c, and 102d. Data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. CN 106 / 115 can provide call control, billing services, location-based services, prepaid calling, internet connectivity, video distribution, etc., and / or perform advanced security functions such as user authentication. Although... Figure 1A Although not shown, it should be understood that RAN 104 / 113 and / or CN 106 / 115 can communicate directly or indirectly with other RANs using the same RAT as RAN 104 / 113 or a different RAT. For example, in addition to connecting to RAN 104 / 113, which can use NR radio technology, CN 106 / 115 can also communicate with another RAN (not shown) using GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0043] CN 106 / 115 can also serve as a gateway for WTRU 102a, 102b, 102c, 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols, such as Transmission Control Protocol (TCP) / Internet Protocol (IP) suite, TCP, User Datagram Protocol (UDP), and / or IP. Network 112 may include wireless and / or wired communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN 104 / 113 or a different RAT.

[0044] Some or all of the WTRUs 102a, 102b, 102c, and 102d in communication system 100 may include multi-mode capability (i.e., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example, Figure 1A The WTRU 102c shown can be configured to communicate with base station 114a, which can use cellular-based radio technology, and with base station 114b, which can use IEEE 802 radio technology.

[0045] Figure 1B This is a system diagram of example WTRU 102. (Example:) Figure 1B As shown, WTRU 102 may include a processor 118, a transceiver 120, a transmitting / receiving element 122, a speaker / microphone 124, a keyboard 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a Global Positioning System (GPS) chipset 136, and other peripheral devices 138, etc. It should be understood that, while remaining consistent with the embodiments, WTRU 102 may include any sub-combination of the above-described elements.

[0046] Processor 118 can be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. Processor 118 can perform signal encoding, data processing, power control, input / output processing, and / or any other function that enables WTRU 102 to operate in a wireless environment. Processor 118 can be coupled to transceiver 120, which can be coupled to transmitting / receiving element 122. Although... Figure 1B The processor 118 and transceiver 120 are described as separate components, but it should be understood that the processor 118 and transceiver 120 can be integrated together into an electronic package or chip.

[0047] Transmitting / receiving element 122 can be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via air interface 116. For example, in one embodiment, transmitting / receiving element 122 can be an antenna configured to transmit and / or receive RF signals. For example, in one embodiment, transmitting / receiving element 122 can be a transmitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, transmitting / receiving element 122 can be configured to transmit and / or receive both RF signals and optical signals. It should be understood that transmitting / receiving element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0048] Furthermore, although the transmitting / receiving element 122 is in Figure 1B While described as a single element, WTRU 102 may include any number of transmitting / receiving elements 122. More specifically, WTRU 102 may use MIMO technology. Thus, in one embodiment, WTRU 102 may include two or more transmitting / receiving elements 122 (e.g., multiple antennas) for transmitting and / or receiving wireless signals via air interface 116.

[0049] Transceiver 120 can be configured to modulate signals to be transmitted by transmitting / receiving element 122 and to demodulate signals received by transmitting / receiving element 122. As described above, WTRU 102 can have multi-mode capability. Thus, transceiver 120 can include multiple transceivers to enable WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0050] The processor 118 of WTRU 102 can be coupled to a speaker / microphone 124, a keyboard 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit), and can receive user input data from the aforementioned devices. The processor 118 can also output user data to the speaker / microphone 124, keyboard 126, and / or display / touchpad 128. Furthermore, the processor 118 can access information from any type of suitable memory, and store data in any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. Non-removable memory 130 can include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 can include a subscriber identification module (SIM) card, memory stick, secure digital storage (SD) card, etc. In other embodiments, processor 118 may access information from memory that is not physically located on WTRU 102 (e.g., on a server or home computer (not shown)) and store data in such memory.

[0051] The processor 118 can receive electrical energy from the power supply 134 and can be configured to distribute that electrical energy to other components in the WTRU 102 and / or control the electrical energy of other components in the WTRU 102. The power supply 134 can be any device used to power the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0052] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. As a supplement to or alternative to the information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via air interface 116, and / or determine its location based on the timing of signals received from two or more adjacent base stations. It should be understood that, while remaining consistent with the embodiments, the WTRU 102 may acquire location information using any suitable location determination method.

[0053] The processor 118 can also be coupled to other peripheral devices 138, which may include one or more software and / or hardware modules providing additional features, functions, and / or wireless or wired connectivity. For example, peripheral devices 138 may include an accelerometer, an electronic compass (e-compass), a satellite transceiver, a digital camera (for photos and / or video), a Universal Serial Bus (USB) port, a vibration device, a television transceiver, a hands-free headset, etc. Modules, FM radio units, digital music players, media players, video game console modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, etc. Peripheral device 138 may include one or more sensors, which may be one or more of the following: gyroscope, accelerometer, Hall effect sensor, magnetometer, orientation sensor, proximity sensor, temperature sensor, time sensor, geolocation sensor, altimeter, light sensor, touch sensor, magnetometer, barometer, gesture sensor, biosensor, and / or humidity sensor.

[0054] WTRU 102 may include a full-duplex radio for which the transmission and reception of some or all signals (e.g., associated with specific subframes for both UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference through hardware (e.g., a choke) or through signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In one embodiment, WTRU 102 may include a half-duplex radio for the transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)).

[0055] Despite WTRU in Figure 1A-1B While described as a wireless terminal, it is anticipated that in some representative embodiments, such a terminal may (e.g., temporarily or permanently) use a wired communication interface with a communication network.

[0056] In a representative embodiment, the other network 112 may be a WLAN.

[0057] refer to Figure 1A-1B As described herein, one or more of the functions described herein can be performed by one or more emulation devices (not shown). An emulation device can be one or more devices configured to emulate one or more of the functions described herein. For example, an emulation device can be used to test other devices and / or emulate network and / or WTRU functions.

[0058] Simulation devices can be designed to perform one or more tests on other devices in a laboratory environment and / or a carrier network environment. For example, one or more simulation devices may perform one or more or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. One or more simulation devices may perform one or more or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. Simulation devices may be directly coupled to another device for testing purposes and / or may be used to perform tests via over-the-air wireless communication.

[0059] One or more analog devices may perform one or more functions, including all functionalities, rather than being implemented / deployed as part of a wired and / or wireless communication network. For example, analog devices may be utilized in test scenarios within test laboratories and / or non-deployed (e.g., test) wired and / or wireless communication networks to perform testing of one or more components. One or more analog devices may be test equipment. Analog devices may transmit and / or receive data using direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas).

[0060] Exemplary System

[0061] Some embodiments use, for example Figure 1C This is achieved through a system. Figure 1C This is a block diagram illustrating examples of systems implementing various aspects and embodiments thereof. System 1000 can be implemented as a device including the various components described below and configured to perform one or more aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000 can be implemented individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described herein.

[0062] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing various aspects, such as those described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, the storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0063] System 1000 includes an encoder / decoder module 1030 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents one or more modules that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Alternatively, the encoder / decoder module 1030 may be implemented as a separate element of system 1000 or may be incorporated into processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0064] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described herein may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more of various items during the execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or portions of the decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0065] In some embodiments, the memory within the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. External memory may be memory 1020 and / or storage device 1040, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video decoding and decoding operations, such as working memory for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Decoding; also known as H.265 and MPEG-H Part 2), or VVC (Universal Video Decoding, a new standard developed by the Joint Video Team experts JVET).

[0066] As shown in box 1130, input to the components of system 1000 can be provided through various input devices. Such input devices include, but are not limited to: (i) an RF section that receives radio frequency (RF) signals transmitted over the air, for example by a broadcaster; (ii) component (COMP) input terminals (or a set of component input terminals); (iii) universal serial bus (USB) input terminals; and / or (iv) high-definition multimedia interface (HDMI) input terminals. Figure 1C Other examples not shown include composite videos.

[0067] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting a signal band to a band), (ii) downconverting the selected signal, (iii) again limiting the band to a narrower band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments), (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements to perform these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various of these functions, including, for example, downconverting received signals to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection to a desired frequency band through filtering, down-conversion, and re-filtering. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0068] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Solomon error correction) can be implemented as needed, for example, within a separate input processing IC or processor 1010. Similarly, various aspects of USB or HDMI interface processing can be implemented as needed within a separate interface IC or within processor 1010. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0069] Various components of system 1000 can be housed within an integrated housing. Within this integrated housing, various components can be interconnected and transmit data therebetween using a suitable connection arrangement 1140 (e.g., internal buses known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0070] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0071] In various embodiments, a wireless network (e.g., a Wi-Fi network, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) is used to stream or otherwise provide data to the system 1000. In these embodiments, the Wi-Fi signal is received via a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other cloud-based communications. Other embodiments use a set-top box that transmits data via an HDMI connection to input box 1130 to provide streaming data to the system 1000. Still other embodiments use an RF connection to input box 1130 to provide streaming data to the system 1000. As described above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0072] The system 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripheral devices 1120. The display 1100 in various embodiments includes one or more of the following: for example, a touchscreen display, an organic light-emitting diode (OLED) display, a flexible display, and / or a foldable display. The display 1100 can be used in televisions, tablets, laptops, cellular phones (mobile phones), or other devices. The display 1100 can also be integrated with other components (e.g., in a smartphone) or stand alone (e.g., an external monitor for a laptop computer). In various examples of embodiments, the other peripheral devices 1120 include one or more of the following: a standalone digital video disc (or digital multifunction disc) (DVR, for both), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functionality based on the output of the system 1000. For example, a disc player performs the function of playing the output of the system 1000.

[0073] In various embodiments, signaling (such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention) is used to transmit control signals between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120. The output devices may be communicatively coupled to system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, the output devices may be connected to system 1000 via communication interface 1050 using communication channel 1060. The display 1100 and speaker 1110 may be integrated into a single unit within an electronic device (e.g., a television set) along with other components of system 1000. In various embodiments, display interface 1070 includes a display driver, such as a timing controller ((TCon) chip).

[0074] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separated from one or more other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, such as an HDMI port, a USB port, or a COMP output.

[0075] These embodiments can be implemented by processor 1010 or by computer software implemented in hardware or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 1020 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 1010 can be of any type suitable for the technical environment and, as a non-limiting example, can include one or more of the following: microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures. Detailed Implementation

[0076] Block-based video decoding

[0077] Similar to HEVC, VVC is built on a block-based hybrid video decoding framework. Figure 2A A block diagram of a block-based hybrid video coding system 200 is given. Variations of this encoder 200 can be envisioned, but for clarity, the encoder 200 is described below without describing all anticipated variations.

[0078] Before being encoded, the video sequence may undergo pre-coding processing (204), such as applying a color transformation to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input image components to obtain a signal distribution more resilient to compression (e.g., using histogram equalization with one of the color components). Metadata may be associated with this pre-processing and appended to the bitstream.

[0079] The input video signal 202, including the image to be encoded, is partitioned (206) and processed block by block. Some blocks may be called decoding units (CUs). Different CUs can have different sizes. In VTM-1.0, CUs can be up to 128×128 pixels. However, unlike HEVC, which is based solely on quadtrees for block partitioning, in VTM-1.0, decoding tree units (CTUs) are divided into multiple CUs based on quadtrees / binaries / tritrees to accommodate varying local characteristics. Furthermore, the concept of multiple partition unit types in HEVC is removed, so in VVC-1.0 there is no longer a separation between CUs, prediction units (PUs), and transform units (TUs); instead, each CU is always used as the basic unit for both prediction and transformation without further partitioning. In the multi-type tree structure, CTUs are first partitioned using a quadtree structure. Then, each quadtree leaf node can be further partitioned using binary and ternary tree structures. There are five partitioning types: quad partitioning, vertical binary partitioning, horizontal binary partitioning, vertical ternary partitioning, and horizontal ternary partitioning.

[0080] exist Figure 2A In the encoder, spatial prediction (208) and / or temporal prediction (210) can be performed. Spatial prediction (or “intra-frame prediction”) uses pixels from samples (referred to as reference samples) of decoded adjacent blocks in the same video picture / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also referred to as “inter-frame prediction” or “motion-compensated prediction”) uses reconstructed pixels from decoded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. A temporal prediction signal for a given CU can be sent by signaling one or more motion vectors (MVs) indicating the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index can be sent additionally to identify which reference picture in the reference picture repository (212) the temporal prediction signal comes from.

[0081] The mode decision box (214) in the encoder selects the optimal prediction mode, for example, based on a rate distortion optimization method. This selection can be made after performing spatial and / or temporal predictions. Intra-frame / inter-frame decisions can be indicated by, for example, a prediction mode flag. The prediction block (216) is subtracted from the current video block to generate a prediction residual. The prediction residual is decorrelated and quantized (220) using a transform (218). (For some blocks, the encoder may bypass both the transform and quantization, in which case the residual can be decoded directly without applying the transform or quantization process). The quantized residual coefficients are inversely quantized (222) and inversely transformed (224) to form a reconstructed residual, which is then added back to the prediction block (226) to form the reconstructed signal of the CU. Further in-loop filtering, such as deblocking / SAO (sample adaptive offset) filtering, can be applied to the reconstructed CU to reduce coding artifacts before the reconstructed CU is placed in the reference picture repository (212) and used to decode future video blocks. In order to form the output video bitstream 230, the decoding mode (inter-frame or intra-frame), prediction mode information, motion information and quantized residual coefficients are sent to the entropy decoding unit (108) for further compression and packing to form the bitstream.

[0082] Figure 2B A block diagram of a block-based video decoder 250 is given. In decoder 250, the bitstream is decoded by decoder elements as described below. Video decoder 250 typically performs operations similar to... Figure 2A The encoding rounds described herein are the inverse of the decoding rounds. Encoder 200 typically also performs video decoding as part of the encoded video data.

[0083] Specifically, the input to the decoder includes a video bitstream 252, which can be generated by the video encoder 200. The video bitstream 252 is first unpacked and entropy decoded in the entropy decoding unit 254 to obtain transform coefficients, motion vectors, and other decoding information. Picture partitioning information indicates how the picture is partitioned. The decoder can therefore partition (256) the picture according to the decoded picture partitioning information. The decoding mode and prediction information are sent to the spatial prediction unit 258 (if intra-frame decoding) or the temporal prediction unit 260 (if inter-frame decoding) to form the prediction block. The residual transform coefficients are sent to the inverse quantization unit 262 and the inverse transform unit 264 to reconstruct the residual block. The prediction block and the residual block are then added together at 266 to generate a reconstructed block. This reconstructed block may undergo further in-loop filtering 268 before it is stored in the reference picture repository 270 for use in predicting future video blocks.

[0084] The decoded image 272 may further undergo post-decoding processing (274), such as inverse color transformation (e.g., conversion from YCbCr4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in pre-encoding processing (204). The post-decoding processing may utilize metadata derived in pre-encoding processing and transmitted as a signal in the bitstream. The decoded and processed video may be sent to display device 276. Display device 276 may be a device separate from decoder 250, or decoder 250 and display device 276 may be components of the same device.

[0085] The various methods and other aspects described in this invention can be used to modify modules of the video encoder 200 or decoder 250. Furthermore, the systems and methods disclosed herein are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future), and any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this disclosure can be used alone or in combination.

[0086] Inter-frame prediction

[0087] Figure 3A and 3B This is a diagram illustrating an example of motion prediction for video blocks (e.g., using inter-frame prediction modules 210 or 260). Figure 3B An example of block-level motion within an image is shown, illustrated in a diagram of an example decoded image buffer including, for example, reference images “Ref pic 0,” “Ref pic 1,” and “Ref pic 2.” Blocks B0, B1, and B2 in the current image can be predicted from blocks in the reference images “Ref pic 0,” “Ref pic 1,” and “Ref pic 2,” respectively. Motion prediction can use video blocks from adjacent video frames to predict the current video block. Motion prediction can leverage temporal correlation and / or remove inherent temporal redundancy in the video signal. For example, in H.264 / AVC and HEVC, temporal prediction can be performed on video blocks of various sizes (e.g., for the luma component, the temporal prediction block size can vary from 16x16 to 4x4 in H.264 / AVC and from 64x64 to 4x4 in HEVC). Using a motion vector of (mvx, mvy), temporal prediction can be performed as provided by the following equation:

[0088] P(x,y)=ref(x-mvx,y-mvy)

[0089] Where ref(x,y) can be the pixel value at position (x,y) in the reference image, and P(x,y) can be the prediction block. A video decoding system can support inter-frame prediction with fractional pixel precision. When the motion vector (mvx,mvy) has fractional pixel values, one or more interpolation filters can be applied to obtain the pixel values ​​at the fractional pixel positions. Block-based video decoding systems can use multi-hypothesis prediction to improve temporal prediction, for example, where a prediction signal can be formed by combining several prediction signals from different reference images. For example, H.264 / AVC and / or HEVC can use bidirectional prediction, which can combine two prediction signals. Bidirectional prediction can combine two prediction signals (each prediction signal comes from a reference image) to form a prediction, for example, the following equation:

[0090]

[0091] P0(x,y) and P1(x,y) can be the first and second prediction blocks, respectively. The two prediction blocks can be obtained by performing motion-compensated prediction based on two reference images ref0(x,y) and ref1(x,y) and using two motion vectors (mvx0,mvy0) and (mvx1,mvy1), respectively. The prediction block P(x,y) can be subtracted from the source video block (e.g., at 216) to form a prediction residual block. This prediction residual block can be transformed (e.g., at transform unit 218) and / or quantized (e.g., at quantization unit 220). The quantized residual transform coefficient block can be sent to an entropy decoding unit (e.g., entropy decoding unit 232) for entropy decoding to reduce the bit rate. The entropy-decoded residual coefficients can be packaged to form a portion of the output video bitstream (e.g., bitstream 230). On the decoder side, the same motion compensation prediction process can be performed (e.g., via inter-frame prediction module 260), and the resulting prediction block can be added to the decoded residual block (e.g., in 266).

[0092] Affine mode

[0093] In HEVC, only the translational motion model is applied to motion compensation prediction. However, in the real world, many types of motion exist, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VTM-2.0, affine motion compensation prediction is applied. The affine motion model is either 4-parameter or 6-parameter. A first flag is sent for each inter-frame decoding block to indicate whether the translational or affine motion model is applied to inter-frame prediction. If it is an affine motion model, a second flag is sent to indicate whether a 4-parameter or 6-parameter model is used.

[0094] A four-parameter affine motion model can have the following parameters: two parameters for translational motion in the horizontal and vertical directions, one parameter for scaling motion in both directions, and one parameter for rotational motion in both directions. A single scaling parameter is used because the horizontal scaling parameter equals the vertical scaling parameter. A single rotation parameter is used because the horizontal rotation parameter equals the vertical rotation parameter. The four-parameter affine motion model is decoded in the VTM using two motion vectors at two control point positions defined at the top left and top right corners of the current block. Figure 4A As shown, the affine motion field of the block is described by two control point motion vectors (V0, V1). Based on the control point motion, the motion field (v0, V1) of the affine decoded block is... x ,v y It is described as:

[0095]

[0096] In Eq(1), (v 0x ,v 0y (v) is the motion vector of the top-left control point. 1x ,v 1y ) is the motion vector of the upper right control point, such as Figure 4A As shown, w is the width of the block. In VTM-2.0, the motion field of the affine decoded block is derived at the 4×4 sub-block level, thus providing the motion field for the 4×4 sub-blocks within the current block ( Figure 4B Each export (v) in ) x ,v y This is then applied to the corresponding 4×4 sub-blocks. (It should be noted that a set of samples referred to as a block in some contexts may be referred to as a sub-block in other contexts. For clarity, different terms may be used in different contexts.)

[0097] The four parameters of the four-parameter affine model can be estimated iteratively. The MV pair at step k is represented as... The original input luminance signal is represented as I(i,j), and the predicted luminance signal is represented as I′. k (i,j). Spatial gradient g x (i,j) and g y (i,j) is the value obtained by applying the prediction signal I′ in the horizontal and vertical directions respectively. k The derivative of Eq.(1) is derived from the Sobel filter of (i,j).

[0098]

[0099] In Eq.(2), (a,b) are the incremental translation parameters at step k, and (c,d) are the incremental scaling and rotation parameters at step k. The incremental MV at the control point can be derived from its coordinates, as shown in Eq.(3) and (4). For example, (0,0) and (w,0) are the coordinates of the top-left and top-right control points, respectively.

[0100]

[0101]

[0102] Based on the optical flow equation, the relationship between brightness variation and spatial gradient and temporal shift is formulated as follows:

[0103]

[0104] Replace with Eq.(2) and Equations for parameters (a, b, c, d) can be given.

[0105]

[0106] Since all samples in the block satisfy Eq.(6), the parameter set (a,b,c,d) can be solved using the least squares error method. The MV at the two control points at step (k+1) Eq.(3) and (4) can be used to solve the problem, and they are rounded to a specific precision (e.g., 1 / 4 pel). Using iteration, the MV at the two control points can be refined until they converge when all parameters (a,b,c,d) are zero or the number of iterations meets a predetermined limit.

[0107] An example affine motion model with six parameters has the following parameters: two parameters for translational motion in the horizontal and vertical directions, one parameter for scaling motion in the horizontal direction, one parameter for rotational motion in the horizontal direction, one parameter for scaling motion in the vertical direction, and one parameter for rotational motion in the vertical direction. This six-parameter affine motion model is decoded using three MVs at three control points. For example... Figure 5 As shown in the example, three control points are defined at the top left, top right, and bottom left corners of the block for a six-parameter affine coding block. Motion at the top left control point is related to translational motion, motion at the top right control point is related to rotational and scaling motion in the horizontal direction, and motion at the bottom left control point is related to rotational and scaling motion in the vertical direction. For a 6-parameter affine motion model, the rotational and scaling motion in the horizontal direction can differ from those in the vertical direction. The motion vector (v) of each sub-block... x ,v y The following can be derived using the three MVs at the control points:

[0108]

[0109]

[0110] In Eq.(7), (v 2x ,v 2y (x, y) is the motion vector of the lower left control point, (x, y) is the center position of the sub-block, and w and h are the width and height of the block.

[0111] The six parameters of the six-parameter affine model can be estimated in a manner similar to that used in the four-parameter model. Eq.(2) can be modified to Eq.(8) as described below.

[0112]

[0113] In Eq.(8), (a,b) are the incremental translation parameters at step k, (c,d) are the incremental scaling and rotation parameters in the horizontal direction at step k, and (e,f) are the incremental scaling and rotation parameters in the vertical direction at step k. Eq.(6) is modified accordingly to obtain Eq.(9).

[0114]

[0115] By considering all samples within the block, the parameter set (a, b, c, d, e, f) can be solved using the least squares method. The upper left control point is calculated using Eq.(3). Calculate the upper right control point using Eq.(10) Calculate the upper right control point using Eq.(11)

[0116]

[0117]

[0118] For affine modes, predictive refinement (PROF) of optical flow is used.

[0119] To achieve finer granularity in motion compensation, a method for refining sub-block-based affine motion compensation predictions using optical flow has been proposed, as described in the following article: Jiancong (Daniel) Luo, Yuwen He, “CE2-related: Prediction refinement with optical flow for affine mode”, JVET-N0236, March 2019, Geneva, Switzerland. After performing sub-block-based affine motion compensation, each brightness prediction sample is refined by adding the difference derived from the optical flow equation. The proposed PROF is described as comprising the following steps.

[0120] In the first step, the sub-block-based affine motion compensation is performed to generate the sub-block prediction I(i,j).

[0121] In the second step, a 3-tap filter [-1, 0, 1] is used to calculate the spatial gradient g of the sub-block prediction at each sample location. x (i,j) and g y (i,j).

[0122] g x (i,j)=I(i+1,j)-I(i-1,j)

[0123] g y (i,j)=I(i,j+1)-I(i,j-1)

[0124] The sub-block prediction extends by one pixel on each side for gradient calculation. To reduce memory bandwidth and complexity, pixels on the extended boundary are copied from the nearest integer pixel position in the reference image. This avoids additional interpolation for filling regions.

[0125] In the third step, the brightness prediction refinement is calculated using the optical flow equation.

[0126] ΔI(i,j)=g x (i,j)*Δv x (i,j)+g y (i,j)*Δv y (i,j) (12)

[0127] Among them, such as Figure 6 As shown, Δv(i,j) is the difference between the pixel MV (denoted by v(i,j)) calculated for sample position (i,j) and the sub-block MV of the sub-block to which pixel (i,j) belongs.

[0128] Because the affine model parameters and the pixel position relative to the sub-block center remain unchanged between sub-blocks, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks within the same block. Assuming x and y are the horizontal and vertical offsets from the pixel position to the sub-block center, Δv(x,y) can be derived using the following equation:

[0129]

[0130] For a four-parameter affine model

[0131]

[0132] For a six-parameter affine model

[0133]

[0134] Among them, (v 0x ,v 0y ),(v 1x ,v 1y ),(v 2x ,v 2y ) are the motion vectors of the top left, top right, and bottom left control points, and w and h are the width and height of the block.

[0135] In the fourth step, the brightness prediction is refined and added to the sub-block prediction I(i,j). The final prediction I' can be generated using the following equation.

[0136] I'(i,j)=I(i,j)+ΔI(i,j) (13)

[0137] Problems solved in some embodiments

[0138] Current motion compensation processes are typically limited by the precision of the motion vectors. For example, the motion vector indicated on the decoder side is used in sample-level, sub-block-level, or block-level motion compensation predictions, which will determine the interpolation filters (if they have a fractional part) at integer reference sample positions and fractional positions. The precision of the relevant motion vector is a factor in the accuracy of the motion compensation prediction at each sample. If four additional bits are used for the fractional part of the motion vector, 1 / 16 PEL precision can be achieved. However, this precision limitation leads to potential problems. One problem is that if the motion compensation prediction is already sufficiently accurate, the additional bits waste signaling overhead. Another problem is that in some cases, the number of additional bits provided may still be insufficient, and even higher precision may be desired. To provide a more efficient precision representation, more flexible and accurate methods may be beneficial in improving motion compensation accuracy.

[0139] In VTM-5.0, if a specific precision granularity is desired, a corresponding interpolation filter (one or more) at that precision or higher is predefined, preventing the use of arbitrary precision. For example, for a decoder used to achieve a precision of 1 / 32 PEL, which is finer than the 1 / 16 PEL precision in VTM-5.0, the interpolation filter at 1 / 32 PEL precision or higher (e.g., 1 / 64 PEL) will be a predetermined interpolation filter (e.g., a filter defined in the video standard specification).

[0140] Overview of Example Implementations

[0141] This disclosure describes systems and methods for refining prediction accuracy for motion-compensated (MC) prediction processes using optical flow in a flexible manner. In some embodiments, after motion compensation is performed, the prediction at each sample is refined by adding a difference derived from the optical flow equation. This refinement may be referred to as motion-compensated precision refinement using optical flow (MCPROF). The optical flow can be sent as a motion vector refinement signal at the block level (which may be at the prediction unit level, such as the CU level or sub-CU level). The difference derived from the optical flow equation can represent different accuracies, allowing for finer precision. Some embodiments described herein can achieve pixel-level granularity without significantly increasing complexity and can maintain the same worst-case memory access bandwidth as conventional block-level motion compensation. The various embodiments described herein are applicable to any sub-block-based inter-frame prediction mode and / or block-based inter-frame prediction mode. The embodiments described herein are applicable to both single and dual prediction modes, and to both inter-frame merging and non-merging modes. A potential benefit of some embodiments is providing precision refinement without requiring any additional interpolation filters.

[0142] Motion compensation precision refinement using optical flow (MCPROF)

[0143] To achieve finer accuracy in motion compensation, some embodiments employ methods to refine the motion compensation predictions using optical flow. After performing motion compensation processing, the luminance and / or chromaticity predictions at each sample are refined by adding the difference derived from the optical flow equation. An example encoding method using MCPROF is as follows.

[0144] In the example method, the motion compensation process is used to generate a prediction I(i,j) at each sample location (i,j) after motion estimation in the inter-frame non-merging mode. This motion compensation process can be performed using existing inter-frame prediction processes, including one-way prediction, two-way prediction, and affine prediction. One or more motion compensation processes can be performed in this step (e.g., multiple motion vector candidates are available). In the case of multiple motion compensation processes, one of the multiple motion compensation processes can be selected according to predetermined criteria (e.g., motion compensation processing with minimum rate distortion cost), as done in VTM 5.0.

[0145] Evaluate one or more selected motion compensation processes to determine whether to use precision refinement. For example, if the existing precision (e.g., 1 / 4 PEL) provided after the MC process is sufficiently accurate, it can be determined that precision refinement should not be performed. This determination can also be made if, for example, the residual value is close to 0 at the current motion compensation precision (e.g., below a predetermined threshold value).

[0146] However, in some cases, a determination is made to refine the accuracy. In such cases, the accuracy difference N used to convey the degree of refinement can be determined. For example, the substantially optimal (or accurate) residual value determined by the motion compensation process would be approximately 9 / 16 PEL. To accommodate this optimal accuracy, a motion compensation accuracy of 1 / 16 PEL might be desired. However, if the current motion compensation process uses an accuracy of 1 / 4 PEL (e.g., provided by an existing defined interpolation filter), then the accuracy difference N between the current accuracy and the desired accuracy is two.

[0147] In some embodiments, optical flow is used to determine the refinement of motion compensation predictions as follows: The motion compensation prediction utilizing one or more existing motion vectors is represented as MCP(i,j), the uncompressed original input sample values ​​are represented as O(i,j), and the horizontal spatial gradient of MCP(i,j) is represented as g. x The vertical spatial gradient of MCP(i,j) is expressed as g. y (i,j). Refinement of optical flow using additional motion vectors (Δmv) x ,Δmv y Motion vector refinement (Δmv) can be selected. x ,Δmv y So as to basically satisfy Eq.14.

[0148] O(i,j)=MCP(i,j)+g x (i,j)*Δmv x +g y (i,j)*Δmv y (14)

[0149] In some embodiments, such as in Eq.(15), the least squares method is used to estimate the motion vector refinement (Δmv). x ,Δmv y ).

[0150]

[0151] If a finer precision is determined (e.g., in terms of bit depth), then the refined value of the motion vector Δmv(i,j) can be sent as a signal in the bitstream (i.e., (Δmv...). x (i,j),Δmv y In some embodiments, the associated precision difference N can also be signaled in the bitstream. To save signaling overhead, N can be signaled at different levels, such as the slice / image level, CTU level, or CU (or other block) level. Similarly, motion vector refinement values ​​can be signaled at different levels, such as the slice / image level, CTU level, CU (or other block) level, or sample level. If motion vector refinement values ​​are not signaled at the sample level, the motion vector refinement value within a block or sub-block may be the same for every sample within that block or sub-block.

[0152] In some embodiments, additional motion vector refinement values ​​can be signaled in a format specific to adjacent sample locations. For example... Figure 7 As shown, neighboring locations can be used to indicate the refinement value of the motion vector, which can be represented by one of the four nearest neighboring locations, the eight nearest neighboring locations (e.g., 1 pixel distance), or even further neighboring locations (e.g., greater than 1 pixel distance). Figure 7 In the example shown, if the adjacent sample positions (i,j-1) are sent by signal, then it has the effect of sending the refined value of the motion vector Δmv(i,j)=(0,-1) by signal, where Δmv x (i,j)=0,Δmv y (i,j)=-1. Alternatively, if the adjacent sample positions (i-1,j-1) are sent using a signal, then it has the effect of sending the refined motion vector value Δmv(i,j)=(-1,-1) using a signal, where Δmv x (i,j)=-1,Δmv y (i,j)=-1.

[0153] In some embodiments, the motion vector refinement value can be signaled in the format of the index value. For example, if four nearest neighbor positions are used, the corresponding above, below, left, and right neighbor positions can be indexed as 0, 1, 2, and 3, respectively. The index can be binary-coded using variable-length codewords. For example, if eight nearest neighbor positions are allowed, the index of four nearest neighbor positions can be decoded using a codeword with a shorter length than the codewords for the other four remaining neighbor positions in the two diagonal directions.

[0154] In some embodiments, the associated accuracy difference N can be an integer value equal to the bit depth difference between the current accuracy and the desired accuracy. For example, if the current accuracy of the motion compensation prediction process is 1 / 4 PEL and the desired accuracy is 1 / 16 PEL, then the accuracy transmitted by signaling can be N = 2. In some embodiments, the encoder transmits a flag indicating whether MCPROF is used.

[0155] Predictions refined using optical flow precision

[0156] In the prediction of generated blocks or sub-blocks, the encoder (e.g., at module 210) or decoder (e.g., at module 260) can use the precision difference N and the motion vector refinement value. In the case of using MCPROF (e.g., signaling an enable flag in the bitstream), the values ​​of Δmv(i,j) and N, sent by signaling, can be obtained. Spatial gradient g x (i,j) and g y (i,j) is calculated for each sample position (i,j). The spatial gradient can be determined using a three-tap filter as described above or other techniques.

[0157] For example, unidirectional prediction, bidirectional prediction, and / or affine prediction are used to generate an initial motion compensation prediction I(i,j) for the current block. Accuracy refinement is then calculated using the scalar product of the motion vector refinement and the spatial gradient, according to Eq. 16.

[0158]

[0159] Where Δmv(i,j) is the refined value of the motion vector transmitted by the signal (e.g., received from the encoder), N is the difference in bit depth transmitted by the signal between the current accuracy and the desired accuracy, and g(i,j) is the spatial gradient calculated as described above.

[0160] The motion-compensated prediction at each sample is refined by adding intensity variations (e.g., brightness or chromaticity). The final prediction I' can be generated according to the following equation.

[0161] I'(i,j)=I(i,j)+ΔI(i,j) (17)

[0162] In the example embodiments described above, four or eight adjacent positions are considered. Each selected adjacent position is used to indicate the direction of precision refinement and is independent of waiting for motion compensation processing of its neighboring samples to complete.

[0163] Figure 8 An example method is illustrated where, at the video encoder, based on motion-compensated prediction, an initial predicted sample value is obtained (802) for at least a first sample position in the current sample block. For at least the first sample position, a motion vector refinement (804) is determined. This motion vector refinement can be encoded (806) in a bitstream, for example, for storage or transmission.

[0164] At the first sample location, the spatial gradient of the sample value is determined (808). Based on the spatial gradient and the scalar product of the refined motion vector, a sample difference is determined (810). In some embodiments, determining the sample difference may include scaling (e.g., bit shifting) the sample difference, and precision information indicating the scaling amount may be encoded in a bitstream. Based on the sample difference, the initial predicted sample value is modified (812), which may be done, for example, by adding the sample difference to the initial predicted sample value to generate a refined sample value.

[0165] In some embodiments, the determination (804) of the motion vector refinement may include selecting the motion vector refinement to substantially minimize the prediction error with respect to the input video block. In embodiments where motion vector refinement is selected sample by sample, the prediction error may be based on the difference (e.g., absolute difference or squared difference) between the refined sample value and the corresponding sample value of the input video block. In embodiments where motion vector refinement is selected block by block (or sub-block by sub-block), the prediction error may be based on the sum of the differences (e.g., the sum of absolute difference or squared difference) between the refined sample value at multiple sample locations in the block (or sub-block) and the corresponding sample value of the input video block.

[0166] In some embodiments, the encoder may use refined sample values, such as those generated at 812, to determine the prediction residuals, and may also encode the prediction residuals in the bitstream.

[0167] In a method performed by a video decoder, the decoder obtains initial predicted sample values ​​(814) for at least a first sample position in the current sample block based on motion-compensated prediction. For example, motion vector thinning at least for the first sample position is determined (816) by decoding the motion vector thinning from the bitstream. At the first sample position, the spatial gradient of the sample value is determined (818). A sample difference is determined (820) by calculating the scalar product of the spatial gradient and the motion vector thinning. In some embodiments, the determination of the sample difference may further include scaling the sample difference based on precision information decoded from the bitstream. Based on the sample difference, the initial predicted sample values ​​are modified (822). For example, the sample difference may be added to the initial predicted sample values ​​to generate refined sample values.

[0168] In some embodiments, the decoder may further decode a prediction residual from the bitstream, and the prediction residual may be used to determine a reconstructed sample value at a first sample location. This reconstructed sample value may be displayed or transmitted to a separate display device for display.

[0169] Example Communication System

[0170] Figure 9 This diagram illustrates an example of a communication system. The communication system 900 may include an encoder 902, a communication network 904, and a decoder 906. The encoder 902 can communicate with the network 904 via a connection 908, which can be a wired or wireless connection. The encoder 902 may be similar to... Figure 2A A block-based video encoder. Encoder 902 may contain a single-layer codec or a multi-layer codec. Decoder 906 may communicate with network 904 via connection 910, which may be a wired or wireless connection. Decoder 906 may be similar to Figure 2B A block-based video decoder. Decoder 906 may include a single-layer codec or a multi-layer codec.

[0171] The encoder 902 and / or decoder 906 can be incorporated into various wired communication devices and / or wireless transmit / receive units (WTRUs), such as, but not limited to, digital televisions, wireless broadcasting systems, network elements / terminals, servers such as content or web servers (such as Hypertext Transfer Protocol (HTTP) servers), personal digital assistants (PDAs), laptops or desktop computers, tablet computers, digital cameras, digital recording devices, video game consoles, cellular or satellite wireless phones, digital media players, and / or similar devices.

[0172] The communication network 904 can be a suitable type of communication network. For example, the communication network 904 can be a multiple access system that provides content such as voice, data, video, messaging, and broadcasting to multiple wireless users. The communication network 904 enables multiple wireless users to access such content by sharing system resources, including wireless bandwidth. For example, the communication network 904 can employ one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), and / or similar methods. The communication network 904 can include multiple interconnected communication networks. The communication network 904 can include the Internet and / or one or more dedicated commercial networks, such as cellular networks, WiFi hotspots, and / or Internet Service Provider (ISP) networks.

[0173] Other embodiments

[0174] A block-based video coding method according to some embodiments includes, at a decoder: generating motion-compensated predictions of sample values ​​in a current sample block; decoding motion vector refinement and precision difference of the current block from a bitstream; and for each predicted sample in the current block: determining a spatial gradient at the sample; calculating a scalar product of the spatial gradient and the motion vector refinement; scaling the scalar product by an amount indicated by the precision difference to generate a sample difference; and adding the sample difference to the predicted sample values ​​to produce refined sample values.

[0175] In some embodiments, motion compensation prediction of sample values ​​in the current block is generated using unidirectional prediction.

[0176] In some embodiments, motion compensation prediction of sample values ​​in the current block is generated using bidirectional prediction.

[0177] In some embodiments, the motion vector thinning is signaled as an index in the bitstream. The index can identify one of a plurality of motion vector thinnings of the form (i,j), where i and j are integers. The index can identify one of the plurality of motion vector thinnings from a group consisting of (0,-1), (1,0), (0,1), and (-1,0). The index can identify one of the plurality of motion vector thinnings from a group consisting of (0,-1), (1,0), (0,1), (-1,0), (-1,-1), (1,-1), (1,1), and (-1,1).

[0178] In some embodiments, the scaling of the scalar product includes bit shifting of the scalar product.

[0179] In some embodiments, the precision difference is N, and scaling the scalar product includes shifting the scalar product to the right by a number of bits equal to the sum of the precision difference N transmitted by the signal and the existing MC precision.

[0180] According to some embodiments, a block-based video coding method includes, at the encoder: for an input video block, generating motion-compensated predictions of sample values ​​in a current sample block; selecting a precision difference; determining a corresponding spatial gradient at the sample; determining a motion vector refinement for the current block, wherein the motion vector refinement is selected to substantially minimize the error between (i) the scalar product of the spatial gradient and the motion vector refinement and (ii) the difference between the input video block and the motion-compensated prediction; and signaling the motion vector refinement and precision difference of the current block in a bitstream.

[0181] In some embodiments, the motion compensation prediction of the sample values ​​in the current block is generated using unidirectional prediction.

[0182] In some embodiments, the motion compensation prediction of the sample values ​​in the current block is generated using bidirectional prediction.

[0183] In some embodiments, the motion vector thinning is signaled as an index in the bitstream. The index can identify one of a plurality of motion vector thinnings of the form (i,j), where i and j are integers. The index can identify one of the plurality of motion vector thinnings from a group consisting of (0,-1), (1,0), (0,1), and (-1,0). The index can identify one of the plurality of motion vector thinnings from a group consisting of (0,-1), (1,0), (0,1), (-1,0), (-1,-1), (1,-1), (1,1), and (-1,1).

[0184] In some embodiments, the motion vector refinement is selected to substantially minimize the sum of (i) the scalar product of the spatial gradient and the motion vector refinement and (ii) the squared difference between the difference between the input video block and the motion-compensated prediction.

[0185] Some embodiments include a processor and a non-transitory computer-readable medium operable to perform any of the functions described herein.

[0186] This disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as specific and are generally described in a manner that may sound restrictive, at least to illustrate individual characteristics. However, this is for the purpose of clarity and does not limit the disclosure or scope of those aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in earlier applications.

[0187] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically shown, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of implementation. At least one aspect relates primarily to video encoding and decoding, and at least one other aspect relates primarily to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions thereon stored thereon for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having a bitstream generated according to any of the described methods stored thereon.

[0188] In this invention, the terms “reconstructed” and “decoded” are used interchangeably, the terms “pixel” and “sample” are used interchangeably, and the terms “image”, “picture” and “frame” are used interchangeably.

[0189] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined. Furthermore, terms such as "first" and "second" may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of these terms does not imply a sequence of modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and may occur, for example, before, during, or in a time period overlapping with the second decoding.

[0190] For example, various numerical values ​​may be used in this disclosure. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0191] The embodiments described herein can be implemented by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The processor can be of any type suitable for the technical environment and can include one or more of microprocessors, general-purpose computers, special-purpose computers, and multi-core architecture-based processors as non-limiting examples.

[0192] Various implementations involve decoding. As used in this disclosure, "decoding" can include, for example, all or part of a process performed on a received encoded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by a decoder of the various implementations described in this disclosure, such as extracting an image from a tiled (encapsulated) image, determining an upsampling filter to use and then upsampling the image, and flipping the image back to its predetermined orientation.

[0193] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to a broader decoding process will be clear based on the specific context of the description.

[0194] Various implementations involve encoding. In a manner similar to the above discussion of “decoding,” “encoding,” as used herein, can include all or part of a process performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy decoding. In various embodiments, such processes also, or alternatively, include processes performed by encoders of the various implementations described herein.

[0195] As a further example, in one embodiment, "encoding" refers only to entropy encoding; in another embodiment, "encoding" refers only to differential encoding; and in yet another embodiment, "encoding" refers to a combination of differential encoding and entropy decoding. Whether the phrase "encoding process" is intended to specifically refer to a subset of operations or generally to a broader encoding process will become clear based on the specific context of the description.

[0196] When the accompanying drawings are presented as flowcharts, it should be understood that they also provide block diagrams of the corresponding devices. Similarly, when the accompanying drawings are presented as block diagrams, it should be understood that they also provide flowcharts of the corresponding methods / processes.

[0197] Various embodiments may involve rate distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, usually with constraints on computational complexity. This rate distortion optimization is generally formulated as minimizing a rate distortion function, which is a weighted sum of rate and distortion. Different approaches exist to address the rate distortion optimization problem. For example, these approaches may be based on extensive testing of all encoding options, including all considered modes or decoding parameter values, and a complete evaluation of their decoding costs and the correlated distortion of the reconstructed signal after decoding and decoding. Faster methods can also be used to save encoding complexity, particularly by calculating approximate distortion based on predicting or predicting the residual signal rather than the reconstructed signal. A hybrid of these two approaches can also be used, for example, by using approximate distortion only for some possible encoding options and full distortion for others. Other methods evaluate only a subset of possible encoding options. More generally, many methods employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a complete evaluation of both decoding costs and correlated distortion.

[0198] The implementations and aspects described herein can be implemented, for example, in a method or process, apparatus, software program, data stream, or signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), implementations of the discussed features can also be implemented in other forms (e.g., apparatus or program). For example, an apparatus can be implemented with appropriate hardware, software, and firmware. The method can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0199] The references to "an embodiment," "an embodiment," "an implementation," or "an implementation," and other variations, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases "in an embodiment," "in one embodiment," "in one implementation," or "in one implementation," and any other variations appearing in various places throughout this disclosure, do not necessarily refer to the same embodiment.

[0200] Additionally, this disclosure may relate to "determining" various types of information. Determining the information may include, for example, one or more of the following: estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0201] Furthermore, this disclosure may relate to “accessing” various types of information. Accessing such information may include, for example, one or more of the following: receiving the information, retrieving the information (e.g., retrieving the information from a memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0202] Additionally, this disclosure can refer to "receiving" various types of information. Like "accessing," receiving is intended to be a broad term. Receiving said information may include, for example, one or more of the following: accessing the information or (e.g., retrieving the information from memory). Furthermore, "receiving" is generally referred to in one or more ways during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0203] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one of” is intended to cover the selection of only the first listed option (A), or only the second listed option (B), or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). This can be extended to multiple listed items.

[0204] Furthermore, as used herein, the term "signal" specifically refers to indicating something to the corresponding decoder. For example, in some embodiments, the encoder signals a particular one of several parameters used for region-based filter parameter selection for artifact removal filtering. Thus, in one embodiment, the same parameter is used on both the encoder and decoder sides. Therefore, for example, the encoder can send (explicitly signal) a specific parameter to the decoder so that the decoder can use that same specific parameter. Conversely, if the decoder already has said specific parameter as well as other parameters, signaling can be used without sending (implicitly signal) to simply allow the decoder to know and select said specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. While the foregoing refers to the verb form of the term "signal," the term "signal" can also be used as a noun herein.

[0205] Implementations can generate various signals formatted to carry information, such as information that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. The formatting may include, for example, encoding the data stream and modulating a carrier wave using the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal can be transmitted via various wired or wireless links. The signal may be stored on a processor-readable medium.

[0206] We have described several embodiments. Features of these embodiments may be provided individually or in any combination across various claim classes and types. Furthermore, across various claim classes and types, embodiments may individually or in any combination include one or more of the following features, devices, or aspects:

[0207] • A bitstream or signal including a syntax that conveys information generated according to any of the described embodiments.

[0208] • Create and / or send and / or receive and / or decode according to any of the embodiments described.

[0209] • Methods, processes, apparatus, media for storing instructions, media for storing data, or signals according to any of the described embodiments.

[0210] TVs, set-top boxes, cellular phones, tablets, or other electronic devices that perform encoding or decoding methods according to any of the described embodiments.

[0211] A TV, set-top box, cellular phone, tablet computer, or other electronic device that performs the decoding method according to any of the described embodiments and displays (e.g., using a monitor, screen, or other type of display) the obtained image.

[0212] TVs, set-top boxes, cellular phones, tablets, or other electronic devices that select (e.g., using a tuner) a channel to receive signals including encoded images and perform decoding according to any of the embodiments described.

[0213] • TVs, set-top boxes, cellular phones, tablets, or other electronic devices that receive signals including encoded images over the air (e.g., using an antenna) and perform decoding according to any of the embodiments described.

[0214] Note that the various hardware elements in one or more of the described embodiments are referred to as “modules”, the execution of which (i.e., operation, implementation, etc.) is in conjunction with the various functions described herein for each module. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that a person skilled in the art would consider suitable for a given implementation. Each described module may also include instructions executable to perform one or more functions described as being performed by the respective module, and note that these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, and / or software instructions, or may be included therein, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.

[0215] While features and elements in a specific combination or order have been described above, those skilled in the art will recognize that each feature or element can be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in computer programs, software, or firmware incorporated into a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, buffer memory, semiconductor storage devices, magnetic media such as internal hard disks and removable discs, magneto-optical media, and optical media such as CD-ROMs and digital multipurpose discs (DVDs). The processor associated with the software can be used to implement radio frequency transceivers used in WTRUs, UEs, terminals, base stations, RNCs, or any host computer.

Claims

1. A method of video encoding, comprising: obtaining, based on motion-compensated prediction, an initial predicted sample value for at least a first sample location in a current sample block; determining a motion vector refinement associated with at least the first sample location, wherein the motion vector refinement is encoded within a bitstream as an index signaled at a sample level; determining, at the first sample location, a spatial gradient of a sample value; determining a sample difference value based on a scalar product of the spatial gradient and the motion vector refinement; and modifying the initial predicted sample value based on the sample difference value.

2. The method of claim 1, wherein determining a motion vector refinement includes selecting the motion vector refinement to substantially minimize a prediction error with respect to an input video block.

3. The method of claim 1, wherein the index identifies one of a plurality of motion vector refinements from a group consisting of (0,-1), (1,0), (0,1), and (-1,0).

4. The method of claim 1, wherein the index identifies one of a plurality of motion vector refinements from a group consisting of (0,-1), (1,0), (0,1), (-1,0), (-1,-1), (1,-1), (1,1), and (-1,1). encoding refinement precision information in the bitstream, wherein determining the sample difference value includes scaling the scalar product by an amount indicated by the precision information.

5. The method of claim 1, further comprising:

6. A method of video decoding, comprising: obtaining, based on motion-compensated prediction, an initial predicted sample value for at least a first sample location in a current sample block; determining a motion vector refinement associated with at least the first sample location, wherein the motion vector refinement is decoded from a bitstream as an index signaled at a sample level; determining, at the first sample location, a spatial gradient of a sample value; determining a sample difference value based on a scalar product of the spatial gradient and the motion vector refinement; and modifying the initial predicted sample value based on the sample difference value. decoding refinement precision information from the bitstream, wherein determining the sample difference value includes scaling the scalar product by an amount indicated by the precision information. bit-shifting the scalar product by an amount indicated by the precision information.

7. The method of claim 6, further comprising:

9. The method of claim 7, wherein the motion-compensated prediction is performed using at least one motion vector having an initial precision, and wherein the refinement precision information includes a precision difference value representing a difference between the initial precision and the refinement precision.

8. The method of claim 7, wherein scaling the scalar product comprises: right-shifting the scalar product by a number of bits equal to a sum of the precision difference value and the initial precision.

11. The method of claim 6, wherein the index identifies one of a plurality of motion vector refinements from a group consisting of (0,-1), (1,0), (0,1), and (-1,0).

10. The method of claim 9, wherein scaling the scalar product comprises: ​ ​ 12. The method of claim 6, wherein the index identifies one of a plurality of motion vector refinements from a group consisting of (0,-1), (1,0), (0,1), (-1,0), (-1,-1), (1,-1), (1,1), and (-1,1).

13. A video encoding device comprising a processor configured to perform at least: obtaining, based on a motion compensated prediction, an initial predicted sample value for at least a first sample location in a current sample block; determining a motion vector refinement associated with at least the first sample location, wherein the motion vector refinement is encoded within a bitstream as an index signaled at a sample level; determining, at the first sample location, a spatial gradient of a sample value; determining a sample difference value based on a scalar product of the spatial gradient and the motion vector refinement; and modifying the initial predicted sample value based on the sample difference value.

14. The device of claim 13, wherein determining a motion vector refinement comprises selecting the motion vector refinement to substantially minimize a prediction error with respect to an input video block.

15. The device of claim 13, wherein the index identifies one of a plurality of motion vector refinements from a group consisting of (0,-1), (1,0), (0,1), and (-1,0).

16. The device of claim 13, wherein the index identifies one of a plurality of motion vector refinements from a group consisting of (0,-1), (1,0), (0,1), (-1,0), (-1,-1), (1,-1), (1,1), and (-1,1). encoding refinement precision information in the bitstream, wherein determining the sample difference value comprises scaling the scalar product by an amount indicated by the precision information.

17. The apparatus of claim 13, further comprising:

18. A video decoding device comprising a processor configured to perform at least: obtaining, based on a motion compensated prediction, an initial predicted sample value for at least a first sample location in a current sample block; determining a motion vector refinement associated with at least the first sample location, wherein the motion vector refinement is decoded from a bitstream as an index signaled at a sample level; determining, at the first sample location, a spatial gradient of a sample value; determining a sample difference value based on a scalar product of the spatial gradient and the motion vector refinement; and modifying the initial predicted sample value based on the sample difference value. decoding refinement precision information from the bitstream, wherein determining the sample difference value comprises scaling the scalar product by an amount indicated by the precision information. bit-shifting the scalar product by an amount indicated by the precision information.

19. The apparatus of claim 18, further comprising:

21. The device of claim 19, wherein the motion compensated prediction is performed using at least one motion vector having an initial precision, and wherein the refinement precision information comprises a precision difference value representing a difference between the initial precision and the refinement precision.

20. The apparatus of claim 19, wherein scaling the scalar product comprises: right-shifting the scalar product by a number of bits equal to a sum of the precision difference value and the initial precision. ​ 22. The apparatus of claim 21, wherein scaling the scalar product comprises: ​ 23. The device of claim 18, wherein the index identifies one of a plurality of motion vector refinements from the group consisting of (0,-1), (1,0), (0,1), and (-1,0).

24. The device of claim 18, wherein the index identifies one of a plurality of motion vector refinements from the group consisting of (0,-1), (1,0), (0,1), (-1,0), (-1,-1), (1,-1), (1,1), and (-1,1).

25. A computer readable medium containing instructions for causing one or more processors to perform operations comprising: obtaining an initial predicted sample value for at least a first sample position in a current sample block based on a motion compensated prediction; determining a motion vector refinement associated with at least the first sample position, wherein the motion vector refinement is encoded within a bitstream as an index signaled at a sample level; determining a spatial gradient of a sample value at the first sample position; determining a sample difference value based on a scalar product of the spatial gradient and the motion vector refinement; and modifying the initial predicted sample value based on the sample difference value.

26. A computer readable medium containing instructions for causing one or more processors to perform operations comprising: obtaining an initial predicted sample value for at least a first sample position in a current sample block based on a motion compensated prediction; determining a motion vector refinement associated with at least the first sample position, wherein the motion vector refinement is decoded from a bitstream as an index signaled at a sample level; determining a spatial gradient of a sample value at the first sample position; determining a sample difference value based on a scalar product of the spatial gradient and the motion vector refinement; and modifying the initial predicted sample value based on the sample difference value. ​ ​

Images