A method for reducing memory access bandwidth for inter-frame prediction using optical flow compensation

Abstract

The present invention describes systems and methods for video encoding. In some embodiments, inter-frame prediction of samples in a current block is performed by rounding an initial motion vector and determining a rounding error vector caused by the rounding. An unrefined prediction for the sample is generated using the rounded motion vector. Unrefined predictions are similarly generated for other samples in the current block. Based on the unrefined prediction, a spatial gradient is determined for each sample position in the block. A refined prediction is generated for each sample position by adding a scalar product between the spatial gradient at the sample position and the rounding error vector to the unrefined prediction. The exemplary method can reduce the number of reference pixels used to predict the current block and, therefore, can reduce memory access bandwidth.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional application of U.S. Provisional Patent Application Serial No. 62 / 819,529, filed on March 16, 2019, entitled “INTER PREDICTION BANDWIDTH REDUCTION METHOD WITH OPTICALFLOW COMPENSATION,” and claims the benefit of that U.S. Provisional Patent Application under 35 U.S.C. §119(e), which is hereby incorporated by reference in its entirety. Background Art

[0003] Video coding systems are widely used to compress digital video signals to reduce the storage requirements and / or transmission bandwidth of such signals. Among the various types of video coding systems (such as block-based systems, wavelet-based systems, and object-based systems), block-based hybrid video coding systems are currently the most widely used and deployed. Examples of block-based video coding systems include international video coding standards such as MPEG-1 / 2 / 4 Part 2, H.264 / MPEG-4 Part 10 AVC, VC-1, and the latest video coding standard called High Efficiency Video Coding (HEVC) developed by ITU-T / SG16 / Q.6 / VCEG and JCT-VC (Joint Collaboration Team on Video Coding) of ISO / IEC / MPEG.

[0004] In October 2017, the ITU-T and ISO / IEC published a Joint Proposal (CfP) for video compression exceeding HEVC. In April 2018, responses to the CfP for the standard dynamic range category were received and evaluated at the 10th JVET meeting, demonstrating approximately 40% improvement in compression efficiency over HEVC. Based on these evaluation results, the Joint Video Experts Team (JVET) launched a new project to develop a next-generation video coding standard called Versatile Video Coding (VVC). That same month, a reference software code library, called the VVC Test Model (VTM), was established to validate a reference implementation of the VVC standard. For the initial VTM-1.0, most coding modules (including intra-frame prediction, inter-frame prediction, transform / inverse transform, quantization / dequantization, and loop filters) followed the existing HEVC design, differing in that VTM's use of a block partitioning structure based on multiple tree types. Furthermore, to facilitate the evaluation of new coding tools, another reference software library, called the Benchmark Set (BMS), was also generated. In the BMS code base, a list of coding tools inherited from JEM is included on top of VTM. These coding tools provide higher coding efficiency and moderate implementation complexity and are used as a benchmark when evaluating similar coding techniques during the VVC standardization process. Specifically, 9 JEM coding tools are integrated in BMS-1.0, including 65-angle intra prediction directions, modified coefficient coding, advanced multiple transform (AMT) + 4×4 non-separable secondary transform (NSST), affine motion model, generalized adaptive loop filter (GALF), advanced temporal motion vector prediction (ATMVP), adaptive motion vector precision, decoding-side motion vector refinement (DMVR) and linear model (LM) chroma mode. Summary of the Invention

[0005]

[0014] Embodiments described herein include methods for video encoding and decoding (collectively, "encoding").

[0006] In some embodiments, a method includes rounding a first motion vector associated with a current sample to obtain a second motion vector; obtaining an unrefined prediction for the current sample using the second motion vector; obtaining a rounding error vector indicating a difference between the first motion vector and the second motion vector; obtaining a spatial gradient at a sample position of the current sample; and obtaining a refined prediction for the current sample based on the unrefined prediction, the spatial gradient, and the rounding error vector. Some embodiments include an apparatus comprising one or more processors configured to perform such methods or other methods described herein.

[0007] In some embodiments, the refined prediction for the current sample is obtained by adding the scalar product between the spatial gradient and the rounding error vector to the unrefined prediction.

[0008] In some embodiments, the second motion vector is rounded to integer precision. In some embodiments, the second motion vector is rounded to fractional precision.

[0009] In some embodiments, the first motion vector is a sub-block motion vector of a sub-block containing the current sample, and the first motion vector is obtained using an affine motion model.

[0010] In some implementations, the first motion vector is a translational motion vector associated with a current block that includes the current sample.

[0011] In some embodiments, obtaining the spatial gradient at the sample position of the current sample is performed based on an unrefined prediction of a plurality of samples neighboring the current sample. In some such embodiments, the horizontal component of the spatial gradient is obtained as the difference between the right and left neighboring samples of the current sample, and the vertical component of the spatial gradient is obtained as the difference between the top and bottom neighboring samples of the current sample.

[0012] In some embodiments, obtaining an unrefined prediction of the current sample is performed using a four-tap interpolation filter.

[0013] In some embodiments, obtaining an unrefined prediction of the current sample is performed using a bilinear interpolation filter.

[0014] In some implementations, the determination to perform rounding of the first motion vector is made based on a size of the current block that includes the current sample. In some such implementations, the determination to perform rounding of the first motion vector is made based on a determination that the current block that includes the current sample is a 4x4, 4x8, or 8x4 block.

[0015] In some implementations, the first motion vector is signaled in the bitstream.

[0016] In some embodiments, the refined prediction of the current sample is subtracted from the input sample value to generate a prediction residual, and the prediction residual is encoded in the bitstream.

[0017] In some embodiments, the prediction residual for the current sample is decoded from the bitstream and added to the refined prediction of the current sample to generate a reconstructed sample value.

[0018] One or more embodiments of the present invention further provide a computer-readable storage medium having stored thereon instructions for performing bidirectional optical flow, encoding, or decoding video data according to any of the above methods. Embodiments of the present invention further provide a computer-readable storage medium having stored thereon a bitstream generated according to the above method. Embodiments of the present invention further provide a method and apparatus for transmitting a bitstream generated according to the above method. Embodiments of the present invention further provide a computer program product comprising instructions for performing any of the above methods.

[0019] In other embodiments, encoder and decoder devices are provided to perform the methods described herein. The encoder or decoder device may include a processor configured to perform the methods described herein. The device may include a computer-readable medium (e.g., a non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, the computer-readable medium (e.g., a non-transitory medium) stores video encoded using any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A is a system diagram illustrating an exemplary communication system in which one or more disclosed embodiments may be implemented.

[0021] Figure 1B It shows that according to the embodiment, Figure 1A A system diagram of an exemplary wireless transmit / receive unit (WTRU) for use within a communication system is shown.

[0022] Figure 1C is a functional block diagram of a system according to some embodiments.

[0023] Figure 2A is a functional block diagram of a block-based video encoder, such as the encoder used for VVC.

[0024] Figure 2B is a functional block diagram of a block-based video decoder, such as the decoder used for VVC.

[0025] Figure 3 is a diagram showing an example of motion compensated prediction.

[0026] Figure 4 is a diagram showing an example of horizontal movement of blocks within a screen.

[0027] Figure 5 is a diagram showing an example of the structure of an encoded bit stream.

[0028] Figure 6is a diagram illustrating an exemplary communication system.

[0029] 7A to 7B Show the four-parameter affine model. Figure 7A The affine model is shown in Figure 7B The sub-block horizontal motion derivation for affine blocks is shown in .

[0030] Figure 8 Shows a six-parameter affine model: V0, V1 and V2 are control points; (MV x , MV y ) is the motion vector of the sub-block centered at position (x, y).

[0031] Figure 9 Shows the sub-block MV V used for optical flow prediction refinement (PROF) sb and pixel ΔV(i, j).

[0032] Figure 10 is a flow chart illustrating a method performed in some embodiments.

[0033] Exemplary Networks and Systems for Implementing Embodiments

[0034] Figure 1A is a schematic diagram illustrating an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content by sharing system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), zero-tail unique word DFT spread OFDM (ZT UWDTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multi-carrier (FBMC), etc.

[0035] like Figure 1AAs shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104, a CN 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112. However, it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular phone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable device, a head-mounted display (HMD), a vehicle, a drone, medical equipment and applications (e.g., remote surgery), industrial equipment and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated process chain environments), a consumer electronic device, a device operating on a commercial and / or industrial wireless network, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0036] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a NodeB, an eNodeB, a Home NodeB, a Home eNodeB, a gNB, an NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it should be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0037] Base station 114a may be part of the RAN 104, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be in a licensed spectrum, an unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage to a specific geographic area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, one for each sector of the cell. In one embodiment, base station 114a may employ multiple-input, multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0038] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0039] More specifically, as noted above, the communication system 100 may be a multiple access system and may employ one or more channel access schemes such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may use Wideband CDMA (WCDMA) to establish the air interface 116. WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0040] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0041] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR radio access, which may establish the air interface 116 using New Radio (NR).

[0042] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement both LTE radio access and NR radio access, for example, using dual connectivity (DC) principles. Thus, the air interface used by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0043] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), GSM Enhanced Data rates for Evolution (EDGE), GSM EDGE (GERAN), etc.

[0044] Figure 1AThe base station 114b in the may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a local area, such as a business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or a femtocell. As Figure 1A As shown, base station 114b may have a direct connection to the Internet 110. Therefore, base station 114b may not need to access the Internet 110 via CN 106.

[0045] The RAN 104 may be in communication with the CN 106, which may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. Data may have different quality of service (QoS) requirements, such as different throughput requirements, delay requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform advanced security functions, such as user authentication. Although not described in detail in the text, the CN 106 may be configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. Figure 1A Although not shown in the figures, it will be appreciated that the RAN 104 and / or the CN 106 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which may utilize NR radio technology, the CN 106 may also be in communication with another RAN (not shown) that employs GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0046] The CN 106 may also act as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), and / or the Internet Protocol (IP) from the TCP / IP internet protocol suite. The networks 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104 or a different RAT.

[0047] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). Figure 1A The illustrated WTRU 102c may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0048] Figure 1B is a system diagram illustrating an exemplary WTRU 102. Figure 1B As shown, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be appreciated that the WTRU 102 may include any subcombination of the foregoing elements while remaining consistent with an embodiment.

[0049] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functions that enable the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, but it is understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0050] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive RF and light signals. It should be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0051] Although the transmit / receive element 122 is Figure 1B Although depicted as a single element in FIG. 1 , the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0052] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive element 122 and demodulate signals received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0053] The processor 118 of the WTRU 102 may be coupled to and may receive user input data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. Furthermore, the processor 118 may access information from and store data in any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from and store data in memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0054] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0055] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or in lieu of the information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by any suitable location-determination method while remaining consistent with an embodiment.

[0056] The processor 118 may also be coupled to other peripherals 138, which may include one or more software modules and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, etc. The peripheral device 138 may include one or more sensors, which may be one or more of the following: a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0057] The WTRU 102 may include a full-duplex radio for which transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference via hardware (e.g., a choke) or via signal processing performed by a processor (e.g., a separate processor (not shown) or via the processor 118). In one embodiment, the WTRU 102 may include a full-duplex radio for which transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous.

[0058] Even though the WTRU Figure 1A to Figure 1B Although described as a wireless terminal, it is contemplated that in certain representative embodiments such a terminal may (eg, temporarily or permanently) employ a wired communications interface with a communications network.

[0059] In a representative embodiment, the other network 112 may be a WLAN.

[0060] Given that Figure 1A to Figure 1B As described above, one or more or all of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more or all of the functions described herein. For example, an emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0061] The emulation device may be designed to implement one or more tests of other devices in a lab environment and / or in a carrier network environment. For example, the one or more emulation devices may perform one or more or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network in order to test other devices within the communication network. The one or more emulation devices may perform one or more or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for testing purposes and / or may use over-the-air wireless communications to perform testing.

[0062] The one or more emulated devices can perform one or more (including all) functions without being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulated device can be used in a test lab and / or in a test scenario in a non-deployed (e.g., testing) wired and / or wireless communication network to enable testing of one or more components. The one or more emulated devices can be test devices. Direct RF coupling and / or wireless communication via RF circuitry (e.g., which can include one or more antennas) can be used by the emulated device to transmit and / or receive data.

[0063] Exemplary Systems .

[0064] Figure 1C A block diagram illustrating an example of a system for implementing various aspects and embodiments is shown. System 1000 may be embodied as a device comprising various components as described below and configured to perform one or more aspects of the aspect described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 1000 may be embodied in a single integrated circuit (IC), multiple ICs and / or discrete components, either individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed over multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to one or more other systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects of the aspect described in this document.

[0065] System 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, various aspects described in this document. Processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes storage devices 1040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drives, and / or optical disk drives. As non-limiting examples, storage devices 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0066] System 1000 includes an encoder / decoder module 1030, which is configured to process data to provide encoded video or decoded video, for example, and may include its own processor and memory. Encoder / decoder module 1030 represents a module that can be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of an encoding module and a decoding module. In addition, encoder / decoder module 1030 can be implemented as a separate component of system 1000, or can be incorporated into processor 1010 as a combination of hardware and software known to those skilled in the art.

[0067] Program code to be loaded onto the processor 1010 or encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operation logic.

[0068] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, the operating system of the television. In at least one embodiment, fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the Joint Video Experts Team JVET).

[0069] Input to the elements of system 1000 may be provided through various input devices, as shown in block 1130. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal transmitted over the air, for example, by a broadcaster; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1C Other examples not shown include composite video.

[0070] In various embodiments, the input device of block 1130 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or band-limiting a signal to a frequency band); (ii) down-converting the selected signal; (iii) again band-limiting to a narrower frequency band to select a signal frequency band, which may be referred to as a channel in some embodiments; (iv) demodulating the down-converted and band-limited signal; (v) performing error correction; and (vi) demultiplexing to select the desired data packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a frequency band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or down-converting to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal by wired (for example, cable) medium transmission, and by filtering, down-conversion and filtering to the desired frequency band again to perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some elements in these elements, and / or add other elements of execution similar or different functions.Adding element can be included in and inserts element between existing element, for example, inserts amplifier and analog to digital converter.In various embodiments, the RF part comprises antenna.

[0071] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within the processor 1010, as needed. Similarly, various aspects of USB or HDMI interface processing may be implemented, for example, within a separate interface IC or within the processor 1010, as needed. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0072] The various components of system 1000 can be disposed within an integrated housing. Within the integrated housing, the various components can be interconnected and data can be transferred between the components using a suitable connection arrangement 1140 (e.g., an internal bus known in the art, including an inter-chip (I2C) bus, wiring, and a printed circuit board).

[0073] System 1000 includes a communication interface 1050 that can communicate with other devices via a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0074] In various embodiments, use wireless network such as Wi-Fi network for example IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers) that data streaming is transmitted or otherwise is offered to system 1000.The Wi-Fi signal of these embodiments is received by communication channel 1060 and communication interface 1050 that are suitable for Wi-Fi communication.The communication channel 1060 of these embodiments is connected to access point or router conventionally, and this access point or router provide the access to the external network that comprises the Internet, to allow streaming application and other cross-top communication.Other embodiments use set-top box to provide streaming data to system 1000, and this set-top box is connected to deliver data by the HDMI of input block 1130.Other embodiments use the RF of input block 1130 to be connected to provide streaming data to system 1000 again.As mentioned above, various embodiments provide data in non-streaming mode.In addition, various embodiments use the wireless network outside Wi-Fi, for example cellular network or Bluetooth network.

[0075] System 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be used for a television, a tablet computer, a laptop computer, a cellular phone (mobile phone), or other devices. Display 1100 can also be integrated with other components (e.g., as in a smartphone) or can be separate (e.g., an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 1120 include one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functions based on the output of system 1000. For example, a disc player performs the function of playing the output of system 1000.

[0076] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to the system 1000 via communication interface 1050 using communication channel 1060. The display 1100 and speakers 1110 can be integrated into a single unit with other components of the system 1000 in an electronic device such as a television. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0077] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection (including, for example, an HDMI port, a USB port, or a COMP output).

[0078] These embodiments may be implemented by computer software implemented by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, these embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and, as a non-limiting example, may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type suitable for the technical environment and, as a non-limiting example, may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture. DETAILED DESCRIPTION

[0079] Block-based video coding .

[0080] Similar to HEVC, VVC is built on a block-based hybrid video coding framework. Figure 2A A block diagram is given of a block-based hybrid video coding system 200. Variations of this encoder 200 are contemplated, but the encoder 200 is described below for clarity without describing all contemplated variations.

[0081] Before encoding, the video sequence may undergo pre-encoding processing (204), such as applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0082] The input video signal 202 including the picture to be encoded is divided (206) and processed block by block, for example, in units of CUs. Different CUs may have different sizes. In VTM-1.0, a CU may be at most 128×128 pixels. However, unlike HEVC, which divides blocks based solely on a quadtree, in VTM-1.0, a coding tree unit (CTU) is divided into CUs to accommodate local characteristics that vary based on quadtrees / binary trees / ternary trees. In addition, the concept of multiple partition unit types in HEVC is removed, so that there is no longer a separation of CUs, prediction units (PUs), and transform units (TUs) in VVC-1.0; instead, each CU is always used as a basic unit for both prediction and transform without further division. In the multi-type tree structure, the CTU is first divided by a quadtree structure. Then, each quadtree leaf node may be further divided by a binary tree structure and a ternary tree structure. Different partition types may be used, such as quad partitioning, vertical binary partitioning, horizontal binary partitioning, vertical ternary partitioning, and horizontal ternary partitioning.

[0083] exist Figure 2A In an encoder, spatial prediction 208 and / or temporal prediction 210 may be performed. Spatial prediction (or "intra-frame prediction") uses pixels from samples of already coded neighboring blocks (which are called reference samples) in the same video picture / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also known as "inter-frame prediction" or "motion compensated prediction") uses reconstructed pixels from already coded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU may be signaled by one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index may be additionally sent that identifies which reference picture in the reference picture store (212) the temporal prediction signal comes from.

[0084] A mode decision block (214) in the encoder selects the best prediction mode, for example, based on a rate-distortion optimization method. This selection may be made after performing spatial and / or temporal prediction. The intra / inter decision may be indicated by, for example, a prediction mode flag. The prediction block is subtracted from the current video block (216) to generate a prediction residual. The prediction residual is decorrelated using a transform (218) and quantization (220). (For some blocks, the encoder may bypass the transform and quantization, in which case the residual may be encoded directly without applying a transform or quantization process.) The quantized residual coefficients are inverse quantized (222) and inverse transformed (224) to form a reconstructed residual, which is then added back to the prediction block (226) to form a reconstructed signal for the CU. In addition, a loop filter (such as a deblocking / SAO (sample adaptive offset) filter) may be applied (228) to the reconstructed CU to reduce coding artifacts, which is then placed in a reference picture store (212) and used to encode future video blocks. To form the output video bitstream 230, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit (108) for further compression and packing to form a bitstream.

[0085] Figure 2B A block diagram of a block-based video decoder 250 is shown. In the decoder 250, the bitstream is decoded by the decoder elements as described below. The video decoder 250 generally performs a decoding process that is the reverse of the encoding process, such as Figure 2A Encoder 200 also typically performs video decoding as part of encoding the video data.

[0086] Specifically, the input to the decoder includes a video bitstream 252, which may be generated by the video encoder 200. The video bitstream 252 is first unpacked and entropy decoded at an entropy decoding unit 254 to obtain transform coefficients, motion vectors, and other encoding information. The picture partition information indicates how the picture is partitioned. Therefore, the decoder can partition (256) the picture according to the decoded picture partition information. The coding mode and prediction information are sent to a spatial prediction unit 258 (if intra-frame coding) or a temporal prediction unit 260 (if inter-frame coding) to form a prediction block. The residual transform coefficients are sent to an inverse quantization unit 262 and an inverse transform unit 264 to reconstruct the residual block. The prediction block and the residual block are then added together at 266 to generate a reconstructed block. The reconstructed block may be further subjected to loop filtering 268 before being stored in a reference picture store 270 for use in predicting future video blocks.

[0087] The decoded picture 272 may also undergo post-decoding processing (274), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse of the remapping process performed in the pre-encoding process (204). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream. The decoded, processed video may be sent to a display device 276. The display device 276 may be a separate device from the decoder 250, or the decoder 250 and the display device 276 may be components of the same device.

[0088] The various methods and other aspects described in this disclosure can be used to modify modules of the video encoder 200 or decoder 250. In addition, the systems and methods disclosed herein are not limited to VVC or HEVC and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future) and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this disclosure can be used alone or in combination.

[0089] Inter-frame prediction .

[0090] Figure 3 and Figure 4 is a diagram showing an example of motion compensated prediction of a video block (eg, using motion compensation module 210 or 260 ). Figure 4 Illustrating an example of horizontal movement of blocks within a picture, is a diagram showing an exemplary decoded picture buffer including, for example, reference pictures "reference picture 0," "reference picture 1," and "reference picture 2." Blocks B0, B1, and B2 in the current picture can be predicted from blocks in reference pictures "reference picture 0," "reference picture 1," and "reference picture 2," respectively. Motion prediction can use video blocks from adjacent video frames to predict the current video block. Motion prediction can exploit temporal correlation and / or remove temporal redundancy inherent in a video signal. For example, in H.264 / AVC and HEVC, temporal prediction can be performed on video blocks of various sizes (e.g., for the luma component, the temporal prediction block size can vary from 16×16 to 4×4 in H.264 / AVC, and from 64×64 to 4×4 in HEVC). Using the motion vector of (mvx, mvy), temporal prediction can be performed as follows:

[0091] P(x, y) = ref(x-mvx, y-mvy)

[0092] Where ref(x, y) may be the pixel value at position (x, y) in the reference picture, and P(x, y) may be the prediction block. The video coding system may support inter-frame prediction with fractional pixel precision. When the motion vector (mvx, mvy) has a fractional pixel value, one or more interpolation filters may be applied to obtain the pixel value at the fractional pixel position. For example, a block-based video coding system may use multi-hypothesis prediction to improve temporal prediction where a prediction signal may be formed by combining multiple prediction signals from different reference pictures. For example, H.264 / AVC and / or HEVC may use bi-prediction that may combine two prediction signals. Bi-prediction may combine two prediction signals (each from a reference picture) to form a prediction, such as in the following equation:

[0093]

[0094] Where P0(x, y) and P1(x, y) may be the first prediction block and the second prediction block, respectively. As shown in the equation, the two prediction blocks may be obtained by performing motion compensated prediction from two reference pictures ref0(x, y) and ref1(x, y) using two motion vectors (mvx0, mvy0) and (mvx1, mvy1), respectively. The prediction block P(x, y) may be subtracted from the source video block (e.g., at 216) to form a prediction residual block. The prediction residual block may be transformed (e.g., at transform unit 218) and / or quantized (e.g., at quantization unit 220). The quantized residual transform coefficient block may be sent to an entropy coding unit (e.g., entropy coding unit 232) for entropy coding to reduce bitrate. The entropy-coded residual coefficients may be packed to form part of an output video bitstream (e.g., bitstream 230).

[0095] Coded bitstream structure .

[0096] Figure 51300 is a diagram illustrating an example of a coded bitstream structure. The coded bitstream 1300 consists of multiple NAL (Network Abstraction Layer) units 1301. NAL units may contain coded sample data, such as coded slices 1306, or higher-level syntax metadata, such as parameter set data, slice header data 1305, or supplemental enhancement information data 1307 (which may be referred to as SEI messages). Parameter sets are high-level syntax structures that contain basic syntax elements that can apply to multiple bitstream layers (e.g., video parameter sets 1302 (VPS)), to coded video sequences within a layer (e.g., sequence parameter sets 1303 (SPS)), or to multiple coded pictures within a coded video sequence (e.g., picture parameter sets 1304 (PPS)). Parameter sets can be transmitted along with the coded pictures of the video bitstream or by other means (including out-of-band transmission using a reliable channel, hard coding, etc.). Slice headers 1305 are also high-level syntax structures that may contain relatively small amounts of picture-related information, or information that is relevant only to certain slices or picture types. SEI message 1307 carries information that may not be required by the decoding process but may be used for various other purposes, such as picture output timing or display and loss detection and concealment.

[0097] Communication equipment and systems .

[0098] Figure 6 14 is a diagram illustrating an example of a communication system. The communication system 1400 may include an encoder 1402, a communication network 1404, and a decoder 1406. The encoder 1402 may communicate with the network 1404 via a connection 1408, which may be a wired connection or a wireless connection. The encoder 1402 may be similar to Figure 2A The encoder 1402 may include a single-layer codec (e.g., Figure 2A ) or multi-layer codec. Decoder 1406 can communicate with network 1404 via connection 1410, which can be a wired connection or a wireless connection. Decoder 1406 can be similar to Figure 2B The decoder 1406 may include a single-layer codec (e.g., Figure 2B ) or multi-layer codec.

[0099] The encoder 1402 and / or decoder 1406 may be incorporated into a wide variety of wired communication devices and / or wireless transmit / receive units (WTRUs), such as, but not limited to, digital televisions, wireless broadcast systems, network elements / terminals, servers, such as content or web servers (such as Hypertext Transfer Protocol (HTTP) servers), personal digital assistants (PDAs), laptop or desktop computers, tablet computers, digital cameras, digital recording devices, video gaming devices, video game consoles, cellular or satellite radio telephones, digital media players, and the like.

[0100] The communication network 1404 may be any suitable type of communication network. For example, the communication network 1404 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication network 1404 may enable multiple wireless users to access such content by sharing system resources (including wireless bandwidth). For example, the communication network 1404 may employ one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), etc. The communication network 1404 may include multiple connected communication networks. The communication network 1404 may include the Internet and / or one or more private commercial networks, such as a cellular network, a WiFi hotspot, an Internet service provider (ISP) network, etc.

[0101] Affine mode .

[0102] In HEVC, only the translational motion model is used for motion-compensated prediction. However, in the real world, many types of motion exist, such as zooming in / out, rotation, perspective motion, and other irregular motion. In VTM-2.0, affine motion-compensated prediction is used. The affine motion model can be either 4-parameter or 6-parameter. A first flag is signaled for each inter-coded CU to indicate whether a translational or affine motion model is used for inter prediction. If the affine motion model is used, a second flag is signaled to indicate whether it is a 4-parameter or 6-parameter model.

[0103] A four-parameter affine motion model has the following parameters: two parameters for translation in the horizontal and vertical directions, one parameter for scaling in both directions, and one parameter for rotation in both directions. The horizontal scaling parameter is equal to the vertical scaling parameter. The horizontal rotation parameter is equal to the vertical rotation parameter. These four-parameter affine motion models are encoded in the VTM using two motion vectors at the two control point locations defined at the top left and top right corners of the current CU. 7A to 7B As shown, the affine motion field of the block is described by two control point motion vectors (V0, V1). Based on the control point motion, the motion field of the affine coded block (v x , v y ) is described as

[0104]

[0105] Where (v 0x , v 0y ) is the motion vector of the upper left control point, and (v 1x , v 1y ) is the motion vector of the upper right corner control point, such as Figure 7A As shown, and w is the width of the CU. In VTM-2.0, the motion field of the affine-coded CU is derived at the 4×4 block level; that is, for each block in the 4×4 block within the current CU, (v x , v y ) and applies it to the corresponding 4×4 block.

[0106] These four parameters of the 4-parameter affine model can be estimated iteratively. The MV pair at step k is represented as Denote the original luminance signal as I(i, j) and the predicted luminance signal as I' k (i, j). Spatial gradient g x (i, j) and g y (i, j) can be applied to the prediction signal I' in the horizontal and vertical directions respectively k The derivative of equation (1) can be expressed as:

[0107]

[0108] Where (a, b) is the delta translation parameter at step k, and (c, d) is the delta scale and rotation parameter. The delta MV at the control point can be derived from its coordinates as equations (3) and (4). For example, (0, 0) and (w, 0) are the coordinates of the upper left and upper right control points, respectively.

[0109]

[0110]

[0111] Based on the optical flow equation, the relationship between brightness change, spatial gradient and temporal movement is expressed as:

[0112]

[0113] Will and Substituting into equation (2), we get the equation for parameters (a, b, c, d).

[0114]

[0115] Since all samples in CU satisfy equation (6), the least square error method can be used to solve the parameter set (a, b, c, d). The two control points at step (k+1) The MV at can be solved using equations (3) and (4), and they are rounded to a certain precision (e.g., 1 / 4 pixel). Using iteration, the MV at the two control points can be refined until the parameters (a, b, c, d) are all zero or the iteration time meets a predefined limit, and the MV converges.

[0116] The affine motion model with six parameters has the following parameters: two parameters for translation in the horizontal and vertical directions, one parameter for scaling in the horizontal direction, one parameter for rotation in the horizontal direction, one parameter for scaling in the vertical direction, and one parameter for rotation in the vertical direction. The 6-parameter affine motion model is encoded by three MVs at three control points. Figure 8 As shown, three control points of a 6-parameter affine coded CU are defined at the upper left, upper right, and lower left corners of the CU. The motion at the upper left control point is related to translation motion, the motion at the upper right control point is related to rotation and scaling motion in the horizontal direction, and the motion at the lower left control point is related to rotation and scaling motion in the vertical direction. For the 6-parameter affine motion model, rotation and scaling motion in the horizontal direction may be different from those in the vertical direction. The motion vector (v) of each sub-block is derived using the three MVs at the control points. x , v y ),as follows:

[0117]

[0118] Where (v 2x , v 2y ) the motion vector of its lower left control point, (x, y) the center position of its sub-block, and w and h the width and height of its CU.

[0119] The six parameters of the 6-parameter affine model are estimated in a similar manner. Equation (2) is modified as follows.

[0120]

[0121] Wherein, at step k, (a, b) are the delta translation parameters, (c, d) are the delta scaling and rotation parameters in the horizontal direction, and (e, f) are the delta scaling and rotation parameters in the vertical direction. Equation (8) is modified accordingly.

[0122]

[0123] By considering all samples in the CU, the parameter set (a, b, c, d, e, f) can be solved using the least squares method. The MV of is calculated using equation (3). The upper right control point The MV of is calculated using equation (10). The upper right control point The MV of is calculated using equation (11).

[0124]

[0125]

[0126] Optical flow prediction refinement (PROF) .

[0127] Affine motion model parameters can be used to derive motion vectors for each pixel in the CU. However, due to the high complexity and memory access bandwidth required to generate pixel-based affine motion compensation predictions, current VVC uses a sub-block-based affine motion compensation approach, where the CU is divided into 4×4 sub-blocks, each assigned an MV derived from the control point MVs of the affine CU. Sub-block-based affine motion compensation is a trade-off between coding efficiency, complexity, and memory access bandwidth. Due to the sub-block-based prediction, it loses prediction accuracy.

[0128] To achieve finer motion compensation granularity, JVET-N0236 proposed Optical Flow Prediction Refinement (PROF) to refine sub-block-based affine motion compensation predictions using optical flow. After performing sub-block-based affine motion compensation, the luma prediction samples are refined by adding the difference derived from the optical flow equation. The proposed PROF is described in the following four steps.

[0129] Step 1: Perform sub-block based affine motion compensation to generate sub-block prediction I(i, j).

[0130] Step 2: Compute the spatial gradient g of the sub-block prediction at each sample position using a 3-tap filter [-l, 0, 1] x (i, j) and g y (i, j).

[0131] g x (i,j)=I(i+1,j)-I(i-1,j)

[0132] g y (i,j)=I(i,j+1)-I(i,j-1)

[0133] For gradient calculation, the sub-block prediction is extended by one pixel on each side. To reduce memory bandwidth and complexity, pixels on the extended boundary are copied from the nearest integer pixel position in the reference picture. Thus, additional interpolation of the padded area is avoided.

[0134] Step 3: Calculate brightness prediction refinement through the optical flow equation.

[0135] ΔI(i, j)=g x (i, j)*Δv x (i, j)+gy (i, j)*Δv y (i, j)

[0136] where Δv(i, j) is the difference between the sample-specific MV (denoted by v(i, j)) calculated for the sample position (i, j) and the sub-block MV of the sub-block to which the pixel (i, j) belongs, as Figure 9 shown.

[0137] Since the affine model parameters and the pixel position relative to the sub-block center do not change between sub-blocks, Δv(i, j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. Let x and y be the horizontal and vertical offsets from the pixel position to the sub-block center, Δv(x, y) can be derived by the following equation:

[0138]

[0139] For a 4-parameter affine model,

[0140]

[0141] For the 6-parameter affine model,

[0142]

[0143] Where (v 0x , v 0y )、(v 1x , v 1y )、(v 2x , v 2y ) are the motion vectors of the upper left control point, the upper right control point and the lower left control point, w and h are the width and height of the CU.

[0144] Step 4: Finally, add the luma prediction refinement to the sub-block prediction I(i, j). The final prediction I' is generated according to the following equation.

[0145] I′(i,j)=I(i,j)+ΔI(i,j)

[0146] Problems Solved in Some Implementations .

[0147] In modern video coding standards such as AVC, HEVC, and VVC, motion vector precision can be higher than integer precision. When a motion vector has fractional components, predictions at fractional pixel positions are generated by interpolation. The interpolation operation can use additional samples from adjacent integer pixel positions, which increases memory access bandwidth. The length of the interpolation filter affects prediction efficiency. In AVC, the interpolation filter for the luma component is 6 taps. The minimum PU size is 8×8. For HEVC, the interpolation filter for the luma component is increased to 8 taps to improve coding efficiency.

[0148] For HEVC, some restrictions are applied to small PUs to reduce the worst-case memory access bandwidth. For example, 4×4 PUs are not allowed for both uni- and bi-prediction, while 4×8 and 8×4 PUs only allow uni-prediction. In VVC, the interpolation filter length for the luma component is maintained at 8 taps. The minimum PU size is 4×4. For non-affine mode, 4×4 PUs only allow uni-prediction, while 4×8 and 8×4 PUs allow both uni- and bi-prediction. The worst-case memory access bandwidth of VVC is 4×8 or 8×4 bi-prediction, which is approximately 47% higher than the worst-case memory access bandwidth of HEVC (8×8 bi-prediction). Memory access bandwidth affects the design of codec chips, such as the area and operating frequency of the data bus. Higher bandwidth generally means higher cost. While maintaining coding efficiency, reducing the worst-case memory access bandwidth can be beneficial. Different solutions have been proposed to address this issue, such as using shorter tap interpolation filters for those small PUs or using padding to limit memory access while maintaining the same interpolation filter length. Both methods compromise coding efficiency. Shorter tap interpolation filters introduce new interpolation filters, which imposes an implementation burden. Padding methods have greater computational complexity because the length of the interpolation filters is not reduced.

[0149] Some systems and methods described herein operate to reduce worst-case memory access bandwidth and reduce computational complexity. Further described are systems and methods for compensating for coding efficiency losses.

[0150] Overview of exemplary embodiments.

[0151] The systems and methods described herein operate to reduce memory access bandwidth for inter-frame prediction, particularly to reduce worst-case memory access bandwidth. Typically, worst-case memory access bandwidth occurs on small sample blocks (such as small PUs). Compared to conventional interpolation-based motion compensation, the systems and methods described herein can further reduce computational complexity.

[0152] In some embodiments, during motion compensation, the motion vector is first rounded to integer precision. An initial (unrefined) prediction is generated by copying from the integer position pointed to by the rounded motion vector in the reference picture without interpolation. The sample intensity error is estimated by applying optical flow as input, using the spatial gradient of the unrefined prediction and the delta motion vector between the actual MV before rounding and the rounded MV as input. The final prediction is refined by adding the pixel intensity error to the unrefined prediction.

[0153] Inter-frame prediction bandwidth reduction method .

[0154] In some embodiments, a method is provided for reducing memory access bandwidth by applying pixel intensity errors determined using optical flow equations. Such a method can reduce memory access bandwidth by avoiding interpolation operations used to generate prediction signals in sub-pixel positions. An example of the method proposed herein can be implemented using the following steps.

[0155] During the rounding process, block-based motion vectors are rounded to integer precision. The rounded motion vectors are used to perform motion compensation by copying those corresponding integer samples from the reference picture. Since the motion vectors are integer precision, no interpolation is required. Reference pixels in integer positions can be used directly to generate predictions (before refinement), referred to herein as unrefined predictions.

[0156] Calculate the spatial gradient g of the unrefined prediction sample at each sample location x (i, j) and g y (i, j). In one example, the spatial gradient can be generated using the same gradient generation process used in bidirectional optical flow. For example, the horizontal gradient at a sample location is calculated as the difference between its right neighbor and its left neighbor, while the vertical gradient at a sample location is calculated as the difference between its bottom neighbor and its top neighbor. In another example, a Sobel filter can be used to generate the spatial gradient.

[0157] The motion vector rounding error is calculated as follows:

[0158]

[0159] Where (v x , v y ) and (v′ x , v′ y ) are the MV and the rounded MV at pixel position (i, j), respectively. For non-affine coded blocks, all pixels in the same block share the same MV. Therefore, the motion vector rounding error is the same for all pixels in the same motion compensated block. For affine coded blocks, each pixel can have a different MV, and therefore the motion vector rounding error can be different for each pixel.

[0160] The pixel intensity change is calculated via the optical flow equation as the scalar product of the gradient and the motion vector rounding error.

[0161] ΔI(i, j)=g x (i, j)*Δv x (i, j)+g y (i, j)*Δv y (i, j) (13)

[0162] The prediction is refined by adding brightness intensity variations. The final prediction I′ is generated according to the following equation.

[0163] I′(i,j)=I(i,j)+ΔI(i,j) (14)

[0164] Where I(i, j) is the unrefined prediction.

[0165] In one embodiment, during rounding, rather than rounding the motion vector to integer precision, the motion vector is rounded to a different precision that is lower than the original precision. For example, if the motion vector precision is 1 / 16 pixel precision, it can be rounded to 1 / 2 pixel precision. In this case, interpolation can still be used. However, a shorter tap filter, such as a 4-tap filter or a bilinear (2-tap) filter, can be used to generate the sub-pixels. In this case, the memory bandwidth reduction ratio can be reduced, but the motion vector error is also reduced. Therefore, the prediction accuracy can be improved.

[0166] In another embodiment, a longer gradient filter can be used to compensate for the accuracy lost by removing interpolation or reducing the length of the interpolation filter. In this case, padding of the extended unrefined prediction block for gradient calculation with a longer gradient filter can be used to avoid increasing memory access bandwidth too much.

[0167] The proposed method can reduce memory bandwidth by avoiding interpolation operations or using filters with shorter lengths in motion compensation. The computational complexity can also be reduced accordingly.

[0168] In some embodiments, to reduce the worst-case memory access bandwidth, the bandwidth reduction methods described herein may be applied only to motion compensation of small blocks (e.g., PUs or CUs), such as 4×4, 4×8, 8×4 uni-predicted / bi-predicted. In some embodiments, the methods described herein are applied only to motion compensation of blocks whose area or number of samples does not exceed a threshold (such as an area threshold of 32 samples). In some embodiments, the methods described herein are applied only to blocks whose horizontal size does not exceed a threshold. In some embodiments, the methods described herein are applied only to blocks whose vertical size does not exceed a threshold. In some embodiments, the methods described herein are applied only to blocks whose horizontal size or vertical size does not exceed a threshold.

[0169] The memory access bandwidth of other larger blocks (eg, PUs or CUs) is not a bottleneck, and conventional motion compensation can still be applied.

[0170] Correction of the use of sub-block motion vectors .

[0171] In some embodiments, the bandwidth reduction method described herein is combined with PROF for affine prediction. In some such embodiments, sub-block motion vectors are derived from control point motion vectors. The sub-block motion vectors are then rounded to integer precision. The prediction is obtained by motion compensation with rounded MVs (e.g., integer precision). The motion vector error Δv(i, j) is the sum of the sub-block motion vector rounding error and the motion vector difference between the pixel MV and the sub-block MV. PROF is then applied with Δv(i, j) and the unrefined prediction as input. When combined with PROF, the memory access bandwidth and computational complexity for affine motion compensation can be greatly reduced.

[0172] In some such embodiments, a motion vector v(i, j) is determined for a current sample in the current block based on an affine motion model. The affine motion model may be determined based on the control point motion vectors of the current block. A sub-block motion vector is also determined for each sub-block in the current block. The sub-block motion vectors may be rounded to generate rounded sub-block motion vectors. An unrefined prediction for the current sample is obtained using the rounded motion sub-block motion vectors. An error vector for the current sample is determined by subtracting the rounded sub-block motion vector from the motion vector v(i, j) of the corresponding sample. A spatial gradient is obtained at the sample position of the current sample. A refined prediction for the current sample is obtained based on the unrefined prediction, the spatial gradient, and the error vector. For example, the refined prediction may be obtained by adding a scalar product between the spatial gradient and the error vector to the unrefined prediction.

[0173] Exemplary methods and systems .

[0174] like Figure 10As shown, the method performed in some embodiments includes rounding (1002) a first motion vector associated with a current sample to generate a second motion vector. An unrefined prediction of the current sample is generated (1004) or otherwise obtained using the rounded second motion vector. A rounding error vector is determined (1006) or otherwise obtained that indicates a difference between the first motion vector and the second motion vector. A spatial gradient of the sample value is determined (1008) or otherwise obtained at a sample position of the current sample. A refined prediction of the current sample is obtained based on the unrefined prediction, the spatial gradient, and the rounding error vector. To obtain the refined prediction, a scalar product between the spatial gradient and the rounding error vector may be determined (1010), and the scalar product may be added (1012) to the unrefined prediction to generate a refined prediction of the current sample. This may be repeated for multiple current samples in a current sample block in the video. Figure 10 method. Figure 10 The method can be implemented as part of the encoding process or as part of the decoding process.

[0175] In some embodiments, a device includes one or more processors configured to: round a first motion vector associated with a current sample to generate a second motion vector; generate an unrefined prediction for the current sample using the second motion vector; determine a rounding error vector representing a difference between the first motion vector and the second motion vector; determine a spatial gradient of sample values ​​at a sample position of the current sample; and generate a refined prediction for the current sample by adding a scalar product between the spatial gradient and the rounding error vector to the unrefined prediction. Such a device may be implemented in an encoder and / or a decoder.

[0176] In some embodiments, an apparatus includes a module for performing an operation on a first motion vector associated with a current sample to generate a second motion vector; a module for generating an unrefined prediction of the current sample using the second motion vector; a module for determining a rounding error vector representing a difference between the first motion vector and the second motion vector; a module for determining a spatial gradient of sample values ​​at a sample position of the current sample; and a module for generating a refined prediction of the current sample by adding a scalar product between the spatial gradient and the rounding error vector to the unrefined prediction. Such an apparatus may be implemented in an encoder and / or a decoder. In an encoder, these modules may be implemented in a motion prediction module 210 ( Figure 2A ) is implemented at the decoder. In the decoder, these modules can be implemented at the motion compensation prediction module 260 ( Figure 2B ) is implemented.

[0177] In some embodiments, a signal for encoding a video is provided, wherein the signal is generated using a method comprising: rounding a first motion vector associated with a current sample to generate a second motion vector; generating an unrefined prediction for the current sample using the second motion vector; determining a rounding error vector representing a difference between the first motion vector and the second motion vector; determining a spatial gradient of sample values ​​at a sample position of the current sample; and generating a refined prediction for the current sample by adding a scalar product between the spatial gradient and the rounding error vector to the unrefined prediction. The signal may encode a residual value representing a difference between a precise prediction for the current sample and an input sample value in the video.

[0178] Some embodiments include an apparatus comprising: an access unit configured to access data comprising a signal as described above; and a transmitter configured to transmit the data. Some embodiments include a method comprising: accessing data comprising a signal as described above; and transmitting the data.

[0179] In some embodiments, an apparatus comprises: a device as described herein, such as configured to perform Figure 10 and at least one of: (i) an antenna configured to receive a signal comprising data representing an image; (ii) a band limiter configured to limit the received signal to a frequency band comprising data representing an image; or (iii) a display configured to display the image. In some such embodiments, the device may be a TV, a cellular phone, a tablet, or a STB.

[0180] In some embodiments, a computer-readable medium is provided that stores a signal as described above.

[0181] In some embodiments, a computer-readable medium is provided that includes instructions for causing one or more processors to perform the following operations: rounding a first motion vector associated with a current sample to generate a second motion vector; generating an unrefined prediction of the current sample using the second motion vector; determining a rounding error vector representing a difference between the first motion vector and the second motion vector; determining a spatial gradient of sample values ​​at a sample position of the current sample; and generating a refined prediction of the current sample by adding a scalar product between the spatial gradient and the rounding error vector to the unrefined prediction.

[0182] In some embodiments, a computer program product is provided that includes instructions that, when the program is executed by one or more processors, cause the one or more processors to perform a method comprising the following operations: rounding a first motion vector associated with a current sample to generate a second motion vector; generating an unrefined prediction of the current sample using the second motion vector; determining a rounding error vector representing a difference between the first motion vector and the second motion vector; determining a spatial gradient of sample values ​​at a sample position of the current sample; and generating a refined prediction of the current sample by adding a scalar product between the spatial gradient and the rounding error vector to the unrefined prediction.

[0183] Other implementation plans .

[0184] In some embodiments, inter-frame prediction of samples (such as luma samples) in a current block is performed by rounding an initial motion vector and determining a rounding error vector caused by the rounding. An unrefined prediction of the sample is generated using the rounded motion vector. In the case where the initial motion vector has been rounded to integer precision, the unrefined prediction can be generated by copying the sample value at the position indicated by the rounded motion vector from a reference picture. Such copying can be performed without interpolation. Unrefined predictions are similarly generated for other samples in the current block. Based on the unrefined prediction, a spatial gradient is determined for each sample position in the block. A refined prediction is generated for each sample position by adding the scalar product between the spatial gradient at the sample position and the rounding error vector to the unrefined prediction. The exemplary method can reduce the number of reference pixels used to predict the current block and, therefore, can reduce memory access bandwidth.

[0185] In some embodiments, a method for encoding a video comprising a plurality of blocks, including a current block, is provided. The method comprises, for each of a plurality of samples in the current block: generating an unrefined prediction for the sample using a rounded motion vector associated with the sample; and generating a refined prediction for the sample by adding to the unrefined prediction a scalar product between (i) a spatial gradient at the sample location and (ii) a rounding error vector associated with the sample. The rounded motion vector may be generated by rounding an initial motion vector associated with the current sample. The rounded motion vector is rounded to integer precision or fractional precision. A rounding error vector may be calculated by subtracting the rounded motion vector from the initial motion vector.

[0186] In some such embodiments, the initial motion vector is a sub-block motion vector of a sub-block containing the sample, and the initial motion vector is generated using an affine motion model. In other embodiments, the initial motion vector is a translation motion vector associated with the current block.

[0187] The spatial gradient of a sample may be based at least in part on an unrefined prediction of a plurality of neighboring samples. For example, the horizontal component of the spatial gradient may be calculated as the difference between the right neighboring sample and the left neighboring sample, and wherein the vertical component of the spatial gradient may be calculated as the difference between the bottom neighboring sample and the top neighboring sample. In some embodiments, a Sobel filter may be used to calculate the spatial gradient. In some embodiments, the current block may be padded to calculate the spatial gradient.

[0188] In some embodiments, the unrefined prediction of the generated samples is performed using a four-tap filter. In some embodiments, the unrefined prediction of the generated samples is performed using a bilinear filter. In some embodiments, the unrefined prediction of the generated samples may be performed by copying sample values ​​from a reference block, provided that the rounded motion vectors are rounded to integer precision.

[0189] In some embodiments, the determination of whether to perform the prediction methods described herein (rather than using conventional translation or affine inter prediction) is based on the size of the current block. For example, in some embodiments, the prediction methods described herein are performed only in response to a determination that the current block is 4×4, 4×8, or 8×4.

[0190] In some embodiments where affine prediction is used, the rounding error vector includes not only an error component due to rounding, but also an error component due to the difference between the sample-specific motion vector and the sub-block motion vector. In some such embodiments, an initial motion vector associated with a sample is generated by determining a sub-block motion vector for a sub-block including the initial motion vector using an affine motion model. A rounded motion vector is generated by rounding the initial motion vector. The rounding error vector can be determined as the sum of: (i) the error generated by rounding the initial motion vector; and (ii) the motion vector difference between the sub-block motion vector and the sample-specific motion vector.

[0191] In some embodiments using affine prediction, an initial motion vector associated with a sample is generated by determining a sub-block motion vector for a sub-block including the initial motion vector using an affine motion model. A rounded motion vector is generated by rounding the initial motion vector. A sample-specific motion vector for the sample is calculated using the affine motion model. A rounding error vector is calculated such that it represents the difference between: (i) the sample-specific motion vector for the sample based on the affine motion model; and (ii) the rounded motion vector.

[0192] In some embodiments, a method for encoding a video comprising a plurality of blocks, including a current block, is provided. The method comprises: identifying at least one initial motion vector, each sample in the current block corresponding to one of the initial motion vectors; rounding each of the initial motion vectors to generate a corresponding rounded motion vector; for each sample, generating an unrefined prediction for each sample in the current block using the rounded motion vector corresponding to the sample; at each sample position in the current block, calculating a spatial gradient based on the unrefined prediction; for each of the rounded motion vectors, calculating a corresponding rounding error vector by subtracting the corresponding rounded motion vector from its corresponding initial motion vector; calculating an intensity change for each sample in the current block, the intensity change being the scalar product of: (i) the spatial gradient at the corresponding sample position; and (ii) the rounding error vector corresponding to the sample; and generating a refined prediction for each sample in the current block by adding the corresponding intensity change to each corresponding unrefined prediction.

[0193] In some embodiments, the current block is predicted using bi-prediction. In such embodiments, the method may further include: generating two refined predictions for each sample in the current block from different reference pictures; and averaging the two refined predictions to generate a bi-prediction for each sample.

[0194] In some embodiments, the method described herein is performed by an encoder. The encoder is operable to: subtract a refined prediction of a current block from an input block to generate a prediction residual; and encode the prediction residual in a bitstream. The initial motion vector may also be signaled in the bitstream.

[0195] In some embodiments, the method described herein is performed by a decoder. The decoder is operable to: decode a prediction residual from a bitstream; and add the prediction residual to a refined prediction of a current block to generate a reconstructed block. The initial motion vector may be signaled in the bitstream.

[0196] Some embodiments include a processor and a non-transitory computer-readable medium storing instructions operable to perform the methods described herein. Some embodiments include a non-transitory computer-readable storage medium storing a video bitstream generated using the methods described herein.

[0197] This disclosure describes a number of aspects, including tools, features, embodiments, models, methods, and the like. Many of these aspects are specifically described, and at least in order to illustrate individual characteristics, are typically described in a manner that may sound restrictive. However, this is for the purpose of describing clearly and does not limit the disclosure or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with aspects described in earlier applications.

[0198] The aspects described and contemplated in this disclosure can be implemented in many different forms. Although some embodiments are specifically shown, other embodiments are contemplated, and the discussion of specific embodiments does not limit the breadth of the specific implementation. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting generated or encoded bitstreams. These and other aspects can be implemented as methods, devices, computer-readable storage media having stored thereon instructions for encoding or decoding video data according to any of the methods described, and / or computer-readable storage media having stored thereon a bitstream generated according to any of the methods described.

[0199] In this disclosure, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image," "picture," and "frame" are used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoding side, while "decoding" is used on the decoding side.

[0200] The terms HDR (high dynamic range) and SDR (standard dynamic range) generally convey specific dynamic range values ​​to those of ordinary skill in the art. However, additional embodiments are also intended in which references to HDR are understood to mean "higher dynamic range" and references to SDR are understood to mean "lower dynamic range." Such additional embodiments are not constrained by any specific values ​​for dynamic range that might typically be associated with the terms "high dynamic range" and "standard dynamic range."

[0201] Various methods are described herein, and each method in these methods includes one or more steps or actions for realizing the method.Unless the correct operation of the method requires specific steps or action sequence, otherwise the order and / or use of specific steps and / or actions can be modified or combined.In addition, in various embodiments, terms such as "first", "second" can be used for modifying elements, parts, steps, operations, etc., such as "first decoding" and "second decoding".Unless otherwise required, using such terms does not mean that the modified operation is sorted.Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can, for example, occur before the second decoding, during the second decoding, or in the time period overlapping with the second decoding.

[0202] For example, various numerical values ​​may be used in the present disclosure. Specific values ​​are used for illustrative purposes, and the described aspects are not limited to these specific values.

[0203] The embodiments described herein may be executed by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, these embodiments may be implemented by one or more integrated circuits. The processor may be of any type suitable for the technical environment, and as a non-limiting example, may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0204] Various implementations involve decoding. As used in this disclosure, "decoding" may encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also include or alternatively include processes performed by the decoder of the various implementations described in this disclosure, such as: extracting a picture from a tiled (packed) picture; determining the upsampling filter to use and then upsampling the picture; and flipping the picture back to its intended orientation.

[0205] As another example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or broadly to a broader decoding process will be clear based on the context of the specific description.

[0206] Various implementations involve encoding. In a manner similar to the discussion above regarding "decoding," "encoding," as used in this disclosure, may encompass, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as partitioning, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also include or alternatively include processes performed by an encoder of the various implementations described in this disclosure.

[0207] As another example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or broadly to a broader encoding process will be clear based on the context of the specific description.

[0208] When the figures are presented in the form of flow charts, it should be understood that the figures also provide block diagrams of the corresponding devices. Similarly, when the figures are presented in the form of block diagrams, it should be understood that the figures also provide flow charts of the corresponding methods / processes.

[0209] Various embodiments relate to rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is often considered, often within the constraints of computational complexity. Rate-distortion optimization is often formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on extensive testing of all coding options (including all considered modes or coding parameter values) and a complete evaluation of their coding costs and associated distortions in the reconstructed signal after encoding and decoding. Faster approaches may also be used to save coding complexity, particularly by computing approximate distortions based on predictions or prediction residual signals rather than reconstructed residual signals. A hybrid of these two approaches may also be used, such as by using approximate distortions for only some of the possible coding options and full distortions for others. Other approaches only evaluate a subset of the possible coding options. More generally, many approaches employ any of a variety of techniques to perform optimization, but optimization does not necessarily involve a complete evaluation of both coding costs and associated distortions.

[0210] The specific implementations and aspects described herein can be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a specific implementation of a single form (e.g., discussed only as a method), the specific implementation of the features discussed can also be implemented in other forms (e.g., a device or program). The device can be implemented in, for example, suitable hardware, software, and firmware. These methods can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication equipment, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other equipment that is convenient for information communication between end users.

[0211] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0212] Additionally, the present disclosure may refer to “determining” various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0213] Furthermore, the present disclosure may involve “accessing” various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0214] Additionally, the present disclosure may involve "receiving" various information. Like "accessing," receiving is a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from a memory device). Furthermore, during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often involved in one way or another.

[0215] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," use of any of the following " / ," "and / or," and "at least one" is intended to encompass selecting only the first-listed option (A), or only the second-listed option (B), or both options (A and B). As further examples, in the case of "A, B, and / or C" and "at least one of A, B, and C," such phrases are intended to encompass selecting only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or only the first-listed option and the second-listed option (A and B), or only the first-listed option and the third-listed option (A and C), or only the second-listed option and the third-listed option (B and C), or all three options (A, B, and C). This can be extended to as many items as are listed.

[0216] Furthermore, as used herein, the term "signal" means, among other things, indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a specific parameter from among multiple parameters for region-based filter parameter selection for de-artifact filtering. Thus, in one embodiment, the same parameters are used at both the encoding and decoding ends. Thus, for example, an encoder may transmit (explicitly signal) specific parameters to a decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has the specific parameters along with other parameters, signaling (implicit signaling) may be used without transmission to simply allow the decoder to know and select the specific parameters. By avoiding the transmission of any actual functionality, bit savings are achieved in various embodiments. It should be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, information is signaled to a corresponding decoder using one or more syntax elements, flags, etc. Although previously described with reference to the verb form of the word "signal," the word "signal" may also be used herein as a noun.

[0217] Specific implementations may generate various signals formatted to carry information, which may be stored or transmitted, for example. The information may include, for example, instructions for performing a method or data generated by one of the described specific implementations. For example, a signal may be formatted to carry a bit stream of the described embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier using the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, signals may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0218] A number of embodiments are described. Features of these embodiments may be provided individually or in any combination across the various claim classes and types. In addition, an embodiment may include one or more of the following features, devices, or aspects, individually or in any combination across the various claim classes and types:

[0219] • Modification of the motion compensated prediction process applied in the decoder and / or encoder.

[0220] • Enabling several advanced motion compensated prediction methods in the decoder and / or encoder.

[0221] • A bitstream or signal comprising syntax conveying information generated according to any of the described embodiments.

[0222] • Creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.

[0223] • A method, process, apparatus, medium storing instructions, medium storing data, or signal according to any of the described embodiments.

[0224] • A television, set-top box, cell phone, tablet or other electronic device that performs motion compensated prediction according to any of the described embodiments.

[0225] • A television, set-top box, cell phone, tablet computer, or other electronic device that performs motion compensated prediction according to any of the described embodiments and displays the resulting image (e.g., using a monitor, screen, or other type of display).

[0226] • A television, set-top box, cell phone, tablet, or other electronic device that selects (e.g., using a tuner) a channel to receive a signal comprising an encoded image and performs motion compensated prediction according to any of the described embodiments.

[0227] • A television, set-top box, cell phone, tablet, or other electronic device that receives (e.g., using an antenna) an over-the-air signal including an encoded image and performs motion compensated prediction according to any of the described embodiments.

[0228] It is noted that the various hardware elements of one or more of the described embodiments are referred to as "modules" that perform (i.e., execute, implement, etc.) the various functions described herein in conjunction with the corresponding modules. As used herein, a module includes hardware that is deemed suitable for a given specific implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices). Each of the described modules may also include executable instructions for performing one or more functions described as being performed by the corresponding module, and it is noted that these instructions may take the form of or include the following instructions: hardware (i.e., hard-wired) instructions, firmware instructions, software instructions, etc., and may be stored in any suitable one or more non-transitory computer-readable media (such as commonly referred to as RAM, ROM, etc.).

[0229] Although features and elements are described above in particular combinations, each feature or element may be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated into a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as internal hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROM disks and digital versatile disks (DVDs)). A processor associated with software may be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A video decoding method, comprising: rounding a first motion vector associated with a current sample to obtain a second motion vector; obtaining an unrefined prediction of the current sample using the second motion vector; obtaining a rounding error vector having a value equal to a difference between the first motion vector and the second motion vector; obtaining a spatial gradient at a sample position of the current sample; and A refined prediction of the current sample is obtained based on the unrefined prediction, the spatial gradient, and the rounding error vector. The method of claim 1 , wherein the second motion vector is rounded to integer precision. The method of claim 1 , wherein the second motion vector is rounded to fractional precision. 4 . The method of claim 1 , wherein the first motion vector is a sub-block motion vector of a sub-block containing the current sample, and wherein the first motion vector is obtained using an affine motion model. 5 . The method of claim 1 , wherein the first motion vector is a translational motion vector associated with a current block containing the current sample, the translational motion vector indicating translational motion in horizontal and vertical directions. The method according to claim 1 , wherein obtaining the spatial gradient at the sample position of the current sample is performed based on an unrefined prediction of a plurality of samples neighboring the current sample.

7. The method of claim 1 , wherein the horizontal component of the spatial gradient is obtained as a difference between a right neighboring sample and a left neighboring sample of the current sample, and wherein the vertical component of the spatial gradient is obtained as a difference between a top neighboring sample and a bottom neighboring sample of the current sample.

8. The method of claim 1, wherein obtaining the unrefined prediction of the current sample is performed using a four-tap interpolation filter.

9. The method of claim 1, wherein obtaining the unrefined prediction of the current sample is performed using a bilinear interpolation filter.

10. The method of claim 1, wherein a determination to perform rounding of the first motion vector is made based on a size of a current block that includes the current sample.

11. The method of claim 1, wherein the determination to perform rounding of the first motion vector is made based on a determination that a current block including the current sample is a 4x4, 4x8, or 8x4 block.

12. The method of claim 1, wherein the first motion vector is signaled in a bitstream.

13. The method of claim 1, wherein the refined prediction of the current sample is subtracted from an input sample value to generate a prediction residual, and wherein the prediction residual is encoded in a bitstream.

14. The method of claim 1, wherein a prediction residual for the current sample is decoded from a bitstream and added to the refined prediction of the current sample to generate a reconstructed sample value.

15. The method of claim 1, wherein the refined prediction of the current sample is obtained by adding a scalar product between the spatial gradient and the rounding error vector to the unrefined prediction.

16. A video decoding device, comprising one or more processors, wherein the one or more processors are configured to perform the following operations: rounding a first motion vector associated with a current sample to obtain a second motion vector; obtaining an unrefined prediction of the current sample using the second motion vector; obtaining a rounding error vector having a value equal to a difference between the first motion vector and the second motion vector; obtaining a spatial gradient at a sample position of the current sample; and A refined prediction of the current sample is obtained based on the unrefined prediction, the spatial gradient, and the rounding error vector. The apparatus of claim 16 , wherein the second motion vector is rounded to integer precision. The apparatus of claim 16 , wherein the second motion vector is rounded to fractional precision.

19. The apparatus of claim 16, wherein the first motion vector is a sub-block motion vector of a sub-block containing the current sample, and wherein the first motion vector is obtained using an affine motion model.

20. The apparatus of claim 16, wherein the first motion vector is a translational motion vector associated with a current block containing the current sample, the translational motion vector indicating translational motion in horizontal and vertical directions.

21. The apparatus of claim 16, wherein obtaining the spatial gradient at a sample position of the current sample is performed based on an unrefined prediction of a plurality of samples neighboring the current sample.

22. The apparatus of claim 16, wherein a horizontal component of the spatial gradient is obtained as a difference between a right-neighboring sample and a left-neighboring sample of the current sample, and wherein a vertical component of the spatial gradient is obtained as a difference between a top-neighboring sample and a bottom-neighboring sample of the current sample.

23. The apparatus of claim 16, wherein obtaining the unrefined prediction of the current sample is performed using a four-tap interpolation filter.

24. The device of claim 16, wherein obtaining the unrefined prediction of the current sample is performed using a bilinear interpolation filter.

25. The apparatus of claim 16, wherein a determination to perform rounding of the first motion vector is made based on a size of a current block that includes the current sample.

26. The apparatus of claim 16, wherein the determination to perform rounding of the first motion vector is made based on a determination that a current block including the current sample is a 4x4, 4x8, or 8x4 block.

27. The apparatus of claim 16, wherein the first motion vector is signaled in a bitstream.

28. The apparatus of claim 16, wherein the refined prediction of the current sample is subtracted from an input sample value to generate a prediction residual, and wherein the prediction residual is encoded in a bitstream.

29. The apparatus of claim 16, wherein a prediction residual for the current sample is decoded from a bitstream and added to the refined prediction of the current sample to generate a reconstructed sample value.

30. The apparatus of claim 16, wherein the refined prediction of the current sample is obtained by adding a scalar product between the spatial gradient and the rounding error vector to the unrefined prediction.

31. A computer-readable medium comprising instructions for causing one or more processors to: rounding a first motion vector associated with a current sample to obtain a second motion vector; obtaining an unrefined prediction of the current sample using the second motion vector; obtaining a rounding error vector having a value equal to a difference between the first motion vector and the second motion vector; obtaining a spatial gradient at a sample position of the current sample; and A refined prediction of the current sample is obtained based on the unrefined prediction, the spatial gradient, and the rounding error vector.

32. The computer-readable medium of claim 31, wherein the first motion vector is a sub-block motion vector of a sub-block containing the current sample, and wherein the first motion vector is obtained using an affine motion model.

33. The computer-readable medium of claim 31, wherein the first motion vector is a translational motion vector associated with a current block containing the current sample, the translational motion vector indicating translational motion in horizontal and vertical directions.

34. The computer-readable medium of claim 31, wherein obtaining the spatial gradient at a sample position of the current sample is performed based on an unrefined prediction of a plurality of samples neighboring the current sample.

Images