Block boundary optical flow prediction refinement

Abstract

Systems, methods, and instrumentalities are disclosed for sub-block / block refinement, including sub-block / block boundary refinement, such as block boundary optical flow prediction refinement (BBPROF). A block including a current sub-block can be decoded based on a sample value of a first pixel obtained based on, for example, a MV of the current sub-block, a MV of a sub-block neighboring the current sub-block, and a sample value of a second pixel neighboring the first pixel. BBPROF can include determining a spatial gradient at a pixel / sample location. A MV difference between the current sub-block and one or more neighboring sub-blocks can be calculated. A MV offset at the pixel / sample location can be determined based on the MV difference. A sample value offset of the pixel in the current sub-block can be determined. A prediction of a reference picture list can be refined by adding the calculated sample value offset to a sub-block prediction.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 856,519, filed June 3, 2019, entitled “Block Boundary Prediction Refinement with Optical Flow,” the entire contents of which are incorporated herein by reference as if fully set forth herein. Background Technology

[0003] Video coding systems can be used to compress digital video signals, for example, to reduce the storage and / or transmission bandwidth required for such signals. Video coding systems can include block-based, wavelet-based, and / or object-based systems. Hybrid block-based video coding systems can be deployed. Summary of the Invention

[0004] Systems, methods, and tools for sub-block / block correction are disclosed, including sub-block / block boundary correction, such as Block Boundary Optical Flow Prediction Correction (BBPROF). A block including the current sub-block can be decoded based on sample values ​​obtained for a first pixel, which can be obtained from, for example, the motion vector (MV) of the current sub-block, the MVs of neighboring sub-blocks, and sample values ​​of a second pixel adjacent to a first pixel. Sub-block / block correction can be applied to decoder-side motion vector correction (DMVR) mode, sub-block-based temporal motion vector prediction (SbTMVP) mode, and / or affine mode. BBPROF can include, for example, sub-block-based motion compensation to generate sub-block-based predictions. Spatial gradients of sub-block-based predictions at one or more pixel / sample locations can be computed. MV differences between the current sub-block and one or more neighboring sub-blocks can be computed. MV differences can be used to compute motion vector offsets at one or more pixel / sample locations. Intensity changes for each pixel in the current sub-block can be computed based on optical flow. Sample value offsets can be used to indicate the intensity changes for each pixel. The prediction of pixel or sample location can be corrected, for example, by adding the calculated intensity change to the sub-block prediction (e.g., motion-compensated prediction).

[0005] In an example, a method can be implemented to perform sub-block / block correction. The method can be implemented, for example, by an apparatus that can include one or more processors configured to execute computer-executable instructions that can be stored on a computer-readable medium or computer program product that, when executed by the one or more processors, perform the method. Thus, an apparatus can include one or more processors configured to perform the method. The computer-readable medium or computer program product can include instructions that cause the one or more processors to perform the method by executing the instructions. The computer-readable medium can contain data content generated according to the method. A signal can include a residual generated based on an original image block and a block predicted using obtained sample values of first pixels according to the method. An apparatus can include an accessing unit and a transmitter configured to perform a second method that includes: accessing data that includes a residual generated according to the apparatus based on obtained sample values of first pixels, the apparatus including one or more processors configured to implement the method (e.g., by executing instructions); and transmitting the data including the residual. A device, such as a television, mobile phone, tablet, or set-top box, can include: an apparatus having one or more processors configured to implement the method (e.g., by executing instructions); and at least one of: (i) an antenna configured to receive a signal that includes data representative of an image; (ii) a frequency band limiter configured to limit the received signal to a frequency band that includes data representative of an image; or (iii) a display configured to display the image.

[0006] A method for performing sub-block / block correction can include, for example, obtaining a sample value of a first pixel based on, for example, a MV of a current sub-block, MVs of sub-blocks neighboring the current sub-block, and sample values of second pixels neighboring the first pixel; and decoding a block including the current sub-block based on the obtained sample value of the first pixel.

[0007] A method for performing sub-block / block correction can include, for example, obtaining a sample value of a first pixel based on, for example, a MV of a current sub-block, MVs of sub-blocks neighboring the current sub-block, and sample values of second pixels neighboring the first pixel; and decoding a block including the current sub-block based on the obtained sample value of the first pixel.

[0008] The block can include, for example, a first pixel, a second pixel, and a third pixel adjacent to the first pixel. Obtaining the sample value of the first pixel can include, for example, determining that the first pixel is adjacent to a boundary of the current sub-block; determining (i) a difference between the MV of the current sub-block and the MV of a sub-block adjacent to the current sub-block, (ii) a gradient of the first pixel based on the sample value of the second pixel and the sample value of the third pixel, and (iii) a sample value offset based on the determined gradient and the difference between the MV of the current sub-block and the MV of the sub-block adjacent to the current sub-block; and obtaining the sample value of the first pixel based on the determined sample value offset.

[0009] The gradient can be determined, for example, based at least on the sample value of the second pixel. The gradient can be used to obtain the sample value of the first pixel.

[0010] The gradient of the optical flow model can be determined, for example, based at least on the sample value of the second pixel. The gradient can be used in the optical flow model to obtain the sample value of the first pixel.

[0011] The sample value of the first pixel can be obtained using the difference between the MV of the current sub-block and the MV of a sub-block adjacent to the current sub-block.

[0012] The sub-block adjacent to the current sub-block can be a first sub-block. The block can include the first sub-block and a second sub-block adjacent to the current sub-block. The sample value of the first pixel can be obtained based on (e.g., further based on) the MV of the second sub-block.

[0013] The sample value of the first pixel can be obtained based on a determination that the first pixel is adjacent to a boundary of the current sub-block.

[0014] The first pixel and the second pixel can be in the current sub-block.

[0015] The sample value of the first pixel can be obtained using a weighting factor. The weighting factor can vary according to a distance of the first pixel from a corresponding boundary of the current sub-block.

[0016] The sample value offset of the first pixel can be determined, for example, based on the MV of the current sub-block, the MV of a sub-block adjacent to the current sub-block, and the sample value of a second pixel adjacent to the first pixel. The sample value of the first pixel can be obtained using the determined sample value offset of the first pixel and the predicted sample value.

[0017] The sample value of the first pixel can be obtained, for example, based on a determination that the first pixel is adjacent to a boundary of the current sub-block. The boundary of the current sub-block can include a common boundary between the current sub-block and a sub-block adjacent to the current sub-block.

[0018] The first pixel can be, for example, located in four rows of pixels from a top boundary of the current sub-block, located in four rows of pixels from a left boundary of the current sub-block, located in four columns of pixels from a left boundary of the current sub-block, or located in four columns of pixels from a right boundary of the current sub-block.

[0019] Each feature disclosed herein can be described in relation to any other feature or combination of features, whether or not expressly disclosed in combination with the other feature(s). Any combination of one or more features disclosed herein can be implemented in any combination with any other feature or features disclosed herein. BRIEF DESCRIPTION OF DRAWINGS

[0020] FIG. 1A is a system diagram illustrating an example communications system in which one or more disclosed embodiments can be implemented.

[0021] FIG. 1B is a system diagram illustrating an example wireless communications system in which FIG. 1A embodiments can be implemented.

[0022] FIG. 1C is a system diagram illustrating an example radio access network (RAN) and an example core network (CN) that can be used within the communications system FIG. 1A illustrated in FIG. 1.

[0023] FIG. 1D is a system diagram illustrating another example RAN and another example CN that can be used within the communications system FIG. 1A illustrated in FIG. 1.

[0024] FIG. 2 is a schematic diagram illustrating an example of a video encoder.

[0025] FIG. 3 is a schematic diagram illustrating an example of a video decoder.

[0026] FIG. 4 is a schematic diagram illustrating an example of a system in which various aspects and examples can be implemented.

[0027] FIG. 5 An example four-parameter affine mode model and sub-block level motion derivation for affine blocks is shown.

[0028] FIG. 6 An example six-parameter affine mode is shown, where V0, V1, and V2 are control points, and (MV x , MV y ) are motion vectors of sub-blocks centered at position (x, y).

[0029] FIG. 7 An example decoded-side motion vector (MV) refinement is shown.

[0030] FIG. 8AAn example of spatial neighboring blocks usable by sub-block based temporal motion vector prediction (SbTMVP) is shown.

[0031] FIG. 8B An example derivation of sub-coding unit (CU) motion field is shown.

[0032] FIG. 9 An example sub-block with overlap block motion compensation (OBMC) applied is shown.

[0033] FIG. 10 An example sub-block MV (V SB ) and pixel delta v(i,j).

[0034] FIG. 11 An example of a method for sub-block / block correction according to one or more of equations (1) to (25) is shown.

[0035] FIG. 12 An example MV difference calculation from selected neighboring sub-blocks is shown. DETAILED DESCRIPTION

[0036] The detailed description now will be described with reference to various drawings. Although this description provides a detailed example of possible implementations, it should be noted that the details are intended to be exemplary and in no way limit the scope of this application.

[0037] FIG. 1A is a diagram illustrating an example communications system 100 in which one or more disclosed embodiments can be implemented. The communications system 100 can be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 100 can enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 100 can employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique-word OFDM (UW-OFDM), resource block-filter OFDM, filter bank multicarrier (FBMC), and the like.

[0038] As FIG. 1AAs shown, the communication system 100 can include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104 / 113, a CN 106 / 115, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d can be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d (any of which can be referred to as a "station" and / or a "STA") can be configured to transmit and / or receive wireless signals, and can include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot or other wireless devices operating in an industrial and / or an automated processing chain environment), a consumer electronics, a device operating on a commercial and / or industrial wireless network, and the like. Any of the WTRUs 102a, 102b, 102c, and 102d can be interchangeably referred to as a UE.

[0039] The communication system 100 can also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b can be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106 / 115, the Internet 110, and / or the other networks 112. By way of example, the base stations 114a, 114b can be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b can include any number of interconnected base stations and / or network elements.

[0040] The base stations 114a can be part of the RAN 104 / 113, which can also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base stations 114a and / or the base stations 114b can be configured to transmit and / or receive wireless signals on one or more carrier frequencies (which can be referred to as a cell (not shown)). These frequencies can be in the licensed spectrum, the unlicensed spectrum, or a combination of the licensed and unlicensed spectrums. A cell can provide wireless service to a particular geographic area that can be relatively fixed or can change over time. The cell can further be divided into cell sectors. For example, a cell associated with a base station 114a can be divided into three sectors. Thus, in one embodiment, the base station 114a can include three transceivers, one for each sector of the cell. In an embodiment, the base station 114a can employ Multiple Input Multiple Output (MIMO) techniques and can utilize multiple transceivers for each sector of the cell. For example, beamforming can be used to transmit and / or receive signals in desired spatial directions.

[0041] The base stations 114a, 114b can communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over the air interface 116, which can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 can be established using any suitable radio access technology (RAT).

[0042] More specifically, as noted above, the communications system 100 can be a multiple access system and can employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104 / 113 and the WTRUs 102a, 102b, 102c can implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can establish the air interface 115 / 116 / 117 using wideband CDMA (WCDMA). WCDMA can include communication protocols such as High-Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA can include High-Speed Downlink (DL) Packet Access (HSDPA) and / or High-Speed UL Packet Access (HSUPA).

[0043] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c can implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which can establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-A Pro.

[0044] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c can implement a radio technology such as NR Radio Access, which can establish the air interface 116 using New Radio (NR).

[0045] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c can implement multiple radio access technologies. For example, the base station 114a and WTRUs 102a, 102b, 102c can implement LTE wireless access and NR wireless access together, for instance using dual connectivity (DC) principles. Thus, the air interface utilized by WTRUs 102a, 102b, 102c can be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., an eNB and a gNB).

[0046] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c can implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 IX, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0047] FIG. 1AThe base station 114b in the RAN 104 / 113 can be a wireless router, Home Node B, Home eNode B, or access point, for example, and can utilize any suitable RAT for facilitating wireless connectivity access. In one embodiment, the base station 114b and the WTRUs 102c, 102d can implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d can implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d can utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or femtocell. As shown, the base station 114b can have a direct connection to the Internet 110. Thus, the base station 114b can not be required to access the Internet 110 via the CN 106 / 115. FIG. 1A

[0048] The RAN 104 / 113 can be in communication with the CN 106 / 115, which can be any type of network configured to provide voice, data, applications, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data can have varying quality of service (QoS) requirements, such as differing throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 106 / 115 can provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions, such as user authentication. Although not shown in FIG. 1A Although not shown in FIG. 10, it will be appreciated that the RAN 104 / 113 and / or the CN 106 / 115 can be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which can employ a NR radio technology, the CN 106 / 115 can also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0049] ​The CN 106 / 115 can also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or the other networks 112. The PSTN 108 can include circuit-switched telephone networks that provide infrastructure for the

[0050] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 can include multi-mode capabilities, e.g., the WTRUs 102a, 102b, 102c, 102d can include multiple transceivers for communicating with different wireless networks over different wireless links. For example, the WTRU 102a, 102b, 102c, 102d can include a transceiver FIG. 1A The WTRU 102c shown in Figure 1A can be configured to communicate as a cellular telephone with the base station 114a and as a wireless local loop (WLL) station with the base station 114b.

[0051] FIG. 1B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1B As shown, the WTRU 102 can include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138, among others. It will be appreciated that the WTRU 102 can include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0052] The processor 118 can be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 can perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 can be coupled to the transceiver 120, which can be coupled to the transmit / receive element 122. While FIG. 1B The processor 118 and the transceiver 120 are depicted as separate components, it is to be understood that the processor 118 and the transceiver 120 can be integrated together in an electronic package or chip.

[0053] The transmit / receive element 122 can be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 can be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 can be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 can be configured to transmit and / or receive both RF and light signals. It will be appreciated that the transmit / receive element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0054] Although the transmit / receive element 122 is depicted in the WTRU 102 FIG. 1B In one embodiment, the WTRU 102 can include two or more transmit / receive elements 122 (e.g., multiple antennas) to enable MIMO technology. Thus, the WTRU 102 can

[0055] The transceiver 120 can be configured to modulate information to be transmitted by the transmit / receive element 122 and to demodulate information received by the transmit / receive element 122. As indicated above, the WTRU 102 can be a multi-mode device. Thus, the transceiver 120 can include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11, for example.

[0056] The processor 118 of the WTRU 102 can be coupled to, and can receive user input data from, the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 can also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 can access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and / or the removable memory 132. The non-removable memory 130 can include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 can include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 can access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0057] The processor 118 can receive power from the power source 134 and can be configured to distribute and / or control the power to the other components in the WTRU 102. The power source 134 can be any suitable device for powering the WTRU 102. For example, the power source 134 can include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0058] The processor 118 can also be coupled to the GPS chipset 136, which can be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 can receive location information over the air interface 116 from a base station (e.g., base stations 114a, 114b) and / or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 can acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0059] The processor 118 can further be coupled to other peripherals 138, which can include one or more software and / or hardware modules that provide additional features, functionality and / or wired or wireless connectivity. For example, the peripherals 138 can include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® The peripheral device 138 can include one or more sensors, which can be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, a compass sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0060] The WTRU 102 can include a full duplex radio for which transmission and reception of some or all signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and the downlink (e.g., for reception) can be concurrent and / or simultaneous. The full duplex radio can include an interference management unit to reduce and / or substantially eliminate self-interference and / or cross- interference by utilizing hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In one embodiment, the WTRU 102 can include a full duplex radio for which transmission and reception of some or all signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and the downlink (e.g., for reception) can be concurrent and / or simultaneous.

[0061] FIG. 1C is a system diagram illustrating the RAN 104 and the CN 106 according to an embodiment. As described above, the RAN 104 can be in communication with the WTRUs 102a, 102b, 102c over the air interface 116 and can include eNode-Bs 160a, 160b, 160c, although it will be appreciated that the RAN 104 can include any number of eNode-Bs while remaining consistent with an embodiment. The eNode-Bs 160a, 160b, 160c can each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the eNode-Bs 160a, 160b, 160c can implement MIMO technology. Thus, the eNode-B 160a, for example, can use multiple antennas to transmit wireless signals to, and / or receive wireless signals from, the WTRU 102a.

[0062] The RAN 104 can include eNode-Bs 160a, 160b, 160c, although it will be appreciated that the RAN 104 can include any number of eNode-Bs while remaining consistent with an embodiment. The eNode-Bs 160a, 160b, 160c can each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the eNode-Bs 160a, 160b, 160c can implement MIMO technology. Thus, the eNode-B 160a, for example, can use multiple antennas to transmit wireless signals to, and / or receive wireless signals from, the WTRU 102a.

[0063] Each of the eNode-Bs 160a, 160b, 160c can be associated with a particular cell (not shown) and can be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, and the like. As shown, the eNode-Bs 160a, 160b, 160c can communicate with one another over an X2 interface. FIG. 1C

[0064] FIG. 1C The CN 106 can include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or PGW) 166. While each of the foregoing elements are depicted as part of the CN 106, it will be appreciated that any of these elements can be owned and / or operated by an entity other than the CN operator.

[0065] The MME 162 can be connected to each of the eNode-Bs 162a, 162b, 162c in the RAN 104 via an SI interface and can serve as a control node. For example, the MME 162 can be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activations / deactivations, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, and the like. The MME 162 can provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM and / or WCDMA.

[0066] The SGW 164 can be connected to each of the eNode-Bs 160a, 160b, 160c in the RAN 104 via the S1 interface. The SGW 164 can generally route and forward user data packets to / from the WTRUs 102a, 102b, 102c. The SGW 164 can perform other functions, such as anchoring user planes during inter-eNode B handovers, triggering paging when DL data is available for the WTRUs 102a, 102b, 102c, managing and storing contexts of the WTRUs 102a, 102b, 102c, and the like.

[0067] The SGW 164 can be connected to the PGW 166, which can provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0068] ​CN 106 can facilitate communications with other networks. For example, the CN 106 can provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional landline

[0069] Although WTRUs are described in FIGS. 1A-1D as being wireless terminals, it is contemplated that in certain representative embodiments that such terminals can use (e.g., temporarily or permanently) wired communication interfaces with the communication network.

[0070] In representative embodiments, the other network 112 can be a WLAN.

[0071] A WLAN in Infrastructure Basic Service Set (BSS) mode can have an access point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP can have an access or an interface to a Distribution System (DS) or another type of wired / wireless network. Traffic to and from the STAs that is not addressed to the AP can be sent to and received from the AP. The AP can be responsible for managing the wireless medium during an association period. Traffic between STAs associated with the AP can be sent through the AP. In an exemplary embodiment, the AP can be a WLAN access point.

[0072] When using an 802.11 ac infrastructure mode of operation or similar mode of operation, an AP can transmit beacons on a fixed channel, such as a primary channel. The primary channel can be a fixed width (e.g., 20 MHz wide bandwidth) or a width that is dynamically set via signaling. The primary channel can be the operating channel of the BSS and can be used by STAs to establish a connection with the AP. In certain representative embodiments, carrier sense multiple access / collision avoidance (CSMA / CA) can be implemented, for example, in 802.11 systems. For CSMA / CA, a STA (e.g., each STA), including the AP, can listen to the primary channel. If the primary channel is sensed / detected as busy by a particular STA, the particular STA can back off. Only one STA can transmit in a given BSS at any given time.

[0073] High Throughput (HT) STAs can use 40 MHz wide channels to communicate, for example, via a combination of the primary 20 MHz channel with an adjacent or nonadjacent 20 MHz channel to form a 40 MHz wide channel.

[0074] Very High Throughput (VHT) STAs can support 20 MHz, 40 MHz, 80 MHz, and / or 160 MHz wide channels. 40 MHz and / or 80 MHz channels can be formed by combining contiguous 20 MHz channels. A 160 MHz channel can be formed by combining 8 contiguous 20 MHz channels, or by combining two noncontiguous 80 MHz channels, which can be referred to as an 80+80 configuration. For the 80+80 configuration, after channel encoding, the data can be parsed by a segment parser that can divide the data into two streams. Inverse Fast Fourier Transform (IFFT) processing and time domain processing can be done on each stream separately. The streams can be mapped to the two 80 MHz channels, and the data can be transmitted by a transmitting STA. At the receiver of the receiving STA, the above described operation for the 80+80 configuration can be reversed, and the combined data can be sent to the Medium Access Control (MAC).

[0075] 802.11af and 802.11ah support sub-1 GHz modes of operation. The channel operating bandwidth and carrier are reduced in 802.11af and 802.11ah relative to those used in 802.11η and 802.1 lac. 802.11af supports 5 MHz, 10 MHz, and 20 MHz bandwidths in the television white space (TVWS) spectrum, and 802.11ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz bandwidths using non-TVWS spectrum. According to a representative implementation, 802.11ah can support meter type control / machine type communications, such as MTC devices in a macro coverage area. MTC devices can have certain capabilities, e.g., limited capabilities, including support for (e.g., only support for) certain bandwidths and / or limited bandwidth. MTC devices can include a battery with a battery life above a threshold (e.g., to maintain a very long battery life).

[0076] WLAN systems that can support multiple channels and channel bandwidths such as 802.11η, 802.1 lac, 802.11af, and 802.11ah include a channel that can be designated as a primary channel. The primary channel can have a bandwidth equal to the largest common operating bandwidth supported by all STAs in a BSS. The bandwidth of the primary channel can be set and / or limited by a STA from all STAs operating in the BSS that supports the smallest bandwidth mode of operation. In the example of 802.11ah, for a STA (e.g., MTC type device) that supports (e.g., only supports) a 1 MHz mode, the primary channel can be 1 MHz wide even though other STAs in the AP and BSS support 2 MHz, 4 MHz, 8 MHz, 16 MHz, and / or other channel bandwidth modes of operation. Carrier sensing and / or network allocation vector (NAV) settings can depend on the status of the primary channel. If the primary channel is busy, e.g., due to a STA (only supporting a 1 MHz mode of operation) transmitting to the AP, the entire available frequency band can be considered busy even though most of the frequency band remains idle and can be available.

[0077] In the United States, the available frequency bands for 802.11ah to use are 902 MHz to 928 MHz. In Korea, the available frequency bands are 917.5 MHz to 923.5 MHz. In Japan, the available frequency bands are 916.5 MHz to 927.5 MHz. The total bandwidth available to 802.11ah is 6 MHz to 26 MHz, depending on the country code.

[0078] FIG. 1D is a system diagram illustrating the RAN 113 and the CN 115 according to an embodiment. As noted above, the RAN 113 can employ an NR radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 113 can also be in communication with the CN 115.

[0079] The RAN 113 can include gNBs 180a, 180b, 180c, although the RAN 113 can include any number of gNBs while remaining consistent with an embodiment. The gNBs 180a, 180b, 180c can each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the gNBs 180a, 180b, 180c can implement MIMO technology. For example, gNBs 180a, 108b can utilize beamforming to transmit signals to and / or receive signals from the gNBs 180a, 180b, 180c. Thus, the gNB 180a, for example, can use multiple antennas to transmit wireless signals to, and / or receive wireless signals from, the WTRU 102a. In an embodiment, the gNBs 180a, 180b, 180c can implement carrier aggregation technology. For example, the gNB 180a can transmit multiple component carriers to the WTRU 102a (not shown). A subset of these component carriers can be on unlicensed spectrum while the remaining component carriers can be on licensed spectrum. In an embodiment, the gNBs 180a, 180b, 180c can implement Coordinated Multi-Point (CoMP) technology. For example, WTRU 102a can receive coordinated transmissions from gNBs 180a and 180b (and / or gNB 180c).

[0080] The WTRUs 102a, 102b, 102c can use transmission associated with scalable numerology to communicate with gNBs 180a, 180b, 180c. For example, OFDM symbol spacing and / or OFDM subcarrier spacing can vary from different transmissions, from different cells, and / or from different portions of the wireless transmission spectrum. The WTRUs 102a, 102b, 102c can use subframe or transmission time interval (TTI) of various or scalable lengths (e.g., containing different quantities of OFDM symbols and / or lasting varying lengths of absolute time) to communicate with gNBs 180a, 180b, 180c.

[0081] The gNBs 180a, 180b, 180c can be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and / or a non-standalone configuration. In the standalone configuration, the WTRUs 102a, 102b, 102c can communicate with one or more of gNBs 180a, 180b, 180c without also accessing other RANs, such as eNode-Bs 160a, 160b, 160c. In the standalone configuration, the WTRUs 102a, 102b, 102c can utilize one or more of gNBs 180a, 180b, 180c as a mobility anchor point. In the standalone configuration, the WTRUs 102a, 102b, 102c can utilize signals

[0082] Each of the gNBs 180a, 180b, 180c can be associated with a particular cell (not shown) and can be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, support of network slicing, dual connectivity, interworking between NR and E-UTRA, routing of user plane data towards user plane functions (UPFs) 184a, 184b, routing of control plane information towards access and mobility management functions (AMFs) 182a, 182b, and / or the like. As shown, the gNBs 180a, 180b, 180c can communicate with one another over an Xn interface. FIG. 1D As shown, the gNBs 180a, 180b, 180c can be in communication with the AN 180a, 180b, 180c over an X2 interface.

[0083] FIG. 1DThe illustrated CN 115 can include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one Session Management Function (SMF) 183a, 183b, and possibly a Data Network (DN) 185a, 185b. While each of the foregoing elements are depicted as part of the CN 115, it will be appreciated that any of these elements can be owned and / or operated by an entity other than the CN operator.

[0084] The AMF 182a, 182b can be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via an N2 interface and can serve as a control node. For example, the AMF 182a, 182b can be responsible for authenticating WTRUs 102a, 102b, 102c, supporting network slicing (e.g., handling different PDU sessions with different requirements), selecting a particular SMF 183a, 183b, managing the WTRU 102a, 102b, 102c registration area, terminating NAS signaling, mobility management, and the like. The AMF 162 can utilize network slicing to customize CN support for WTRUs 102a, 102b, 102c based on the type of services utilized by the WTRUs 102a, 102b, 102c. For example, different network slices can be established for different use cases such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced mobile broadband (eMBB) access, services for machine type communication (MTC) access, and / or the like. The AMF 162 can provide control plane functionality for 5G NR and non-3GPP access network technologies such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies such as WiFi.

[0085] The SMF 183a, 183b can be connected to AMF 182a, 182b in the CN 115 via an N11 interface. The SMF 183a, 183b can also be connected to UPF 184a, 184b in the CN 115 via an N4 interface. The SMF 183a, 183b can select and control the UPF 184a, 184b and configure the routing of traffic through the UPF 184a, 184b. The SMF 183a, 183b can perform other functions, such as managing and allocating IP address, managing PDU sessions, controlling policy enforcement and QoS, providing downlink data notifications, and the like. A PDU session type can be IP-based, non-IP based, Ethernet-based, and the like.

[0086] The UPF 184a, 184b can be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via an N3 interface, which can provide WTRUs 102a, 102b, 102c with access to packet- switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The UPF 184, 184b can perform other functions, such as routing and forwarding packets, enforcing user plane policies, supporting multi-homed PDU sessions, handling user plane QoS, buffering of downlink packets, providing mobility anchoring, and the like.

[0087] The CN 115 can facilitate communications with other networks. For example, the CN 115 can include, or can communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the CN 115 and the PSTN 108. Further, the CN 115 can provide the WTRUs 102a, 102b, 102c with access to the other networks 112, which can include other wired and / or wireless networks that are owned and / or operated by other service providers. In one embodiment, the WTRUs 102a, 102b, 102c can be connected to a DN 185a, 185b through the UPF 184a, 184b via the N3 interface between the UPF 184a, 184b and the UPF 184a, 184b and an N6 interface between the UPF 184a, 184b and the DN 185a, 185b.

[0088] In view of FIGS. 1A-1D And FIGS. 1A-1D In view of the corresponding descriptions of FIGS. 1 through 18, one or more or all of the functions described with reference to one or more of the WTRUs 102a-d, base stations 114a-b, RANs 104 / 106 / 113, core network 106 / 112 / 115, other networks 112, and / or any other devices described herein can be performed by one or more emulation devices (not shown). The emulation devices can be one or more devices configured to emulate one or more or all of the functions described herein. For example, emulation devices can be used to test other devices and / or to simulate network and / or WTRU functionality.

[0089] The one or more emulation devices can perform the one or more, including all, functions while not implemented / deployed as part of a wired and / or wireless communication network. For example, an emulation device can be utilized in a testing scenario in a testing laboratory and / or a non-deployed (e.g., testing) wired and / or wireless communication network in order to implement testing of one or more components. The one or more emulation devices can be test equipment. Direct RF coupling and / or wireless communications via RF circuitry (e.g., which can include one or more antennas) can be used by an emulation device to transmit and / or receive data.

[0090] The one or more emulation devices can perform the one or more, including all, functions while not implemented / deployed as part of a wired and / or wireless communication network. For example, an emulation device can be utilized in a testing scenario in a testing laboratory and / or a non-deployed (e.g., testing) wired and / or wireless communication network in order to implement testing of one or more components. The one or more emulation devices can be test equipment. Direct RF coupling and / or wireless communications via RF circuitry (e.g., which can include one or more antennas) can be used by an emulation device to transmit and / or receive data.

[0091] This application describes a number of aspects, including tools, features, examples or embodiments, models, methods, etc. Many of these aspects are described in a particular manner, and at least to illustrate individual features, often in a manner that can sound limiting. However, this is for description’s sake, and does not limit the application or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. Moreover, these aspects can also be combined and interchanged with aspects described in earlier filings.

[0092] The aspects described and contemplated in this application can be implemented in many different forms. The aspects described herein are described with respect to video coding and decoding, and at least one other aspect is described with respect to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatuses, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having a bitstream generated according to any of the described methods stored thereon. FIGS. 5-12 Some embodiments can be provided, but other embodiments are contemplated. FIGS. 5-12 The discussion of the aspects contained in this application is not limiting of the breadth of the specific implementations. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatuses, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having a bitstream generated according to any of the described methods stored thereon.

[0093] In this application, the terms “reconstruct” and “decode” can be used interchangeably, the terms “pixel” and “sample” can be used interchangeably, and the terms “image,” “picture,” and “frame” can be used interchangeably.

[0094] Various methods are described herein, and each of the methods includes one or more steps or actions for accomplishing the described method. Unless a specific order of steps or actions is required for proper operation of a method, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, the terms “first,” “second,” and the like, if used in the description, are used to modify the elements, components, steps, operations, and the like, such as “first decoding” and “second decoding.” Unless specifically required, the use of such terms is not intended to imply a sequencing of the modified operations. Thus, in this example, the first decoding need not be performed before the second decoding, and can occur, for example, before, during, or overlapping in time with the second decoding.

[0095] Various methods and other aspects described in this application can be used to modify modules of video encoder 200 and decoder 300 (e.g., intra prediction and entropy encoding and / or decoding modules (260, 360, 245, 330)), as shown in FIG. 2 and FIG. 3 The subject matter disclosed herein presents aspects that are not limited to VVC or HEVC, and can be applied to, for example, any type, format, or version of video coding (whether described in a standard or in a proposal, whether pre-existing or future-developed), as well as extensions of any such standards and proposals (e.g., including VVC and HEVC). Unless otherwise indicated or technically precluded, aspects described in this application can be used individually or in combination.

[0096] Various numerical values are used in describing examples of this application, such as weighting factors (such as {1 / 4, 1 / 8, 1 / 16, 1 / 32} or {3 / 4, 7 / 8, 15 / 16, 31 / 32}), filters (such as a 3-tap filter [-1, 0, 1]), and the like. These and other specific values are used for the purpose of describing examples, and the described aspects are not limited to these specific values.

[0097] FIG. 2 FIG. 2 is a schematic diagram illustrating an example video encoder. Variations of the example encoder 200 are contemplated, but the encoder 200 is described below for the sake of clarity without describing all contemplated variations.

[0098] Before encoding, the video sequence can undergo pre-encoding processing (201), e.g., applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to get a signal distribution more resilient to compression (e.g., using a histogram equalization of one of the color components). Metadata can be associated with the pre-processing and attached to the bitstream.

[0099] In the encoder 200, a picture is encoded by the encoder elements as described below. The picture to be encoded is partitioned (202) and processed in units of, for example, coding units (CUs). Each unit is encoded using, for example, intra mode or inter mode. When a unit is encoded in intra mode, it performs intra prediction (260). In inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which of intra mode or inter mode to use for encoding the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. The prediction residual is calculated, for example, by subtracting (210) the prediction block from the original image block.

[0100] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, as well as the motion vectors and other syntax elements, are entropy encoded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly on the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization processes.

[0101] The encoder decodes the encoded blocks to provide references for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (255) to reconstruct the image block. A loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce encoding artifacts. The filtered image is stored in a reference picture buffer (280).

[0102] FIG. 3 is a schematic diagram illustrating an example of a video decoder. In the example decoder 300, a bitstream is decoded by the decoder elements as described below. The video decoder 300 generally performs a decoding process that is the inverse of the encoding process as described in FIG. 2 The encoder 200 can generally also perform video decoding as part of encoding video data. For example, the encoder 200 can perform one or more of the video decoding steps presented herein. The encoder, for example, reconstructs the decoded images to maintain synchronization with the decoder with respect to one or more of the following: reference pictures, entropy coding contexts, and other decoder-related state variables.

[0103] In particular, the input to the decoder includes a video bitstream, which can be generated by video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other coded information. Picture partitioning information indicates how the pictures are partitioned. Thus, the decoder can partition (335) the pictures according to the decoded picture partitioning information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (355) to reconstruct the image blocks. The prediction blocks can be obtained (370) from intra prediction (360) or motion-compensated prediction (i.e., inter prediction) (375). Loop filters are applied (365) to the reconstructed pictures. The filtered pictures are stored in the reference picture buffer (380).

[0104] The decoded pictures can also undergo post-decoding processing (385), e.g., inverse color transform (e.g., transform from YCbCr 4:2:0 to RGB 4:4:4), or inverse remapping, which performs the inverse of the remapping process performed in the pre-encoding processing (201). The post-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0105] FIG. 4 FIG. 4 is a schematic diagram illustrating an example of a system in which various aspects and embodiments described herein can be implemented. System 400 can embody a device that includes the various components described below and is configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 400, singly or in combination, can be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one example, the processing and encoder / decoder elements of system 400 are distributed across multiple ICs and / or discrete components. In various embodiments, system 400 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, communication buses or by special-purpose input and / or output ports. In various embodiments, system 400 is configured to implement one or more of the aspects described in this document.

[0106] The system 400 includes at least one processor 410 configured to execute instructions loaded thereinto for implementing the various aspects described in this document, for example. The processor 410 can include embedded memory, input output interface, and various other circuitry known in the art. The system 400 includes at least one memory 420 (e.g., a volatile memory device and / or a non-volatile memory device). The system 400 includes a storage device 440, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. By way of non-limiting example only, the storage device 440 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and / or a network accessible storage device.

[0107] The system 400 includes an encoder / decoder module 430 configured, for example, to process data to provide encoded video or decoded video, and the encoder / decoder module 430 can include its own processor and memory. The encoder / decoder module 430 represents module(s) that can be included in a device to perform the encoding functions and / or the decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module 430 can be implemented as a separate element of the system 400, or can be incorporated in the processor 410 as a combination of hardware and software as known to those skilled in the art.

[0108] Program code to be loaded onto the processor 410 or the encoder / decoder 430 to perform the various aspects described in this document can be stored in the storage device 440 and then loaded onto the memory 420 for execution by the processor 410. In accordance with various embodiments, one or more of the processor 410, the memory 420, the storage device 440, and the encoder / decoder module 430 can store one or more of various items during the performance of the processes described in this document. Such storage items can include, but are not limited to, input video, decoded video or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0109] In some implementations, the processor 410 and / or memory internal to the encoder / decoder module 430 are used to store instructions and provide working memory for processing needed during encoding or decoding. However, in other implementations, memory external to the processing device (e.g., the processing device can be the processor 410 or the encoder / decoder module 430) is used for one or more of these functions. The external memory can be the memory 420 and / or storage device 440, such as dynamic volatile memory and / or non-volatile flash memory. In several implementations, the external non-volatile flash memory is used to store, for example, the operating system of the television. In at least one implementation, fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as, for example, MPEG-2 (MPEG refers to Moving Pictures Expert Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the Joint Video Expert Team, JVET).

[0110] As shown in block 445, input to elements of the system 400 can be provided through various input devices. Such input devices include, but are not limited to: (i) a radio frequency (RF) portion that receives RF signals transmitted, for example, over the air by a broadcaster; (ii) a component (COMP) input terminal (or set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. FIG. 4 Other examples, not shown in FIG. 4, include composite video.

[0111] In various embodiments, the input devices of block 445 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements adapted to (i) select a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band), (ii) down-convert the selected signal, (iii) band-limit again to a narrower frequency band to select a signal frequency band which can be referred to as a channel in certain embodiments, for example, (iv) demodulate the down-converted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired data packet stream. The RF portion of various embodiments includes one or more elements used to perform these functions, such as frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various ones of these functions, including, for example, down-converting a received signal to a lower frequency (such as an intermediate frequency or a near-baseband frequency) or to a baseband frequency. In one set-top box embodiment, the RF portion and its associated input processing elements receive an RF signal transmitted via a wired (e.g., cable) medium, and perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements can include inserting elements between existing elements, such as amplifiers and analog-to-digital converters. In various embodiments, the RF portion includes an antenna.

[0112] Further, the USB and / or HDMI terminals can include respective interface processors for connecting the system 400 to other electronic devices across USB and / or HDMI connections. It will be appreciated that various aspects of input processing (e.g., Reed-Solomon error correction) can be implemented as desired within, for example, a separate input processing IC or within the processor 410. Similarly, aspects of USB or HDMI interface processing can be implemented within a separate interface IC or within the processor 410 as desired. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 410 and the encoder / decoder 430, which operate in conjunction with memory and storage elements to process the data streams as desired for presentation on output devices.

[0113] The various elements of the system 400 can be disposed within an integrated housing. Within the integrated housing, the various elements can be interconnected and transmit data between these elements using suitable connection arrangements 425, such as internal buses known in the art, including Inter-Integrated Circuit (I2C) buses, wiring, and printed circuit boards.

[0114] The system 400 includes a communication interface 450 that enables communication with other devices via a communication channel 460. The communication interface 450 can include, but is not limited to, a transceiver configured to transmit and to receive data via the communication channel 460. The communication interface 450 can include, but is not limited to, a modem or network card, and the communication channel 460 can be implemented, for example, within a wired and / or wireless medium.

[0115] In various embodiments, data is streamed or otherwise provided to the system 400 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals of these examples are received through the communication channel 460 and the communication interface 450 adapted for Wi-Fi communication. The communication channel 460 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet for allowing streaming applications and other over-the- cloud communications. Other embodiments provide streaming data to the system 400 using a set-top box that delivers data through an HDMI connection of the input block 445. Still other embodiments provide streaming data to the system 400 using an RF connection of the input block 445. As noted above, various embodiments provide data in a non-streaming manner. Moreover, various embodiments use wireless networks other than Wi-Fi, e.g., a cellular network or a Bluetooth network.

[0116] The system 400 can provide output signals to various output devices including a display 475, speakers 485, and other peripheral devices 495. The display 475 of various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 475 can be used in a television set, a tablet, a laptop, a mobile phone (a smartphone), or other device. The display 475 can also be integrated with other components (e.g., as in a smartphone), or separate (e.g., an external monitor for a notebook). In various examples of embodiments, the other peripheral devices 495 include one or more of a standalone digital video recorder (or digital versatile recorder, both terms are DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 495 that provide a function based on the output of the system 400. For example, a disc player performs the function of playing the output of the system 400.

[0117] In various embodiments, control signals use signaling such as AV.Link, Consumer Electronics Control (CEC) or other communications protocols that enable device-to- device control without or with minimal user intervention. The output devices can be communicatively coupled to system 400 via dedicated connections, through the respective interfaces 470, 480, and 490. Alternatively, output devices can be connected to system 400 using the communication channel 460 via the communication interface 450. Display 475 and speakers 485 can be integrated into a single unit with other components of system 400 in an electronic device, such as a television. In various embodiments, display interface 470 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0118] Alternatively, if the RF portion of input 445 is part of a separate set-top box, display 475 and speakers 485 can be separate from one or more of the other components. In various embodiments in which display 475 and speakers 485 are external components, output signals can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0119] These embodiments can be implemented by computer software implemented by the processor 410 or by hardware, or by a combination of hardware and software. As a non-limiting example, these embodiments can be implemented by one or more integrated circuits. The memory 420 can be of any type appropriate for the technical environment and can be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory and removable memory, as non-limiting examples. The processor 410 can be of any type appropriate for the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0120] Various implementations participate in decoding. As used in this application, “decoding” can encompass all or a portion of the processes performed on a received encoded sequence in order to produce a final output suitable for display, for example. In various embodiments, such processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and difference decoding. In various embodiments, such processes also or alternatively include processes performed by decoders of various implementations described in this application, such as decoding a block including a current sub-block based on sample values obtained for a first pixel of the current sub-block including sample values that can be obtained based on a motion vector (MV) of the current sub-block, an MV of a sub-block neighboring the current sub-block, and a sample value of a second pixel neighboring the first pixel, for example.

[0121] As a further example, in one implementation“encoding” refers only to differential encoding, in another implementation“encoding” refers only to entropy encoding, and in yet another implementation“encoding” refers to a combination of differential encoding and entropy encoding. Whether the phrase“encoding process” refers specifically to a subset of operations or refers broadly to a more extensive encoding process will be clear based on the context of the specific description, and is believed to be well understood by those skilled in the art.

[0122] Various implementations participate in encoding. In a similar manner as discussed above with respect to“decoding,”“encoding” as used in this application can encompass, for example, the entire or partial process performed on an input video sequence in order to produce an encoded bitstream. In various implementations, such a process includes one or more processes typically performed by an encoder, e.g., partitioning, differential encoding, transform, quantization, and entropy encoding. In various implementations, such a process also or alternatively includes processes performed by the encoders of the various implementations described in this application, e.g., encoding a block including a current sub-block based on sample values obtained for a first pixel of the current sub-block including sample values that can be obtained based on a motion vector (MV) of the current sub-block, an MV of a sub-block neighboring the current sub-block, and a sample value of a second pixel neighboring the first pixel, etc.

[0123] As a further example, in one implementation“encoding” refers only to differential encoding, in another implementation“encoding” refers only to entropy encoding, and in yet another implementation“encoding” refers to a combination of differential encoding and entropy encoding. Whether the phrase“encoding process” refers specifically to a subset of operations or refers broadly to a more extensive encoding process will be clear based on the context of the specific description, and is believed to be well understood by those skilled in the art.

[0124] It is noted that syntax elements as used herein are descriptive terms. Therefore, they do not exclude other syntax element designations.

[0125] When an accompanying drawing figure is presented as a flow chart, it will also be understood that one or more steps can be embodied in a block diagram in which the corresponding blocks represent one or more steps in the process. Similarly, when an accompanying drawing figure is presented as a block diagram, it will also be understood that one or more blocks can be embodied in a flow chart in which each block of the block diagram represents one or more steps in the process.

[0126] During the encoding process, a balance or trade-off between rate and distortion is typically considered, often taking into account constraints on computational complexity. Rate-distortion optimization is often formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solve the rate-distortion optimization problem. For example, the approaches can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, and full evaluation of their encoding cost as well as the relevant distortion of the reconstructed signal after encoding and decoding. Faster approaches can also be used to reduce the encoding complexity, in particular on the computation of an approximate distortion based on a prediction or predicted residual signal rather than the reconstructed residual signal. A mix of the two approaches can also be used, such as by using an approximate distortion for only some of the possible encoding options, while using a full distortion for other encoding options. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a full evaluation of both the encoding cost and the relevant distortion.

[0127] The implementations and aspects described herein can be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), implementation of the discussed features can also occur in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, phones, portable / personal digital assistants (“PDAs”), and other devices that have processors.

[0128] References to “one implementation,” “an implementation,” “an example,” “one specific implementation,” or “in a specific implementation,” and other variants thereof, mean that a particular feature, structure, characteristic, and so forth being described is included in at least one implementation. Therefore, appearances of the phrases “in one implementation,” “in an implementation,” “in an example,” or “in one specific implementation,” or other variants thereof, in various places in the specification are not necessarily all referring to the same implementation or example.

[0129] Furthermore, the present application can relate to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory. Obtaining can include receiving, retrieving, constructing, generating, and / or determining.

[0130] Also, the application can relate to “accessing” various pieces of information. Accessing information can include one or more of, for example, receiving information, retrieving information (e.g., from storage), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0131] In addition, the application can relate to “receiving” various pieces of information. As with “accessing,” receiving is intended to be a broad term. Receiving information can include one or more of, for example, accessing information or retrieving information (e.g., from storage). Also, “receiving” is typically involved in one way or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0132] It should be understood that any of the use of the following “ / ”, “and / or”, and “at least one of” are intended to encompass only the options selected from a first listed option, or only the options selected from a second listed option, or only the options selected from a third listed option, or only the options selected from a first listed option and a second listed option, or only the options selected from a first listed option and a third listed option, or only the options selected from a second listed option and a third listed option, or only the options selected from a first listed option, a second listed option, and a third listed option, into a larger combination of options. As would be apparent to one of ordinary skill in the art and relevant field(s) this can be expanded to encompass any number of items selected from any number of listed options.

[0133] Also, as used herein, the word "signal" refers to, among other things, indicating something to a corresponding decoder. For example, in some implementations, an encoder signals (e.g., to a decoder) a weight index, etc. In this way, in implementations, the same parameters are used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to a decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter along with other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmitting any actual functionality, bit savings are achieved in various implementations. It will be appreciated that signaling can be implemented in a variety of ways. For example, in various implementations, information is signaled to a corresponding decoder using one or more syntax elements, flags, etc. While the foregoing involves the verb form of the word "signal," the word "signal" can also be used as a noun herein.

[0134] It will be apparent to those skilled in the art that a specific embodiment can generate various signals having information formatted in accordance with a variety of communication protocols or conventions. For example, such information can be formatted into a data frame, packet, or other appropriate information signal. A given protocol or convention may

[0135] Bidirectional motion compensated prediction (MCP) can be performed. MCP can bring high efficiency in removing temporal redundancy, e.g., by exploiting temporal correlation between pictures. A bi-prediction signal can be formed, e.g., by combining two uni-prediction signals (e.g., using a weight value equal to 0.5). Combining uni-prediction signals can be suboptimal, e.g., when illumination changes rapidly from one reference picture to another. Prediction techniques can compensate for changes in illumination over time, e.g., by applying global or local weight and / or offset values to one or more (e.g., each) of the sample values in the reference pictures.

[0136] Encoding modules (e.g., associated with temporal prediction) can be extended and / or enhanced. Affine motion compensation can be used as an inter coding tool.

[0137] This document describes a specific implementation using affine modes. Translational motion models can be applied to motion-compensated prediction. Many types of motion may exist (e.g., zooming in or out, rotation, perspective motion, and / or other irregular motion). Simplified affine transformation motion-compensated prediction can be applied. Flags for inter-frame coding CUs (e.g., per inter-frame coding CU) can be signaled, for example, to indicate whether a translational or affine motion model is applied to inter-frame prediction. Flags (e.g., if affine motion is used) can be signaled to indicate the number of parameters used in the affine motion model (e.g., four or six).

[0138] The affine motion model can be a four-parameter model. Two parameters can be used for translational movement (e.g., one parameter each in the horizontal and vertical directions). One parameter can be used for scaling motion. One parameter can be used for rotational motion. The horizontal scaling parameter can be equal to the vertical scaling parameter. The horizontal rotation parameter can be equal to the vertical rotation parameter. The four-parameter motion model can be encoded using two motion vectors (MVs) as a pair (e.g., one) at two control point positions defined at the top left and top right corners of the current CU. FIG. 5 An exemplary four-parameter affine pattern model of an affine block and the derivation of sub-block-level motion are shown. For example... FIG. 5 As shown, the affine motion field of the block can be described by two control point motion vectors (V0, V1). Based on the control point motion, the motion field (v0, V1) is... x v y This can be described, for example, according to Equation 1:

[0139]

[0140]

[0141] Where (v 0x v 0y (v) can be the motion vector of the top-left control point. 1x v 1y () can be the motion vector of the upper right control point, such as FIG. 5 As shown, w can be the width of CU.

[0142] Affine motion models can be six-parameter models. Two parameters can be used for translational motion (e.g., one parameter each in the horizontal and vertical directions). Two parameters can be used for scaling motion (e.g., one parameter each in the horizontal and vertical directions). Two parameters can be used for rotational motion (e.g., one parameter each in the horizontal and vertical directions). The six-parameter motion model can be encoded using three MVs at three control points. FIG. 6 An exemplary six-parameter affine pattern is shown, where V0, V1, and V2 are control points, and (MV x MV y) is the motion vector of the sub-block centered at position (x, y). For example... FIG. 6 As shown, control points for a six-parameter affine-coded CU can be defined at the top left, top right, and bottom left corners of the CU. Motion at the top left control point can be related to translational motion. Motion at the top right control point can be related to rotational and scaling motion in the horizontal direction. Motion at the bottom left control point can be related to rotational and scaling motion in the vertical direction. The rotational and scaling motion in the horizontal direction can differ from the motion in the vertical direction. For example, the MV(v) of a sub-block (e.g., each sub-block) can be derived using the three MVs at the control points according to Equations 2 and 3. x v y ):

[0143]

[0144]

[0145] Where (v 2x v 2y (x, y) can be the motion vector of the lower left control point, (x, y) can be the center position of the sub-block, and w and h can be the width and height of the CU, respectively.

[0146] The motion field for blocks encoded using an affine motion model can be derived, for example, at the granularity of sub-blocks. This can be achieved, for example, by calculating the MV of the center samples of the sub-blocks (e.g., as shown in the image). FIG. 5 The MV (value) of each sub-block is derived (e.g., according to equation (1)). The calculation can be rounded to, for example, 1 / 16-pel accuracy. The derived MV can be used in the motion compensation stage to generate prediction signals for sub-blocks (e.g., each sub-block) within the current block. The size of the sub-block applied to affine motion compensation can be, for example, 4×4. The four parameters of the 4-parameter affine model can be estimated iteratively, for example. For example, one or more MVs at step k can be represented as The original illuminance signal can be represented as I(i,j). The predicted illuminance signal can be represented as I′. k (i, j). Spatial gradient g x (i, j) and g y (i, j) can be used, for example, in the horizontal and vertical directions respectively, to predict the signal I′. k Derivation of the Sobel filter on (i, j). The derivative of equation (1) can be expressed, for example, according to equation 4:

[0147]

[0148] where at step k, (a, b) can be the delta translation parameters and (c, d) can be the delta scaling and rotation parameters. For example, the delta MV at the control points can be derived from the coordinates according to equations 5 and 6. For example, (0, 0) and (w, 0) can be the coordinates of the top-left control point and the top-right control point, respectively.

[0149]

[0150]

[0151] The relationship between the illumination change and the spatial gradient and the temporal movement can be adapted, for example, according to equation 7:

[0152]

[0153] where and can be replaced by the values in equation (4), for example, to obtain an equation for the parameters (a, b, c, d), for example, as shown in equation 8:

[0154] I′ k (i, j) - I(i, j) = (g x (i, j) * i + g y (i, j) * j) * c + (-g x (i, j) * j + g y (i, j) * i) * d + g x (i, j) * a + g y (i, j) * b (8)

[0155] The parameter set (a, b, c, d) can be derived, for example, using least squares (e.g., because the samples in the CU satisfy equation 8). At step (k+1), the can be solved with equations 5 and 6, and they are rounded to a certain precision (e.g., 1 / 4 pel). The MVs at the two control points can be corrected (e.g., using iterations) until the parameters (a, b, c, d) are (e.g., all) zero or the number of iterations has reached (e.g., a predefined) limit.

[0156] The six parameters of the six-parameter affine model can be estimated. Equation 4 can be changed, for example, according to equation 9:

[0157]

[0158] where at step k, (a, b) can be the delta translation parameters, (c, d) can be the delta scaling and rotation parameters in the horizontal direction, and (e, f) can be the delta scaling and rotation parameters in the vertical direction. For example, equation 8 can be changed according to equation 10:

[0159] I' = I + (a * i + b * j) * c + (a * i + b * j) * d + (a * i + b * j) * e + (a * i + b * j) * f k (i, j) = (g x (i, j) * i) * c + (g x (i, j) * j) * d + (g y (i, j) * i) * e + (g y (i, j) * j) * f + g x (i, j) * a + g y (i, j) * b (10)

[0160] The parameter set (a, b, c, d, e, f) can be derived, for example, by considering samples (e.g., multiple samples) within the CU, for example, using least squares. The MV of the top-right control point can be calculated, for example, according to equations 11 and 12 and the

[0161]

[0162]

[0163] Decoding-side motion vector refinement (DMVR) can be provided. DMVR based on bilateral matching (BM) can be applied, for example, to improve the accuracy of the MV of the merge mode. In bi-prediction operation, a refined MV can be searched around the initial MV in reference picture list L0 and / or reference picture list L1. BM based DMVR can calculate the distortion between two candidate blocks in reference picture list L0 and list L1. FIG. 7 An example decoding-side motion vector (MV) refinement is shown. As FIG. 7 shown, an absolute difference sum (SAD) between collocated blocks can be calculated, for example, based on one or more (e.g., each) MV candidate around the initial MV. The MV candidate with the lowest SAD can become the refined MV and can be used to generate the bi-predicted signal.

[0164] The refined MV derived by DMVR can be used, for example, to generate inter-predicted samples. The refined MV derived by DMVR can be used for temporal motion vector prediction, for example, for future picture encoding. The initial MV can be used for de-blocking and / or spatial motion vector prediction for future CU encoding, for example, to avoid any MV dependency between the current CU and neighboring CUs.

[0165] like FIG. 7 As shown, the search points around the initial MV and MV offset can follow the MV difference mirroring (e.g., symmetric) rule. The points checked by DMVR, represented by the candidate MV pair (MV0, MV1), can be determined according to Equations 13 and / or 14:

[0166]

[0167] MV offset This can represent the corrected offset between the initial MV and the corrected MV in one of the reference images. The corrected search range can be, for example, two integer illuminance samples from the initial MV. Fast search methods with early termination mechanisms can be applied, for example, to reduce search complexity.

[0168] Sub-block-based temporal motion vector prediction (SbTMVP) is available. SbTMVP can use motion fields in juxtaposed images to improve motion vector prediction and merging patterns of CUs in the current image. The same juxtaposed image used by temporal motion vector prediction (TMVP) can be used for SbTMVP. SbTMVP may differ from TMVP in one or more of the following ways: TMVP predicts motion at the CU level. SbTMVP predicts motion at the sub-CU level. TMVP obtains temporal motion vectors from juxtaposed blocks in the juxtaposed image. The juxtaposed block can be the lower right or center block relative to the current CU. SbTMVP may apply a motion offset, for example, before obtaining temporal motion information from the juxtaposed image. The motion offset may be obtained from, for example, the motion vector of a spatially neighboring block from the current CU's spatially neighboring blocks.

[0169] FIG. 8A and FIG. 8B An exemplary SbTMVP process is shown. FIG. 8A An example spatial neighbor block that can be used in SbTMVP is shown. FIG. 8B An exemplary derivation of the motion field of a sub-coding unit (CU) is shown. For example... FIG. 8B As shown, the motion field of a sub-CU can be derived by applying motion offsets from spatial neighbors and scaling motion information from the corresponding juxtaposed sub-CU. SbTMVP can predict the motion vectors of sub-CUs within the current CU. Spatial neighbors can be checked (e.g., FIG. 8AA1). For example, if A1 has a motion vector that uses a collocated picture as its reference picture, the motion vector of A1 can be selected (e.g., for the motion offset to be applied). For example, if no motion is identified, the motion offset can be selected to be (0, 0). The selected motion offset can be applied, e.g., to obtain sub-CU level motion information (e.g., such as motion vectors and / or reference indices) from the collocated picture. For example, the selected motion offset can be added to the coordinates of the current block. The motion offset can be set to the motion of block A1 (e.g., after being identified) to the coordinates of the center sample of the sub-CU in the collocated picture (e.g., in the depicted example). The motion information of the corresponding block (e.g., covering the smallest motion grid of the center sample) of (e.g., each) sub-CU in the collocated picture can be used to derive the motion information of the sub-CU. The motion information of the collocated sub-CU (e.g., after being identified) can be converted to a motion vector and reference indices of the current sub-CU. For example, temporal motion scaling can be applied to align the reference picture of the temporal motion vector to the reference picture of the current CU. FIG. 8B The motion information of the corresponding block (e.g., covering the smallest motion grid of the center sample) of (e.g., each) sub-CU in the collocated picture can be used to derive the motion information of the sub-CU. The motion information of the collocated sub-CU (e.g., after being identified) can be converted to a motion vector and reference indices of the current sub-CU. For example, temporal motion scaling can be applied to align the reference picture of the temporal motion vector to the reference picture of the current CU.

[0170] The combined sub-block based merge list including the SbTMVP candidate and the affine merge candidate can be used for the signaling of the sub-block based merge mode. The SbTMVP mode can be enabled and / or disabled by a sequence parameter set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor can be added as (e.g., the first) entry of the list of sub-block based merge candidates and can be followed by the affine merge candidate. The size of the sub-block based merge list can be signaled in the SPS. For example, the maximum allowed size of the sub-block based merge list can be 5.

[0171] The sub-CU size used in SbTMVP can be fixed, e.g., can be 8x8. The SbTMVP mode can be (e.g., only) applicable to CUs with a width and height greater than or equal to 8.

[0172] The coding logic of the additional SbTMVP merge candidate can be the same as the coding logic for other merge candidates. For example, for each CU in a P or B slice, an additional RD check can be performed. The additional RD check can be used to decide whether to use the SbTMVP candidate.

[0173] Overlapped block motion compensation (OBMC) can be provided. Syntax can be used, e.g., at the CU level, to turn on and off OBMC. OBMC can be performed for motion compensation (MC) block boundaries (e.g., except for the right and bottom boundaries of a CU). OBMC can be applied to luma and chroma components. An MC block can correspond to a coding block. A CU can be coded with sub-CU modes (e.g., including sub-CU merge, affine, and FRUC modes). One or more sub-blocks (e.g., each sub-block) of a CU coded with a sub-CU mode can be an MC block. OBMC can be performed at the sub-block level of (e.g., all) MC block boundaries, e.g., to (e.g., uniformly) handle CU boundaries. FIG. 9 An example of sub-blocks to which OBMC is applied is shown. The sub-block size can be set to equal 4x4, e.g., as shown in FIG. 9 .

[0174] OBMC can be applied to a current sub-block. Four motion vectors (e.g., in addition to the current motion vector) of neighboring sub-blocks (e.g., if available and different from the current motion vector) can be used to derive a prediction block for the current sub-block. In one or more examples, "neighboring" can be used interchangeably with "adjacent." Multiple prediction blocks based on the multiple motion vectors can be combined, e.g., to generate a final prediction signal for the current sub-block.

[0175] A prediction block based on a motion vector of a neighboring sub-block can be denoted as P N , where N indicates an index of a neighboring sub-block above, below, left, and right of the current sub-block. A prediction block based on a motion vector of the current sub-block can be denoted as P C . For example, if P N is based on motion information of a neighboring sub-block that includes the same motion information as the current sub-block, OBMC can be skipped. Otherwise, one or more (e.g., each) sample of P N can be added to a same sample in P C , e.g., four rows / four columns of P N can be added to P C . In an example, weighting factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} can be used for P N , and weighting factors {3 / 4, 7 / 8, 15 / 16, 3l / 32} can be used for P C . For small MC blocks, two rows and / or two columns of P N can be added to P C , e.g., if a height or width of a coding block is equal to four (4) or a CU is coded with a sub-CU mode. For small MC blocks, weighting factors {1 / 4, 1 / 8} can be used for P N , and weighting factors {3 / 4, 7 / 8} can be used for P CFor P N Samples in the same row (e.g., and / or column) P N may be added to P C For example, the overlap region pixels can use a weighting factor different from the weighting factor used for the non-overlap region pixels.

[0176] A CU-level flag can be signaled to indicate whether OBMC is applied to the current CU, e.g., a CU with a size smaller than or equal to 256 luma samples. For example, for a CU with a size larger than 256 luma samples or not coded with AMVP mode, OBMC can be applied by default. For example, during the motion estimation stage, the impact of OMBC can be taken into account at the encoder. The prediction signal formed by the motion information of the top and left neighboring blocks used by OBMC can be used to compensate the top and left boundaries of the original signal of the current CU. The (e.g., normal) motion estimation process can be applied (e.g., thereafter) (e.g., otherwise).

[0177] Optical flow (PROF) prediction refinement can be applied to affine mode. PROF can refine the subblock-based affine motion compensation prediction with optical flow, e.g., to achieve finer-granular motion compensation. For example, the luma prediction samples can be refined (e.g., after the subblock-based affine motion compensation) by adding a difference value derived through an optical flow equation. PROF can include one or more of the following. Subblock-based affine motion compensation can be performed to generate a subblock prediction I(i,j). A spatial gradient g x (i,j) and g y (i,j) of the subblock prediction at one or more sample locations (e.g., each sample location) can be calculated. For example, the spatial gradient can be calculated using one or more pixels that can or can not be partially or completely contiguous. In an example, a gradient of a first pixel can be based on a sample value of a second pixel and a sample value of a third pixel, where the second pixel and the third pixel are adjacent to the first pixel. In some examples, the first pixel calculating the gradient can abut either or both of the second pixel or the third pixel adjacent to the first pixel. In other examples, the first pixel calculating the gradient can be proximate to but not abutting either or either of the second pixel or the third pixel adjacent to the first pixel. The calculation can be performed using a 3-tap filter such as [-1, 0, 1], e.g., as shown in Equations 15 and 16:

[0178] g x (i,j) = I(i+1,j) - I(i-1,j) (15)

[0179] g y (i,j) = I(i,j+1) - I(i,j-1) (16)

[0180] For gradient computation, the sub-block prediction can be extended (e.g., by one pixel on each side). For example, the pixels on the extended boundaries can be copied from the nearest integer pixel positions in the reference picture. For example, if the pixels on the extended boundaries are copied from the nearest integer pixel positions in the reference picture, additional interpolation of the padded area can be avoided. FIG. 10 An exemplary sub-block MV V SB and pixel Δv(i,j). The illumination prediction correction can be computed by an optical flow equation, e.g., as shown in Equation 17:

[0181] Δl(i,j) = g x (i,j)*Δv x (i,j) + g y (i,j)*Δv y (i,j) (17)

[0182] where Δv(i,j) is the difference between the pixel MV (denoted by v(i,j)) and the sub-block MV of the sub-block to which the pixel (i,j) belongs, computed for sample position (i,j), as shown in Equation 18. FIG. 10

[0183] The affine model parameters and the pixel position relative to the sub-block center can not change across sub-blocks. Δv(i,j) can be computed for (e.g., the first) sub-block, and can be reused for other sub-blocks (e.g., in the same CU). Let x and y be the horizontal and vertical offsets from the pixel position to the sub-block center, Δv(x,y) can be derived, e.g., according to Equation 18:

[0184]

[0185] where, for the 4-parameter affine model, c and e can be determined according to Equation 19:

[0186]

[0187] where, for the 6-parameter affine model, c, d, e, and f can be determined according to Equation 20:

[0188]

[0189] and where (v 0x , v 0y ), (v 1x , v 1y ), (v 2x , v 2y ​) are the left-top control point motion vector, right-top control point motion vector, and left-bottom control point motion vector, respectively, and w and h are the width and height of the CU. The illumination prediction refinement is added to the subblock prediction I(i, j). The final prediction I' can be generated, for example, according to Equation 21:

[0190] I'(i, j) = I(i, j) + AI(i, j) (21)

[0191] DMVR and SbTMVP can be used in different prediction modes to improve the accuracy of the prediction MVs. The refined MVs after DMVR or SbTMVP can be used, for example, only, to perform subblock-based motion compensation thereafter. OBMC can include pixel-level refinement. OBMC can be used to reduce the boundary discontinuity at the subblocks of a CU or subCU. OBMC can include multiple motion compensation operations for one or more (e.g., each) subblock. For example, if the MVs of four connected neighboring subblocks are available and different from the MV of the current subblock, they can be used to derive the prediction block for the current subblock.

[0192] Methods for subblock / block refinement (e.g., pixel-level refinement) can be provided. For example, the methods can be used to reduce boundary discontinuity. FIG. 11 Examples of methods are provided. As FIG. 11 The methods described can be applied in decoders and / or encoders.

[0193] FIG. 11 Examples of methods for subblock / block refinement according to one or more of Equations (1) to (25) are shown. The examples disclosed herein, as well as other examples, can operate according to the example method 1100 shown. FIG. 11 The example method 1100 operates. The method 1100 includes 1102 and 1104. In 1102, a sample value of a first pixel can be obtained based on, for example, (1) a motion vector (MV) of a current subblock, (2) MVs of subblocks neighboring the current subblock, and (3) sample values of second pixels neighboring the first pixel. In 1104, a block including the current subblock can be encoded or decoded based on the obtained sample value of the first pixel. When the method as FIG. 11 described is applied in a decoder, FIG. 11 1104 in the method as FIG. 11 described is applied in an encoder, FIG. 11 1104 in the method as

[0194] Examples of methods for sub-block / block refinement are provided, e.g., for encoding and decoding. Examples can refer to "boundaries" that can include different types of boundaries, such as boundaries of blocks, sub-blocks, CUs, and / or PUs. Examples can refer to "neighbors" that can include different types of neighbors, such as spatial neighbors and temporal neighbors of blocks, sub-blocks, CUs, and / or PUs. Examples can refer to "adjacents" that can include different types of adjacents, such as adjacent blocks, adjacent sub-blocks, adjacent pixels, and / or pixels adjacent to a boundary. Spatial neighbors can be adjacent in the same frame, while temporal neighbors can be located in the same position in adjacent frames. For example, an adjacent sub-block is a sub-block that can be a spatial or temporal neighbor. A boundary pixel is a pixel that is adjacent to a boundary, where the boundary can be any type of boundary. For example, a boundary pixel can be adjacent to a boundary of a block, sub-block, CU, and / or PU.

[0195] Sub-block / block refinement can include sub-block / block boundary refinement. For example, a difference in MVs between a current block and / or sub-block and a neighboring block and / or sub-block can be calculated and converted to a difference in sample values derived by an optical flow equation. A pixel intensity (e.g., luma and / or chroma) of a boundary pixel of the current block and / or sub-block can be refined, e.g., by adding the derived difference. The derived difference can be referred to as a block boundary optical flow prediction refinement (BBPROF). A sample value offset of the boundary pixel can indicate the derived difference. A sample value of the boundary pixel can indicate the pixel intensity of the boundary pixel. BBPROF (e.g., as described herein) can provide pixel-level granularity for sub-block and block boundary refinement. BBPROF (e.g., as described herein) can be applied to any sub-block-based inter prediction mode and / or CU-based inter prediction mode.

[0196] A boundary pixel can include a pixel at a boundary of a block and / or sub-block. For example, a square sub-block can have four boundaries, including a left boundary, a right boundary, a top boundary, and a bottom boundary. A boundary can include a common boundary shared between two sub-blocks. The two sub-blocks can abut each other at the boundary. A pixel can be at a boundary when the pixel is located near the boundary. For example, a pixel can be at a boundary when the pixel is in a certain (e.g., 4, 3, 2, or 1) row of pixels from a top boundary of a sub-block, in a certain row of pixels from a bottom boundary of the sub-block, in a certain (e.g., 4, 3, 2, or 1) column of pixels from a left boundary of the sub-block, or in a certain column of pixels from a right boundary of the sub-block. In certain examples, when a first pixel is inside a first sub-block, the first pixel can be in the first sub-block or at a common boundary of the first sub-block, and abut a pixel inside a second sub-block that shares the common boundary with the first sub-block. In certain examples, a pixel can be at a boundary of a sub-block but outside the sub-block.

[0197] BBPROF can be applied, for example, in a DMVR mode. BBPROF can reduce block boundary discontinuity for DMVR-based subblock-level motion compensation prediction. A change in pixel intensity can be applied by BBPROF. The change in pixel intensity can be derived, for example, from an optical flow equation. A sample value offset of a pixel can indicate the change in pixel intensity. BBPROF can be used to perform one (e.g., only one) motion compensation operation per subblock. Motion compensation in a DMVR mode can perform one motion compensation operation per subblock.

[0198] A refined motion vector for a subblock in a CU can be derived, for example, by performing DMVR (e.g., as described herein). Subblock-based motion compensation (e.g., as described herein) can be performed to generate a subblock-based prediction.

[0199] A spatial gradient g x (i,j) of a subblock prediction at one or more (e.g., each) pixel / sample location (e.g., as described herein) can be calculated. y (i,j) can be calculated.

[0200] A motion vector difference MV diff between a current subblock and one or more neighboring subblocks (considered as neighboring subblocks) can be calculated. The MV difference can be at a subblock level. Each candidate neighboring subblock that is not far away (e.g., close or near) from the current subblock can be considered. Various quantities and / or locations of subblocks can be selected as neighboring subblocks to calculate MV diff . In an example, BBPROF in a DMVR mode can use four neighboring subblocks (e.g., left, top, right, and bottom neighboring subblocks), two neighboring subblocks (e.g., top and left neighboring subblocks), corner neighboring subblocks (e.g., top-left, bottom-right), or other quantities and locations of neighboring subblocks to calculate MV diff .

[0201] In an example, four neighboring subblocks (e.g., at the top, bottom, left, and right subblocks adjacent to the current subblock) can be considered, where MV diff can be a set of MV differences including four different MV differences. For example, MV diff can be calculated as MV diff = {MV diff (A), MV diff (B), MV diff (L), MV diff (R)}, where A, B, L, R can represent MV differences between the current subblock and the top, left, and right subblocks, respectively.

[0202] FIG. 12 Example MV difference calculations from selected neighboring subblocks in a DMVR mode are depicted, for example. As FIG. 12As shown, the sub-blocks can have their own MV difference values after DMVR. A current sub-block within a CU (e.g., FIG. 12 The current DMVR sub-block in) can have four connected neighboring sub-blocks, e.g., except for the current sub-block located at the boundaries.

[0203] The calculated sub-block level MV difference MV diff may be used to calculate a motion vector offset Av(i,j) at one or more (e.g., each) pixel / sample location within the current sub-block, e.g., as shown in Equation 21.

[0204]

[0205] where n can be an index of a particular neighboring sub-block, N can be a total number of the considered neighboring sub-blocks, and w(i,j,n) can be a weighting factor when the neighboring MV diff (n) is applied to a particular pixel at location (i,j). For example, if left, top, right, and bottom neighboring sub-blocks are considered, N can be equal to 4.

[0206] The set of weighting factors can be, e.g., {1 / 4, 1 / 8, 1 / 16, 1 / 32}. The weighting factors can be used by, e.g., four rows / columns of pixels at one or more (e.g., each) side of the current sub-block, respectively. The MV difference MV diff may be calculated, e.g., based on the motion vectors of the vertical and / or horizontal neighboring sub-blocks. Pixels in the same row and / or column of the current sub-block can use the same weighting factor. For example, pixels in a first column of the left side of the current sub-block can use the same weighting factor (e.g., 1 / 4), and pixels in a second column can use the same weighting factor (e.g., 1 / 8), etc. The weighting factors can be determined, e.g., based on a distance from the current location to a block boundary between the current block and its neighboring blocks. For example, the weighting factors can be smaller when the column and / or row is further away from the block boundary.

[0207] The weighting factors can be adjusted (e.g., dynamically) based on, e.g., the pixel location. In an example, the pixel can be located at the top-left of the current sub-block. The MV differences from the left and top neighboring sub-blocks can be weighted / combined together to generate the final MV offset at the pixel, e.g., if there are left and top neighboring sub-blocks and if neither of the MV differences from the left and top neighbors is zero. For example, if the current sub-block is on the left boundary or the top boundary of the current CU, the left or top neighboring sub-block can not be available.

[0208] The intensity change of each pixel within the current sub-block can be calculated, e.g., according to optical flow Equation 22:

[0209] AI(i,j) = g x (i,j) * Av x (i,j) + g y(i,j) * Δv y (i,j) (22)

[0210] where Δv(i,j) and g(i,j) can be MV offsets and spatial gradients at one or more (e.g., each) sample locations (i,j), which can be computed, e.g., in a previous step.

[0211] The prediction of a pixel or sample location can be corrected, e.g., by adding the computed intensity change (e.g., luminance or chrominance) to the subblock prediction. The corrected prediction of a pixel or sample location can be associated with a certain reference picture list (e.g., list L0 or list L1). The final prediction I’ can be generated, e.g., according to equation 23:

[0212] I'(i,j) = I(i,j) + ΔI(i,j) (23)

[0213] When applying BBPROF in DMVR mode, four neighboring subblocks (e.g., at most) can be considered. The inner subblock can wait for the DMVR process of the neighboring subblocks to complete. In an example, when applying BBPROF in DMVR mode, two neighboring subblocks can be considered. In an example, e.g., two neighbors (e.g., only top and left neighbors) can be considered, such that the BBPROF of the current subblock can depend on the DMVR process of the two neighboring subblocks.

[0214] BBPROF can be applied in SbTMVP mode. For example, BBPROF can reduce discontinuities at subblock boundaries of SbTMVP-based subblock-level motion compensation prediction. One or more examples of applying BBPROF in DMVR mode herein can be applicable to performing BBPROF in SbTMVP mode. Applying BBPROF in SbTMVP mode can include performing one (e.g., only one) motion compensation operation per subblock. SbTMVP motion compensation can perform one motion compensation operation per subblock. Applying BBPROF in SbTMVP mode can include one or more of the following.

[0215] A corrected motion vector (e.g., as described herein) of one or more (e.g., each) subblock in a CU can be derived, e.g., by performing SbTMVP. Motion information can be obtained from a collocated sub-CU. Appropriate temporal scaling can be applied to the motion information. Subblock-based motion compensation can be performed, e.g., to generate a subblock-based prediction.

[0216] A spatial gradient g of the subblock prediction at one or more (e.g., all) pixel / sample locations (e.g., as described herein) can be computed x (i,j) and g y (i,j).

[0217] A motion vector difference MV between the current sub-block and, for example, one or more considered neighboring sub-blocks can be computed diff .

[0218] The prediction refinement as described herein can be applied in ATMVP. In an example, the prediction refinement can use the motion vectors of the four neighboring sub-blocks of the current sub-block as illustrated in FIG. 22. The motion vectors can be used to derive a sub-block level MV difference MV FIG. 9 between the current sub-block and the spatial neighboring sub-blocks. diff .

[0219] The computed sub-block level MV difference MV diff may be used to compute a motion vector offset Av(i,j) at one or more (e.g., each) of the pixel / sample locations within the current sub-block.

[0220] The intensity change of each pixel within the current sub-block can be computed by the optical flow equation (e.g., based on equation 22).

[0221] The prediction of one or more (e.g., each) of the reference picture lists can be refined, for example, by adding the intensity change (e.g., luminance or chrominance). The final prediction I' can be generated, for example, according to equation 24:

[0222] I'(i,j) = I(i,j) + AI(i,j) (24)

[0223] BBPROF can be applied in affine mode. BBPROF can be applied to an affine coded CU (e.g., similar to SbTMVP). The affine coded CU can include multiple sub-blocks. A block level MV of one or more (e.g., each) of the sub-blocks can be derived by an affine motion model (e.g., as described herein). The four parameters of a 4-parameter affine model and / or the six parameters of a 6-parameter affine model can be estimated, for example, with two or three control point motion vectors. The block level motion vectors of the sub-blocks within the affine coded CU can be derived, for example, using the four or six estimated affine model parameters. A sub-block level MV difference MV diff and / or a motion vector offset Av(i,j) at one or more (e.g., each) of the pixel / sample locations within the sub-block can be computed, for example, with different sub-block motion vectors. The sub-block level MV difference MV diff and / or the motion vector offset Av(i,j) can be computed, for example, where BBPROF can be performed as described herein.

[0224] BBPROF can be applied to pixels adjacent to a CU boundary, for example, as described herein for pixels adjacent to a boundary of a sub-block. BBPROF can be applied, for example, at a CU level. The reference pictures can be the same or different, for example, for adjacent CUs.

[0225] For example, if the selected neighboring CUs (e.g., the above, below, left, and right CUs) have the same reference picture as the current CU, a CU-level MV difference MVCU can be computed (e.g., directly computed) for the particular CU diff The prediction of the boundary pixels of the particular CU can be corrected, e.g., by applying BBPROF (e.g., directly).

[0226] For example, if (i) one or more (e.g., all) of the selected neighboring CUs (e.g., the above, below, left, and right CUs) and (ii) the current CU have different reference pictures in the reference picture list, temporal motion scaling can be applied for the particular CU. Appropriate temporal motion scaling can align the reference pictures of the temporal motion vectors of the selected neighboring CUs with the reference picture of the particular CU. A CU-level MV difference MVCUmay be computed (e.g., based on the scaled MVs of the selected neighboring CUs), and the prediction correction at the CU boundary can be implemented, e.g., by applying BBPROF. diff

[0227] A number of implementation variations of BBPROF are provided as additional examples. The number and location of neighboring (e.g., adjacent) subblocks that can be selected for subblock / block correction such as subblock / block boundary correction (e.g., BBPROF) can not be limited to the examples described herein, such as the examples described with reference to FIG. 12 Other numbers and / or locations of subblocks can be selected. Every candidate neighboring subblock that is not far (e.g., close or near) from the current subblock can be considered. For example, BBPROF can use corner neighboring subblocks (e.g., top-left, bottom-right), the four subblocks shown, FIG. 12 or other numbers and locations of neighboring subblocks to compute the MV diff .

[0228] The location of the considered neighboring subblocks can not be limited to being within the same CU as the current subblock. The aspect ratio of the considered neighboring subblocks from neighboring CUs can or can not be limited to being the same as the current subblock. Subblock / block correction such as subblock / block boundary correction (e.g., BBPROF) can allow for different aspect ratios.

[0229] A MV offset at a pixel / sample location can be derived, e.g., based on a subblock MV difference. The number of rows and / or columns of pixels at one or more (e.g., each) side of the current subblock can be configurable and / or dynamically changed, e.g., based on one or more (e.g., predefined) criteria. For example, two or more columns of pixels at the left side of the current subblock can include the MV difference of the left neighboring subblock, e.g., instead of a default number of columns of pixels, such as, e.g., four columns of pixels.

[0230] ​The MV difference can be based on vertically and / or horizontally neighboring subblocks. In an example, pixels in the same row and / or column of the current subblock can use the same weighting factor. In an example, pixels in the same row and / or column of the current subblock can use different weighting factors.

[0231] The weighting factor can vary. For example, the weighting factor can increase with a decrease in the spatial distance between the pixel and the vertical or horizontal boundary.

[0232] The intensity difference (e.g., derived from equation 22) can be multiplied by a weight factor w, e.g., before adding the intensity difference to the prediction, as shown, e.g., in equation 25:

[0233] I'(i, j) = I(i, j) + w • AI(i, j) (25)

[0234] where w can be set to a value between 0 and 1, inclusive. w can be signaled, e.g., at the CU level or picture level. For example, w can be signaled by a weight index. Equation 25 can be a variant of equation 23 and / or equation 24.

[0235] BBPROF can be used, e.g., after the L0 and L1 predictions based on DMVR are combined with the weight. BBPROF can be applied to, e.g., one prediction, e.g., L0 or L1, to reduce complexity. In an example, BBPROF can be applied to one prediction where, e.g., the reference picture is closer in time to the current picture. In an example, BBPROF can be applied to one prediction where, e.g., the reference picture is further in time from the current picture.

[0236] Many implementations are described herein. Features of the implementations can be provided alone or in any combination, across various claim classes and types. Further, implementations can include one or more of the features, devices, or aspects described herein across various claim classes and types, such as, for example, any of the following.

[0237] As described in FIG. 11 The methods as described in FIG. 11 When the methods as described in FIG. 11 1104 in can be performed by an encoder and 1104 can be necessary to encode a block including the current subblock based on the obtained sample values of the first pixels. As FIG. 11The method can be based on one or more of equations (1)-(25). For example, a decoder can decode a current sub-block based on a sample value of a pixel. The pixel can be at one of the boundaries of the current sub-block. The sample value can be a revised sample value obtained based on one or more of equations (1)-(25). As indicated in one or more of equations (1)-(25), the decoder can obtain the sample value of the pixel based on, for example, an MV of the current sub-block, MVs of sub-blocks neighboring the current sub-block, and sample values of pixels neighboring the pixel for which the sample value is obtained. The decoder can obtain, for example, a prediction of the sample value of the pixel before the decoder revises the prediction of the sample value. The prediction of the sample value of the pixel can be referred to as I(i,j), for example, as indicated in equation (23). As indicated in equation (23), the decoder can obtain the sample value of the pixel based on a sample value offset and the prediction of the sample value (e.g., a sum of the sample value offset and the prediction of the sample value). The sample value offset of the pixel can be referred to as ΔΙ(ί, j), for example, as indicated in equation (23). As indicated in one or more of equations (1)-(25), the sample value offset can be obtained based on, for example, the MV of the current sub-block, the MVs of the sub-blocks neighboring the current sub-block, and the sample values of the pixels neighboring the pixel for which the sample value is obtained. The decoder can obtain an MV difference (e.g., the MV difference in equation (21)) using the MV of the current sub-block and the MVs of the sub-blocks neighboring the current sub-block, as described herein. The MV difference can be referred to as MV diff (n), for example, as indicated in equation 21. The decoder can obtain the MV difference using one or more MVs associated with one or more respective sub-blocks neighboring the current sub-block. The decoder can obtain a gradient using the sample values of the pixels neighboring the pixel for which the sample value is obtained, for example, as indicated in equations (15) and (16). As in the example shown in equations (15) and (16), the decoder can obtain the gradient using one or more sample values of one or more respective pixels neighboring the pixel for which the sample value is obtained. The decoder can obtain the sample value of the pixel based on the gradient and the MV difference. In an example, the decoder can obtain an MV offset based on the MV difference, as indicated in equation (21). The decoder can obtain the sample value of the pixel using the MV offset and the gradient, as indicated in equations (22) and (23). The decoder can obtain the sample value offset based on the gradient and the MV difference, and obtain the sample value of the pixel using the sample value offset. The decoder can determine a weighting factor and obtain the sample value of the pixel using the weighting factor, for example, as indicated in equation (21). The decoder can decode a block including the current sub-block based on the obtained sample value of the pixel.

[0238] The method as described can be implemented in a decoder using decoding tools and techniques including one or more of entropy decoding, inverse quantization, inverse transform, and differential decoding. These decoding tools and techniques can be used to decode a block according to, for example, the following equation: FIG. 11 The method as described can be implemented in a decoder using decoding tools and techniques including one or more of entropy decoding, inverse quantization, inverse transform, and differential decoding. These decoding tools and techniques can be used to decode a block according to, for example, the following equation:FIG. 11 the method to implement one or more of the subblock / block modifications; the subblock / block boundary modification of the method as in FIG. 11 the method to implement one or more of the subblock / block modifications; the subblock / block boundary modification of the method as in FIG. 11 the method to implement one or more of the subblock / block modifications; the subblock / block boundary modification of the method as in FIG. 11 the method to implement one or more of the subblock / block modifications; the subblock / block boundary modification of the method as in FIG. 11 the method to implement one or more of the subblock / block modifications; the subblock / block boundary modification of the method as in

[0239] An encoder can encode a current subblock based on a sample value of a pixel. The pixel can be located at one of the boundaries of the current subblock. The sample value can be a modified sample value obtained based on one or more of equations (1)-(25). As indicated by one or more of equations (1)-(25), the encoder can obtain the sample value of the pixel based on, for example, an MV of the current subblock, MVs of subblocks neighboring the current subblock, and sample values of pixels neighboring the pixel for which the sample value is obtained. The encoder can obtain, for example, a prediction of the sample value of the pixel before the encoder modifies the prediction of the sample value. The prediction of the sample value of the pixel can be referred to as I(i,j), for example, as indicated by equation (23). As indicated by equation (23), the encoder can obtain the sample value of the pixel based on a sample value offset and the prediction of the sample value (e.g., a sum of the sample value offset and the prediction of the sample value). The sample value offset of the pixel can be referred to as ΔΙ(ί, j), for example, as indicated in equation (23). As indicated in one or more of equations (1)-(25), the sample value offset can be obtained based on, for example, an MV of the current subblock, MVs of subblocks neighboring the current subblock, and sample values of pixels neighboring the pixel for which the sample value is obtained. The encoder can obtain an MV difference (e.g., the MV difference in equation (21)) using the MV of the current subblock and the MVs of the subblocks neighboring the current subblock, as described herein. The MV difference can be referred to as MV diff(n), e.g., as shown in equation (21). The encoder can obtain the MV difference using one or more MVs associated with one or more respective sub-blocks neighboring the current sub-block. The encoder can obtain the gradient using sample values of pixels neighboring the pixel for which the sample value is obtained, as shown in equations (15) and (16). The encoder can obtain the gradient using one or more sample values of one or more respective pixels neighboring the pixel for which the sample value is obtained, as in the example shown in equations (15) and (16). The encoder can obtain the sample value of the pixel based on the gradient and the MV difference. In an example, the encoder can obtain the MV offset based on the MV difference, as shown in equation (21). The encoder can obtain the sample value of the pixel using the MV offset and the gradient, as shown in equations (22) and (23). The encoder can obtain the sample value offset based on the gradient and the MV difference, and obtain the sample value of the pixel using the sample value offset. The encoder can determine the weighting factor and obtain the sample value of the pixel using the weighting factor, e.g., as shown in equation (21). The encoder can encode the block including the current sub-block based on the obtained sample value of the pixel.

[0240] The methods described above can be implemented in an encoder using encoding tools and techniques including one or more of quantization, entropy coding, inverse quantization, inverse transform, and differential coding. These encoding tools and techniques can be used to implement one or more of the sub-block / block modifications according to the methods described above; the sub-block / block boundary modifications according to the methods described above; the BBPROF according to the methods described above; the sub-block / block modifications in the DMVR mode; the sub-block / block modifications in the SbTMVP mode; the sub-block / block modifications in the affine mode; the obtaining of the sample value according to the methods described above; the obtaining of the sample value offset according to the methods described above; the obtaining of the gradient as described herein; the obtaining of the MV difference as described herein; the obtaining of the prediction of the sample value; and other encoder behaviors related to any of the above. FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11

[0241] Syntax elements can be inserted into the signaling, e.g., to enable the decoder to identify indications associated with performing or using the methods described above. For example, the syntax elements can include indications of one or more of the BBPROF, DMVR, SbTMVP mode, affine mode, e.g., to indicate to the decoder whether one or more of them is enabled or disabled. As an example, the syntax elements can include indications of one or more weighting factors as described herein, and / or parameters used by the decoder to perform one or more of the examples herein. FIG. 11

[0242] ​​​​​​​For example, the method as described can be selected and / or applied based on a syntax element applied at a decoder. For example, the decoder can receive an indication indicating that BBPROF is enabled. Based on the indication, the decoder can perform the method as described on pixels located at or near a boundary of a sub-block. FIG. 11 For example, the method as described can be selected and / or applied based on a syntax element applied at a decoder. For example, the decoder can receive an indication indicating that BBPROF is enabled. Based on the indication, the decoder can perform the method as described on pixels located at or near a boundary of a sub-block. FIG. 11 For example, the method as described can be selected and / or applied based on a syntax element applied at a decoder. For example, the decoder can receive an indication indicating that BBPROF is enabled. Based on the indication, the decoder can perform the method as described on pixels located at or near a boundary of a sub-block.

[0243] The encoder can adjust the prediction residual based on one or more examples herein. For example, a residual can be obtained by subtracting a predicted video block from an original image block. For example, the encoder can predict the video block based on sample values of pixels obtained as described herein. The encoder can obtain the original image block and subtract the predicted video block from the original image block to generate the prediction residual.

[0244] The bitstream or signal can include one or more syntax elements or variations thereof. For example, the bitstream or signal can include a syntax element indicating that any of BBPROF, DMVR, SbTMVP mode, affine mode is enabled or disabled.

[0245] The bitstream or signal can include syntax conveying information generated according to one or more examples herein. For example, the information or data can be generated in performing the example as described. FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG The generated information or data can be conveyed in syntax included in the bitstream or signal.

[0246] A syntax element enabling a decoder to adapt a residual in a manner corresponding to that used by an encoder can be inserted in a signal. For example, the residual can be generated using one or more examples herein.

[0247] A method, process, apparatus, medium storing instructions, medium storing data, or signal for creating and / or transmitting and / or receiving and / or decoding a bitstream or signal including one or more of the syntax elements as described or variations thereof.

[0248] A method, process, apparatus, medium storing instructions, medium storing data, or signal for creating and / or transmitting and / or receiving and / or decoding according to any of the examples as described.

[0249] A method, process, apparatus, medium storing instructions, medium storing data, or signal according to but not limited to one or more of the following: determining a spatial gradient of a sub-block based prediction at one or more pixel / sample locations; using a MV difference to calculate a motion vector offset at one or more pixel / sample locations; determining an intensity change per pixel in a current sub-block, e.g., based on optical flow; modifying a prediction of a reference picture list, e.g., by adding the calculated intensity change to a sub-block prediction; determining that a first pixel is adjacent to a boundary of a current sub-block; determining a difference between a MV of the current sub-block and a MV of a sub-block adjacent to the current sub-block; determining a gradient for the first pixel based on a sample value of a second pixel adjacent to the first pixel and a sample value of a third pixel adjacent to the first pixel; determining a sample value offset based on the determined gradient and the difference between the MV of the current sub-block and the MV of the sub-block adjacent to the current sub-block; obtaining a sample value for the first pixel based on the determined sample value offset; determining a gradient, e.g., based on at least a sample value of a second pixel; using the gradient to determine a sample value for the first pixel; determining a gradient of an optical flow model, e.g., based on at least a sample value of a second pixel; using the gradient in the optical flow model to obtain a sample value for the first pixel; using a difference between a MV of a current sub-block and a MV of a sub-block adjacent to the current sub-block to obtain a sample value for the first pixel; obtaining a sample value for the first pixel further based on, e.g., a MV of a second sub-block adjacent to the current sub-block; obtaining a sample value for the first pixel based on a determination that the first pixel is adjacent to a boundary of a current sub-block; obtaining a sample value for the first pixel using a weighting factor, where the weighting factor can or can not vary as a function of a distance of the first pixel from a corresponding boundary of the current sub-block; determining a sample value offset for the first pixel based on, e.g., a MV of a current sub-block, a MV of a sub-block adjacent to the current sub-block, and a sample value of a second pixel adjacent to the first pixel; obtaining a sample value for the first pixel using the determined sample value offset for the first pixel and a predicted sample value; and obtaining a sample value for the first pixel based on a determination that the first pixel is adjacent to a boundary of a current sub-block, e.g., where the boundary of the current sub-block can include a common boundary between the current sub-block and a sub-block adjacent to the current sub-block.

[0250] A TV, set-top box, mobile phone, tablet, or other electronic device performing block / sub-block / CU modification according to any of the recited examples.

[0251] A TV, set-top box, mobile phone, tablet, or other electronic device performing block / sub-block / CU modification and displaying the resulting image (e.g., using a monitor, screen, or other type of display) according to any of the recited examples.

[0252] A TV, set-top box, mobile phone, tablet or other electronic device that selects (e.g., using a tuner) a channel to receive a signal including an encoded image and performs block / sub-block / CU modification according to any of the described examples.

[0253] A TV, set-top box, mobile phone, tablet or other electronic device that receives (e.g., using an antenna) an over-the-air signal including an encoded image and performs block / sub-block / CU modification according to any of the described examples.

[0254] While features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein can be implemented in computer-readable medium for execution by a computer or processor, such as is the case with the computer programs, software, or firmware containing the program code. Examples of computer-readable media include electronic signals (optical, electrical or electromagnetic) that are transmittable through a wired or wireless connection. Examples of computer-readable media include, but are not limited to, removable tapes, magnetic disks, magnetic disks, optical disks, semiconductor memories, etc. The processor in association with software can be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. An apparatus for video decoding, comprising one or more processors, wherein the one or more processors are configured to: Obtain the motion vector MV of the current sub-block; The sample value of the first pixel is obtained based on the MV of the current sub-block, the MV of the sub-blocks adjacent to the current sub-block, and the sample value of the second pixel adjacent to the first pixel, wherein the first pixel is the boundary pixel of the current sub-block; and The block including the current sub-block is decoded based on the obtained sample value of the first pixel.

2. An apparatus for video encoding, comprising one or more processors, wherein the one or more processors are configured to: Obtain the motion vector MV of the current sub-block; The sample value of the first pixel is obtained based on the MV of the current sub-block, the MV of the sub-blocks adjacent to the current sub-block, and the sample value of the second pixel adjacent to the first pixel, wherein the first pixel is the boundary pixel of the current sub-block; and The block including the current sub-block is encoded based on the obtained sample value of the first pixel.

3. The apparatus of claim 1 or 2, wherein the block comprises the first pixel, the second pixel, and a third pixel adjacent to the first pixel, wherein the one or more processors are configured to: Determine that the first pixel is the boundary pixel of the current sub-block; and Based on the determination that the first pixel is the boundary pixel of the current sub-block, Determine the difference between the MV of the current sub-block and the MV of the sub-blocks adjacent to the current sub-block. The gradient of the first pixel is determined based on the sample value of the second pixel and the sample value of the third pixel, and The sample value offset is determined based on the determined gradient and the difference between the MV of the current sub-block and the MV of the sub-block adjacent to the current sub-block; Obtaining the sample value of the first pixel based on the MV of the current sub-block, the MV of the sub-block adjacent to the current sub-block, and the sample value of the second pixel adjacent to the first pixel includes obtaining the sample value of the first pixel based on a determined sample value offset.

4. The apparatus of claim 1 or 2, wherein a gradient for an optical flow model is determined based at least on the sample value of the second pixel, and the gradient is used in the optical flow model to obtain the sample value of the first pixel.

5. The apparatus of claim 1 or 2, wherein obtaining the sample value of the first pixel based on the MV of the current sub-block, the MV of the sub-block adjacent to the current sub-block, and the sample value of the second pixel adjacent to the first pixel comprises obtaining the sample value of the second pixel adjacent to the first pixel based on the sample value of the second pixel and based on the MV difference between the MV of the current sub-block and the MV of the sub-block adjacent to the current sub-block.

6. The apparatus of claim 1 or 2, wherein the sub-block adjacent to the current sub-block is a first sub-block, and the block comprises the first sub-block and a second sub-block adjacent to the current sub-block, wherein the sample value of the first pixel is further obtained based on the MV of the second sub-block.

7. The apparatus of claim 1 or 2, wherein the first pixel and the second pixel are in the current sub-block.

8. The apparatus of claim 1 or 2, wherein the sample value of the first pixel is obtained using a weighting factor that varies according to the distance of the first pixel from the corresponding boundary of the current sub-block.

9. A method for video decoding, comprising: Obtain the motion vector MV of the current sub-block; The sample value of the first pixel is obtained based on the MV of the current sub-block, the MV of the sub-blocks adjacent to the current sub-block, and the sample value of the second pixel adjacent to the first pixel, wherein the first pixel is the boundary pixel of the current sub-block; and The block including the current sub-block is decoded based on the obtained sample value of the first pixel.

10. A method for video encoding, comprising: Obtain the motion vector MV of the current sub-block; The sample value of the first pixel is obtained based on the MV of the current sub-block, the MV of the sub-blocks adjacent to the current sub-block, and the sample value of the second pixel adjacent to the first pixel, wherein the first pixel is the boundary pixel of the current sub-block; and The block including the current sub-block is encoded based on the obtained sample value of the first pixel.

11. The method of claim 9 or 10, wherein the block comprises the first pixel, the second pixel, and a third pixel adjacent to the first pixel, wherein obtaining the sample value of the first pixel comprises: The first pixel is determined to be the boundary pixel of the current sub-block; Based on the determination that the first pixel is the boundary pixel of the current sub-block, Determine the difference between the MV of the current sub-block and the MV of the sub-blocks adjacent to the current sub-block. The gradient of the first pixel is determined based on the sample value of the second pixel and the sample value of the third pixel, and The sample value offset is determined based on the determined gradient and the difference between the MV of the current sub-block and the MV of the sub-block adjacent to the current sub-block; as well as Obtaining the sample value of the first pixel based on the MV of the current sub-block, the MV of the sub-block adjacent to the current sub-block, and the sample value of the second pixel adjacent to the first pixel includes obtaining the sample value of the first pixel based on a determined sample value offset.

12. The method of claim 9 or 10, wherein a gradient for an optical flow model is determined based at least on the sample value of the second pixel, and the gradient is used in the optical flow model to obtain the sample value of the first pixel.

13. The method of claim 9 or 10, wherein obtaining the sample value of the first pixel based on the MV of the current sub-block, the MV of the sub-block adjacent to the current sub-block, and the sample value of the second pixel adjacent to the first pixel comprises obtaining the sample value of the second pixel adjacent to the first pixel based on the sample value of the second pixel and based on the MV difference between the MV of the current sub-block and the MV of the sub-block adjacent to the current sub-block.

14. The method of claim 9 or 10, wherein the sub-block adjacent to the current sub-block is a first sub-block, and the block comprises the first sub-block and a second sub-block adjacent to the current sub-block, wherein the sample value of the first pixel is further obtained based on the MV of the second sub-block.

15. The method of claim 9 or 10, wherein the first pixel and the second pixel are in the current sub-block.

16. The method of claim 9 or 10, wherein the sample value of the first pixel is obtained using a weighting factor that varies according to the distance of the first pixel from the corresponding boundary of the current sub-block.

17. A non-transitory computer-readable medium containing data content generated according to any one of claims 9 to 16.

18. A computer-readable medium comprising instructions for causing one or more processors to execute any one of claims 9 to 16.

19. The apparatus according to any one of claims 1 to 8, further comprising: At least one of the following: (i) an antenna configured to receive a signal, the signal including data representing an image; (ii) a bandwidth limiter configured to limit the received signal to a bandwidth including the data representing the image; or (iii) a display configured to display the image.

20. A computer program product comprising computer program instructions, wherein, When the computer program instructions are executed by a processor, they implement the steps of the method according to any one of claims 9 to 16.

Images