Motion compensation for video blocks

Abstract

Systems, methods, and tools associated with performing motion compensation for video blocks are disclosed herein. A video processing device (e.g., an encoder or decoder, etc.) according to embodiments of the present invention may obtain a block of video data (e.g., a 4 * 4 sub-block or a 8 * 8 sub-block) and determine whether to partition the block of video data for motion compensation. Based on a determination to segment the block of video data, the video processing device may be further configured to segment the block of video data into a plurality of units (e.g., sub-block partitions or sub-blocks) and perform the motion compensation for the block of video data based on the plurality of units obtained from the segmentation. Based on a determination that the block of video data is not partitioned, the video processing device may be configured to perform the motion compensation for the block of video data at a block level.

Description

Cross - reference to related applications

[0001] This application claims priority to European Provisional Patent Application No. 22306988.1, filed on December 22, 2022, which is hereby incorporated by reference in its entirety. Background art

[0002] Inter - frame prediction can be used as a tool in video compression. An encoder can select a block in a reference frame after applying motion compensation. Systems, methods, and means for improving the quality of motion compensation may be required. Summary of the invention

[0003] Systems, methods, and means associated with decoding (e.g., encoding and / or decoding) video blocks are disclosed herein. A video encoding or decoding device may include a processor configured to obtain a video data block (e.g., a 4×4 sub - block, an 8×8 sub - block, etc.) and determine whether to split the video data block for performing motion compensation. Based on determining to split the video data block, the processor may further be configured to split the video data block into multiple units smaller than the block (e.g., sub - block partitions or sub - sub - blocks), perform motion compensation for the video data block based on the multiple units obtained from the split, and decode the video data block at least based on the motion compensation. The split may be performed in the horizontal direction, the vertical direction, and any other suitable manner (e.g., such as in the diagonal direction). For example, the split mode (e.g., whether to split) and / or split direction (e.g., along the straight direction or the horizontal direction) may be determined based on the difference between the motion vector associated with the video data block and the motion model associated with the block (e.g., represented by multiple control - point motion vectors or CPMV). Based on determining not to split the video data block, the processor may further be configured to perform the motion compensation for the block based on the motion vector calculated for the block (e.g., the motion vector calculated at the center of the block).

[0004] In an example, the processor being configured to perform the motion compensation for the video data block based on the multiple units obtained from the split may include: the processor being configured to determine a corresponding motion vector associated with each of the multiple units and apply the motion compensation for each of the multiple units based on the motion vector. In an example, whether to split the video data block for motion compensation may be determined based on the maximum difference between the motion vector associated with the block and one or more CPMV associated with the block. In an example, whether to split the video data block for the motion compensation may be determined based on the maximum difference between the motion vector associated with the block and the corresponding motion associated with one or more samples of the block.

[0005] A video decoding apparatus may be configured to partition a video block into a plurality of sub - blocks. The video decoding device may determine, for a sub - block among the plurality of sub - blocks (e.g., each sub - block among the plurality of sub - blocks), whether to split the sub - block for motion compensation. The video decoding device may split the sub - block into a plurality of units smaller than the sub - block (e.g., based on determining to split the sub - block). For example, each of the plurality of units may include a plurality of pixels. The video decoding device may perform motion compensation for the plurality of units of the sub - block. The video decoding device may decode the video block based at least on the plurality of motion - compensated units.

[0006] The sub - block may be a 4×4 sub - block or an 8×8 sub - block. The video decoding device configured to split the sub - block into the plurality of units may include splitting the sub - block horizontally and / or vertically.

[0007] The video decoding device may perform motion compensation for the plurality of units of the sub - block including at least one of the following operations: determining a first unit motion vector associated with a first unit among the plurality of units; obtaining a first motion - compensated unit based on the first unit motion vector; determining a second unit motion vector associated with a second unit among the plurality of units; and obtaining a second motion - compensated unit based on the second unit motion vector.

[0008] The video decoding device may perform, for example, optical flow prediction refinement (PROF) for the first unit among the plurality of units. Performing the PROF for the first unit among the plurality of units may include: determining a difference between a motion vector associated with the sub - block and the first unit motion vector associated with the first unit among the plurality of units, and / or refining the first motion - compensated unit among the plurality of motion - compensated units based on the difference.

[0009] Whether to split the sub - block for motion compensation may be determined based on a maximum difference between a motion vector associated with the sub - block and one or more control point motion vectors (CPMVs) associated with the sub - block, or between a motion vector associated with the sub - block and corresponding motion vectors associated with one or more samples of the sub - block.

[0010] A video encoding device may be configured to partition a video block into a plurality of sub - blocks. The video encoding device may determine, for a sub - block among the plurality of sub - blocks (e.g., each sub - block among the plurality of sub - blocks), whether to split the sub - block for motion compensation. The video encoding device may (e.g., based on determining to split the sub - block) split the sub - block into a plurality of units smaller than the sub - block. For example, each of the plurality of units may include a plurality of pixels. The video encoding device may perform motion compensation for the plurality of units of the sub - block. The video encoding device may encode the video block based at least on the plurality of motion - compensated units.

[0011] The sub - block may be a 4×4 sub - block or an 8×8 sub - block. The video encoding device configured to split the sub - block into the plurality of units may include splitting the sub - block horizontally and / or vertically.

[0012] The video encoding device performing motion compensation for the plurality of units of the sub - block may include at least one of the following operations: determining a first unit motion vector associated with a first unit among the plurality of units; obtaining a first motion - compensated unit based on the first unit motion vector; determining a second unit motion vector associated with a second unit among the plurality of units; and obtaining a second motion - compensated unit based on the second unit motion vector.

[0013] The video encoding device may (e.g.) perform optical flow prediction refinement (PROF) for a first unit among the plurality of units. Performing the PROF for the first unit among the plurality of units may include: determining a difference between a motion vector associated with the sub - block and the first unit motion vector associated with the first unit among the plurality of units, and / or refining the first motion - compensated unit among the plurality of motion - compensated units based on the difference.

[0014] Whether to split the sub - block for motion compensation may be determined based on a maximum difference between a motion vector associated with the sub - block and one or more control point motion vectors (CPMVs) associated with the sub - block, or between a motion vector associated with the sub - block and corresponding motion vectors associated with one or more samples of the sub - block. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A is a system diagram showing an example communication system in which one or more of the disclosed embodiments may be implemented.

[0016] Figure 1B is a system diagram showing an example wireless transmit / receive unit (WTRU) that may be used within the Figure 1A shown communication system according to an embodiment.

[0017] Figure 1C is a system diagram showing an example radio access network (RAN) and an example core network (CN) that can be used within the Figure 1A communication system shown in

[0018] Figure 1D is a system diagram showing another example RAN and another example CN that can be used within the Figure 1A communication system shown in

[0019] Figure 2 shows an example video encoder.

[0020] Figure 3 shows an example video decoder.

[0021] Figure 4 shows an example of a system in which various aspects and examples can be implemented.

[0022] Figure 5A shows an example of a 4-parameter motion model, and Figure 5B shows an example of a 6-parameter motion model.

[0023] Figure 6 shows an example motion vector that can be associated with a sub-block of video data.

[0024] Figure 7A shows an example of spatially adjacent blocks that can be used in a temporal motion vector prediction mode.

[0025] Figure 7B shows an example of deriving a sub-decoding unit motion field by applying a motion shift.

[0026] Figure 8 shows an example of motion vector usage.

[0027] Figure 9 shows an example of luminance prediction refinement based on sub-block motion vectors.

[0028] Figure 10 shows an example of sub-block partitioning.

[0029] Figure 11 shows another example of sub-block partitioning. DETAILED DESCRIPTION

[0030] The present invention can be understood in more detail from the following description given by way of example in conjunction with the accompanying drawings.

[0031] Figure 1AFIG. is a diagram illustrating an exemplary communication system 100 in which one or more of the disclosed embodiments may be implemented. The communication system 100 may be a multi-access system that provides content such as voice, data, video, messaging, broadcast, etc. to a plurality of wireless users. The communication system 100 may enable the plurality of wireless users to access such content by sharing system resources including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique word DFT-spread OFDM (ZT-UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multi-carrier (FBMC), etc.

[0032] As Figure 1A shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, Internet 110, and other networks 112, but it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Any of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile station, fixed or mobile subscriber unit, subscription-based unit, pager, cellular phone, personal digital assistant (PDA), smart phone, laptop computer, netbook, personal computer, wireless sensor, hot spot or MiFi device, Internet of Things (IoT) device, watch or other wearable device, head-mounted display (HMD), vehicle, drone, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in an industrial and / or automated processing chain environment), consumer electronic devices, devices operating on commercial and / or industrial wireless networks, etc. Any of the wireless transmit / receive units 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0033] The communication system 100 may further include base station 114a and / or base station 114b. Each of base stations 114a, 114b may be any type of device configured to wirelessly connect to at least one of WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as CN 106 / 115, Internet 110, and / or other networks 112. As an example, base stations 114a, 114b may be base transceiver stations (BTSs), NodeBs, eNode Bs, home NodeBs, home eNode Bs, gNBs, NR NodeBs, site controllers, access points (APs), wireless routers, etc. Although base stations 114a, 114b are each depicted as a single element, it will be understood that base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0034] Base station 114a may be part of RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage of wireless services to a specific geographical area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In one embodiment, base station 114a may employ multiple-input multiple-output (MIMO) technology and may use multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0035] Base stations 114a, 114b may communicate with one or more of WTRUs 102a, 102b, 102c, 102d via air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, millimeter wave, infrared (IR), ultraviolet (UV), visible light, etc.). Air interface 116 may be established using any suitable radio access technology (RAT).

[0036] More specifically, as described above, the communication system 100 can be a multi-access system and can employ one or more channel access schemes such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base stations 114a and the WTRUs 102a, 102b, 102c in the RAN 104 / 113 can implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can use Wideband CDMA (WCDMA) to establish the air interfaces 115 / 116 / 117. WCDMA can include communication protocols such as High-Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA can include High-Speed Downlink (DL) Packet Access (HSDPA) and / or High-Speed UL Packet Access (HSUPA).

[0037] In one embodiment, the base stations 114a and the WTRUs 102a, 102b, 102c can implement radio technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which can use Long-Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-APro) to establish the air interface 116.

[0038] In one embodiment, the base stations 114a and the WTRUs 102a, 102b, 102c can implement radio technologies such as NR radio access, which can use New Radio (NR) to establish the air interface 116.

[0039] In one embodiment, the base stations 114a and the WTRUs 102a, 102b, 102c can implement multiple radio access technologies. For example, the base stations 114a and the WTRUs 102a, 102b, 102c can implement LTE radio access and NR radio access together using, for example, the Dual Connectivity (DC) principle. Thus, the air interfaces used by the WTRUs 102a, 102b, 102c can be characterized by multiple types of radio access technologies and / or transmissions to / from multiple types of base stations (such as eNBs and gNBs).

[0040] In other embodiments, the base station 114a and the wireless transmit / receive units 102a, 102b, 102c may implement wireless technologies such as IEEE 802.11 (i.e., Wi-Fi (Wireless Fidelity)), IEEE 802.16 (i.e., WiMAX (Worldwide Interoperability for Microwave Access)), CDMA 2000, CDMA 2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), GSM EDGE (GERAN), etc.

[0041] Figure 1A The base station 114b in can be, for example, a wireless router, a Home NodeB, a Home eNode B, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a local area, such as a business premises, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for drones), a road, etc. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology, such as IEEE 802.15, to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA 2000, GSM, LTE-A, LTE-A Pro, NR, etc.) to establish a pico cell or a femto cell. As Figure 1A shown, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b does not need to access the Internet 110 via the CN 106 / 115.

[0042] The RAN 104 / 113 may communicate with the CN 106 / 115, and the CN 106 / 115 may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have varying quality of service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 / 115 may provide call control, billing services, location-based services for mobile devices, prepaid calling, Internet connectivity, video distribution, etc., and / or perform advanced security functions (e.g., user authentication). Although in Figure 1AAlthough not shown in the figure, it should be understood that RAN 104 / 113 and / or CN 106 / 115 can communicate directly or indirectly with other RANs that use the same RAT as RAN 104 / 113 or a different RAT. For example, in addition to being connected to RAN 104 / 113 that can utilize NR radio technology, CN 106 / 115 can also communicate with another RAN (not shown) that uses GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0043] CN 106 / 115 can also be used as a gateway for WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 can include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 can include a global system of interconnected computer networks and devices that use common communication protocols, such as the Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) in the TCP / IP Internet protocol family. The network 112 can include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the network 112 can include another CN that is connected to one or more RANs, and the RANs can use the same RAT as RAN 104 / 113 or a different RAT.

[0044] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 can include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d can include multiple transceivers to communicate with different wireless networks via different wireless links). For example, Figure 1A the illustrated WTRU 102c can be configured to communicate with a base station 114a that can use cellular-based radio technology, and with a base station 114b that can use IEEE 802 radio technology.

[0045] Figure 1B is a system diagram showing an exemplary WTRU 102. As Figure 1B shown, the WTRU 102 can include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keyboard 126, a display / touchpad 128, a non-removable memory 130, a removable memory 132, a power supply 134, a Global Positioning System (GPS) chipset 136, and / or other peripheral devices 138, etc. It can be understood that the WTRU 102 can include any sub-combination of the foregoing elements while remaining consistent with the embodiment.

[0046] The processor 118 can be a general-purpose processor, a dedicated processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 can perform signal decoding, data processing, power control, input / output processing, and / or any other functions that enable the WTRU 102 to operate in a wireless environment. The processor 118 can be coupled to the transceiver 120, and the transceiver 120 can be coupled to the transmit / receive element 122. Although Figure 1B the processor 118 and the transceiver 120 are depicted as separate components, it should be understood that the processor 118 and the transceiver 120 can be integrated together in an electronic package or chip.

[0047] The transmit / receive element 122 can be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via the air interface 116. For example, in one embodiment, the transmit / receive element 122 can be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 can be a transmitter / detector configured to transmit and / or receive signals such as IR, UV, or visible light signals. In yet another embodiment, the transmit / receive element 122 can be configured to transmit and / or receive both RF and optical signals. It should be understood that the transmit / receive element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0048] Although the transmit / receive element 122 is described as a single element in Figure 1B the WTRU 102 can include any number of transmit / receive elements 122. More specifically, the WTRU 102 can use MIMO technology. Thus, in one embodiment, the WTRU 102 can include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals via the air interface 116.

[0049] The transceiver 120 can be configured to modulate the signals to be transmitted by the transmit / receive element 122 and demodulate the signals received by the transmit / receive element 122. As described above, the WTRU 102 can have multi-mode capabilities. Thus, the transceiver 120 can include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0050] The processor 118 of the WTRU 102 may be coupled to a speaker / microphone 124, a keyboard 126, and / or a display / touchpad 128 (such as a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), and may receive user input data therefrom. The processor 118 may also output user data to the speaker / microphone 124, the keyboard 126, and / or the display / touchpad 128. Additionally, the processor 118 may access information from any type of suitable memory and store data in the memory, such as non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In other embodiments, the processor 118 may access information from a memory that is not physically located on the WTRU 102 and store data in the memory (such as located on a server or a home computer (not shown)).

[0051] The processor 118 may receive power from a power source 134 and may be configured to distribute and / or control power to other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cells (such as nickel cadmium (NiCd), nickel zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li-ion), etc.), a solar cell, a fuel cell, etc.

[0052] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (such as longitude and latitude) regarding the current location of the WTRU 102. In addition to, or as an alternative to, the information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (such as base stations 114a, 114b) via an air interface 116, and / or determine its location based on the timing of signals received from two or more neighboring base stations. It should be understood that the WTRU 102 may obtain location information by any suitable location determination method while remaining consistent with the embodiments.

[0053] The processor 118 may also be coupled to other peripheral devices 138, which may include one or more software and / or hardware modules that provide additional features, functions, and / or wired or wireless connections. For example, the peripheral devices 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, Modules, FM radio units, digital music players, media players, video game player modules, Internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, etc. The peripheral device 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors; geolocation sensors; altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors.

[0054] The WTRU 102 may include a full-duplex radio device for which the transmission and reception of some or all of the signals (e.g., signals associated with a particular subframe for uplink (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference through signal processing by hardware (e.g., chokes) or by a processor (e.g., a separate processor (not shown) or the processor 118). In one embodiment, the WTRU 102 may include a half-duplex radio for which the transmission and reception of some or all of the signals (e.g., associated with an uplink (e.g., for transmission) or a downlink (e.g., for reception) particular subframe).

[0055] Figure 1C is a system diagram showing the RAN 104 and the CN 106 according to an embodiment. As described above, the RAN 104 may employ E-UTRA radio technology to communicate with the WTRU 102a, 102b, 102c via the air interface 116. The RAN 104 may also communicate with the CN 106.

[0056] The RAN 104 may include eNode-Bs 160a, 160b, 160c, but it should be understood that the RAN 104 may include any number of eNode-Bs while remaining consistent with the embodiment. The eNode-Bs 160a, 160b, 160c may each include one or more transceivers to communicate with the WTRU 102a, 102b, 102c via the air interface 116. In one embodiment, the eNode-Bs 160a, 160b, 160c may implement MIMO technology. Thus, for example, the eNode-B 160a may use multiple antennas to transmit wireless signals to and / or receive wireless signals from the WTRU 102a.

[0057] Each of eNode-Bs 160a, 160b, and 160c may be associated with a specific cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, user scheduling in UL and / or DL, etc. As Figure 1C shown, eNode-Bs 160a, 160b, and 160c may communicate with each other via the X2 interface.

[0058] Figure 1C The CN 106 shown in Figure 1C may include a Mobility Management Entity (MME) 162, a Serving Gateway (SGW) 164, and a Packet Data Network (PDN) Gateway (or PGW) 166. Although each of the foregoing elements is depicted as part of the CN 106, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0059] The MME 162 may be connected to each of eNode-Bs 162a, 162b, and 162c in the RAN 104 via the S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of WTRUs 102a, 102b, and 102c, bearer activation / deactivation, selecting a specific serving gateway during the initial attachment of WTRUs 102a, 102b, and 102c, etc. The MME 162 may provide control plane functions for handover between the RAN 104 and other RANs (not shown) employing other radio technologies (such as GSM and / or WCDMA).

[0060] The SGW 164 may be connected to each of eNode-Bs 160a, 160b, and 160c in the RAN 104 via the S1 interface. The SGW 164 may generally route and forward user data packets to / from the WTRUs 102a, 102b, and 102c. The SGW 164 may perform other functions, such as anchoring the user plane during handover between eNode Bs, triggering paging when DL data is available for the WTRUs 102a, 102b, and 102c, managing and storing the context of the WTRUs 102a, 102b, and 102c, etc.

[0061] The SGW 164 may be connected to the PGW 166, and the PGW 166 may provide the WTRUs 102a, 102b, and 102c with access to a packet switched network (such as the Internet 110) to facilitate communication between the WTRUs 102a, 102b, and 102c and IP-enabled devices.

[0062] CN 106 can facilitate communication with other networks. For example, CN 106 can provide the WTRUs 102a, 102b, 102c with access to a circuit-switched network (such as the PSTN 108) to facilitate communication between the WTRUs 102a, 102b, 102c and traditional landline communication devices. For example, CN 106 can include an IP gateway (such as an IP Multimedia Subsystem (IMS) server) or can communicate with an IP gateway that serves as an interface between CN 106 and the PSTN 108. Additionally, CN 106 can provide the WTRUs 102a, 102b, 102c with access to other networks 112, which can include other wired and / or wireless networks owned and / or operated by other service providers.

[0063] Although the WTRU is described as a wireless terminal in Figures 1A - 1D it is contemplated that in some representative embodiments, such a terminal can use a wired communication interface with a communication network (such as temporarily or permanently).

[0064] In a representative embodiment, the other network 112 can be a WLAN.

[0065] A WLAN in infrastructure basic service set (BSS) mode can have an access point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP can have access or an interface to a distribution system (DS) or another type of wired / wireless network that carries traffic into and / or out of the BSS. Traffic going to an STA from outside the BSS can arrive through the AP and can be delivered to the STA. Traffic from an STA going to a destination outside the BSS can be sent to the AP to be delivered to the corresponding destination. Traffic between STAs within the BSS can be sent through the AP. For example, where the source STA can send traffic to the AP and the AP can deliver the traffic to the destination STA. Traffic between STAs within the BSS can be considered and / or referred to as peer-to-peer traffic. The peer-to-peer traffic can be sent between the source and destination STAs (such as directly between the source and destination STAs) using direct link setup (DLS). In some representative embodiments, the DLS can use 802.11e DLS or 802.11z tunnel DLS (TDLS). A WLAN using independent BSS (IBSS) mode does not have an AP, and STAs within the IBSS or using the IBSS (such as all STAs) can communicate directly with each other. The IBSS communication mode can sometimes be referred to as an "ad-hoc" communication mode in this document.

[0066] When operating in an 802.11ac infrastructure mode or a similar operating mode, the AP can send beacons on a fixed channel (such as the primary channel). The primary channel can be of a fixed width (e.g., a bandwidth of 20 MHz wide) or a width dynamically set via signaling. The primary channel can be the operating channel of the BSS and can be used by the STA to establish a connection with the AP. In some representative embodiments, for example, in an 802.11 system, Carrier Sense Multiple Access with Collision Avoidance (CSMA / CA) can be implemented. For CSMA / CA, the STA including the AP (e.g., each STA) can listen to the primary channel. If the primary channel is listened to / detected and / or determined to be busy by a particular STA, that particular STA can back off. One STA (e.g., only one station) can transmit at any given time in a given BSS.

[0067] High Throughput (HT) STAs can communicate using a 40 MHz wide channel, for example, by combining the primary 20 MHz channel with an adjacent or non - adjacent 20 MHz channel to form a 40 MHz wide channel.

[0068] Very High Throughput (VHT) STAs can support channels that are 20 MHz, 40 MHz, 80 MHz, and / or 160 MHz wide. The 40 MHz and / or 80 MHz channels can be formed by combining consecutive 20 MHz channels. The 160 MHz channel can be formed by combining eight consecutive 20 MHz channels, or by combining two non - consecutive 80 MHz channels, which can be referred to as an 80 + 80 configuration. For the 80 + 80 configuration, after channel coding, the data can pass through a segment parser, which can divide the data into two streams. The Inverse Fast Fourier Transform (IFFT) processing and time - domain processing can be performed separately on each stream. The streams can be mapped to two 80 MHz channels, and the data can be transmitted by the transmitting STA. At the receiver of the receiving STA, the operation of the above 80 + 80 configuration can be reversed, and the combined data can be sent to the Media Access Control (MAC).

[0069] Operating modes below 1 GHz are supported by 802.11af and 802.11ah. The channel operating bandwidth and carrier in 802.11af and 802.11ah are reduced as compared to the channel operating bandwidth and carrier used in 802.11n and 802.11ac. 802.11af supports 5 MHz, 10 MHz, and 20 MHz bandwidths in the TV white space (TVWS) spectrum, while 802.11ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz bandwidths using the non-TVWS spectrum. According to a representative embodiment, 802.11ah may support meter type control / machine type communication, such as MTC devices in a macro coverage area. The MTC device may have certain capabilities, for example, including limited capabilities such as support for certain and / or limited bandwidths (e.g., support only). The MTC device may include a battery having a battery life above a threshold (e.g., to maintain a very long battery life).

[0070] A WLAN system that can support multiple channels and channel bandwidths (e.g., 802.11n, 802.11ac, 802.11af, and 802.11ah) includes a channel that can be designated as a primary channel. The primary channel may have a bandwidth equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel may be set and / or restricted by the STA that supports the minimum bandwidth operating mode among all STAs operating in the BSS. In an example of 802.11ah, for an STA that supports (e.g., supports only) the 1 MHz mode (e.g., an MTC type device), even if the AP and other STAs in the BSS support 2 MHz, 4 MHz, 8 MHz, 16 MHz, and / or other channel bandwidth operating modes, the primary channel may be 1 MHz wide. Carrier sensing and / or network allocation vector (NAV) setting may depend on the state of the primary channel. If the primary channel is busy, for example, busy due to an STA (which supports only the 1 MHz operating mode) sending to the AP, the entire available frequency band may be considered busy even if most of the frequency band remains idle and available.

[0071] In the United States, the available frequency band that can be used by 802.11ah is from 902 MHz to 928 MHz. In Korea, the available frequency band is from 917.5 MHz to 923.5 MHz. In Japan, the available frequency band is from 916.5 MHz to 927.5 MHz. The total bandwidth available for 802.11ah is 6 MHz to 26 MHz, depending on the country code.

[0072] Figure 1Dis a system diagram showing RAN 113 and CN 115 according to an embodiment. As described above, RAN 113 can communicate with WTRUs 102a, 102b, 102c via air interface 116 using NR radio technology. RAN 113 can also communicate with CN 115.

[0073] RAN 113 may include gNBs 180a, 180b, 180c, but it should be understood that RAN 113 may include any number of gNBs while remaining consistent with the embodiment. Each of gNBs 180a, 180b, 180c includes one or more transceivers for communicating with WTRUs 102a, 102b, 102c via air interface 116. In one embodiment, gNBs 180a, 180b, 180c may implement MIMO technology. For example, gNBs 180a, 108b may utilize beamforming to transmit signals to and / or receive signals from gNBs 180a, 180b, 180c. Thus, gNB 180a, for example, may use multiple antennas to transmit wireless signals to and / or receive wireless signals from WTRU102a. In one embodiment, gNBs 180a, 180b, 180c may implement carrier aggregation technology. For example, gNB 180a may transmit multiple component carriers to WTRU 102a (not shown). A subset of these component carriers may be on unlicensed spectrum, while the remaining component carriers may be on licensed spectrum. In one embodiment, gNBs 180a, 180b, 180c may implement coordinated multi-point (CoMP) technology. For example, WTRU 102a may receive coordinated transmissions from gNB 180a and gNB 180b (and / or gNB 180c).

[0074] WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using transmissions associated with scalable numerology. For example, the OFDM symbol interval and / or the OFDM subcarrier interval may vary for different transmissions, different cells, and / or different parts of the radio transmission spectrum. WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using subframes or transmission time intervals (TTIs) having different lengths or scalable lengths (e.g., containing different numbers of OFDM symbols and / or an absolute time of continuously varying length).

[0075] gNBs 180a, 180b, 180c can be configured to communicate with WTRUs 102a, 102b, 102c in a stand-alone configuration and / or a non-stand-alone configuration. In a stand-alone configuration, WTRUs 102a, 102b, 102c can communicate with gNBs 180a, 180b, 180c without also accessing another RAN (e.g., eNode-Bs 160a, 160b, 160c). In a stand-alone configuration, WTRUs 102a, 102b, 102c can utilize one or more of gNBs 180a, 180b, 180c as a mobility anchor. In a stand-alone configuration, WTRUs 102a, 102b, 102c can communicate with gNBs 180a, 180b, 180c using signals in an unlicensed band. In a non-stand-alone configuration, WTRUs 102a, 102b, 102c can communicate / connect with gNBs 180a, 180b, 180c while also communicating / connecting with another RAN (such as eNode-Bs 160a, 160b, 160c). For example, WTRUs 102a, 102b, 102c can implement the DC principle in order to communicate with one or more gNBs 180a, 180b, 180c and one or more eNode-Bs 160a, 160b, 160c substantially simultaneously. In a non-stand-alone configuration, eNode-Bs 160a, 160b, 160c can be used as a mobility anchor for WTRUs 102a, 102b, 102c, while gNBs 180a, 180b, 180c can provide additional coverage and / or throughput for serving WTRUs 102a, 102b, 102c.

[0076] Each of gNBs 180a, 180b, 180c can be associated with a specific cell (not shown) and can be configured to handle radio resource management decisions, handover decisions, user scheduling in UL and / or DL, support for network slicing, dual connectivity, interworking between NR and E-UTRA, routing of user plane data to user plane functions (UPFs) 184a, 184b, routing of control plane information to access and mobility management functions (AMFs) 182a, 182b, etc. As Figure 1D shown, gNBs 180a, 180b, 180c can communicate with each other via the Xn interface.

[0077] Figure 1DThe CN 115 shown in the figure may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one session management function (SMF) 183a, 183b, and may include data networks (DN) 185a, 185b. Although each of the foregoing elements is depicted as part of the CN 115, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0078] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via the N2 interface and may act as a control node. For example, the AMF 182a, 182b may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, supporting network slicing (e.g., handling different PDU sessions with different requirements), selecting a particular SMF 183a, 183b, managing the registration area, terminating NAS signaling, mobility management, etc. The AMF 182a, 182b may use network slicing to customize the CN support for the WTRUs 102a, 102b, 102c according to the type of service being used by the WTRUs 102a, 102b, 102c. For example, different network slices may be able to be established for different use cases, such as services that rely on ultra-reliable low-latency (URLLC) access, services that rely on enhanced massive mobile broadband (eMBB) access, services for machine type communication (MTC) access, etc. The AMF 162 may provide control plane functions for handovers between the RAN 113 and other RANs (not shown) that employ other radio technologies (e.g., LTE, LTE-A, LTE-A Pro and / or non-3GPP access technologies such as WiFi).

[0079] The SMF 183a, 183b may be connected to the AMF 182a, 182b in the CN 115 via the N11 interface. The SMF 183a, 183b may also be connected to the UPF 184a, 184b in the CN 115 via the N4 interface. The SMF 183a, 183b may select and control the UPF 184a, 184b and configure the routing of traffic through the UPF 184a, 184b. The SMF 183a, 183b may perform other functions, such as managing and allocating UE IP addresses, managing PDU sessions, controlling policy enforcement and QoS, providing downlink data notifications, etc. The PDU session type may be IP-based, non-IP-based, Ethernet-based, etc.

[0080] UPF 184a and 184b can be connected to one or more of gNBs 180a, 180b, 180c in RAN 113 via the N3 interface. The N3 interface can provide access to a packet switched network (such as the Internet 110) for WTRU 102a, 102b, 102c to facilitate communication between WTRU 102a, 102b, 102c and IP-enabled devices. UPF 184a, 184b can perform other functions, such as routing and forwarding packets, enforcing user plane policies, supporting multi-homed PDU sessions, handling user plane QoS, buffering downlink packets, providing mobility anchoring, etc.

[0081] CN 115 can facilitate communication with other networks. For example, CN 115 can include an IP gateway (such as an IP Multimedia Subsystem (IMS) server) that serves as an interface between CN 115 and the PSTN 108, or can communicate with the IP gateway. Additionally, CN 115 can provide access to other networks 112 for WTRU 102a, 102b, 102c. Other networks 112 can include other wired and / or wireless networks owned and / or operated by other service providers. In one embodiment, WTRU 102a, 102b, 102c can be connected to DNs 185a, 185b via the N3 interface to UPF 184a, 184b and the N6 interface between UPF 184a, 184b and local data networks (DNs) 185a, 185b.

[0082] In view of Figures 1A - 1D and Figures 1A - 1D the corresponding descriptions, one or more or all of the functions described herein with respect to one or more of the following: WTRU 102a-d, base stations 114a-b, eNode-Bs 160a-c, MME 162, SGW 164, PGW 166, gNBs 180a-c, AMF 182a-b, UPF 184a-b, SMF 183a-b, DNs 185a-b, and / or any other device described herein can be performed by one or more emulation devices (not shown). The emulation device(s) can be one or more devices configured to emulate one or more or all of the functions described herein. For example, the emulation device(s) can be used to test other devices and / or simulate network and / or WTRU functionality.

[0083] Emulation devices can be designed to implement one or more tests on other devices in a laboratory environment and / or an operator network environment. For example, one or more emulation devices can perform one or more or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network, in order to test other devices within the communication network. The one or more emulation devices can perform one or more or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. An emulation device can be directly coupled to another device for testing purposes and / or can perform tests using over-the-air wireless communication.

[0084] One or more emulation devices can perform one or more functions (including all functions) while not being implemented / deployed as part of a wired and / or wireless communication network. For example, an emulation device can be used in a test laboratory and / or a test scenario in a non-deployed (e.g., test) wired and / or wireless communication network, in order to implement tests on one or more components. One or more emulation devices can be test devices. An emulation device can transmit and / or receive data using direct RF coupling and / or wireless communication via an RF circuit (e.g., which can include one or more antennas).

[0085] This application describes multiple aspects, including tools, features, examples, models, methods, etc. Many of these aspects are described as having particularity, and at least for the purpose of showing each characteristic, they are usually described in a way that may sound restrictive. However, this is for the purpose of clear description and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide additional aspects. In addition, these aspects can also be combined and interchanged with aspects described in earlier documents.

[0086] The aspects described and contemplated in this application can be implemented in many different forms. The accompanying drawings provided herein can provide some examples, but other examples can be contemplated. The discussion of the drawings does not limit the breadth of the implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting the generated or encoded bitstream. These and other aspects can be implemented as methods, apparatuses, computer-readable media (e.g., storage media) including (e.g., storing thereon) instructions for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media storing a bitstream generated according to any of the described methods. When mentioned herein, a bitstream can refer to the data being transmitted, but can also refer to data stored, generated, and / or accessed without being transmitted (e.g., non-transitory data).

[0087] In the present application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture", and "frame" may be used interchangeably.

[0088] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires steps or actions in a specific order, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as "first", "second", etc. may be used in various examples to modify elements, components, steps, operations, etc., such as "first decoding" and "second decoding". Unless specifically required, the use of these terms does not imply an ordering of the modified operations. Thus, in this example, the first decoding does not need to be performed before the second decoding, and may occur, for example, before, during, or in a time period overlapping with the second decoding.

[0089] The various methods and other aspects described in the present application can be used to modify modules (such as the decoding module) of the video encoder 200 and decoder 300 as shown in Figure 2 and Figure 3 In addition, the subject matter disclosed herein can be applied to, for example, any type, format, or version of video coding, whether described as a standard or a recommendation, whether pre - existing or future - developed, and extensions of any such standards and recommendations. Unless otherwise indicated or technically excluded, the aspects described in the present application can be used alone or in combination.

[0090] Various numerical values are used in the examples described in the present application. These and other specific values are for the purpose of describing the examples, and the described aspects are not limited to these specific values.

[0091] Figure 2 is a diagram showing an example video encoder. Variations of the example encoder 200 can be envisioned, but for clarity, encoder 200 is described below without describing all envisioned variations.

[0092] Before being encoded, the video sequence may undergo a pre - encoding process 201. For example, a color transformation (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0) may be applied to the input color picture, or a remapping of the input picture components may be performed to obtain a signal distribution that is more tolerant to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre - processing and appended to the bitstream.

[0093] In encoder 200, as described below, pictures are encoded by encoder elements. The picture to be encoded is partitioned 202, for example, in units of coding units (CUs), and processed. Each unit is encoded using, for example, an intra or inter mode. When a unit is encoded in the intra mode, intra prediction 260 is performed. In the inter mode, motion estimation 275 and compensation 270 are performed. The encoder determines 205 which of the intra mode or the inter mode is used to encode the unit, and indicates the intra / inter determination, for example, by a prediction mode flag. For example, a prediction residual is calculated by subtracting 210 a predicted block from the original image block.

[0094] Then, the prediction residual is transformed 225 and quantized 230. The quantized transform coefficients, motion vectors, and other syntax elements are entropy decoded 245 to output a bitstream. The encoder may skip the transformation and directly apply quantization to the untransformed residual signal. The encoder may bypass the transformation and quantization, i.e., directly decode the residual without applying the transformation or quantization process.

[0095] The encoder decodes the encoded blocks to provide references for further prediction. The quantized transform coefficients are dequantized 240 and inverse-transformed 250 to decode the prediction residual. The decoded prediction residual and the predicted block are combined 255 to reconstruct the image block. An in-loop filter 265 is applied to the reconstructed image to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce encoding artifacts. The filtered image is stored in a reference picture buffer (280).

[0096] Figure 3 FIG. is a diagram showing an example of a video decoder. In an example decoder 300, as described below, a bitstream is decoded by decoder elements. Video decoder 300 generally performs decoding rounds that are reciprocal to the Figure 2 encoding rounds described in. Encoder 200 generally also performs video decoding as part of encoding video data.

[0097] In particular, the input to the decoder includes a video bitstream, which may be generated by video encoder 200. The bitstream is first entropy decoded 330 to obtain transform coefficients, motion vectors, and other decoded information. Picture partitioning information indicates how the picture is partitioned. Thus, the decoder can partition 335 the picture according to the decoded picture partitioning information. The transform coefficients are dequantized 340 and inverse-transformed 350 to decode the prediction residual. The decoded prediction residual and the predicted block are combined 355 to reconstruct the image block. The predicted block may be obtained 370 from intra prediction 360 or motion compensated prediction (i.e., inter prediction) 375. An in-loop filter 365 is applied to the reconstructed image. The filtered image is stored in reference picture buffer 380.

[0098] The decoded picture may further undergo post - decoding processing 385. For example, an inverse color transformation (e.g., a conversion from YCbCr 4:2:0 to RGB 4:4:4) or performing an inverse remapping of the remapping process performed in the pre - encoding processing 201. The post - decoding processing may use metadata derived in the pre - encoding processing and signaled in the bitstream. In one example, the decoded image (e.g., after applying the in - loop filter 365 and / or, if post - decoding processing is used, after the post - decoding processing 385) may be sent to a display device for presentation to a user.

[0099] Figure 4 FIG. is an example of a system in which aspects and examples described herein can be implemented. System 400 can be implemented as a device including various components described below and configured to perform one or more aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set - top boxes, digital television receivers, personal video recording systems, connected household appliances, and servers. The elements of system 400 can be implemented singly or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one example, the processing and encoder / decoder elements of system 400 are distributed across multiple ICs and / or discrete components. In various examples, system 400 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various examples, system 400 is configured to implement one or more aspects described herein.

[0100] System 400 includes at least one processor 410 configured to execute instructions loaded therein for implementing, for example, the various aspects described herein. Processor 410 may include embedded memory, input - output interfaces, and various other circuits known in the art. System 400 includes at least one memory 420 (e.g., volatile memory devices and / or non - volatile memory devices). System 400 includes a storage device 440, which may include non - volatile memory and / or volatile memory, and storage device 440 includes, but is not limited to, electrically erasable programmable read - only memory (EEPROM), read - only memory (ROM), programmable read - only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non - limiting example, storage device 440 may include internal storage devices, attached storage devices (including removable and non - removable storage devices), and / or network - accessible storage devices.

[0101] System 400 includes an encoder / decoder module 430, which is configured to process data, for example, to provide encoded video or decoded video, and the encoder / decoder module 430 may include its own processor and memory. The encoder / decoder module 430 represents the module(s) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding and a decoding module. Additionally, the encoder / decoder module 430 may be implemented as a separate element of the system 400 or may be incorporated within the processor 410 as a combination of hardware and software known to those skilled in the art.

[0102] The program code to be loaded onto the processor 410 or the encoder / decoder 430 to perform the various aspects described in this document may be stored in the storage device 440 and subsequently loaded onto the memory 420 for execution by the processor 410. According to various examples, one or more of the processor 410, the memory 420, the storage device 440, and the encoder / decoder module 430 may store one or more of the various items during the execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operation logic.

[0103] In some examples, the memory internal to the processor 410 and / or the encoder / decoder module 430 is used to store instructions and provide a working memory for the processing operations required during encoding or decoding. However, in other instances, memory external to the processing device (e.g., the processing device may be the processor 410 or the encoder / decoder module 430) is used for one or more of these functions. The external memory may be the memory 420 and / or the storage device 440, for example, dynamic volatile memory and / or non-volatile flash memory. In several examples, the external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one example, fast external dynamic volatile memory such as RAM is used as a working memory for video encoding and decoding operations.

[0104] As shown in block 445, input to the elements of the system 400 may be provided through various input devices. Such input devices include, but are not limited to: (i) an RF section that receives RF signals transmitted over the air, for example, by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 4 Other examples not shown include composite video.

[0105] In various examples, the input device of block 445 has corresponding input processing elements associated therewith known in the art. For example, the RF section may be associated with elements adapted for: (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some examples, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and / or (vi) demultiplexing to select a desired data packet stream. The RF section of various examples includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In an example of a set-top box, the RF section and its associated input processing elements receive an RF signal transmitted through a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band. Various examples rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements (e.g., inserting an amplifier and an analog-to-digital converter). In various examples, the RF section includes an antenna.

[0106] The USB and / or HDMI terminals may include corresponding interface processors for connecting system 400 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented within, for example, a separate input processing IC or within processor 410 as needed. Similarly, various aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within processor 410 as needed. The streams of demodulation, error correction, and demultiplexing are provided to various processing elements (including, for example, processor 410) and encoder / decoder 430 (which operates in conjunction with memory and storage elements) to process the data stream as needed for presentation on an output device.

[0107] The various elements of system 400 may be disposed within an integrated housing. Within the integrated housing, the various elements may be interconnected using suitable connection components 425 and data may be transmitted therebetween, where connection components 425 may be, for example, internal buses known in the art, and connection components 425 include inter-integrated circuit (I2C) buses, wiring, and printed circuit boards.

[0108] System 400 includes a communication interface 450 that enables communication with other devices via a communication channel 460. The communication interface 450 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 460. The communication interface 450 may include, but is not limited to, a modem or a network card, and the communication channel 460 may be implemented, for example, within a wired and / or wireless medium.

[0109] In various examples, a wireless network such as a Wi-Fi network (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) is used to stream or otherwise provide data to System 400. The Wi-Fi signals of these examples are received via a communication channel 460 and a communication interface 450 suitable for Wi-Fi communication. The communication channel 460 of these examples is typically connected to an access point or a router that provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other examples use a set-top box that delivers data via an HDMI connection of an input block 445 to provide streaming data to System 400. Still other examples use an RF connection of the input block 445 to provide streaming data to System 400. As described above, various examples provide data in a non-streaming manner. Additionally, various examples use a wireless network other than Wi-Fi (e.g., a cellular network or a network).

[0110] System 400 may provide output signals to various output devices, including a display 475, speakers 485, and other peripheral devices 495. The display 475 of various instances includes one or more of the following: for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 475 may be used for a television, a tablet, a laptop computer, a cellular phone (mobile phone), or other devices. The display 475 may also be integrated with other components (e.g., as in a smart phone) or separated (e.g., an external monitor for a laptop computer). In various examples, the other peripheral devices 495 include one or more of the following: a standalone digital video disc (or digital versatile disc) (for both, DVD), a disc player, a stereo system, and / or a lighting system. Various examples use one or more peripheral devices 495 that function based on the output of System 400. For example, a disc player performs the function of playing the output of System 400.

[0111] In various examples, control signals are transmitted between the system 400 and the display 475, the speaker 485, or other peripheral devices 495 using signaling such as AV.Link, consumer electronics control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 400 via dedicated connections through the respective interfaces 470, 480, and 490. Alternatively, the output devices can be connected to the system 400 using the communication channel 460 via the communication interface 450. The display 475 and the speaker 485 can be integrated into a single unit within an electronic device (such as a television) with other components of the system 400. In various examples, the display interface 470 includes a display driver, such as a timing controller (TCon) chip.

[0112] For example, if the RF portion of the input 445 is part of a separate set-top box, the display 475 and the speaker 485 can alternatively be separated from one or more of the other components. In various examples where the display 475 and the speaker 485 are external components, the output signal can be provided via a dedicated output connection, which includes, for example, an HDMI port, a USB port, or a COMP output.

[0113] These examples can be executed by computer software implemented by the processor 410 or a combination of hardware and software. As a non-limiting example, these examples can be implemented by one or more integrated circuits. The memory 420 can be of any type suitable for the technical environment and can be implemented using any appropriate data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 410 can be of any type suitable for the technical environment and can include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0114] Various implementations include decoding. As used in this application, "decoding" can include, for example, all or part of the processing performed on a received coded sequence to produce a final output suitable for display. In various examples, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various examples, such a process also or alternatively includes processes performed by the decoders of the various implementations described in this application.

[0115] As a further example, in one example, "decoding" refers only to entropy decoding, and in another example, "Decoding" refers only to differential decoding, and in another example, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally refer to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0116] Various implementations include encoding. In a manner similar to the above discussion regarding "decoding", "encoding" as used in this application can include, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various examples, such a process includes one or more processes typically performed by an encoder, such as, partitioning, differential encoding, transformation, quantization, and entropy encoding. In various examples, such a process also or alternatively includes processes performed by the encoders of the various implementations described in this application.

[0117] As a further example, in one example, "encoding" refers only to entropy encoding, and in another example, "encoding" refers only to differential encoding, and in another example, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to specifically refer to a subset of operations or generally refer to a broader encoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0118] Note that the grammatical elements used here are descriptive terms. Thus, they do not exclude the use of other grammatical element names.

[0119] When the drawings are presented as flowcharts, it should be understood that they also provide a block diagram of the corresponding apparatus. Similarly, when the drawings are presented as block diagrams, it should be understood that they also provide a flowchart of the corresponding method / process.

[0120] The implementations and aspects described herein can be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., an apparatus or a program). For example, an apparatus can be implemented with appropriate hardware, software, and firmware. The method can be implemented in, for example, a processor, which generally refers to a processing device, which includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. A processor also includes a communication device, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate communication of information between end users.

[0121] References to "an example" or "examples" or "an implementation" or "implementations" and other variations thereof mean that the particular features, structures, characteristics, etc. described in connection with that example are included in at least one example. Thus, the phrases "in an example" or "in examples" or "in an implementation" or "in implementations" and any other variations that occur throughout this application do not necessarily all refer to the same example.

[0122] Additionally, this application may relate to "determining" various information. Determining information may include one or more of the following: for example, estimating information, calculating information, predicting information, or retrieving information from a memory. Obtaining may include receiving, retrieving, constructing, generating, and / or determining.

[0123] Furthermore, this application may relate to "accessing" various information. Accessing information may include one or more of the following: for example, receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0124] Additionally, this application may refer to "receiving" various information. Like "accessing", receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Further, "receiving" is typically involved in one way or another during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0125] It should be understood that the use of any of the following, " / ", "and / or", and "at least one of the following", e.g., in the cases of "A / B", "A and / or B", and "at least one of A and B", is intended to cover the selection of only the first-listed option (A), or only the second-listed option (B), or the selection of both options (A and B). As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include the selection of only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or the selection of only the first and second-listed options (A and B), or the selection of only the first and third-listed options (A and C), or the selection of only the second and third-listed options (B and C), or the selection of all three options (A and B and C). As will be clear to those of ordinary skill in the art and related fields, this can be extended to any number of listed items.

[0126] In addition, as used herein, the term "signal" particularly refers to indicating something to a corresponding decoder. In this way, in an example, the same parameters are used at both the encoder side and the decoder side. Thus, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder such that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters and other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select the specific parameters. By avoiding the transmission of any actual functionality, bit savings are achieved in various examples. It should be understood that signaling can be implemented in various ways. For example, in various examples, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. Although the foregoing relates to the verb form of the term "signal", the term "signal" can also be used as a noun herein.

[0127] As will be apparent to those of ordinary skill in the art, implementations can generate various signals that are formatted to carry information such as can be stored or transmitted. This information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry the bitstream of the described example. Such a signal can be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, signals can be transmitted over various different wired or wireless links. A signal can be stored on a processor-readable medium or can be accessed or received from a processor-readable medium.

[0128] This document describes many examples. The features of the examples can be provided individually or in any combination across various claim categories and types. Additionally, the examples can span various claim categories and types to include one or more of the features, devices, or aspects described herein, either individually or in any combination. For example, the features described herein can be implemented in a bitstream or signal that includes information generated as described herein. This information can allow a decoder to decode the bitstream, with the encoder, bitstream, and / or decoder according to any of the described embodiments. For example, the features described herein can be implemented by creating and / or transmitting and / or receiving and / or decoding a bitstream or signal. For example, the features described herein can be implemented as a method, process, apparatus, medium storing instructions (e.g., a computer-readable medium), medium storing data, or signal. For example, the features described herein can be implemented by a TV, set-top box, cellular phone, tablet, or other electronic device that performs decoding. The TV, set-top box, cellular phone, tablet, or other electronic device can display (e.g., using a monitor, screen, or other type of display) the generated image (e.g., an image reconstructed from residuals of a video bitstream). The TV, set-top box, cellular phone, tablet, or other electronic device can receive a signal including an encoded image and perform decoding.

[0129] Affine motion compensation prediction can be performed during video coding (e.g., encoding and / or decoding). A translational motion model (e.g., a translational-only motion model) can be applied to motion compensation prediction (MCP). Different kinds of motion can be associated with a video, such as zooming in / out, rotation, perspective motion, other irregular motions, etc. Block-based affine motion compensation prediction can perform warping with motion compensation.

[0130] Figure 5A An example is shown where an affine motion field of a video block can be described by motion information of a bi-control point vector (e.g., a 4-parameter model). Figure 5B An example is shown where an affine motion field of a video block can be described by motion information of a tri-control point motion vector (e.g., a 6-parameter model).

[0131] For a 4-parameter affine motion model, for example, as Figure 5A shown, the motion vector at the sample position (x,y) in a block can be derived by the following formula:

[0132] For a 6-parameter affine motion model, for example, as Figure 5B shown, the motion vector at the sample position (x,y) in a block can be derived by the following formula: where (mv 0x ,mv 0y) can be the motion vector of the upper left control point, (mv 1x , mv 1y ) can be the motion vector of the upper right control point, and (mv 2x , mv 2y ) can be the motion vector of the lower left control point, and H and W can indicate the size of the block (e.g., H can represent the height of the block and W can represent the width of the block).

[0133] For example, by applying block-based affine transformation prediction, motion compensation prediction can be simplified. To derive the motion vector (MV) associated with a sub-block (e.g., each sub-block, e.g., each 4×4×luma sub-block), the motion vector of the central sample of the sub-block can be calculated according to equations (1) and (2) (e.g., as shown in Figure 6 ) and / or can be rounded to 1 / 16 fractional pixel accuracy. One or more motion compensation interpolation filters can be applied to generate the prediction of the sub-block with the derived motion vector. The size of the sub-block of the chrominance component can also be set (e.g., set to 4×4). The MV of a sub-block (e.g., a 4×4 chrominance sub-block) can be calculated as, for example, the average of the MVs of the upper left and lower right luma sub-blocks in a co-located region (e.g., a co-located 8×8 luma region). For translational motion inter-frame prediction, there can be multiple (e.g., two) affine motion inter-frame prediction modes, such as the affine merge mode and the affine motion vector prediction (AMVP) mode.

[0134] Sub-block-based temporal motion vector prediction (Sb TMVP) can be supported for video coding (e.g., encoding and / or decoding). Sb TMVP can be similar to temporal motion vector prediction (TMVP). Sb TMVP can use the motion field in the co-located picture to improve the motion vector prediction and merge mode of the coding unit CU in the current picture. The same co-located image used by TMVP can be used for Sb TMVP. Sb TMVP can be different from TMVP in multiple aspects. For example, TMVP can predict the motion at the CU level, while Sb TMVP can predict the motion at the sub-CU level. As another example, TMVP can obtain the temporal motion vector from a co-located block in the co-located picture (e.g., the co-located block can be the lower right or central block relative to the current CU), while Sb TMVP can apply a motion shift before obtaining the temporal motion information from the co-located picture (e.g., the motion shift can be obtained based on the motion vectors of the spatially adjacent blocks of the current CU).

[0135] Figure 7A and Figure 7B show an example of the Sb TMVP process. Sb TMVP can predict the motion vector of the sub-CU within the current CU in multiple operations (e.g., two operations). As shown in Figure 7AAs shown, the spatial neighbor A1 of the current CU can be examined. If A1 has a motion vector that uses the co-located picture as its reference picture, then that motion vector can be selected as the motion shift to be applied. If no such motion is identified, the motion shift can be set to (0, 0). As Figure 7B shown, the motion shift identified in the first step can be applied (e.g., the motion shift can be added to the coordinates of the current block) to obtain sub-CU level motion information (e.g., motion vector and / or reference index) from the co-located picture. Figure 7B The example in can assume that the motion shift is set to block the motion of A1. For a sub-CU (e.g., each sub-CU), the motion information of its corresponding block (e.g., the smallest motion grid that can cover the central sample) located in the co-located picture can be used to derive the motion information of the sub-CU. After identifying the motion information of the co-located sub-CU, the motion information can be converted to the motion vector and / or reference index of the current sub-CU (e.g., in a manner similar to TMVP), where temporal motion scaling can be applied to align the reference picture of the temporal motion vector with the reference picture of the current CU.

[0136] The combined sub-block based merge list can contain Sb TMVP candidates and / or affine merge candidates. The combined sub-block based merge list can be used to signal the sub-block based merge mode. For example, the Sb TMVP mode can be enabled and / or disabled by a sequence parameter set (SPS) flag. If the Sb TMVP mode is enabled, then the Sb TMVP predictor can be added to the list of sub-block based merge candidates (e.g., as the first entry in the list followed by one or more affine merge candidates). The size of the sub-block based merge list can be signaled (e.g., in the SPS). The maximum allowed size of the sub-block based merge list can be set (e.g., set to a fixed value, e.g., 5).

[0137] The sub-CU size used in Sb TMVP can be fixed (e.g., is 8×8). The Sb TMVP mode can be applicable to (e.g., only applicable to) CUs having a width and / or height greater than or equal to 8 (e.g., as in the affine merge mode). The encoding logic for additional Sb TMVP merge candidates can be the same as that for other merge candidates. For example, for a CU in a P or B slice (e.g., each CU), an additional RD check can be performed to determine whether to use an Sb TMVP candidate.

[0138] Affine motion information can be stored (e.g., in a buffer). The control point motion vector (CPMV) of an affine CU can be stored in a buffer (e.g., stored in a separate buffer designated for this storage). The stored CPMV can be used (e.g., only used) to generate an inherited control point motion vector predictor (CPMVP) in affine merge mode and / or affine AMVP mode (e.g., for the most recently decoded CU). Sub-block MVs derived from the CPMV can be used for various purposes, which include (e.g.) motion compensation, merging of translational MVs / MV derivation from the AMVP list, and / or deblocking.

[0139] The inheritance of affine motion data from the CU associated with the upper CTU can be treated differently from the inheritance from normal neighboring CUs (e.g., to avoid the picture line buffer for additional CPMVs). If the candidate CU for affine motion data inheritance is in the upper CTU row, then the lower left and lower right sub-block MVs (e.g., instead of CPMVs) in the row buffer can be used for affine MVP derivation. In this way, the CPMV can be stored (e.g., only stored) in the local buffer. If the candidate CU is decoded with a 6-parameter affine, the affine model can be degraded to a 4-parameter model. As Figure 8 shown, along the top CTU boundary, the lower left and lower right sub-block motion vectors of the CU can be used for affine inheritance of the CU in the lower CTU.

[0140] Optical flow prediction refinement (PROF) can be performed in affine mode. Sub-block based affine motion compensation can save memory access bandwidth and / or reduce computational complexity (e.g., compared to pixel-based motion compensation and / or at the cost of prediction accuracy). PROF can be used to refine the sub-block based affine motion compensation prediction. The use of PROF can achieve a finer granularity of motion compensation without increasing the memory access bandwidth for motion compensation. For example, after performing sub-block based affine motion compensation, the luminance prediction samples can be refined by adding the differences that can be derived using the optical flow equation.

[0141] PROF can be performed as follows. In the first step, sub-block based affine motion compensation can be performed to generate the sub-block prediction I(i,j). In the second step, the spatial gradient g x (i,j) and g y (i,j) of the sub-block prediction can be calculated at (e.g., each) sample position using, for example, a 3-tap filter [-1,0,1]. The gradient calculation can be the same as the gradient calculation performed in the bidirectional optical flow (BDOF) mode, e.g., as shown in the following equation. g x (i,j) = (I(i + 1,j) >> shift1) - (I(i - 1,j) >> shift1) (3) gy (i,j) = (I(i,j+1) << shift1) - (I(i,j-1) >> shift1) (4) Where shift1 can be used to control the precision of the gradient. Sub-block (e.g., 4×4 sub-block) prediction can be extended by samples on one or more edges (e.g., on each edge) for gradient calculation. For example, those extended samples on the extended boundary can be copied from the nearest integer pixel positions in the reference picture to avoid additional memory bandwidth and / or additional interpolation calculations.

[0142] In the third step, the luminance prediction refinement can be determined by the following optical flow equation. ΔI(i,j) = g x (i,j) * Δv x (i,j) + g y (i,j) * Δv y (i,j) (5) Where Δv(i,j) can represent the difference between the sample MV calculated for the sample position (i,j) (represented by v(i,j)) and the sub-block MV of the sub-block to which the sample (i,j) belongs (e.g., V SB ) as shown Figure 9 . Δv(i,j) (e.g., represented by the short arrow in the figure) can be quantized in units of 1 / 32 luminance samples.

[0143] Since the affine model parameters and / or sample positions relative to the sub-block center may not change sub-block by sub-block, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. Let dx(i,j) and dy(i,j) be the horizontal and vertical offsets from the sample position (i,j) to the center (x SB ,y SB ) of the sub-block, and Δv(x,y) can be derived by the following equation: Where C, D, E, and F can be defined as follows.

[0144] The center (x SB ,y SB ) of the sub-block can be calculated as ((W SB - 1) / 2, (H SB - 1) / 2) (e.g., to maintain the precision of the calculation), where W SB and W SB can represent the sub-block width and height respectively. For a 4-parameter affine model, the following may hold:

[0145] For a 6-parameter affine model, the following may hold: where (v 0x , v 0y ), (v 1x , v 1y ), (v 2x , v 2y ) can represent the motion vectors of the upper-left, upper-right, and lower-left control points, and e and h can represent the width and height of the CU.

[0146] In the fourth step, the refined luminance prediction ΔI(i, j) can be added to the sub-block prediction I(i, j). The final prediction I′ can be generated using the following equation: I′(i, j) = I(i, j) + ΔI(i, j) (10)

[0147] PROF may not be applied to affine decoded CUs in some cases. For example, if multiple (e.g., all) control point MVs are the same, then PROF may not be applied, which may indicate that the CU only has translational motion. As another example, if the affine motion parameters are greater than a specified limit (e.g., because sub-block based affine MC can degrade to CU based MC to avoid using large memory access bandwidth), then PROF may not be applied.

[0148] Fast encoding methods can be applied to reduce the encoding complexity of affine motion estimation using PROF. In some cases, PROF may not be applied to the affine motion estimation stage. For example, if the CU is not a root block and its parent block does not select the affine mode as its best mode, then PROF may not be applied because the probability that the current CU selects the affine mode as the best mode is low. As another example, if the value of one or more (e.g., all) of the four affine parameters (C, D, E, F) is less than a predefined threshold and the current picture is not a low-latency picture, then PROF may not be applied (e.g., because the improvement introduced by PROF may be small for this case). In this way, the affine motion estimation using PROF can be accelerated.

[0149] PROF (e.g., as described herein) can be applied to the motion vector(s) of a sub-block (e.g., multiple pixels).

[0150] Inter-frame prediction can be used as a decoding tool in video compression. An encoder can select a block (e.g., the best block) in a reference frame after applying a motion model (e.g., a translational, affine, or sub-block based motion compensation model). Motion compensation can be performed based on the uniform motion (e.g., translation) of each sub-block (e.g., which can have a size of 4×4, 8×8, etc.). Blocks using an affine motion model can undergo motion compensation, where one or more sub-blocks (e.g., all sub-blocks) can be motion compensated using a unique motion vector associated with the sub-block (e.g., the motion vector can be used for the entire sub-block). Various techniques can be used to improve the quality of motion compensation for these blocks. For example, the quality of motion compensation can be improved by using PROF to correct the motion compensated sub-block(s). For example, if the affine motion model does not produce a large enough appropriate motion (e.g., the angular motion is not large relative to the motion of the top left corner), then this technique can work well. As another example, the quality of motion compensation can be improved by performing motion compensation at the pixel granularity (e.g., on a per-pixel basis, e.g., for each 1×1 pixel). This method can increase the complexity of motion compensation (e.g., requires more motion derivation and / or more independent motion compensations, which can imply more memory reads, reduced possibility of parallel processing, etc.).

[0151] One or more of the following techniques can be used to reduce the complexity of affine motion compensation, such as motion compensation performed at the 1x1 level (e.g., pixel level). Motion compensation can be performed at the sub-block level (e.g., based on smaller partitions or units within the sub-block), for example, at a level below the 4×4 pixel or 8×8 pixel sub-block size level but above the 1×1 (e.g., pixel) level. For example, motion compensation can be performed for multiple units (e.g., where each unit contains multiple pixels). Motion compensation at the sub-sub-block level can be performed without adding signaling (e.g., motion compensation can be performed based on an automatic decision). Block level splitting (e.g., sub-block level splitting) can be performed without additional complexity. For example, these techniques may not increase the complexity of motion compensation operations because they may re-use computations that have already been performed during PROF. The techniques may be described herein in the context of an affine motion model, but those skilled in the art should understand that the techniques can be applied to any motion model in which motion compensation can be performed at a level below the sub-block (e.g., 4×4 sub-block or 8×8 sub-block) level.

[0152] The sub-blocks can be split (e.g., into smaller partitions or units) to reduce the complexity of affine motion compensation. For example, affine motion compensation can be performed (e.g., only performed) on a finite number of sub-sub-blocks (e.g., a finite number of smaller partitions or units within the sub-block) (e.g., for two sub-blocks). For example, the sub-blocks can be split horizontally, vertically, and / or diagonally. For example, the sub-blocks can be split along the direction of the maximum motion.

[0153] Figure 10 An example of splitting a video data block (e.g., a 4×4 sub-block of video data, etc.) is shown. In the figure, A, B, and C can represent the CPMV (Control Point Motion Vector) for the sub-block, and can represent the motion model for the sub-block (e.g., other equivalent motion parameterizations can also be used). The CPMV can define the motion model for this particular sub-block (e.g., it can be within the original block), and thus, the CPMV may or may not be signaled at the block level.

[0154] Decisions can be made (e.g., during encoding and / or decoding) regarding the following: whether to not split the sub-block, whether to split the sub-block horizontally, whether to split the sub-block vertically, whether to split the sub-block diagonally, etc. If it is determined not to split the sub-block, affine motion compensation can be performed at the 4×4 sub-block level, where multiple (e.g., all) pixels in the sub-block can be motion compensated using the same motion vector (such as the motion vector calculated at the center of the sub-block). If it is determined to split the sub-block horizontally or vertically, then the corresponding motion vectors can be determined and applied (e.g., applied to the sub-block or sub-blocks) for each horizontally or vertically obtained sub-block (e.g., at the center of this sub-block). For example, compared to performing motion compensation based on a 1×1 (pixel) affine model, using Figure 10 the splitting method shown in, the complexity of motion calculation can be reduced to 1 / 8. The memory read burden can also be reduced.

[0155] One or more of the following can be applied during the splitting decision process. The error (e.g., difference) between the motion vector of the sub-block (e.g., the common MV for the entire sub-block) and the motion model (e.g., which can be represented by one or more CPMVs) can be calculated. If the error is higher than a threshold, the sub-block can be split. To determine whether to split the sub-block along the vertical or horizontal direction, the maximum error (e.g., maximum difference) associated with the sub-block can be calculated for each potential splitting direction, and the direction with the lower maximum error can be selected as the actual splitting direction.

[0156] The error (e.g., difference) between the motion vector and the motion model can be calculated in various ways. For example, the error can be calculated (e.g., for potential segmentation directions) as the maximum error (e.g., maximum difference) between each CPMV of the motion model and the motion vector of the sub-block. As another example, the error can be calculated as the maximum error between the motion vector of the sub-block and the corresponding motion of one or more pixels or samples (e.g., each pixel or sample) of the sub-block. Such a value may already have been calculated in the PROF process as shown in Equation (7) (e.g., the error can be Δv x and Δv y which is the sum of squares). The motion of a sample (e.g., pixel) can be calculated by applying the motion model to the pixel center and obtaining the motion vector of the pixel.

[0157] For example, the segmentation decision segmentation process described herein can be accelerated by directly deriving the segmentation decision from the motion model parameters themselves. For example, in Equation 7, Δv x and Δv y can be the maximum values in any direction (e.g., horizontal or vertical). If |C| + |D| > ∈, or |E| + |F| > ∈, then the sub-block can be segmented. In the case of segmentation, if |C| + |D| > |E| + |F|, then for at least some pixels, Δv x can be greater than Δv y . In this case, the sub-block can be segmented vertically; otherwise, the sub-block can be segmented horizontally.

[0158] For at least a 4-parameter affine model, the decision can be further simplified as follows. If |C| > ∈' or |D| > ∈, then the sub-block can be segmented. In the case of segmentation, if |C| > |E|, then for at least some pixels, Δv x can be greater than Δv y . In this case, the sub-block can be segmented vertically; otherwise, the sub-block can be segmented horizontally.

[0159] Although examples for segmenting a 4×4 pixel sub-block are described herein, it should be noted that segmentation can also be performed for other sub-block or unit sizes, which includes, for example, if the allowable number of segments is 4, such as at an 8×8 sub-block level (e.g., keeping the complexity constant compared to 4×4 level motion compensation) or at a 16×16 sub-block level.

[0160] Figure 11 An example is shown where an 8×8 video data block can be segmented into 4 sub-blocks each containing 16 pixels. The decision process described herein can be applied to this example, during which the maximum error can be calculated in the 4 sub-blocks (e.g., instead of the 2 sub-sub-blocks and the unsegmented sub-block shown in Figure 10 ). Figure 11The "no split" determination shown can correspond to the default case, where motion compensation can be performed based on 4x4. In an example, sub-sub blocks of size 8x1 or 1x8 can be allowed, which can split an 8x8 block into 8 sub-sub blocks. The number of splits can depend on the original block size.

[0161] If the sub-blocks are split, the PROF process described herein can be adapted to consider the motion vectors of the sub-sub blocks. One or more of the equations provided herein can remain the same. One or more terms calculated for making the split determination (e.g., C, D, E, F used in the equations provided herein) can be re-used for the PROF processing.

[0162] The performance of PROF can be improved, for example, by adjusting the PROF gradient calculation to enhance the level of precision involved. For example, during the gradient step size calculation, full precision can be used (e.g., instead of the reduced precision of using sample values as in Equation 3) to calculate the gradient. The resulting g x gradient can be shifted as per the following formula, and the same can be applied to the g y gradient. g x (i,j) = (I(i + 1,j) - I(i - 1,j)) >> shift1

[0163] Although the examples provided herein can assume that media content is streamed to a display device, there is no specific limitation on the type of display device, which can benefit from the example techniques described herein. For example, the display device can be a television, a projector, a mobile phone, a tablet, etc. Additionally, the example techniques described herein can be applied not only to streaming use cases but also to teleconference settings. Further, the decoder and the display as described herein can be separate devices or can be part of the same device. For example, a set-top box can decode an incoming video stream and (e.g., subsequently) provide the decoded stream to the display device (e.g., via HDMI), and information about viewing conditions such as viewing distance is transmitted from the display device to the set-top box (e.g., via HDMI).

[0164] Adjusting the PROF process described herein to account for motion vectors of sub - blocks (e.g., units, pixels, multiple pixels) can include one or more of the following. Block - based affine motion compensation can be performed to generate a sub - block prediction I(i,j). Sub - sub - block - based affine motion compensation can involve applying motion compensation to multiple units (e.g., multiple pixels) within a sub - block and using an affine motion model to represent the motion within each of the multiple units. The affine motion model can include translation, rotation, scaling, and / or shear components (e.g., to provide a more flexible representation of motion compared to simpler models such as only translation or rigid motion models). Applying motion compensation to sub - sub - blocks can allow for a finer - grained representation of the motion within a block. For example, as part of this motion compensation process, different motion vectors and affine motion parameters can be estimated for each of the multiple units within a sub - block. This method can be useful when there is variation in the motion within a sub - block, such as when different parts of an object are moving in different directions or undergoing different types of motion (rotation, scaling, etc.). The sub - sub - block - based method can allow for a more accurate representation of complex motion patterns and / or can improve the quality of motion compensation prediction.

[0165] The spatial gradients gx(i,j) and gy(i,j) of the sub - sub - block prediction can be calculated at each sample position (e.g., using a 3 - tap filter [-1,0,1]). The filter can be convolved with the pixel values at sample positions adjacent in the horizontal and vertical directions. The convolution operation can include multiplying each filter coefficient by the corresponding pixel value and summing the results. The resulting sum can provide an approximation of the spatial gradients in the horizontal and vertical directions. For example, the gradient calculation can use equations (3) and (4). The process of determining the spatial gradients for the sub - block prediction process can capture the rate of intensity change in the horizontal and / or vertical directions within the local neighborhood of each unit position in the sub - block prediction. The resulting gradients can be used for luminance prediction refinement (e.g., as shown in equation (5)).

[0166] In an example, a first unit motion vector can be calculated for a first unit of multiple units (e.g., the first multiple pixels of a sub - block), and the first unit motion vector can be reused for other units of the multiple units (e.g., within the same sub - block). The difference between the first unit motion vector associated with the first unit of the multiple units and the motion vector associated with the sub - block can be determined. The first motion - compensated unit among the multiple motion - compensated units can be refined based on the difference.

[0167] Although the features and elements have been described above in specific combinations, one of ordinary skill in the art will understand that each feature or element can be used separately or in any combination with other features and elements. Additionally, the methods described herein can be implemented in a computer program, software, or firmware that is incorporated into a computer-readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted via wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor associated with the software can be used to implement a radio frequency transceiver used in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A video decoding device, the device comprising: A processor configured to: Partition a video block into a plurality of sub - blocks; For a sub - block among the plurality of sub - blocks, determine whether to split the sub - block for motion compensation; Based on determining to split the sub - block, split the sub - block into a plurality of units smaller than the sub - block, wherein each of the plurality of units includes a plurality of pixels; Perform motion compensation for the plurality of units of the sub - block; And Decode the video block based at least on the plurality of motion - compensated units.

2. The video decoding device according to claim 1, wherein the sub - block is a 4×4 sub - block or an 8×8 sub - block.

3. The video decoding device according to any one of claims 1-2, wherein the processor is configured to split the sub-block into the plurality of units including: The processor is configured to split the sub - block horizontally.

4. The video decoding device according to any one of claims 1-2, wherein the processor is configured to divide the sub-block into the plurality of units, including: The processor is configured to split the sub - block vertically.

5. The video decoding device according to any one of claims 1-4, wherein the processor is configured to perform motion compensation for the plurality of units of the sub-blocks, including: The processor is configured to: Determine a first unit motion vector associated with a first unit among the plurality of units; Obtain a first motion - compensated unit based on the first unit motion vector; Determine a second unit motion vector associated with a second unit among the plurality of units; And Obtain a second motion - compensated unit based on the second unit motion vector.

6. The video decoding device according to claim 5, wherein the processor is further configured to perform optical flow prediction refinement (PROF) for the first unit among the plurality of units, and performing PROF for the first unit includes: Determine a difference between a motion vector associated with the sub - block and the first unit motion vector associated with the first unit among the plurality of units, and Refine the first motion - compensated unit among the plurality of motion - compensated units based on the determined difference.

7. The video decoding device according to any one of claims 1 - 6, wherein the determination of whether to split the sub - block for motion compensation is made based on a maximum difference between a motion vector associated with the sub - block and one or more control point motion vectors (CPMVs) associated with the sub - block.

8. The video decoding device according to claim 1, wherein the determination of whether to split the sub - block for the motion compensation is made based on a maximum difference between a motion vector associated with the sub - block and corresponding motion vectors associated with one or more samples of the sub - block.

9. A video decoding method, the method comprising: Partition a video block into a plurality of sub - blocks; For a sub - block among the plurality of sub - blocks, determine whether to split the sub - block for motion compensation; Based on determining to split the sub - block, split the sub - block into a plurality of units smaller than the sub - block, wherein each of the plurality of units includes a plurality of pixels; Perform motion compensation for the plurality of units of the sub - block; And Decode the video block based at least on the plurality of motion - compensated units.

10. The video decoding method according to claim 9, wherein the sub - block is a 4×4 sub - block or an 8×8 sub - block.

11. The video decoding method according to any one of claims 9-10, wherein splitting the sub-block into the plurality of units comprises: Split the sub - block horizontally.

12. The video decoding method according to any one of claims 9-10, wherein splitting the sub-block into the plurality of units comprises: Split the sub - block vertically.

13. The video decoding method according to any one of claims 9 - 12, wherein performing motion compensation for the plurality of units obtained from the splitting includes: Determine a first unit motion vector associated with a first unit among the plurality of units; Obtain a first motion-compensated unit based on the first unit motion vector; Determine a second unit motion vector associated with a second unit among the plurality of units; And Obtain a second motion-compensated unit based on the second unit motion vector.

14. The video decoding method according to claim 13, the method further comprising: Perform optical flow prediction refinement (PROF) for the first unit among the plurality of units, where performing PROF for the first unit includes: Determine a difference between a sample motion vector associated with the sub-block and the first unit motion vector associated with the first unit among the plurality of units, and Refine the first motion-compensated unit among the plurality of motion-compensated units based on the determined difference.

15. The video decoding method according to any one of claims 9-14, wherein the determination of whether to partition the sub-block for motion compensation is based on a maximum difference between a motion vector associated with the sub-block and one or more control point motion vectors (CPMVs) associated with the sub-block.

16. The video decoding method according to claim 9, wherein the determination of whether to partition the sub-block for the motion compensation is based on a maximum difference between a motion vector associated with the sub-block and corresponding motion vectors associated with one or more samples of the sub-block.

17. A video encoding device, the device comprising: A processor configured to: Partition a video block into a plurality of sub-blocks; For a sub-block among the plurality of sub-blocks, determine whether to partition the sub-block for motion compensation; Based on the determination to partition the sub-block, partition the sub-block into a plurality of units smaller than the sub-block, wherein each of the plurality of units includes a plurality of pixels; Perform motion compensation for the plurality of units of the sub-block; And Encode the video block based at least on the plurality of motion-compensated units.

18. The video encoding device according to claim 17, wherein the sub-block is a 4×4 sub-block or an 8×8 sub-block, and wherein each of the plurality of units includes a plurality of pixels.

19. The video encoding device according to any one of claims 17-18, wherein the processor is configured to partition the sub-block into the plurality of units includes; the processor is configured to horizontally partition the sub-block.

20. The video encoding device according to any one of claims 17-18, wherein the processor is configured to divide the sub-block into the plurality of units including: The processor is configured to vertically partition the sub-block.

21. The video encoding device according to any one of claims 17-20, wherein the processor is configured to perform motion compensation for the plurality of units obtained from the segmentation, including: The processor is configured to: Determine a first unit motion vector associated with a first unit among the plurality of units; Obtain a first motion-compensated unit based on the first unit motion vector; Determine a second unit motion vector associated with a second unit among the plurality of units; And Obtain a second motion-compensated unit based on the second unit motion vector.

22. The video encoding device according to claim 21, wherein the processor is further configured to perform optical flow prediction refinement (PROF) for the first unit among the plurality of units, where performing PROF for the first unit includes: Determine the difference between the sample motion vector associated with the sub-block and the first unit motion vector associated with the first unit among the plurality of units, and Refine the first motion-compensated unit among the plurality of motion-compensated units based on the determined difference.

23. The video encoding device according to any one of claims 17-22, wherein the determination of whether to partition the sub-block for motion compensation is based on the maximum difference between the motion vector associated with the sub-block and one or more control point motion vectors (CPMVs) associated with the sub-block.

24. The video encoding device according to claim 17, wherein the determination of whether to partition the sub-block for motion compensation is based on the maximum difference between the motion vector associated with the sub-block and the corresponding motion vectors associated with one or more samples of the sub-block.

25. A video encoding method, the method comprising: Partition a video block into a plurality of sub-blocks; For a sub-block among the plurality of sub-blocks, determine whether to partition the sub-block for motion compensation; Based on the determination to partition the sub-block, partition the sub-block into a plurality of units smaller than the sub-block; Perform motion compensation for the plurality of units of the sub-block; And Encode the video block based at least on the plurality of motion-compensated units.

26. The video encoding method according to claim 25, wherein the sub-block is a 4×4 sub-block or an 8×8 sub-block, and wherein each of the plurality of units includes a plurality of pixels.

27. The video encoding method according to any one of claims 25-26, wherein splitting the sub-block into the plurality of units includes: Partition the sub-block horizontally.

28. The video encoding method according to any one of claims 25-26, wherein splitting the sub-block into the plurality of units comprises: Partition the sub-block vertically.

29. The video encoding method according to any one of claims 25-28, wherein performing motion compensation for the plurality of units obtained from the partitioning includes: Determine a first unit motion vector associated with a first unit among the plurality of units; Obtain a first motion-compensated unit based on the first unit motion vector; Determine a second unit motion vector associated with a second unit among the plurality of units; And Obtain a second motion-compensated unit based on the second unit motion vector.

30. The video encoding method according to claim 29, the method further comprising: Perform optical flow prediction refinement (PROF) for the first unit among the plurality of units, wherein performing PROF for the first unit includes: Determine the difference between the sample motion vector associated with the sub-block and the first unit motion vector associated with the first unit among the plurality of units, and Refine the first motion-compensated unit among the plurality of motion-compensated units based on the determined difference.

31. The video encoding method according to any one of claims 25-30, wherein the determination of whether to partition the sub-block for motion compensation is based on the maximum difference between the motion vector associated with the sub-block and one or more control point motion vectors (CPMVs) associated with the sub-block.

32. The video encoding method according to claim 25, wherein the determination of whether to partition the sub-block for the motion compensation is based on the maximum difference between the motion vector associated with the sub-block and the corresponding motion vectors associated with one or more samples of the sub-block.

33. A computer program product, which is stored on a non-transitory computer-readable medium and includes program code instructions for implementing the steps of the method according to any one of claims 9-16 or claims 25-32 when executed by a processor.

34. A computer program, which includes program code instructions for implementing the steps of the method according to any one of claims 9-16 or claims 25-32 when executed by a processor.

35. A video data, which includes information representing the video block encoded in accordance with one of the methods according to any one of claims 25 to 34.

Images