Adaptive control point selection for affine motion model-based video coding

Through affine motion model and adaptive control point selection technology, the problem of video block motion vectors exceeding the range is solved, the video decoding efficiency and accuracy are improved, and it is suitable for video blocks of different shapes.

CN119011869APending Publication Date: 2024-11-22INTERDIGITAL VC HOLDINGS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411208546.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-12-11
Filing Date
2019-06-27
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In block-based video decoding systems, the motion vectors of video blocks may exceed the expected range, resulting in unintended results, which are difficult for the prior art to effectively handle.

Method used

Using an affine motion model, control point positions are adaptively selected to optimize the export and prediction of motion vectors by cropping and predicting the control point motion vectors associated with video blocks, including selecting control points based on block shape and distances of adjacent blocks, using Affine merge mode and sports field to reduce signaling overhead.

Benefits of technology

It improves video decoding efficiency and accuracy, reduces the signaling overhead of motion vectors, enhances the motion prediction ability of video blocks, and is suitable for video blocks of different shapes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011869A_ABST
    Figure CN119011869A_ABST
Patent Text Reader

Abstract

Disclosed are adaptive control point selection for affine motion model-based video coding, in particular systems, methods, and instrumentalities for motion vector cropping when affine motion mode is enabled for video blocks. A video coding device may determine to enable an affine mode for a video block. The video coding device may determine a plurality of control point affine motion vectors associated with the video block. The video coding device may store the plurality of cropped control point affine motion vectors for motion vector prediction of adjacent control point affine motion vectors. The video coding device may derive a sub-block motion vector associated with a sub-block of the video block, clip the derived sub-block motion vector, and store the derived sub-block motion vector for spatial motion vector prediction or temporal motion vector prediction. For example, the video coding device may clip the derived sub-block motion vector based on a range of motion fields based on bit depth values.
Need to check novelty before this filing date? Find Prior Art

Description

This application is a divisional application of Chinese invention patent application No. 201980044216.9, filed on June 27, 2019, entitled “Adaptive Control Point Selection for Video Decoding Based on Affine Motion Model”. The contents of the parent application are incorporated herein by reference. CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 691,770 filed on June 29, 2018, U.S. Provisional Patent Application No. 62 / 734,728 filed on September 21, 2018, and U.S. Provisional Patent Application No. 62 / 778,055 filed on December 11, 2018, the contents of which are incorporated herein by reference. Background Art

[0002] Video coding systems can be used to compress digital video signals, for example, to reduce the bandwidth required to store and / or transmit these signals. Video coding systems can include block-based, wavelet-based, and / or object-based systems. Block-based hybrid video coding systems can be deployed. In block-based video coding systems, motion vectors associated with sub-blocks of a video block may have values ​​that may be outside a certain range. Using such values ​​may lead to unexpected results. Summary of the invention

[0003] Disclosed are systems, methods, and means for performing motion vector clipping when an affine motion mode is enabled for a video block (e.g., a coding unit (CU)). A video coding device may determine to enable an affine mode for a video block (e.g., a current video block). The video block may include a plurality of sub-blocks. The video coding device may determine a plurality of control point affine motion vectors associated with the video block. At least one of the control point affine motion vectors associated with the current video block may be determined using one or more control point affine motion vectors associated with one or more neighboring video blocks. The video coding device may clip the control point affine motion vector associated with the current video block. For example, the control point affine motion vector may be clipped based on a bit depth for motion field storage. The video coding device may store the clipped control point affine motion vector for motion vector prediction of neighboring control point affine motion vectors.

[0004] The video coding device may derive a sub-block motion vector associated with a sub-block. The video coding device may derive the sub-block motion vector based on one or more control point affine motion vectors. The video coding device may clip the derived sub-block motion vector. For example, the video coding device may clip the derived sub-block motion vector based on a motion field range. The motion field range may be used for motion field storage. The motion field range may be based on a bit depth value. The video coding device may store the clipped sub-block motion vector for spatial motion vector prediction or temporal motion vector prediction. The video coding device may use the clipped sub-block motion vector to predict the sub-block.

[0005] The video decoding device may determine the control point position associated with the control point affine motion vector of the video block based on the shape of the video block. For example, the control point position may be determined based on the length and / or width of the video block.

[0006] For example, if the width of the current video block is greater than the length of the current video block, the control point positions may include an upper left control point and an upper right control point. The video decoding device may classify this video block as a horizontal rectangular video block. For example, if the width of the current video block is less than the length of the current video block, the control point positions may include an upper left control point and a lower left control point. The video decoding device may classify the current video block as a vertical rectangular video block. For example, if the width of the current video block is equal to the length of the current video block, the control point positions may include a lower left control point and an upper right control point. The video decoding device may classify the current video block as a square video block. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1A is a system diagram illustrating an exemplary communication system.

[0008] Figure 1B It shows that it can be Figure 1A A system diagram of an exemplary wireless transmit / receive unit (WTRU) for use within a communication system is shown.

[0009] Figure 1C It shows that it can be Figure 1A A system diagram of an exemplary radio access network (RAN) and an exemplary core network (CN) used within a communication system is shown.

[0010] Figure 1D It shows that it can be Figure 1A A system diagram of another exemplary RAN and another exemplary CN used within the communication system shown.

[0011] Figure 2An example diagram of a block-based video encoder is shown.

[0012] Figure 3 An example block diagram of a video decoder is shown.

[0013] Figure 4 An example block partitioning in a multi-type tree structure is shown.

[0014] Figure 5 An example of a four-parameter affine pattern is shown.

[0015] Figure 6 An example of an affine merge candidate is shown.

[0016] Figure 7 An example motion vector derivation at the control points of an affine motion model is shown.

[0017] Figure 8 An example of affine motion predictor construction is shown.

[0018] Fig. 9 An example of affine motion vector (MV) temporal scaling produced by the MV predictor is shown.

[0019] Fig.10 An example of adaptive control point selection based on patch shape is shown.

[0020] Fig.11 An example of affine merge selection using maximum control point distance is shown.

[0021] Fig.12 An example workflow for motion field generation for an affine model is shown.

[0022] Fig.13 An example workflow is shown for reusing motion fields used for MV prediction and deblocking to generate prediction samples for an affine coding unit (CU).

[0023] Fig.14 An example workflow is shown that reuses the motion field used to generate prediction samples to perform MV prediction and deblocking on an affine CU.

[0024] Fig.15 An example of modifying one or more control points MV to scale a reference block is shown.

[0025] Fig.16 An example of modifying a control point MV to include a reference block is shown. DETAILED DESCRIPTION

[0026] The present invention may be understood in more detail from the following description given by way of example in conjunction with the accompanying drawings, in which:

[0027] Figure 1A 1 is a schematic diagram showing an exemplary communication system 100 in which one or more disclosed examples may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content by sharing system resources including wireless bandwidth. For example, the communication system 100 may use one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), zero tail unique word DFT-spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, and filter bank multi-carrier (FBMC), etc.

[0028] like Figure 1A As shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, Internet 110, and other networks 112, although it should be appreciated that the disclosed examples contemplate any number of WTRUs, base stations, networks, and / or network components. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, any of the WTRUs 102a, 102b, 102c, 102d may be referred to as a “station” and / or “STA”, which may be configured to transmit and / or receive wireless signals, and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular phone, a personal digital assistant (PDA), a smart phone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable device, a head-mounted display (HMD), a vehicle, a drone, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices working in an industrial and / or automated process chain environment), consumer electronic devices, and devices working on a commercial and / or industrial wireless network, etc. Any of the WTRUs 102a, 102b, 102c, 102d may be interchangeably referred to as a UE.

[0029] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to facilitate access to one or more communication networks (e.g., CN 106 / 115, Internet 110, and / or other networks 112) by wirelessly interfacing with at least one of the WTRUs 102a, 102b, 102c, 102d. For example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNode B, a Home Node B, a Home eNode B, a gNB, a New Radio (NR) Node B, a site controller, an access point (AP), a wireless router, and the like. Although each of the base stations 114a, 114b is depicted as a single component, it should be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network components.

[0030] The base station 114a may be part of the RAN 104 / 113, and the RAN may also include other base stations and / or network components (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, and the like. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies called cells (not shown). These frequencies may be in a licensed spectrum, an unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage for a specific geographic area that is relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, a cell associated with the base station 114a may be divided into three sectors. Thus, in one example, the base station 114a may include three transceivers, i.e., each transceiver corresponds to a sector of the cell. In the example, the base station 114a may use multiple-input multiple-output (MIMO) technology, and may use multiple transceivers for each sector of the cell. For example, by using beamforming, signals may be transmitted and / or received in a desired spatial direction.

[0031] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, millimeter wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0032] More specifically, as described above, the communication system 100 may be a multiple access system and may use one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, and SC-FDMA, among others. For example, the base station 114a in the RAN 104 / 113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may use Wideband CDMA (WCDMA) to establish the air interface 115 / 116 / 117. WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0033] In an example, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0034] In an example, the base station 114a and the WTRUs 102a, 102b, 102c may implement some radio technology that may establish the air interface 116 using new radio (NR), such as NR radio access.

[0035] In an example, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may jointly implement LTE radio access and NR radio access (e.g., using dual connectivity (DC) principles). Thus, the air interface used by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0036] In an example, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), and GSM EDGE (GERAN), among others.

[0037] Figure 1A The base station 114b in the example can be, for example, a wireless router, a Home Node B, a Home eNode B, or an access point, and can use any appropriate RAT to facilitate wireless connectivity in a local area, such as a business place, a residence, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), and a road, etc. In one example, the base station 114b and the WTRUs 102c, 102d can establish a wireless local area network (WLAN) by implementing a radio technology such as IEEE 802.11. In an example, the base station 114b and the WTRUs 102c, 102d can establish a wireless personal area network (WPAN) by implementing a radio technology such as IEEE 802.15. In an example, the base station 114b and the WTRUs 102c, 102d can establish a picocell or a femtocell by using a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). Figure 1A As shown, the base station 114b may be directly connected to the Internet 110. Thus, the base station 114b does not need to access the Internet 110 via the CN 106 / 115.

[0038] The RAN 104 / 113 may be in communication with the CN 106 / 115, which may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have different quality of service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 / 115 may provide call control, billing services, mobile location-based services, prepaid calls, Internet connectivity, video distribution, etc., and / or may perform advanced security functions such as user authentication. Although in Figure 1AAlthough not shown, it will be appreciated that the RAN 104 / 113 and / or the CN 106 / 115 may be in direct or indirect communication with other RANs that employ the same RAT or a different RAT as the RAN 104 / 113. For example, in addition to being connected to the RAN 104 / 113 employing NR radio technology, the CN 106 / 115 may also be in communication with another RAN (not shown) employing GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0039] The CN 106 / 115 may also act as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer network devices that use common communication protocols, such as TCP, User Datagram Protocol (UDP), and / or IP in the Transmission Control Protocol / Internet Protocol (TCP / IP) Internet protocol suite. The networks 112 may include wired or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, where the one or more RANs may use the same RAT or a different RAT as the RAN 104 / 113.

[0040] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). Figure 1A The illustrated WTRU 102c may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0041] Figure 1B is a system diagram illustrating an exemplary WTRU 102. Figure 1B As shown, the WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a numeric keypad 126, a display / touchpad 128, a non-removable memory 130, a removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or peripherals 138. It should be appreciated that the WTRU 102 may also include any sub-combination of the foregoing components.

[0042] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal decoding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, however it should be appreciated that the processor 118 and the transceiver 120 may also be integrated together in one electronic component or chip.

[0043] The transmit / receive component 122 may be configured to transmit or receive signals to or from a base station (e.g., base station 114a) via the air interface 116. For example, in one example, the transmit / receive component 122 may be an antenna configured to transmit and / or receive RF signals. As an example, in an example, the transmit / receive component 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals. In an example, the transmit / receive component 122 may be configured to transmit and / or receive RF and light signals. It should be appreciated that the transmit / receive component 122 may be configured to transmit and / or receive any combination of wireless signals.

[0044] Although in Figure 1B 102 as a single component, but the WTRU 102 may include any number of transmit / receive components 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one example, the WTRU 102 may include two or more transmit / receive components 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0045] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive component 122 and to demodulate signals received by the transmit / receive component 122. As described above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers that allow the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0046] The processor 118 of the WTRU 102 may be coupled to a speaker / microphone 124, a numeric keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), and may receive user input data from these components. The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from and store information in any suitable memory such as a non-removable memory 130 and / or a removable memory 132. The non-removable memory 130 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In an example, the processor 118 may access information from and store data in memory that is not physically located in the WTRU 102, such as, for example, a server or a home computer (not shown).

[0047] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control power for use by other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (Ni-Cd), nickel-zinc (Ni-Zn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0048] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) related to the current location of the WTRU 102. In addition to or in lieu of the information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) via the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information using any suitable positioning method.

[0049] The processor 118 may also be coupled to other peripheral devices 138, wherein the peripheral devices may include one or more software and / or hardware modules that provide additional features, functions, and / or wired or wireless connectivity. For example, the peripheral devices 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, Modules, frequency modulation (FM) radio units, digital music players, media players, video game console modules, Internet browsers, virtual reality and / or augmented reality (VR / AR) devices, and activity trackers, etc. The peripheral device 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors, geographic location sensors, altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors, etc.

[0050] The WTRU 102 may include a full-duplex radio in which reception or transmission of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous for the radio. A full-duplex radio may include an interference management unit that reduces and / or substantially eliminates self-interference by means of hardware (e.g., chokes) or by signal processing by a processor (e.g., a separate processor (not shown) or by the processor 118). In an example, the WTRU 102 may include a half-duplex radio that transmits and receives some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)).

[0051] Figure 1C 1 is a system diagram illustrating an exemplary RAN 104 and CN 106. As described above, the RAN 104 may be in communication with the WTRUs 102a, 102b, 102c using an E-UTRA radio technology over the air interface 116. The RAN 104 may also be in communication with the CN 106.

[0052] The RAN 104 may include eNode-Bs 160a, 160b, 160c, however it will be appreciated that the RAN 104 may include any number of eNode-Bs. The eNode-Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one example, the eNode-Bs 160a, 160b, 160c may implement MIMO technology. Thus, for example, the eNode-B 160a may use multiple antennas to transmit wireless signals to, and / or receive wireless signals from, the WTRU 102a.

[0053] Each of the eNodeBs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, user scheduling in the UL and / or DL, etc. Figure 1C As shown, the eNode-Bs 160a, 160b, 160c may communicate with each other via an X2 interface.

[0054] Figure 1C The illustrated CN 106 may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or PGW) 166. Although each of the aforementioned components is described as being part of the CN 106, it should be appreciated that any of these components may be owned and / or operated by an entity other than the CN operator.

[0055] The MME 162 may be connected to each of the eNode-Bs 162a, 162b, 162c in the RAN 104 via an S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, performing bearer activation / deactivation processing, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, and the like. The MME 162 may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM and / or WCDMA.

[0056] The SGW 164 may be connected to each of the eNode-Bs 160a, 160b, 160c in the RAN 104 via an S1 interface. The SGW 164 may generally route and forward user data packets to / from the WTRUs 102a, 102b, 102c. The SGW 164 may also perform other functions, such as anchoring the user plane during an inter-eNB handover, triggering paging processing when DL data is available for the WTRUs 102a, 102b, 102c, and managing and storing the contexts of the WTRUs 102a, 102b, 102c, and the like.

[0057] The SGW 164 may be connected to the PGW 146, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0058] The CN 106 may facilitate communications with other networks. For example, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the CN 106 may include or communicate with an IP gateway, such as an IP Multimedia Subsystem (IMS) server, and the IP gateway may serve as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers.

[0059] Although in Figures 1A-1D The WTRU is described as a wireless terminal, however it should be appreciated that in some examples such a terminal may use a (eg, temporary or permanent) wired communication interface with a communication network.

[0060] In an example, the other network 112 may be a WLAN.

[0061] A WLAN using an infrastructure basic service set (BSS) mode may have an access point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP may access or be connected to a distributed system (DS) or other types of wired / wireless networks that send traffic into and / or out of the BSS. Traffic originating from outside the BSS and destined for the STA may arrive through the AP and be delivered to the STA. Traffic originating from the STA and destined for a destination outside the BSS may be sent to the AP for delivery to the corresponding destination. Traffic between STAs within the BSS may be sent through the AP, for example, when the source STA may send traffic to the AP and the AP may deliver traffic to the destination STA. Traffic between STAs within the BSS may be considered and / or referred to as point-to-point traffic. The point-to-point traffic may be sent using a direct link establishment (DLS) between the source and destination STAs (e.g., directly therebetween). In an example, the DLS may use 802.11eDLS or 802.11z channelized DLS (TDLS)). For example, a WLAN using an independent BSS (IBSS) mode does not have an AP, and STAs (eg, all STAs) within or using the IBSS can communicate directly with each other. Here, the IBSS communication mode may sometimes be referred to as an "ad-hoc" communication mode.

[0062] When using the 802.11ac infrastructure mode of operation or a similar mode of operation, the AP may transmit a beacon on a fixed channel (e.g., a primary channel). The primary channel may have a fixed width (e.g., a bandwidth of 20 MHz) or a width dynamically set via signaling. The primary channel may be an operating channel of the BSS and may be used by the STA to establish a connection with the AP. In an example, carrier sense multiple access (CSMA / CA) with collision avoidance (e.g., in an 802.11 system) may be implemented. For CSMA / CA, STAs including the AP (e.g., each STA) may sense the primary channel. If a particular STA senses / detects and / or determines that the primary channel is busy, the particular STA may back off. In a specified BSS, there is one STA (e.g., only one station) transmitting at any given time.

[0063] A high throughput (HT) STA may communicate using a 40 MHz wide channel (eg, by combining a 20 MHz wide primary channel with a 20 MHz wide adjacent or non-adjacent channel to form a 40 MHz wide channel).

[0064] Very high throughput (VHT) STA can support channels with widths of 20MHz, 40MHz, 80MHz and / or 160MHz. 40MHz and / or 80MHz channels can be formed by combining continuous 20MHz channels. A 160MHz channel can be formed by combining 8 continuous 20MHz channels or by combining two discontinuous 80MHz channels (this combination can be referred to as an 80+80 configuration). For the 80+80 configuration, after channel coding, the data can be transmitted and passed through a segment parser, which can separate the data into two streams. Inverse Fast Fourier Transform (IFFT) processing and time domain processing can be performed separately on each stream. The stream can be mapped on two 80MHz channels, and the data can be transmitted by the STA performing the transmission. At the receiver of the STA performing the reception, the above operations for the 80+80 configuration can be reversed, and the combined data can be sent to the medium access control (MAC).

[0065] 802.11af and 802.11ah support working modes below 1 GHz. Compared with 802.11n and 802.11ac, the channel working bandwidth and carrier used in 802.11af and 802.11ah are reduced. 802.11af supports 5MHz, 10MHz and 20MHz bandwidths in the TV White Space (TVWS) spectrum, and 802.11ah supports 1MHz, 2MHz, 4MHz, 8MHz and 16MHz bandwidths using non-TVWS spectrum. According to an example, 802.11ah can support meter type control / machine type communication (MTC) (e.g., MTC devices in macro coverage areas). MTC devices can have certain capabilities, such as limited capabilities including support (e.g., only support) certain and / or limited bandwidths. MTC devices can include a battery, and the battery life of the battery is higher than a threshold (e.g., for maintaining a very long battery life).

[0066] For WLAN systems that can support multiple channels and channel bandwidths (e.g., 802.11n, 802.11ac, 802.11af, and 802.11ah), these systems include channels that can be designated as primary channels. The bandwidth of the primary channel can be equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel can be set and / or limited by a certain STA, where the STA originates from all STAs operating in the BSS that supports the minimum bandwidth operating mode. In the example of 802.11ah, even if the AP and other STAs in the BSS support 2MHz, 4MHz, 8MHz, 16MHz, and / or other channel bandwidth operating modes, for STAs (e.g., MTC-type devices) that support (e.g., only support) 1MHz mode, the width of the primary channel can be 1MHz. Carrier sensing and / or network allocation vector (NAV) settings can depend on the status of the primary channel. If the primary channel is busy (e.g., because a STA (which only supports the 1MHz operating mode) transmits to the AP), then even if most of the available frequency bands remain idle and available for use, the entire available frequency band can be considered busy.

[0067] In the United States, the available frequency band for 802.11ah is 902MHz to 928MHz. In South Korea, the available frequency band is 917.5MHz to 923.5MHz. In Japan, the available frequency band is 916.5MHz to 927.5MHz. Depending on the country code, the total bandwidth available for 802.11ah is 6MHz to 26MHz.

[0068] Figure 1D1 is a system diagram showing an exemplary RAN 113 and CN 115. As described above, the RAN 113 may communicate with the WTRUs 102a, 102b, 102c using NR radio technology over the air interface 116. The RAN 113 may also communicate with the CN 115.

[0069] The RAN 113 may include gNBs 180a, 180b, 180c, though it will be appreciated that the RAN 113 may include any number of gNBs. The gNBs 180a, 180b, 180c may each include one or more transceivers to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. In an example, the gNBs 180a, 180b, 180c may implement MIMO technology. For example, the gNBs 180a, 180b may use beamforming processing to transmit and / or receive signals to and / or from the gNBs 180a, 180b, 180c. Thus, for example, the gNB 180a may use multiple antennas to transmit wireless signals to the WTRU 102a and receive wireless signals from the WTRU 102a. In an example, the gNBs 180a, 180b, 180c may implement carrier aggregation technology. For example, the gNB 180a may transmit a plurality of component carriers to the WTRU 102a (not shown). A subset of these component carriers may be on an unlicensed spectrum, while the remaining component carriers may be on a licensed spectrum. In an example, the gNBs 180a, 180b, 180c may implement coordinated multi-point (CoMP) technology. For example, the WTRU 102a may receive coordinated transmissions from the gNB 180a and the gNB 180b (and / or the gNB 180c).

[0070] The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using transmissions associated with a scalable digital configuration. For example, the OFDM symbol spacing and / or OFDM subcarrier spacing may be different for different transmissions, different cells, and / or different portions of the radio transmission spectrum. The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using subframes or transmission time intervals (TTIs) of different or scalable lengths (e.g., containing different numbers of OFDM symbols and / or lasting different absolute time lengths).

[0071] The gNBs 180a, 180b, 180c may be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and / or a non-standalone configuration. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c without accessing other RANs (e.g., the eNodeBs 160a, 160b, 160c). In a standalone configuration, the WTRUs 102a, 102b, 102c may use one or more of the gNBs 180a, 180b, 180c as mobility anchors. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using signals in an unlicensed band. In a non-standalone configuration, the WTRU 102a, 102b, 102c may communicate / connect to the gNB 180a, 180b, 180c while communicating / connecting to another RAN (e.g., the eNode-B 160a, 160b, 160c). For example, the WTRU 102a, 102b, 102c may communicate with one or more gNBs 180a, 180b, 180c and one or more eNode-Bs 160a, 160b, 160c in a substantially simultaneous manner by implementing the DC principle. In a non-standalone configuration, the eNode-B 160a, 160b, 160c may act as a mobility anchor for the WTRU 102a, 102b, 102c, and the gNB 180a, 180b, 180c may provide additional coverage and / or throughput to serve the WTRU 102a, 102b, 102c.

[0072] Each of the gNBs 180a, 180b, 180c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, user scheduling in UL and / or DL, support network slicing, dual connectivity, implement interworking between NR and E-UTRA, route user plane data to a user plane function (UPF) 184a, 184b, and route control plane information to an access and mobility management function (AMF) 182a, 182b, and the like. Figure 1D As shown, gNBs 180a, 180b, and 180c can communicate with each other through the Xn interface.

[0073] Figure 1DThe CN 115 shown may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one session management function (SMF) 183a, 183b, and may include a data network (DN) 185a, 185b. Although each of the aforementioned components is described as part of the CN 115, it should be understood that any of these components may be owned and / or operated by an entity other than the CN operator.

[0074] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via the N2 interface and may act as a control node. For example, the AMF 182a, 182b may be responsible for authenticating users of the WTRU 102a, 102b, 102c, supporting network slicing (e.g., handling different PDU sessions with different requirements), selecting a specific SMF 183a, 183b, managing registration areas, terminating NAS signaling, and mobility management, etc. The AMF 182a, 182b may use network slicing processing to customize the CN support provided to the WTRU 102a, 102b, 102c based on the type of service used by the WTRU 102a, 102b, 102c. As an example, different network slices may be established for different use cases, such as services relying on Ultra Reliable Low Latency (URLLC) access, services relying on enhanced massive mobile broadband (eMBB) access, and / or services for machine type communication (MTC) access, etc. The AMF 182 may provide a control plane function for switching between the RAN 113 and other RANs (not shown) using other radio technologies (e.g., LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies such as WiFi).

[0075] The SMF 183a, 183b may be connected to the AMF 182a, 182b in the CN 115 via the N11 interface. The SMF 183a, 183b may also be connected to the UPF 184a, 184b in the CN 115 via the N4 interface. The SMF 183a, 183b may select and control the UPF 184a, 184b, and may configure traffic routing through the UPF 184a, 184b. The SMF 183a, 183b may perform other functions such as managing and allocating WTRU or UE IP addresses, managing PDU sessions, controlling policy enforcement and QoS, and providing downlink data notifications, etc. The PDU session type may be IP-based, non-IP-based, Ethernet-based, etc.

[0076] The UPF 184a, 184b can be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via the N3 interface, which can provide the WTRUs 102a, 102b, 102c with access to packet-switched networks (e.g., the Internet 110) to facilitate communication between the WTRUs 102a, 102b, 102c and IP-enabled devices. The UPF 184, 184b can perform other functions, such as routing and forwarding packets, implementing user plane policies, supporting multi-host PDU sessions, processing user plane QoS, buffering downlink packets, and providing mobility anchor processing, etc.

[0077] The CN 115 may facilitate communications with other networks. For example, the CN 115 may include or may communicate with an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that acts as an interface between the CN 115 and the PSTN 108. In addition, the CN 115 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers. In one example, the WTRUs 102a, 102b, 102c may be connected to the local data networks (DNs) 185a, 185b via an N3 interface connected to the UPFs 184a, 184b and an N6 interface between the UPFs 184a, 184b and the DNs 185a, 185b and through the UPFs 184a, 184b.

[0078] In view of Figures 1A-1D and about Figures 1A-1D In accordance with the corresponding description of the present invention, one or more or all functions described herein in comparison with one or more of the following may be performed by one or more simulation devices (not shown): WTRU 102a-d, base station 114a-b, eNodeB 160a-c, MME 162, SGW 164, PGW 166, gNB 180a-c, AMF 182a-b, UPF 184a-b, SMF 183a-b, DN185ab and / or one or more other devices described herein. These simulation devices may be one or more devices configured to simulate one or more or all functions described herein. For example, these simulation devices may be used to test other devices and / or simulate network and / or WTRU functions.

[0079] The simulation device can be designed to implement one or more tests about other devices in a laboratory environment and / or an operator network environment. For example, the one or more simulation devices can perform one or more or all functions while being implemented and / or deployed as a part of a wired and / or wireless communication network in whole or in part to test other devices inside the communication network. The one or more simulation devices can perform one or more or all functions while being temporarily implemented or deployed as a part of a wired and / or wireless communication network. The simulation device can be directly coupled to other equipment to perform the test, and / or can use over-the-air wireless communication to perform the test.

[0080] The one or more simulation devices can perform one or more functions, including all functions, while not being implemented or deployed as part of a wired and / or wireless communication network. For example, the simulation device can be used in a test lab and / or a test scenario of a wired and / or wireless communication network that is not deployed (e.g., tested) to implement tests on one or more components. The one or more simulation devices can be test devices. The simulation device can transmit and / or receive data using direct RF coupling and / or wireless communication with the aid of RF circuits (e.g., which can include one or more antennas).

[0081] Video coding systems can compress digital video signals, for example, to reduce storage space and / or transmission bandwidth associated with storing and / or transmitting such signals. Video coding systems can include block-based systems, wavelet-based systems, object-based systems, and the like.

[0082] The video decoding device may be based on a block-based hybrid video decoding framework. A block partition structure based on multiple types of trees may be adopted. One or more decoding modules may be included, such as an intra-frame prediction module, an inter-frame prediction module, a transform / inverse transform module, and a quantization / dequantization module. The video decoding device may also include an in-loop filter.

[0083] The video decoding device may include one or more decoding tools, such as 65° angle intra-frame prediction direction, modified coefficient decoding, advanced multiple transform (AMT) + 4×4 non-separable secondary transform (NSST), affine motion model, generalized adaptive loop filter (GALF), advanced temporal motion vector prediction (ATMVP), adaptive motion vector precision, decoder-side motion vector refinement (DMVR) and / or linear model (LM) chroma mode.

[0084] An exemplary block-based video coding system may include a block-based hybrid video coding framework. Figure 2 An exemplary block-based hybrid video coding framework 200 for an encoder is shown. Figure 2As shown, the input video signal 202 can be processed block by block. The block can be called a decoding unit (CU). The CU can also be called a video block. For example, the size of a CU can be up to 128×128 pixels. In the decoding framework, the CU can be partitioned into prediction units (PUs) and / or separate predictions can be used. In the decoding framework, the CU can be used as a basic unit for both prediction and transformation without further partitioning. The CTU can be partitioned into multiple CUs, for example to adapt to changing local characteristics based on a quad / bin / ternary tree structure. In a multi-type tree structure, the CTU can be partitioned by a quadtree structure. The quadtree leaf nodes can be further partitioned by binary and ternary tree structures. As Figure 4 As described in , one or more partition types may be provided, including, for example, quaternary partitioning ( Figure 4 (a)), horizontal binary partitioning ( Figure 4 (c)), vertical binary partitioning ( Figure 4 (b)), vertical ternary partitioning ( Figure 4 (d)) and horizontal ternary partitioning ( Figure 4 (e)).

[0085] like Figure 2 , for an input video block (e.g., a macroblock (MB) and / or a CU), spatial prediction 260 and / or temporal prediction 262 may be performed. Spatial prediction 260 (e.g., intra prediction) may use pixels of samples (e.g., reference samples) from coded neighboring blocks in a video picture / slice to predict a current video block. The spatial prediction 260 may reduce spatial redundancy that may be inherent in, for example, a video signal. Motion prediction 262 (e.g., inter prediction and / or temporal prediction) may use reconstructed pixels (e.g., from a coded video picture) to predict a current video block. The motion prediction 262 may reduce temporal redundancy that may be inherent in, for example, a video signal. A motion prediction signal (e.g., a temporal prediction signal) for a video block (e.g., a CU) may be signaled by one or more motion vectors (MVs). The MV may indicate an amount and / or direction of motion between a current block and / or a reference block of the current block or its temporal reference. If multiple reference pictures are supported for a (e.g., each) video block, a reference picture index for the video block may be sent by the encoder. The reference picture index may be used to identify from which reference picture in the reference picture store 264 the motion prediction signal may be derived.

[0086] After spatial prediction 260 and / or motion prediction 262, a mode decision block 280 in the encoder may determine a prediction mode (e.g., an optimal prediction mode), e.g., based on rate-distortion optimization. At 216, the prediction block may be subtracted from the current video block, and / or the prediction residual may be decorrelated using transform 204 and / or quantization 206 to achieve a bit rate, such as a target bit rate. The quantized residual coefficients may be inverse quantized at inverse quantization 210 and / or inverse transformed at transform 212, e.g., to form a reconstructed residual, which may be added to the prediction block at 226, e.g., to form a reconstructed video block. In-loop filtering (e.g., a deblocking filter and / or an adaptive loop filter) may be applied to the reconstructed video block at loop filter 266 before the reconstructed video block may be placed in a reference picture store 264 and / or used to decode a video block (e.g., a future video block). To form the output video bitstream 220, the coding mode (e.g., inter or intra), prediction mode information, motion information, and / or quantized residual coefficients may be sent (e.g., may all be sent) to the entropy decoding module 208, e.g., to be compressed and / or packed to form a bitstream.

[0087] Figure 3 A block diagram of an exemplary block-based video decoding framework for a decoder is shown. A video bitstream 302 (e.g., Figure 2 The video bitstream 220 in (e.g., first unpacked) may be unpacked (e.g., first unpacked) and / or entropy decoded at the entropy decoding module 308. The coding mode and prediction information may be sent to the spatial prediction module 360 ​​(e.g., if intra-coded) and / or the motion compensated prediction module 362 (e.g., if inter-coded and / or temporal coded) to form a prediction block. The residual transform coefficients may be sent to the inverse quantization module 310 and / or the inverse transform module 312, for example, to reconstruct the residual block. The prediction block and / or residual block may be added together at 326. The reconstructed block may be subjected to in-loop filtering at a loop filter 366 before, for example, storing the reconstructed block in a reference picture repository 364. The reconstructed video 320 in the reference picture repository 364 may be sent to drive a display device and / or used to predict a video block (e.g., a future video block).

[0088] As described herein, affine motion compensation may be used as an inter-coding tool.

[0089] As described herein, various affine modes and affine motion models for video coding may be used. A translational motion model may be applied to motion compensated prediction. Various motions may exist (e.g., zooming in or out, rotation, perspective motion, and / or other irregular motions). Affine transform (e.g., simplified affine transform) motion compensated prediction may be applied to prediction. A flag for an inter-coded CU (e.g., each inter-coded CU) may be signaled, for example, to indicate whether a translational motion or affine motion model is applied to inter prediction.

[0090] The simplified affine motion model may be a four-parameter model. Of the four parameters of the model, two parameters may be used for translational movement (e.g., in the horizontal and vertical directions), one parameter may be used for scaling movement, and one parameter may be used for rotational movement. The horizontal scaling parameter value may be equal to the vertical scaling parameter value. The horizontal rotation parameter value may be equal to the vertical rotation parameter value. The four-parameter motion model may be coded using two motion vectors as a pair of motion vectors at two control point locations (e.g., at the upper left corner location and the upper right corner location of the current video block or current CU). Figure 5 As described in, the affine motion field of a CU or block can be composed of two control point motion vectors (e.g., ) description. Based on the motion of the control points, the motion field (v x ,v y ) can be determined as follows: Where (v 0x ,v 0y ) can be the motion vector of the upper left control point, and (v 1x ,v 1y ) may be the motion vector of the upper right control point. When a block is decoded in affine mode, its motion field may be derived based on, for example, the granularity of a sub-block. For example, the motion vector of a sub-block (e.g., each sub-block) may be derived by calculating the motion vector of the center sample of the sub-block using equation (1). The motion vector may be rounded to an accuracy value (e.g., 1 / 16 pixel accuracy). The derived motion vector may be used in the motion compensation stage to generate a prediction signal for a sub-block (e.g., each sub-block) inside the current block. The size of the sub-block applied for affine motion compensation may be calculated using the following equation: Where (v 2x ,v 2y ) may be the motion vector of the lower left control point, w and h may be the CU width and CU height, for example, as calculated by equation (1), and M and N may be the width and height of the derived sub-block size.

[0091] Affine merge mode coding can be used to code a CU. Predictive coding can be used to signal two sets of motion vectors associated with two control points of each reference picture list. Affine merge mode can be applied, and the difference between a motion vector and its predictor can be coded using a lossless coding scheme. A potentially significant (e.g., at a low bit rate) signaling overhead can be signaled. Affine merge mode can be applied to reduce signaling overhead, for example, by considering local continuity of a motion field. Motion vectors at two control points of the current CU can be derived. The motion vector of the current CU can be derived using the affine motion of the affine merge candidate of the CU, which can be selected from its neighboring blocks.

[0092] like Figure 6 As illustrated in FIG. 1 , for example, a current CU coded in affine merge mode may have five neighboring blocks (N0 to N4). The neighboring blocks may be checked in the order of N0 to N4—N0, N1, N2, N3, N4. The first affine coded neighboring block may be used as an affine merge candidate. Figure 7 As described in , the current CU may be coded in affine merge mode. The lower left neighboring block (e.g., N0) of the current CU may be selected as an affine merge candidate. The lower left neighboring block N0 may belong to a neighboring CU, CU0. The width and height of the CU including block N0 may be denoted as nw and nh. The width and height of the current CU may be denoted as cw and ch. At position P i The MV of can be expressed as (v ix ,v iy ). The MV (v 0x ,v 0y ) can be derived according to the following equation: V(v 1x ,v 1y ) can be derived according to the following equation: The MV(v 2x ,v 2y ) can be derived according to the following equation: Once the MVs at two control points (e.g., P0 and P1) are determined, the MVs of sub-blocks (e.g., each sub-block) in the current CU can be derived. The derived MVs of the sub-blocks can be used for sub-block-based motion compensation and temporal motion vector prediction for future picture decoding.

[0093] Affine MV prediction can be performed. For non-merged affine coded CUs, signaling of MVs at control points can be associated with high signaling costs. Predictive coding can be used to reduce signaling overhead. Affine MV predictors can be generated from the motion of its neighboring coded blocks. For MV prediction of affine coded CUs, various predictors can be supported. For example, affine motion predictors generated from neighboring blocks of control points and / or translational motion predictors for MV prediction. The translational motion predictor can be used as a supplement to the affine motion predictor.

[0094] Multiple MV sets can be obtained and used to generate multiple affine motion predictors. Figure 8 As shown, the MV set may include the following: MVs of neighboring blocks {A, B, C} from corner P0, which may include set S1 and may be expressed as {MV A ,MV B ,MV C}; The MV of the neighboring block {D, E} from the corner P1, which may include the set S2 and may be represented as {MV D ,MV E}; and / or the MV of the neighboring block {F, G} from the corner P2, which may include the set S3 and may be represented as {MV F ,MV G}. The MV from the neighboring block can be derived as follows. For example, it can be checked whether the spatial neighboring block is an inter-frame decoding block. The MV can be used directly. The reference picture of the neighboring block can be the same as the reference picture of the current CU. For example, if the reference picture of the neighboring block is different from the reference picture of the current CU, the MV can be scaled according to the temporal distance. Fig. 9 As shown, the temporal distance between the current picture 902 of the current CU and the reference picture 904 can be represented as TB. The temporal distance between the current picture 902 and the reference picture of the neighboring block 906 can be represented as TD. The MV1 of the neighboring block can be scaled using: Among them, MV2 can be used for motion vector set.

[0095] For example, if the neighboring block is not an inter-coded block, the collocated block in the collocated reference picture may be checked. For example, if the temporally collocated block is an inter-coded block, the MV may be scaled according to equation (9) based on the temporal distance. For example, if the temporally collocated block is not an inter-coded block, the MV in the neighboring block may be set to zero.

[0096] The affine MV predictor can be generated by selecting an MV from a set of MVs. For example, there can be three MV sets, such as S1, S2, and S3. The sizes of S1, S2, and S3 can be 3, 2, and 2, respectively. In such an example, there can be 12 (e.g., 3×2×2) possible combinations. For example, if the magnitude of the scaling or rotation-related parameters represented by one or more MVs is greater than a threshold, the candidate MV can be discarded. The threshold can be predefined. For three corners of the CU (e.g., top left, top right, and bottom left), the combination can be expressed as (MV0, MV1, MV2). The conditional MV can be checked as follows: Where T may be 1 / 2. For example, if the condition is met (eg, the scaling or rotation is too large), the candidate MV may be discarded.

[0097] The remaining candidates can be sorted. A triple of three MVs can represent a 6-parameter motion model (e.g., including translation, scaling, and rotation in the horizontal and vertical directions). The sorting criterion can be the difference between the 6-parameter motion model and the 4-parameter motion model represented by (MV0, MV1). Candidates with smaller differences can have smaller indexes in the sorted candidate list. The difference between the affine motion represented by (MV0, MV1, MV2) and the affine motion represented by (MV0, MV1) can be estimated according to the following equation:

[0098] Affine motion models can be used to increase coding efficiency. For example, MVs at two control points may be signaled for large CUs. Motion vectors of sub-blocks within a CU may be interpolated. The motion of a sub-block (e.g., each sub-block) may be different, for example, due to scaling or rotational movement. When an affine motion model is selected for a coding block instead of a translational motion model, the control points may be fixed, for example, in the affine motion model. The control points used may be fixed, such as the upper left and upper right corners of the coding block. The motion vector accuracy of the affine MV may be fixed (e.g., ¼ pixel). When the vertical position y of a sub-block is greater than the block width (w) in equation (1), magnification (y / w) may be utilized.

[0099] When using affine merge mode, the first available neighboring block from {N0, N1, N2, N3, N4} may not be the best neighboring block. From the affine MV derived from the merge candidate (e.g., as provided in equations () to (), the accuracy may be related to the width of the merge candidate (e.g., as shown by "nw" in equations () to (). The first affine merge candidate may not have the best accuracy for affine MV derivation. In affine MV prediction, candidates with large scaling or rotation may be discarded based on the conditional check of equation (10). The discarded candidates may be added back to the list.

[0100] Systems, methods, and means for decoding based on affine motion models may be disclosed herein. As disclosed herein, affine motion decoding based on adaptive control point selection may be used. In affine motion decoding based on adaptive control point selection, control point positions may be adaptively selected based on the shape of the block. For example, one or more control points may be selected based on whether the block is a horizontal rectangular block, a vertical rectangular block, or a square block. Affine merge candidates may be selected from adjacent blocks based on the distance between two control points. For example, the affine merge candidate with the highest control point distance may be selected. Affine predictor generation may be performed such that candidates with large scaling or rotational movement may be placed at the back of the predictor list.

[0101] Affine motion based coding with adaptive control point selection may be used. For a video block coded in affine mode, for example, the upper left corner and the upper right corner of the video block may be used as control points. The motion of a sub-block (e.g., each sub-block associated with the video block) may be derived using the MV at the two control points (e.g., based on equation (1)). The derivation accuracy may be related to the block width (e.g., the distance between the two control points). Some sub-blocks may be far away from the two control points (e.g., as determined by Fig.10 Therefore, the motion obtained using the MVs at P0 and P1 may be affected.

[0102] Shape-dependent control point selection may be performed. Video blocks may be classified into several categories, such as horizontal rectangular blocks, vertical rectangular blocks, or square blocks. For example, a block may be classified as a horizontal rectangle if its width is greater than its height. The control points of a horizontal rectangular block may be defined by the upper left corner (e.g., P0) and the upper right corner (e.g., P1), for example, as Fig.10 If the width of the block is less than the height, the block can be classified as a vertical rectangle. The control point of the vertical rectangular block can be the upper left corner (e.g., P0) and a lower left corner (e.g., P2), for example, as defined by Fig.10 For example, if the width of the block is equal to the height, the block can be classified as a square block. Fig.10 As shown in box 1030 of , the control points of the square block can be defined by the upper right corner (e.g., P1) and the lower left corner (e.g., P2).

[0103] For a horizontal rectangular block, control points P0 and P1 may be used. The MV of a sub-block of a horizontal rectangular block may be derived based on equation (1).

[0104] For a vertical rectangular block, control points P0 and P2 can be used. The MV of a sub-block of a vertical rectangular block can be derived as follows. Assume that the position of the center of the sub-block relative to the upper left corner of the block is represented by (x, y), and the MV of the sub-block centered at (x, y) is represented by (v x ,v y ). In addition, suppose the block width is denoted as w, the block height is denoted as h, and the MV at P0 and P2 is denoted as (v 0x ,v 0y ),(v 2x ,v 2y ), then the MV of the sub-block of the horizontal rectangular block centered at (x, y) is derived as follows:

[0105] For a square block, control points P1 and P2 can be used. The MV of the sub-block belonging to the square block can be derived as follows: v x =v 1x +a*(xw)-b*y (14) v y =v 1y +b*(xw)+a*y (15) where a and b can be calculated as follows: a=(-(v 2x -v 1x )*w+(v 2y -v 1y )*h) / (w*w+h*h) (16) b=(-(v 2x -v 1x )*h-(v 2y -v 1y )*w) / (w*w+h*h) (17). Assuming that w is equal to h in the case of square blocks, a and b can be simplified as follows: a=(-(v 2x -v 1x )+(v 2y -v 1y )) / (2w) (18) b=(-(v 2x -v 1x )-(v 2y -v 1y )) / (2w) (19).

[0106] A mode indicating the control point selection for an affine coded CU may be signaled. For example, for an affine coded CU, a mode indicating which control points are being used may be signaled. For example, the mode may indicate that control points P0, P1 are being used in the horizontal direction, or control points P0, P2 are being used in the vertical direction, or control points P1, P2 are being used in the diagonal direction. The control point mode may be determined based on motion estimation cost or rate distortion cost. For example, for a video block (e.g., each block), the encoder may try affine motion estimation using different control point selection modes to obtain prediction errors for each possible control point selection. The encoder may select the mode with the lowest motion estimation cost, for example, by adding the motion prediction distortion and the control point MV bit cost.

[0107] Affine merge candidate selection with maximum control point distance may be used. The MV at the control point of the current video block may be derived from the MV of the merge candidate using equations (3)-(8). The accuracy of motion vector derivation may depend on the distance between two control points of its neighboring blocks. The distance between two control points may be the width of the block. In shape-dependent control point selection as described herein, the distance between two control points may be measured based on the block shape. The square of the distance between two control points may be used to obtain the distance from neighboring blocks (e.g., {N0, N1, N2, N3, N4}, as Figure 6 For example, if the distance is large, the accuracy of the motion derivation of the current block may be higher. The affine merge candidate with the largest control point distance may be selected to derive the MV, for example, as Fig.11 As shown. The affine merge candidates in the candidate list (e.g., all affine merge candidates) may be checked sequentially, for example, as Figure 6 As shown. Fig.11 As shown, at 1102, a candidate neighboring block Nk may be selected from a list of available neighboring blocks. At 1104, an affine mode of the selected neighboring block Nk may be checked. After checking that the affine mode is enabled for the selected neighboring block Nk, at 1106, a distance D between two control points may be calculated. The distance D may be calculated based on the block shape. At 1106, a merge candidate with a maximum control point distance may be selected as an affine merge candidate for the current block to derive an MV at the control point of the current block. At 1110, it is checked that all candidates have been evaluated.

[0108] An affine merge index may be signaled. The distance between two control points may be used to sort the available merge candidates in a merge candidate list. A final affine merge candidate may be derived in the following manner. Available affine merge candidates (e.g., all available affine merge candidates) may be obtained from neighboring blocks. The distance between two control points may be calculated for the candidates in the list (e.g., each candidate). The affine merge candidate list may be sorted, for example, in descending order of control point distances. A final affine merge candidate may be selected from the sorted list using a merge index signaled for a coded block.

[0109] Affine MV prediction can be performed as described herein. Candidate sorting in affine MV predictor generation can be performed, for example, by checking condition (10) or using a criterion as provided in the following equation: If the scaling or rotational movement is too large, A1 and A2 can be adjusted. A1 and A2 can be calculated using the following equation: Where T1 and T2 can be predefined thresholds (e.g., T1=3, T2=1 / 4), and w and h can be the width and height of the decoding block. Using the sorting criteria provided in equation (20), candidates with large scaling or rotational movement can be placed at the back of the predictor list.

[0110] A unified control point MV for affine motion compensation, motion vector prediction, and / or deblocking may be used. As described herein, when affine mode is enabled, a CU may be divided into several sub-blocks of equal size (e.g., 4×4 sub-blocks). A sub-block (e.g., each sub-block) may be assigned an MV (e.g., a unique MV) that may be derived using an affine mode. For example, the affine mode may be a 4-parameter affine mode or a 6-parameter affine mode. The affine mode may be signaled at the CU level. The center position of a sub-block (e.g., each sub-block) may be used to derive the corresponding MV of the sub-block based on the selected affine mode. MV of (i, j) sub-block The three control points MVv0, v1 and v2 at the top left, top right and bottom left corners of the affine CU can be derived as follows: Where (i, j) can be the horizontal index and vertical index of the sub-block in the CU; sb and h sbIt can be the width and height of (e.g., one) sub-block (e.g., it can be equal to 4). The CU may have one or more sub-blocks that do not include the control point position. For example, the upper left and upper right positions of the 4-parameter affine mode, and the upper left, upper right, and lower left positions of the 6-parameter affine mode may not include the control point position. The MV in this case can be calculated as provided in equation (23). These MVs can be used to generate prediction samples of sub-blocks during motion compensation. The MVs can be used to predict the MVs of spatial and temporal neighboring blocks of the CU. The MVs can be used to calculate boundary strength values ​​for deblocking filters. For the sub-block located at the control point position, its MV can be used as a seed to derive the control point MV of its neighboring blocks through the affine merge mode. In order to maintain the MV accuracy of the affine merge mode, the MV in equation (23) can be used for control point sub-blocks (e.g., each control point sub-block) during motion compensation. For spatial / temporal MV prediction and deblocking, their MVs can be replaced by corresponding control point MVs. For example, for a CU coded by a 4-parameter affine model, the MVs of its top-left and top-right sub-blocks that can be used for MV prediction and deblocking can be calculated as follows:

[0111] For a CU coded in 6-parameter affine mode, the MVs of the top-left, top-right, and bottom-left sub-blocks that can be used for MV prediction and / or deblocking can be calculated as follows:

[0112] Fig.12 An example of generating a motion field for a CU that can be coded using an affine mode is shown. Fig.12 The workflow shown in can maintain the MV accuracy of affine motion compensation and MV prediction. Fig.12 The workflow shown in can be used in many ways. For example, for a sub-block including a control point position of a CU (e.g., each sub-block associated with the CU), one or more different MVs can be derived and / or stored. In one example, the MV can be derived based on equation (23) and can be used to generate prediction samples for the sub-block. In an example, the MV can be derived based on equations (24) and (25) and can be used for MV prediction and deblocking.

[0113] For a sub-block (e.g., each sub-block) at a control point position, its MV may be set (e.g., initially set) to the corresponding control point MV. The MV may be set to the corresponding control point MV to derive the MV of its neighboring blocks in the parsing stage. In the motion compensation stage, the MV of the sub-block may be recalculated by using the center position as input to the selected affine model. One or more different MVs may be stored for the sub-block at the control point position (e.g., each control point position). The MV of the sub-block at the control point position (e.g., each control point position) may be derived twice.

[0114] The motion fields used at different coding processes for CUs coded by affine mode may be unified. Fig.13 As illustrated in FIG. 2 , the MV used for spatial / temporal MV prediction and deblocking (e.g., as indicated by equations (24) and (25)) can be reused to generate prediction samples for control point subblocks within an affine CU. For a subblock located at a control point position of an affine CU, an MV derived based on the center position of the subblock (e.g., according to equation (23)) can be reused in the motion compensation stage. The MV can be an MV predictor used for spatial / temporal MV prediction. The MV can be used to calculate the boundary strength of the deblocking process. Fig.14 The workflow for deriving the motion field of an affine CU is shown.

[0115] Motion vector clipping may be used. For example, when an affine mode associated with a video block or CU is enabled, motion vector clipping may be used. When affine mode is enabled, the CU may be divided into one or more sub-blocks. The sub-blocks associated with the CU may be equal in size (e.g., 4×4). An MV may be assigned to a sub-block associated with the CU. For example, the MV assigned to each MV of the CU may be a unique MV. For example, the assigned MV may be derived using a 4-parameter affine mode or a 6-parameter affine mode. The type of affine mode (4-parameter affine mode or 6-parameter affine mode) may be signaled at the CU level. The derived MV associated with the CU may be stored in a motion field and may be represented using a limited bit depth (e.g., 16 bits in VVC). When deriving a sub-block MV, the value of the calculated MV may be outside the range of values ​​that can be represented based on the motion field bit depth. The calculated MV may cause arithmetic underflow and / or overflow problems outside the range of values. This underflow and / or overflow problem may occur even when the control point MV is within the range specified by the motion field bit depth. For example, after the MV is derived, the MV may be cropped. Cropping the MV may result in similar behavior between various systems that may use varying bit depth values. For example, a video encoding device may use a bit depth value that may be higher than the bit depth value used by a video decoding device, or vice versa.

[0116] The MV of sub-block (i, j) can be clipped according to the following equation 26 where N may be the bit depth used for motion field storage (eg, N=16). As indicated in equation (26), the MV of sub-block (i, j) may be clipped based on the motion field range The motion field range may be a motion field storage bit depth (MFSBD) value. The MFSBD may be represented by the number of bits (eg, 16 bits, 18 bits).

[0117] One or more control point MVs may be clipped based on a bit depth value that may be the same as the bit depth value used for motion field storage. The control point MVs may be clipped. For example, the control point MVs may be clipped after sub-block MV derivation. The control point MVs may be clipped to maintain the precision of the derived MVs. The control point MVs may have a precision higher than the motion field storage bit depth. For example, the control point MVs used for sub-block derivation may have a higher precision (e.g., may have more bits) than the range of values ​​that can be represented taking into account the motion field storage bit depth. The control point MVs may be clipped and stored for affine merging derivation of adjacent blocks. For example, the control point MVs may be clipped and stored after derivation.

[0118] Various mechanisms can be used to derive sub-block MVs. For example, planar motion vector prediction and / or regression-based motion vector fields can be used. The MV associated with each sub-block in a CU can be derived from the MVs of neighboring blocks of the CU. For example, the MV associated with each sub-block can be derived based on the control point MVs of neighboring blocks of the CU. The derived sub-block MVs can be stored in a motion field for future decoding. The derived MVs can be clipped based on the value of the motion field storage bit depth to avoid overflow and / or underflow problems.

[0119] The control point MV and / or sub-block MV of an affine-coded CU may be used for MV prediction, for example, when predicting neighboring blocks. The reference area pointed to by the MV may be outside the picture boundary and / or may be far from the picture boundary, for example, this may occur even when the MV is cropped based on the motion field storage bit depth.

[0120] The affine control point MV and / or the affine sub-block MV may be clipped within a range of values. The range value may be specified by a picture boundary plus a margin to allow a portion of the sub-block to be outside the picture when deriving the MV of the affine decoding CU. For example, the control point MV may be clipped and / or scaled so that the reference block obtained after affine motion compensation is bounded by the picture boundary (e.g., taking into account the additional margin). The sub-block MV may be clipped so that the reference sub-block obtained after motion compensation overlaps the reference picture by at least one sample.

[0121] Fig.15 An example of modifying one or more control points MV to scale the reference block is shown. Fig.15 As shown, one or more initial control points MV v associated with the initial reference block 1506 may be modified. i , so that the reference block is scaled to be completely contained within the desired range. For example, the range can be based on the picture boundary 1502 plus the margin 1504. Fig.15 As further shown in FIG. 15 , the modified control point MV v may be determined based on the coordinates of the scaled reference block 1508. m In one example, the initial control point MV may be modified so that the reference block is scaled to be completely contained within a range value. The range value may be based on the picture boundary 1502 plus a margin 1504.

[0122] Fig.16 An example of modifying the control point MV to include a reference block is shown. Fig.16 As shown, the initial control point MV v2 i The initial control point MV v2 may be modified so that the initial reference block 1606 is outside the valid area 1604. The valid area 1604 may be based on the picture boundary 1602 plus the margin 1604. i , so that the lower left corner of the modified reference block 1608 is selected as the intersection between the initial reference block 1606 and the valid area 1604. Various mechanisms can be used to modify the control point MV (e.g., all) control point MVs. Various techniques can be evaluated, and the technique that produces the best performance can be selected. For example, a sub-block MV (e.g., each sub-block MV) can be derived from the affine control point MV. Cropping can be applied to the derived sub-block MV. For example, cropping can be applied based on the position of the sub-block relative to the picture boundary. In one example, the sub-block MV (e.g., each sub-block MV) can be cropped so that the associated reference sub-block overlaps one or more samples with the reference picture. For example, in and The horizontal component of the sub-block MV is cropped by using equations (27) and (28): Where W pic and W SB Can be picture width and sub-block width respectively. SB can be the horizontal coordinate of the upper left corner of the sub-block within the picture. The vertical component of the sub-block MV can be obtained by using equations (29) and (30) in and Between are clipped: Among them, H pic and HSB can be the picture height and sub-block height respectively. o can be the offset used for filtering operation. y SB It may be the vertical coordinate of the upper left corner of the sub-block within the picture. The upper left position of the CU relative to the picture boundary may be used (eg, instead of the sub-block position).

[0123] Although the features and elements are described above in specific combinations, it will be understood by those skilled in the art that each feature or element may be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware embedded in a computer-readable medium and executed by a computer or processor. Examples of computer-readable media include electronic signals (transmitted via a wired or wireless connection) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, buffer memory, semiconductor storage devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM discs and digital versatile discs (DVDs). A processor associated with the software may be used to implement a radio frequency transceiver used in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method for video decoding, the method comprising: determining that an affine mode is enabled for a current block including a plurality of sub-blocks; obtaining a sub-block motion vector associated with a sub-block among the plurality of sub-blocks; Based on the bit depth, clipping the sub-block motion vector; as well as Based on the clipped sub-block motion vector, the sub-block is decoded. 2 . The method of claim 1 , wherein the sub-block motion vector is based on a control point motion vector associated with the current block.

3. The method according to claim 2, wherein: The control point motion vector is one of a plurality of control point motion vectors of the current block, and The plurality of control point motion vectors are associated with a plurality of control point positions. 4 . The method of claim 3 , wherein based on a width of the current block being greater than a length of the current block, the plurality of control point positions include an upper left control point and an upper right control point. 5 . The method of claim 3 , wherein based on a width of the current block being smaller than a length of the current block, the plurality of control point positions include an upper left control point and a lower left control point. 6 . The method of claim 3 , wherein based on a width of the current block being equal to a length of the current block, the plurality of control point positions include a lower left control point and an upper right control point. The method of claim 1 , wherein the clipping of the sub-block motion vectors is further based on a motion field range.

8. The method according to claim 1, further comprising: The cropped sub-block motion vector is stored for spatial motion vector prediction or temporal motion vector prediction.

9. The method of claim 1, wherein the bit depth is used for motion field storage.

10. A method for video encoding, the method comprising: determining that an affine mode is enabled for a current block including a plurality of sub-blocks; obtaining a sub-block motion vector associated with a sub-block among the plurality of sub-blocks; Based on the bit depth, clipping the sub-block motion vector; as well as The sub-block is encoded based on the clipped sub-block motion vector.

11. The method of claim 10, wherein the sub-block motion vector is based on a control point motion vector associated with the current block.

12. The method according to claim 11, wherein: The control point motion vector is one of a plurality of control point motion vectors of the current block, and The plurality of control point motion vectors are associated with a plurality of control point positions. 13 . The method of claim 12 , wherein based on a width of the current block being greater than a length of the current block, the plurality of control point positions include an upper left control point and an upper right control point.

14. The method of claim 12, wherein based on a width of the current block being smaller than a length of the current block, the plurality of control point positions include an upper left control point and a lower left control point.

15. The method of claim 12, wherein based on a width of the current block being equal to a length of the current block, the plurality of control point positions include a lower left control point and an upper right control point.

16. The method of claim 10, wherein the clipping of the sub-block motion vectors is further based on a motion field range.

17. The method according to claim 10, further comprising: The cropped sub-block motion vector is stored for spatial motion vector prediction or temporal motion vector prediction. The method of claim 10 , wherein the bit depth is used for motion field storage.

19. A device for video decoding, comprising: A processor configured to: Determine the list of candidates for merger; calculating a control point distance associated with a merge candidate from the merge candidate list; Based on the calculation of the control point distance, determining that the merge candidate has a maximum control point distance; selecting the merge candidate from the merge candidate list based on the determination that the merge candidate has the maximum control point distance; as well as Based on the selected merge candidate, the current block is decoded.

20. The apparatus of claim 19, wherein the processor is further configured to: deriving a control point motion vector of the current block based on the selected merge candidate; and The current block is predicted based on the control point motion vector.

21. The apparatus of claim 19, wherein the processor is further configured to: calculating a corresponding control point distance associated with each merge candidate from the merge candidate list; and Based on the determination that the merge candidate has a maximum control point distance among the calculated respective control point distances, the merge candidate is selected from the merge candidate list.

22. The apparatus of claim 21, wherein the processor is further configured to: The merge candidate list is sorted based on the descending order of the calculated corresponding control point distances.

23. The apparatus of claim 19, wherein the merge candidate is associated with a neighboring block, and wherein the processor is further configured to: It is determined that an affine mode is enabled for the neighboring block, wherein the control point distance associated with the merge candidate is calculated based on the determination that an affine mode is enabled for the neighboring block.

24. The apparatus of claim 19, wherein the merge candidate is associated with a neighboring block, and wherein the control point distance associated with the merge candidate is calculated based on the square of a distance between two control points of the neighboring block.

25. The apparatus of claim 24, wherein: The distance between the two control points of the neighboring blocks is based on the shape of the neighboring blocks.