Adaptive motion vector precision for affine motion model-based video coding.

Abstract

A system and method for video encoding using an affine motion model with adaptive accuracy is described. In one example, a block of video is coded into a bitstream using an affine motion model, characterized by at least two motion vectors. A precision is selected for each of the motion vectors, and the selected precision is signaled in the bitstream. In some embodiments, the precision is signaled by including information in the bitstream that identifies one of multiple elements in a selected predetermined precision set. The identified element indicates the precision of each of the motion vectors that characterize the affine motion model. In some embodiments, the precision set used is explicitly signaled in the bitstream; in other embodiments, the precision set may be inferred from, for example, block size, block shape, or temporal layer.

Description

[Technical Field]

[0001] Cross-reference of related applications

[0001] This application is a regular patent application claiming benefits under Section 119(e) of the United States Patent Act from U.S. Provisional Patent Application No. 62 / 724,500 (filed on 29 August 2018), U.S. Provisional Patent Application No. 62 / 773,069 (filed on 29 November 2018), and U.S. Provisional Patent Application No. 62 / 786,768 (filed on 31 December 2018), all entitled “Adaptive Motion Vector Precision for Affine Motion Model Based Video Coding,” which are incorporated herein by reference in their entirety. [Background technology]

[0002] background

[0002] Video coding systems are widely used to compress digital video signals to reduce the storage needs and / or transmission bandwidth of such signals. Among the various types of video coding systems, such as block-based systems, wavelet-based systems, and object-based systems, block-based hybrid video coding systems are currently the most widely used and deployed. Examples of block-based video coding systems include international video coding standards such as MPEG1 / 2 / 4 Part 2, H.264 / MPEG-4 Part 10 AVC, VC-1, and the latest video coding standard called High Efficiency Video Coding (HEVC). HEVC was developed by ITU-T / SG16 / Q.6 / VCEG and ISO / IEC / MPEG's JCT-VC (Joint Collaborative Team on Video Coding).

[0003]

[0003] The first version of the HEVC standard was completed in October 2013, offering approximately 50% bitrate reduction or equivalent perceived quality compared to the previous generation video encoding standard H.264 / MPEG AVC. While the HEVC standard brings significant encoding improvements over its predecessor, there is evidence that even better encoding efficiency can be achieved with further encoding tools. Based on this, both VECG and MPEG have begun research into new encoding technologies for future video encoding standardization. In October 2015, ITU-T VECG and ISO / IEC MPEG formed JVET (Joint Video Exploration Team) to begin important research into advanced technologies that could enable significantly improved encoding efficiency compared to HEVC. In the same month, a software codebase called JEM (Joint Exploration Model) was established for future video encoding research. The JEM reference software is based on the HEVC Test Model (HM) developed by JCT-VC for HEVC. Any additional proposed coding tools can be integrated into the JEM software and tested using the JVET Common Test Conditions (CTC).

[0004]

[0004] In October 2017, the ITU-T and ISO / IEC issued a joint Call for Proposals (CfP) for video compression with features exceeding HEVC. In April 2018, 22 CfP responses for the Standard Dynamic Range category were accepted and evaluated at the 10th JVET meeting, demonstrating an improvement in compression efficiency of approximately 40% compared to HEVC. Based on these evaluation results, JVET (Joint Video Expert Team) launched a new project to develop a next-generation video coding standard called VVC (Versatile Video Coding). In the same month, a reference software codebase called the VVC Test Model (VTM) was established to demonstrate a reference implementation of the VVC standard. In the initial VTM-1.0, most coding modules, including intra-prediction, inter-prediction, transform / inverse transform and quantization / dequantization, as well as in-loop filters, follow the existing HEVC design, except that a multi-type tree-based block partitioning structure is used in VTM. Meanwhile, to facilitate the evaluation of new coding tools, another reference software base called the Benchmark Set (BMS) has also been created. The BMS codebase includes a set of coding tools inherited from JEM on top of VTM, resulting in higher coding efficiency and a moderate level of implementation complexity, and is used as a benchmark when evaluating similar coding techniques during the VVC standardization process. Specifically, there are nine JEM coding tools integrated into BMS-1.0, which include 65 angle intra-prediction directions, correction factor coding, advanced multiple transformation (AMT) + 4x4 non-separable retransform (NSST), affine motion models, generalized adaptive loop filters (GALF), advanced temporal motion vector prediction (ATMVP), adaptive motion vector precision, decoder-side motion vector refinement (DMVR), and linear model (LM) chroma modes. [Overview of the project] [Means for solving the problem]

[0005] overview

[0005] Embodiments described herein include methods used for encoding and decoding video (collectively, “encoding”). In some embodiments, a method is provided for decoding video from a bitstream, the method comprising: reading information from the bitstream for at least one current block in the video that identifies at least a first motion vector predictor and a second motion vector predictor; reading information from the bitstream that identifies one of a plurality of precisions within a given set of precisions; reading at least a first differential motion vector and a second differential motion vector from the bitstream, wherein the first and second differential motion vectors have the precisions identified; generating at least (i) a first control point motion vector from the first motion vector predictor and the first differential motion vector, and (ii) a second control point motion vector from the second motion vector predictor and the second differential motion vector; and generating a prediction for the current block using an affine motion model, wherein the affine motion model is characterized by at least a first control point motion vector and a second control point motion vector.

[0006]

[0006] Multiple precisions within a predetermined precision set may include 1 / 4 pixel precision, 1 / 16 pixel precision, and 1 pixel precision. The predetermined precision set is different from the predetermined precision set used for non-affine intercoding in the same video.

[0007]

[0007] The affine motion model can be a 4-parameter motion model or a 6-parameter motion model. If the affine motion model is a 6-parameter motion model, the method may further include reading information from the bitstream to identify a third motion vector predictor, reading a third differential motion vector having the identified precision from the bitstream, and generating a third control point motion vector from the third motion vector predictor and the third differential motion vector, wherein the affine motion model is characterized by a first control point motion vector, a second control point motion vector, and a third control point motion vector.

[0008]

[0008] Information that identifies one of several precisions may be read from the bitstream in block units, allowing different blocks in the image to use different precisions.

[0009]

[0009] In some embodiments, the motion vector predictors are rounded to a specified precision. Each of the control point motion vectors may be generated by adding the corresponding difference motion vector to each motion vector predictor.

[0010]

[0010] In some embodiments, the prediction of the current block is generated by using an affine motion model to determine the respective subblock motion vector for each of the multiple subblocks of the current block, and then using the respective subblock motion vector to generate the respective interpretation of the subblock.

[0011]

[0011] In some embodiments, the method further includes reading the residuals for the current block from the bitstream and reconstructing the current block by adding the residuals to a prediction of the current block.

[0012]

[0012] Systems and methods for adaptively selecting the accuracy of affine motion vectors and for performing motion estimation on affine motion models are also described.

[0013]

[0013] In additional embodiments, an encoder and decoder system is provided for performing the method described herein. The encoder or decoder system may comprise a processor and a non-temporary computer-readable medium for storing instructions for performing the method described herein. Further embodiments include a non-temporary computer-readable storage medium for storing video encoded using any of the methods disclosed herein. [Brief explanation of the drawing]

[0014] Brief explanation of the drawing [Figure 1A]

[0014] This is a system diagram showing an exemplary communication system in which one or more of the disclosed embodiments may be implemented. [Figure 1B]

[0015] This is a system diagram showing an exemplary wireless transmit / receive unit (WTRU) that may be used in the communication system shown in Figure 1A, according to one embodiment. [Figure 2A]

[0016] This is a functional block diagram of a block-based video encoder, such as the encoder used in VVC. [Figure 2B]

[0017] This is a functional block diagram of a block-based video decoder, such as the decoder used in VVC. [Figure 3A]

[0018] Figure 3A is a diagram illustrating block partitioning using a multi-type tree structure, and is a diagram showing quadtree partitioning. [Figure 3B]

[0018] Figure 3B is a diagram showing a block partition in a multi-type tree structure, and is a diagram showing a vertical binary tree partition. [Figure 3C]

[0018] Figure 3C is a diagram showing a block partition in a multi-type tree structure, and is a diagram showing a horizontal binary tree partition. [Figure 3D]

[0018] Figure 3D is a diagram showing a block partition in a multi-type tree structure, and is a diagram showing a vertical ternary tree partition. [Figure 3E]

[0018] Figure 3E is a diagram showing a block partition in a multi-type tree structure, and is a diagram showing a horizontal ternary tree partition. [Figure 4A]

[0019] Figure 4A is a diagram showing a four-parameter affine motion model, illustrating the affine model. [Figure 4B]

[0019] Figure 4B is a diagram showing a four-parameter affine motion model, which is a diagram showing the derivation of subblock-level motion in an affine block. [Figure 5]

[0020] This diagram shows the affine merge candidates. The candidate availability check order is N0, N1, N2, N3, N4. [Figure 6]

[0021] This figure shows the derivation of motion vectors at control points in an affine motion model. [Figure 7]

[0022] This figure shows the construction of affine motion vector predictors from motion vectors in blocks {A,B,C}, {D,E}, and {F,G}. [Figure 8]

[0023] This figure shows an example of MV time scaling for affine motion vector (MV) predictor generation. [Figure 9]

[0024] This diagram shows the adjacent blocks used to derive the context of block BC. [Figure 10]

[0025] This figure shows the mode determination method in CU coding without partitioning. [Figure 11]

[0026] This diagram shows motion models and methods for selecting accuracies other than the default accuracies (1 / 4 pixel for translational motion models, and (1 / 4 pixel, 1 / 4 pixel) for affine motion models). [Figure 12]

[0027] This figure shows a method for estimating affine motion with respect to the accuracy of (p0 pixel, p1 pixel). [Figure 13]

[0028] This diagram shows the refinement of MV0 using the eight nearest positions. Step 1: Select the best position in {P1, P2, P3, P4}. Step 2: If MV0 was updated in Step 1, select the best position from two adjacent positions. [Figure 14]

[0029] This diagram shows a 6-parameter affine mode. V0, V1, and V2 are control points, and (MVx,MVy) is the motion vector of a subblock centered at position (x,y). [Figure 15]

[0030] This figure shows an example of an encoded bitstream structure. [Figure 16]

[0031] This is a diagram illustrating an exemplary communication system. [Figure 17]

[0032] This figure shows the derivation of motion vectors in the subblocks of an 8x4 coding unit. [Figure 18]

[0033] This figure shows how the decoder performs the operation in several embodiments. [Modes for carrying out the invention]

[0015] Exemplary network for implementation of the embodiment

[0034] Figure 1A shows an exemplary communication system 100 that may implement one or more disclosed embodiments. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, and broadcast to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communication system 100 may utilize one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), quadrature FDMA (OFDMA), and single-carrier FDMA (SC-FDMA), zero-tail unique word DFT spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, and filter bank multicarrier (FBMC).

[0016]

[0035] As shown in Figure 1A, the communication system 100 may include radio transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104, CN 106, public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, but it should be understood that the disclosed embodiments intend any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d can be any type of device configured to operate and / or communicate in a radio environment. For example, WTRU102a, 102b, 102c, and 102d, which may all be called “stations” and / or “STAs,” may be configured to transmit and / or receive radio signals and may include user equipment (UEs), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, radio sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearables, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other radio devices operating in industrial and / or automated processing chain situations), consumer electronic devices, and devices operating on commercial and / or industrial radio networks. Any of WTRU102a, 102b, 102c, and 102d may be interchangeably referred to as UEs.

[0017]

[0036] The communication system 100 may also include base stations 114a and / or base stations 114b. Each of the base stations 114a and 114b can be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks such as CN 106, the Internet 110, and / or other networks 112. For example, base stations 114a and 114b may be base transceiver base stations (BTS), Node-B, eNodeB, home NodeB, home eNodeB, gNB, NR NodeB, site controller, access point (AP), and wireless router. Although base stations 114a and 114b are shown as single elements, it will be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0018]

[0037] Base station 114a may be part of RAN 104, which may also include other base stations and / or network elements (not shown) such as base station controllers (BSCs), radio network controllers (RNCs), and relay nodes. Base stations 114a and / or base stations 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be called cells (not shown). These frequencies may be in the licensed spectrum, the unlicensed spectrum, or a combination of the licensed and unlicensed spectrum. Cells may provide coverage of radio services to a particular geographic area, which may be relatively fixed or may change over time. Cells may be further divided into cell sectors. For example, a cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one for each sector of the cell. In one embodiment, base station 114a may utilize multiple-input multiple-output (MIMO) technology, and may utilize multiple transceivers per sector of the cell. For example, beamforming can be used to transmit and / or receive signals in a desired spatial direction.

[0019]

[0038] Base stations 114a and 114b can communicate with one or more WTRUs 102a, 102b, 102c, and 102d via a radio interface 116, the radio interface 116 can be any suitable radio communication link (e.g., radio frequency (RF), microwave, centimeter wave, millimeter wave, infrared (IR), ultraviolet (UV), visible light, etc.). The radio interface 116 can be established using any suitable radio access technology (RAT).

[0020]

[0039] More specifically, as described above, the communication system 100 can be a multiple access system and can utilize one or more channel access schemes such as CDMA, TDMA, FDMA, OFDMA, and SC-FDMA. For example, base stations 114a and WTRU 102a, 102b, and 102c in RAN 104 can implement radio technologies such as Universal Mobile Communications System (UMTS) Terrestrial Radio Access (UTRA), which can establish a radio interface 116 using broadband CDMA (WCDMA®). WCDMA® may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Advanced HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0021]

[0040] In one embodiment, base stations 114a and WTRUs 102a, 102b, and 102c can implement radio technologies such as Advanced UMTS Terrestrial Radio Access (E-UTRA), which can establish a radio interface 116 using Long-Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0022]

[0041] In one embodiment, base stations 114a and WTRUs 102a, 102b, and 102c can implement radio technologies such as NR radio access, which can establish a radio interface 116 using New Radio (NR).

[0023]

[0042] In one embodiment, base station 114a and WTRU 102a, 102b, 102c can implement multiple radio access technologies. For example, base station 114a and WTRU 102a, 102b, 102c can implement LTE radio access and NR radio access together, for example, using the principle of dual connectivity (DC). Thus, the radio interface used by WTRU 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions transmitted to and from multiple types of base stations (e.g., eNB and gNB).

[0024]

[0043] In other embodiments, base stations 114a and WTRUs 102a, 102b, and 102c can implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (Wi-Fi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Pan-European Digital Mobile Communications System (GSM), Enhanced Data Rate for GSM Evolution (EDGE), and GSM EDGE (GERAN).

[0025]

[0044] The base station 114b in Figure 1A can be, for example, a wireless router, home NodeB, home eNodeB, or access point, and can utilize any suitable RAT to facilitate wireless connectivity in localized areas such as workplaces, homes, vehicles, campuses, industrial facilities, air corridors (e.g., for drone use), and roads. In one embodiment, the base station 114b and WTRU 102c, 102d can implement wireless technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, the base station 114b and WTRU 102c, 102d can implement wireless technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and WTRU 102c, 102d can utilize cellular-based RATs (e.g., WCDMA®, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or femtocell. As shown in Figure 1A, base station 114b may have a direct connection to the internet 110. Therefore, base station 114b may not need to access the internet 110 via CN 106.

[0026]

[0045] RAN104 can communicate with CN106, which can be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRU102a, 102b, 102c, and 102d. The data may have various Quality of Service (QoS) requirements, including different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, and mobility requirements. CN106 can provide call control, billing services, mobile location-based services, prepaid calls, internet connectivity, video distribution, and / or perform high-level security functions such as user authentication. Although not shown in Figure 1A, it will be understood that RAN104 and / or CN106 can communicate directly or indirectly with other RANs utilizing the same or different RAT as RAN104. For example, in addition to connecting to RAN104 which can utilize NR radio technology, CN106 can also communicate with other RANs (not shown) that utilize GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or Wi-Fi radio technology.

[0027]

[0046] CN106 can also function as a gateway for WTRU102a, 102b, 102c, and 102d to access PSTN108, the Internet 110, and / or other networks 112. PSTN108 may include a circuit-switched telephone network providing basic telephone services (POTS). The Internet 110 may include a global system consisting of interconnected computer networks and devices using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) within the TCP / IP Internet Protocol Suite. Network 112 may include wired and / or wireless networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs that may utilize the same or different RAT as RAN104.

[0028]

[0047] Some or all of the WTRUs 102a, 102b, 102c, and 102d in the communication system 100 can include multimode functionality (for example, WTRUs 102a, 102b, 102c, and 102d can include multiple transceivers for communicating with different radio networks via different radio links). For example, WTRU 102c shown in Figure 1A can be configured to communicate with base station 114a which can utilize cellular-based radio technology, and also with base station 114b which can utilize IEEE 802 radio technology.

[0029]

[0048] Figure 1B is a system diagram showing an exemplary WTRU102. As shown in Figure 1B, the WTRU102 may comprise, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, a non-removable memory 130, a removable memory 132, a power supply 134, a Global Positioning System (GPS) chipset 136, and / or other peripherals 138. It will be understood that the WTRU102 may include any subcombinations of the above elements while maintaining consistency with the embodiment.

[0030]

[0049] The processor 118 can be a general-purpose processor, a dedicated processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors working with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and a state machine. The processor 118 can perform signal coding, data processing, power control, input / output processing, and / or any other functions that enable the WTRU 102 to operate in a wireless environment. The processor 118 can be coupled to the transceiver 120, and the transceiver 120 can be coupled to the transmit / receive element 122. Although Figure 1B shows the processor 118 and the transceiver 120 as separate components, it will be understood that the processor 118 and the transceiver 120 can be integrated together in an electronic package or chip.

[0031]

[0050] The transmitting / receiving element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via the radio interface 116. For example, in one embodiment, the transmitting / receiving element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmitting / receiving element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In another embodiment, the transmitting / receiving element 122 may be configured to transmit and / or receive both RF signals and optical signals. It will be understood that the transmitting / receiving element 122 may be configured to transmit and / or receive any combination of radio signals.

[0032]

[0051] In Figure 1B, the transmit / receive element 122 is shown as a single element, but the WTRU 102 can include any number of transmit / receive elements 122. More specifically, the WTRU 102 can utilize MIMO technology. Thus, in one embodiment, the WTRU 102 can include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving radio signals via the radio interface 116.

[0033]

[0052] The transceiver 120 may be configured to modulate the signal transmitted by the transmit / receive element 122 and to demodulate the signal received by the transmit / receive element 122. As described above, the WTRU 102 may have multimode capabilities. Therefore, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0034]

[0053] The processor 118 of the WTRU102 can be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit) and can receive user input data from them. The processor 118 can also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 can obtain information from any type of suitable memory, such as a non-removable memory 130 and / or removable memory 132, and can store data in them. The non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identification module (SIM) card, a memory stick, and a secure digital (SD) memory card, etc. In other embodiments, the processor 118 can access information from memory located on a server or home computer (not shown), rather than from memory physically located on the WTRU 102, and can store data in such memory.

[0035]

[0054] The processor 118 can receive power from the power supply 134 and may be configured to distribute and / or control power to other components within the WTRU 102. The power supply 134 can be any suitable device for supplying power to the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, and fuel cells.

[0036]

[0055] The processor 118 can also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or instead of, the information from the GPS chipset 136, the WTRU 102 can receive location information from base stations (e.g., base stations 114a, 114b) via the radio interface 116 and / or determine its own position based on the timing of signals received from two or more nearby base stations. It will be understood that the WTRU 102 can acquire location information using any suitable positioning method while maintaining consistency with the embodiments.

[0037]

[0056] The processor 118 can be further coupled to other peripherals 138, which may include one or more software modules and / or hardware modules that provide additional features, functions, and / or wired or wireless connectivity. For example, peripherals 138 may include an accelerometer, e-compass, satellite transceiver, digital camera (for photos and / or videos), Universal Serial Bus (USB) port, vibration device, television transceiver, hands-free headset, Bluetooth® module, frequency modulation (FM) radio unit, digital music player, media player, video game player module, internet browser, virtual reality and / or augmented reality (VR / AR) device, and activity tracker. Peripherals 138 may include one or more sensors, which may be one or more of a gyroscope, accelerometer, Hall effect sensor, magnetometer, orientation sensor, proximity sensor, temperature sensor, time sensor, geolocation sensor, altimeter, light sensor, touch sensor, magnetometer, barometer, gesture sensor, biometric sensor, and / or humidity sensor.

[0038]

[0057] WTRU102 may include a full-duplex radio (for example, one in which the transmission and reception of some or all of the signals associated with a particular subframe of both UL (e.g., for transmission) and downlink (e.g., for reception) may occur in parallel and / or simultaneously.) The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference via signal processing either through hardware (e.g., chokes) or through a processor (e.g., another processor (not shown) or processor 118). In one embodiment, WTRU102 may include a half-duplex radio for the transmission and reception of some or all of the signals (for example, one in which the transmission and reception of either a particular subframe of UL (e.g., for transmission) or downlink (e.g., for reception) may occur in parallel and / or simultaneously.

[0039]

[0058] In Figures 1A and 1B, the WTRU is described as a wireless terminal, but in certain representative embodiments, it is intended that such a terminal may have access to a wired communication interface with a communication network (e.g., temporarily or permanently).

[0040]

[0059] In a typical embodiment, the other network 112 may be a WLAN.

[0041]

[0060] Considering Figures 1A to 1B and the corresponding descriptions, one or more, or all, of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, an emulation device may be used to test other devices and / or to simulate network and / or WTRU functions.

[0042]

[0061] Emulation devices may be designed to perform tests on one or more other devices in a laboratory and / or carrier network environment. For example, one or more emulation devices may be fully or partially implemented and / or deployed as part of a wired and / or wireless network to test other devices in a communications network, and may perform one or more or all of their functions. One or more emulation devices may be temporarily implemented / deployed as part of a wired and / or wireless network, and may perform one or more or all of their functions. Emulation devices may be directly coupled to another device for testing purposes and / or may perform tests using wireless communication.

[0043]

[0062] One or more emulation devices may perform one or more functions, including all of them, without being implemented / deployed as part of a wired and / or wireless communication network. For example, an emulation device may be used in a test scenario in a test lab and / or in an undeployed (e.g., being tested) wired and / or wireless communication network to perform testing of one or more components. One or more emulation devices may serve as test equipment. Wireless communication via direct RF coupling and / or RF circuitry (e.g., which may include one or more antennas) may be used by an emulation device to transmit and / or receive data.

[0044] Detailed explanation Block-based video coding

[0063] Like HEVC, VVC is built on a block-based hybrid video coding framework. Figure 2A shows a block diagram of an example block-based hybrid video coding system. The input video signal 103 is processed in block units (called coding units (CUs)). In VTM-1.0, a CU can be up to 128 × 128 pixels. However, unlike HEVC, which divides blocks based solely on quadtrees, in VTM-1.0, one coding tree unit (CTU) is divided into CUs to adapt to various local characteristics based on quadtree / binary / ternary trees. In addition, the concept of multi-type unit types in HEVC is removed; that is, the distinction between CU, prediction unit (PU), and transformation unit (TU) no longer exists in VVC-1.0, and instead, each CU is always used as the basic unit for both prediction and transformation without further division. In a multi-type tree structure, one CTU is first divided by a quadtree structure. Then, each quadtree leaf node can be further divided by binary and ternary tree structures. As shown in Figures 3A to 3E, there are five partitioning types: quadtree partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal ternary partitioning, and vertical ternary partitioning. In Figure 2A, spatial prediction (160) and / or temporal prediction (162) can be performed. Spatial prediction (or "intra prediction") predicts the current video block using pixels from samples of already encoded adjacent blocks (called reference samples) within the same video image / slice. Spatial prediction reduces the spatial redundancy inherent in video signals. Temporal prediction (also called "inter prediction" or "motion-compensated prediction") predicts the current video block using reconstructed pixels from already encoded video images. Temporal prediction reduces the temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs) indicating the amount and direction of motion between the current CU and its temporal reference.Additionally, if multiple reference images are supported, one reference image index is transmitted, and this single reference image index is used to identify from which reference image in the reference image store (164) the temporal prediction signal originated. After spatial and / or temporal prediction, a mode determination block (180) in the encoder selects the best prediction mode, for example, based on a rate-distortion optimization method. The prediction block is then subtracted from the current video block (117), and the prediction residual is decorrelated and quantized (107) using a transform (105). The quantized residual coefficients are inversely quantized (111), inversely transformed (113) to form a reconstructed residual, which is then added back to the prediction block (127) to form a reconstructed signal of the CU. Further in-loop filtering, such as a deblocking filter, may be applied to the reconstructed CU before the reconstructed CU is placed in the reference image store (164) and used to encode future video blocks (166). To form the output video bitstream 121, the encoding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy encoding unit (109), where they are further compressed and packed to form the bitstream.

[0045]

[0064] Figure 2B shows a block diagram of an example block-based video decoder. The video bitstream 202 is first unpacked and entropy-decoded in the entropy decoding unit 208. The encoding mode and prediction information are sent to the spatial prediction unit 260 (if intra-encoded) or the temporal prediction unit 262 (if inter-encoded) to form a prediction block. The residual transformation coefficients are sent to the inverse quantization unit 210 and the inverse transformation unit 212 to reconstruct the residual block. The prediction block and the residual block are then added together in 226. The reconstructed block can undergo further in-loop filtering before being stored in the reference image store 264. The reconstructed video in the reference image store can then be sent out to drive a display device and can also be used to predict future video blocks.

[0046]

[0065] As described above, BMS-1.0 conforms to the same VTM-1.0 encoding / decoding workflow as shown in Figures 2A and 2B. However, several encoding modules, particularly those related to temporal prediction, are further extended and enhanced. Below, we briefly describe affine motion compensation as one of the intercoding tools included in BMS-1.0 or earlier JEMs.

[0047] Affine Mode

[0066] In HEVC, only translational motion models are applied to motion-compensated predictions. However, in reality, there are many types of motion, such as zoom in / out, rotation, projection, and other irregular motions. In BMS, a simplified affine transform motion-compensated prediction is applied. Flags in each intercoded CU are signaled to indicate whether a translational or affine motion model is applied to the inter-prediction.

[0048]

[0067] The simplified affine motion model is a four-parameter model, namely two parameters for translational motion in the horizontal and vertical directions, one parameter for zoom motion, and one parameter for rotational motion. The horizontal zoom parameter is equal to the vertical zoom parameter. The horizontal rotation parameter is equal to the vertical rotation parameter. The four-parameter affine motion model is encoded in the BMS using two motion vectors as a pair at two control point positions defined at the upper left and upper right corners of the current CU. As shown in Figure 4A, the affine motion field of the block is described by two control point motion vectors (V0,V1). Based on the control point motion, the affine-encoded motion field of the block (v x ,v y ) can be expressed as follows:

number

number

[0049]

[0068] The four affine model parameters can be repeatedly estimated. The MV pair at step k is

Equation

Equation

Equation

[0050]

[0069] Based on the optical flow equation, the relationship between the luminance change, spatial gradient, and temporal movement is formulated as follows.

Equation

[0051]

[0070]

Equation

Equation

[0052]

[0071] Since all samples in CU satisfy equation (7), the parameter set (a, b, c, d) can be solved using the least squares method. Two control points at step (k+1)

number

[0053]

[0072] As shown in Figure 14, a 6-parameter affine-encoded CU has three control points: upper left, upper right, and lower left. The motion of the upper left control point is translational motion, the motion of the upper right control point is related to horizontal rotation and zoom motion, and the motion of the lower left control point is related to vertical rotation and zoom motion. In the case of a 4-parameter affine motion model, the horizontal and vertical rotation and zoom motions are the same. Motion vector (MV) of each subblock x MV y ) is derived using three MVs at the control points as follows:

number

[0054] Affine Merge Mode

[0073] When a CU is encoded in affine mode, two sets of motion vectors for the two control points of each reference list are signaled by predictive coding. The difference between the MV and its predictor is encoded losslessly, and this signaling overhead can be considerable, especially at low bit rates. To reduce signaling overhead, affine merge mode is also applied to BMS by considering the local continuity of the motion field. The motion vectors at the two control points of the current CU are derived using the affine motion of an affine merge candidate selected from an adjacent block. When the current CU is encoded in affine merge mode, five adjacent blocks are checked in the order N0 to N4, as shown in Figure 5. The first affine-encoded adjacent block is then used as the affine merge candidate. For example, as shown in Figure 6, the current CU is encoded in affine merge mode, and its lower left adjacent block (N0) is selected as the affine merge candidate. The width and height of the CU containing block N0 are represented by nw and nh. The width and height of the current CU are represented by cw and ch. Position P i The MV in (v ix ,v iy It is expressed as MV(v) at control point P0. 0x ,v 0y ) is derived as follows:

number

[0055]

[0074] MV(v) at control point P1 1x ,v 1y ) is derived as follows:

number

[0056]

[0075] MV(v) at control point P2 2x ,v 2y ) is derived as follows:

number

[0057]

[0076] After the MVs at the two control points (P0 and P1) are derived, the MVs for each subblock within the current CU are derived as described above, and these derived subblock MVs can be used for subblock-based motion compensation and for predicting temporal motion vectors for future image coding.

[0058] Affine MV prediction

[0077] For non-merged affine coded CUs, signaling MV at control points is costly, and predictive coding is used to reduce signaling overhead. In BMS, affine MV predictors are generated from the motion of adjacent coded blocks. For affine coded CUs, there are two types of predictors for MV prediction: (a) affine motion generated from adjacent blocks of control points, and (b) translational motion used in conventional MV prediction, with (b) being used only when the number of affine predictors from (a) is insufficient (less than 2 in BMS).

[0059]

[0078] Three sets of MVs are used to generate multiple affine motion predictors. As shown in Figure 7, the three sets of MVs are (1){MV A MV B MV C (2) {MV} from adjacent blocks {A, B, C} at corner P0 composed of set S1, (2) {MV} D MV E (3) {MV} from adjacent blocks {D,E} in corner P1, which is made up of set S2, and (3) {MV} F MV GThis is the MV from adjacent blocks {F,G} in corner P2, which consists of set S3, represented as}. The MV from adjacent blocks is derived as follows: First, spatially adjacent blocks are checked. If the adjacent block is an intercoded block, the MV is used directly. If the reference image of the adjacent block is the same as the reference image of the current CU, or if the reference image of the adjacent block is different from the reference image of the current CU, the MV is scaled according to the temporal distance. As shown in Figure 8, the temporal distance between the current image and the reference image of the current CU is denoted as TB, and the temporal distance between the current image and the reference image of the adjacent block is denoted as TD. The MV1 of the adjacent block is scaled as follows.

number

[0060]

[0079] MV2 is used in motion vector sets.

[0061]

[0080] If adjacent blocks are not intercoded blocks, blocks at the same location in the reference image are checked. If blocks at the same temporal location are intercoded blocks, the MV is scaled according to equation (18) based on the temporal distance. If blocks at the same temporal location are not intercoded blocks, the MV of its adjacent blocks is set to zero.

[0062]

[0081] After three sets of MVs are obtained, an affine MV predictor is generated by selecting one MV from each of the three sets of MVs. The sizes of S1, S2, and S3 are 3, 2, and 2, respectively. In total, 12 (3 × 2 × 2) combinations are obtained. In BMS, if the zoom or rotation-related parameters represented by the three MVs are greater than a predefined threshold, the candidate is discarded. For the three corners of the CU, i.e., top left, top right, and bottom left, one combination is represented as (MV0, MV1, MV2). The following conditions are checked: (|(v 1x-v 0x )|>T*w) or (|(v 1y -v 0y )|>T*h) or (|(v 2x -v 0x )|>T*w) or (|(v 2y -v 0y )|>T*h) (17) T is (1 / 2). If the condition is met, i.e., the zoom or rotation is too large, the candidate is discarded.

[0063]

[0082] In BMS, all remaining candidates are sorted. A triplet of three MVs represents a 6-parameter motion model including horizontal and vertical translation, zoom, and rotation. The ordering criterion is the difference between this 6-parameter motion model, represented as (MV0,MV1), and a 4-parameter motion model. Candidates with smaller differences have lower indices in the ordered candidate list. The difference between the affine motion represented by (MV0,MV1,MV2) and the affine motion model represented by (MV0,MV1) is evaluated by equation (18). D=|(v 1x -v 0x )*h-(v 2y -v 0y )*w|+|(v 1y -v 0y )*h+(v 2x -v 0x )*w| (18)

[0064] Affine MV coding

[0083] When a control point (CU) is encoded as an affine mode, it can be either an affine merged mode or an affine non-merged mode. In the affine merged mode, the affine MV at a control point is derived from the affine MV of the adjacent affine-encoded CU. Therefore, signaling MV information is not necessary for the affine merged mode. In the affine non-merged mode, the MV at a control point is encoded using differential coding. The MV predictor is generated using the adjacent MV as described above, and the difference between the current MV and its predictor is encoded. The signaled differential MV is called the MVD. Since an affine 4-parameter model has two control points, two MVDs are used for unidirectional prediction signaling, and four MVDs are used for bidirectional prediction signaling. Since an affine 6-parameter model has three control points, three MVDs are used for unidirectional prediction signaling, and six MVDs are used for bidirectional prediction signaling. The MVD is a two-dimensional vector (horizontal and vertical components) and is losslessly encoded, making it difficult to compress. In the current VVC design (VTM-1.0 / BMS-1.0), the accuracy of the MVD for signaling is 1 / 4 pixel accuracy.

[0065] Adaptive MVD accuracy

[0084] For CUs encoded as non-merged and non-affine intermode, the MVD between the current CU's MV and its predictor can be encoded at different resolutions. This can be 1 / 4 pixel precision, 1 pixel precision, or 4 pixel precision. 1 / 4 pixel precision is fractional precision. Both 1 pixel precision and 4 pixel precision are integer precision. Precision is signaled for each CU with two flags to indicate the MVD precision. The first flag indicates whether the precision is 1 / 4 pixel precision or not. If the precision is not 1 / 4 pixel precision, the second flag is signaled to indicate 1 pixel precision or 4 pixel precision. In motion estimation, the difference MV is typically searched around an initial MV, which is treated as the starting position. The starting position can be selected from its spatial and temporal predictors. For ease of implementation, the starting MV is rounded to the precision required for MVD signaling, and only MVD candidates with the desired precision are searched. MV predictors are also rounded to the MVD precision. The VTM / BMS reference software checks the rate distortion (RD) cost of different MVD accuracies and selects the optimal MVD accuracy with the lowest RD cost. RD cost is calculated as a weighted sum of sample value distortion and coding rate, and is a measure of coding performance. A lower RD cost results in better overall coding performance. To reduce signaling overhead, MVD accuracy-related flags are only signaled if the signaled MVD is not zero. If the signaled MVD is zero, it is assumed to be 1 / 4 pixel accuracy.

[0066] MVD encoding

[0085] In VVC, the MVD entropy coding method is the same in both affine and non-affine coding modes. This method codes the two components independently. The MVD code for each component is coded with 1 bit. The coding of the absolute value is done in two parts: (1) The values ​​0 and 1 are coded with flags. The first flag indicates whether the absolute value is greater than 0, and if the value is greater than 0, the second flag indicates whether the absolute value is greater than 1. (2) If the absolute value v is greater than 1, the remaining part (v-2) is binarized with first-order exponential Golomb (EG) coding, and these binarized bins are coded with fixed-length coding. For example, the binarization of the remaining part (v-2) using first-order EG coding is shown in Table 1.

[0067] [Table 1]

[0068] For the same value being encoded, the codeword lengths of EG codes of different orders can differ. Lower orders generally result in shorter codeword lengths for smaller values ​​and longer codeword lengths for larger values. In affine coding modes, the MVD statistics for control points can differ. EG codes of the same order may not be optimal for MVD coding of all control points.

[0069] Challenges addressed in some embodiments

[0086] As mentioned above, MVD signaling involves more MVDs being signaled—two sets of MVDs in a four-parameter affine model and three sets in a six-parameter affine model—resulting in a signaling overhead that is not negligible compared to explicitly affine-encoded CUs, compared to interCUs encoded in translational motion models. Adaptive MVD accuracy for signaling helps to achieve a better trade-off between motion compensation efficiency and signaling overhead. However, the use of control point motion vectors in affine models differs from the motion vectors in conventional translational motion models; that is, the MV at the control point is not used directly for motion compensation but is used to derive the MV of a subblock, and the MV of the subblock is used for motion compensation of that subblock.

[0070]

[0087] The motion estimation (ME) process for the affine motion model described above differs from the motion search methods for conventional translational motion models in VTM / BMS. The ME process used to find the optimal MV at two control points is based on the estimation of the optical flow field. With each iteration, the difference MV derived from the optical flow estimation is different, and it is difficult to control the step size in each iteration. In contrast, the ME for translational motion models to find the optimal MV for a coded block is typically a position-unit search method within a specific range. Within the search range centered on the starting MV, the ME cost can be evaluated and compared for each possible position in the complete search scheme, and then the optimal position with the smallest ME cost can be selected. The ME cost is typically evaluated as a weighted sum of the prediction error and bits of the MV-related signaling, including the reference image index and MVD. The prediction error can be measured by the sum of the absolute differences (SAD) between the original signal and the predicted signal of the coded block.

[0071]

[0088] In the ME process for this decision for translational motion models, there are numerous fast search methods for adaptively adjusting the search step size during iteration. For example, a search can be started with a coarse step size within the search window. Once an optimal position is obtained with a coarse step size, the step size can be reduced, and the search window is also reduced to a smaller window centered on the last optimal position obtained from the previous search window. This iterative search can be terminated when the search step size is reduced to a value below a predefined threshold, or when the total number of searches reaches a predefined threshold.

[0072]

[0089] The ME process for affine models differs from the ME process for translational motion models. This disclosure describes ME methods for affine models with different MVD accuracies.

[0073] Overview of several embodiments

[0090] To provide motion estimation for affine models, this disclosure describes adaptive MVD accuracy methods that improve the coding efficiency of affine motion models. Several embodiments provide an improved trade-off between signaling and motion compensation prediction efficiency. A determination method for adaptive MVD accuracy is also proposed.

[0074]

[0091] In some embodiments, the MVD accuracy for an affine model is adaptively selected from multiple accuracy sets for two control points. The MVD accuracy may differ at different control points.

[0075]

[0092] In some embodiments, MV search methods are proposed for affine models at different MVD accuracies to improve accuracy and reduce coding complexity.

[0076]

[0093] In some embodiments, the affine control point motion vector predictor (MVP) and MV are kept highly accurate, while the MVD is rounded to a lower accuracy. This allows for improved accuracy of motion compensation using high-accuracy MVs.

[0077]

[0094] For simplicity, the following explanation uses a four-parameter affine motion model as an example. However, the proposed method can also be directly extended to a six-parameter affine motion model.

[0078] Adaptive MVD accuracy for affine models

[0095] In VTM / BMS, MVD at control points for an affine model is always signaled with 1 / 4 pixel precision. Fixed precision does not provide a good trade-off between MVD signaling overhead and the efficiency of affine motion compensation. By increasing the precision of the MVD at those control points, the MV derived by equation (1) for each subblock becomes more accurate. Thus, motion prediction can be improved. However, this uses more bits for MVD signaling. In this disclosure, a method for adaptive MVD precision at control points is proposed. The motion of the upper left control point is related to the translational motion of each subblock within its CU, and the motion difference between two control points is related to the zoom and rotational motion of each subblock. Blocks encoded in an affine motion model may have different motion characteristics. Some affine blocks may have translational and rotational motion with high precision, while some affine blocks may have translational motion with low precision. In some embodiments, the translational and rotational / zoom motions of affine blocks may have different precisions. Based on this, several exemplary embodiments signal different accuracies for MVD coding at different control points.

[0079]

[0096] Signaling precision individually for each control point increases the signaling overhead of affine-encoded control units. One embodiment involves signaling the precision of two control points together. Only frequently used combinations are signaled. For example, a precision pair (prec0, prec1) may be used to indicate the precision of the upper-left control point "prec0" and the precision of the upper-right control point "prec1". An exemplary embodiment uses the following four precision sets: S1{(1 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 4 pixel)}, S2{(1 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 8 pixel)}, S3{(1 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 4 pixel),(1 / 8 pixel, 1 / 8 pixel)}, and S4{(1 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 8 pixel),(1 / 8 pixel, 1 / 8 pixel)}.

[0080]

[0097] (1 / 4 pixel, 1 / 4 pixel) precision is used for affine blocks as normal precision. (1 pixel, 1 / 4 pixel) is used for affine blocks with low precision and translational motion, but rotation / zoom still has normal precision. (1 / 4 pixel, 1 / 8 pixel) is used for affine blocks with high precision and rotation / zoom. (1 / 8 pixel, 1 / 8 pixel) is used for affine blocks with high precision and both translational motion and rotation / zoom. Precision sets can be signaled, for example, in sequence parameter sets, image parameter sets, or slice headers.

[0081]

[0098] In some embodiments, the precision of one control point applies to the MVDs of two lists when the current affine CU is encoded in bidirectional predictive mode. In some embodiments, to reduce signaling redundancy, precision is signaled only if the MVD at that control point is not zero. If the MVD at a control point is zero, precision does not affect the zero MVD, so there is no need to signal precision information for that control point. For example, if the top-left MVD is zero, the (1 pixel, 1 / 4 pixel) precision is not valid for the current CU. Therefore, in this case there is additional precision signaling when the precision set is S1. (1 / 4 pixel, 1 / 4 pixel) and (1 / 8 pixel, 1 / 8 pixel) are valid when the precision set is S3. The precision of a zero MVD may be inferred as a default precision such as (1 / 4 pixel, 1 / 4 pixel). Another embodiment may always signal precision because even if the MVD is zero, it may lead to a high-precision MV from that predictor. For example, the MV predictor is derived from adjacent affine-encoded CUs. High accuracy leads to a high-accuracy MV predictor, and therefore the final MV accuracy is high.

[0082]

[0099] Tables 2, 3, 4, and 5 propose the binarization of these precision sets, and the binarized bins are encoded.

[0083] [Table 2]

[0084] [Table 3]

[0085] [Table 4]

[0086] [Table 5]

[0087]

[0100] For precision coding, we use S3 as an example. According to Table 4, there are two bins that are coded for the S3 set after binarization. The second bin is coded only if the first bin is 0. The bins are coded using context-adaptive binary arithmetic coding (CABAC). The context of one bin in CABAC is used to record the probability of being 0 or 1. The context of the first bin can be derived from its adjacent left and top positions, as shown in Figure 9. Here we define two functions, namely, (1) Model(CU) which indicates whether the motion model of CU is an affine model, and (2) Prec(CU) which indicates whether precision (1 pixel, 1 / 4 pixel) is used in CU.

number

[0088]

[0101] The accuracy of adjacent CUs and the current CU is compared and evaluated using equations (21) and (22), namely equalPrec(B L ), equalPrec(B A ) obtain.

number

[0089]

[0102] The index of the first bin's context is constructed as shown in equation (23). Context_idx(B C )=equalPrec(B A )+equalPrec(B L ) (twenty three)

[0090]

[0103] The second bin can be encoded using a single fixed context, or it can be encoded using a fixed-length 1-bit encoding.

[0091]

[0104] Alternatively, in the aforementioned precision-based signaling scheme, the 1-pixel precision of the upper-left control point can be replaced with 1 / 2-pixel precision.

[0092]

[0105] Another embodiment involves individually signaling the precision for each control point. For example, one precision is signaled for the top-left control point, selected from a set {1 pixel, 1 / 4 pixel, 1 / 8 pixel}, and one precision is signaled for the top-right control point, selected from a set {1 / 2 pixel, 1 / 4 pixel, 1 / 8 pixel}. The reason the precision sets for the two control points are different is that 1-pixel precision is too coarse for the top-right MV associated with rotational and zoom motion, because rotational and zoom motions have more complex warping effects than translational motion. If the affine block has low precision and translational motion, 1-pixel precision can be selected for the top-left control point; if the affine block has high precision and translational motion, 1 / 8 pixel precision can be selected for the top-left control point; if the affine block has high precision and rotational or zoom motion, 1 / 8 pixel precision can be selected for the top-right control point. Based on statistical values, the following binarization tables (Tables 6 and 7) may be used to encode the precision selected for the two control points. The binarization code is a codeword and can be encoded using various entropy coding methods such as CABAC. On the decoder side, the affine MV predictor at each control point is rounded to the precision of the MVD and scaled to high precision for MV field storage (e.g., 1 / 16 pixels for VVC). The decoded MVD is first scaled to high precision for MV field storage based on its precision. The scaled MVD is then added to the MV predictor to obtain a reconstructed MV with the precision used for motion field storage. The reconstructed MV at the control points is used to derive the MV of each subblock in equation (1) in order to obtain the sample value prediction for that subblock in order to obtain the motion compensation for that subblock.

[0093] [Table 6]

[0094] [Table 7]

[0095]

[0106] In another embodiment, the precision sets of both control points may be the same as {1 / 2 pixel, 1 / 4 pixel, 1 / 8 pixel}, but the binarization of the precision coding of the two control points may be different. An example of the binarization of the precision coding of the two control points is proposed in Table 8.

[0096] [Table 8]

[0097]

[0107] In some embodiments, precision control of control points is applied only to large CUs to save signaling overhead. This is because affine motion models are typically used more frequently for larger CUs. For example, in some embodiments, MVD precision of control points may only be signaled if the CU has an area greater than a threshold (e.g., 16x16). For small CUs, precision may be inferred as (1 / 4 pixel) for both control points.

[0098]

[0108] In some embodiments, the precision set is modified at the image level. In a random access configuration, there may be different temporal layers, and different quantization parameters (QPs) may be used for different layers. For example, for low-temporal layer images with small QPs, there may be more precision options, and high precision such as 1 / 8 pixel may be preferred. Alternatively, the precision set {1 / 2 pixel, 1 / 4 pixel, 1 / 8 pixel} may be used. For high-temporal layer images with large QPs, there may be fewer precision options, and low precision such as 1 pixel may be preferred. Alternatively, the precision set {1 pixel, 1 / 4 pixel} or {1 pixel, 1 / 2 pixel, 1 / 4 pixel} may be used.

[0099]

[0109] In a 6-parameter affine model, the motion at the top left is related to translational motion, the difference in motion between the top right and top left is related to horizontal rotation and zoom, and the difference in motion between the bottom left and top left is related to vertical rotation and zoom. A triplet precision (p0, p1, p2) is specified for the 6-parameter affine model, where p0, p1, and p2 are the precisions of the top left, top right, and bottom left control points. One embodiment is to set the same precision for MVD signaling at both the top right and bottom left control points. For example, the precision of the three control points can be one of the sets {(1 pixel, 1 / 4 pixel, 1 / 4 pixel), (1 / 4 pixel, 1 / 4 pixel, 1 / 4 pixel), (1 / 8 pixel, 1 / 8 pixel, 1 / 8 pixel)}. Another embodiment is to set different precisions for the top right and bottom left control points. To reduce signaling overhead, it is best to have as few precision set choices as possible. In some embodiments, the precision set is selected based on the shape of the CU. If the width is equal to the height (i.e., for a square CU), the precision of the top right and bottom left can be the same, for example, the precision set is {(1 pixel, 1 / 4 pixel, 1 / 4 pixel), (1 / 4 pixel, 1 / 4 pixel, 1 / 4 pixel), (1 / 8 pixel, 1 / 8 pixel, 1 / 8 pixel)}. If the width is greater than the height (i.e., for a long CU), the precision of the top right control point can be greater than or equal to the precision of the bottom left control point, for example, the precision set is {(1 pixel, 1 / 4 pixel, 1 / 4 pixel), (1 / 4 pixel, 1 / 4 pixel, 1 / 4 pixel), (1 / 8 pixel, 1 / 8 pixel, 1 / 4 pixel)}. If the width is smaller than the height (i.e., for a tall CU), the precision of the upper right control point can be less than or equal to the precision of the lower left control point, for example, the precision set is {(1 pixel, 1 / 4 pixel, 1 / 4 pixel), (1 / 4 pixel, 1 / 4 pixel, 1 / 4 pixel), (1 / 8 pixel, 1 / 4 pixel, 1 / 8 pixel)}.

[0100]

[0110] Figure 18 shows examples of how the decoder performs the operation in several embodiments. The decoder receives a bitstream (block 1802) and reads from the bitstream information identifying at least a first motion vector predictor (block 1804) and a second motion vector predictor (block 1806), information identifying one of several precisions within a given precision set (block 1808), and at least a first differential motion vector (block 1810) and a second differential motion vector (block 1812). The first and second differential motion vectors have the precisions identified by the information read in block 1808. The syntax and semantics of how the information is encoded in the bitstream may differ depending on the embodiment. The decoder generates a first control point motion vector (block 1814) from at least the first motion vector predictor and the first differential motion vector, and a second control point motion vector (block 1816) from the second motion vector predictor and the second differential motion vector. The decoder then uses an affine motion model to generate a prediction for the current block (block 1818). The affine motion model is characterized by at least a first control point motion vector and a second control point motion vector.

[0101] Motion estimation for affine motion models using adaptive MVD accuracy

[0111] When adaptive MVD accuracy is applied to two affine control points, the encoder operates to determine the optimal accuracy that affects the coding performance of the affine motion model. The encoder also operates to apply a good motion estimation method with a given accuracy to determine the affine model parameters.

[0102]

[0112] In VVC, the flowchart for CU mode determination is shown in Figure 10, where the encoder checks different encoding modes and selects the best encoding mode with the lowest RD cost. For explicit intermodes with different translational motion model accuracies, i.e., 1 / 4 pixel, 1 pixel, and 4 pixels, there are three RD cost checking processes. To reduce encoding complexity, the 4-pixel accuracy-based RD cost is calculated only if the 1-pixel accuracy RD cost is less than or comparable to the 1 / 4 pixel RD cost. In the RD cost calculation process for 1 / 4 pixel accuracy, the encoder compares the motion estimation costs of the translational motion model and the affine motion model and selects the motion model with the lowest ME cost. The accuracy of the affine motion model is (1 / 4 pixel, 1 / 4 pixel) for two control points.

[0103]

[0113] In some embodiments, more precisions are introduced for the adaptive MVD precision of the affine motion model. For example, in addition to the precision (1 / 4 pixel, 1 / 4 pixel), (1 pixel, 1 / 4 pixel) and (1 / 8 pixel, 1 / 8 pixel) are added to the affine model. In the following description, these three precisions are used for the affine model as an example. However, other embodiments may use other precisions or combinations of more precisions. The (1 / 4 pixel, 1 / 4 pixel) precision of the affine model can be used as the default precision. To reduce complexity, a 1 / 4 pixel RD cost check process is maintained in which the (1 / 4 pixel, 1 / 4 pixel) precision affine model is evaluated. The remaining affine precision checks are added to the 1 pixel precision RD cost checks.

[0104]

[0114] Figure 11 shows a flowchart of an embodiment using RD cost checking with 1-pixel precision. One motion estimation is performed for a translational motion model with precision 1 pixel (block 1102), and two affine motion estimations are performed for precisions (1 pixel, 1 / 4 pixel) (block 1104) and (1 / 8 pixel, 1 / 8 pixel) (block 1106), respectively. The motion models and corresponding precisions are selected by comparing their ME costs (block 1108). To reduce coding complexity, these two affine motion estimations with precisions are performed only if the current best mode is an inter-encoded mode with an affine motion model, after the encoder has already checked the (1 / 4 pixel, 1 / 4 pixel) precision for the affine model. This is because different affine model precisions are only valid if the current CU has affine motion. To further reduce coding complexity, in some embodiments, the encoder may check the ME cost of the affine model only if the current best encoding mode is an affine non-merged mode or an affine non-skipped mode. This is because the merge and skip modes indicate that the current CU is already coded efficiently, and improvements may be very limited.

[0105]

[0115] The (1 pixel, 1 / 4 pixel) and (1 / 2 pixel, 1 / 4 pixel) precisions are lower than the default precision (1 / 4 pixel, 1 / 4 pixel). The precision of the upper-left control point is coarse, and it is easy for the encoder to obtain a local minimum, so optical flow-based iterative search methods have proven insufficient. Therefore, we propose a composite search method for this type of low precision. Figure 12 is a flowchart of an example of the search method.

[0106]

[0116] The optical flow-based iterative search described above in the "Affine Mode" section is first applied. Next, (MV0,MV1) is obtained as input for the next step, where MV0 is the MV of the top-left control point and MV1 is the MV of the top-right control point (block 1202). The next step is to refine MV0 by checking the eight closest adjacent positions (block 1204). An example is shown in Figure 13. If P0 is the position pointed to by MV0, then P0 has eight closest adjacent positions. The distance between P0 and P4, P1 is the precision of MV0, such as 1 pixel or 1 / 2 pixel. If MV0 is changed to point to an adjacent position, the corresponding MV1 is estimated using the optical flow-based search method, and the ME cost is calculated using the updated (MV0,MV1). These eight adjacent positions are grouped into two groups. The first group is the four closest adjacent positions {P1,P2,P3,P4}, and {P5,P6,P7,P8} is the second group. First, the ME cost at position P0 is compared with the ME costs of adjacent positions from {P1, P2, P3, P4}. If the cost at P0 is the minimum, the refinement of MV0 stops. If the ME costs of adjacent positions in the first group are lower than the ME cost at P0, then two more adjacent positions from {P5, P6, P7, P8} are further compared. For example, if the cost at P2 is the minimum in the first round, then P5 and P6 are further checked. Thus, the maximum number of cost checks is 6, not 8.

[0107]

[0117] Once MV0 is determined, MV1 is further refined (block 1204). Refinement is an iterative search of a square pattern. In each iteration, there is a center position that was in the best position in the last iteration. The encoder calculates the ME cost of eight adjacent positions, compares it to the current best ME cost, and moves the center position to a new position that has the minimum ME cost among the center position and the eight adjacent positions. If an adjacent position has already been checked in the previous iteration, that position check is skipped in the current iteration. If there is no update in the current iteration, i.e., the center is in the best position, the search ends. Alternatively, the search ends when the number of iterations reaches a predefined threshold (e.g., 8 or 16).

[0108]

[0118] For 6-parameter affine models, the search method proposed for 4-parameter affine models can be extended. Consider the case of searching for (MV0, MV1, MV2) of a 6-parameter affine motion. The search can be performed using at least three steps: initial motion search, refinement of translational motion parameters, and refinement of rotational and zoom motion parameters. The first and second steps are the same as the steps in the 4-parameter affine search. The third step is to refine both MV1 and MV2. To reduce the complexity of the search, these two can be refined using iterative refinement. For example, fix MV0 and MV2 and refine MV1 using the same scheme as the refinement of MV1 in the 4-parameter affine model. After MV1 is refined, fix MV0 and MV1 and refine MV2 using the same scheme. Then refine MV1 again. In this way, these two MVs related to rotational and zooming motions can be iteratively refined until one MV remains unchanged or the number of iterations meets a predefined threshold. To achieve rapid convergence, in this iterative refinement scheme, the starting MV for refinement may be selected as follows: The choice of MV1 or MV2 to refine first may depend on their respective precisions. Typically, the MV with lower precision is refined first. If both have the same precision, the MV with a greater distance from its control point to its top-left control point may be selected.

[0109]

[0119] To further reduce coding complexity, CU size and temporal layers may be considered when the encoder tests various accuracies at control points for affine model-based coding. Accuracy determination may only be performed for large CUs. For example, an exemplary accuracy determination method may only be applied to CUs with an area larger than a predefined threshold (e.g., 16x16). For CUs with an area smaller than the threshold, (1 / 4 pixel, 1 / 4 pixel) accuracy is used for the two control points. For images with different temporal layers and different QP settings, the encoder can only test their possible accuracies at each temporal layer. For example, only (1 pixel, 1 / 4 pixel)(1 / 4 pixel, 1 / 4 pixel) may be tested for images of higher temporal layers (e.g., the top temporal layer image). Also, only (1 / 4 pixel, 1 / 4 pixel)(1 / 8 pixel, 1 / 8 pixel) may be tested for images of lower temporal layers (e.g., the bottom temporal layer image). For these intermediate layer images, the entire set of accuracies may be tested.

[0110] Subblock-based affine motion compensation and estimation

[0120] Affine motion estimation is an iterative estimation process. In each iteration, the temporal difference between the original signal and the motion-compensated prediction signal using the current motion vector, the spatial gradient, and the relationships between the local affine parameters (a, b, c, d in equation (3)) are expressed by equation (7) based on the optical flow equation. However, to reduce the memory access bandwidth on the decoder side, affine motion compensation prediction is based on subblocks (e.g., 4x4) rather than samples. This is because, typically, when the motion vector points to a decimal position, there is an interpolation filter to derive the sample value in motion compensation. This interpolation process can significantly improve predictions compared to directly using the sample value of the nearest adjacent integer position. However, the interpolation refers to multiple adjacent samples at integer positions. Given the MV at a control point, the MV of each subblock can be derived based on the center position of the subblock using equation (1). If the subblock size is 1x1, motion compensation is sample-based, meaning that the motion of each sample can be different. Suppose there is a separable interpolation filter with tap length N and subblock size SxS. For each sample, it operates to fetch (S+N-1)x(S+N-1) integer samples enclosing the reference position pointed to by the MV for interpolation in both the horizontal and vertical directions. On average, it operates to fetch ((S+N-1)x(S+N-1) / (SxS)) reference samples at integer positions for each sample. In the case of sample-based affine motion compensation where S is equal to 1, this is NxN. For example, in HEVC and VTM, N is 8 and with a subblock size of 4x4, the memory access per sample is (121 / 16). In the case of sample-based interpolation, the memory access per sample is 64, which is 8.5 times more than 4x4 subblock-based motion compensation. Therefore, subblock-based motion compensation is used for affine motion prediction. The affine motion estimation methods described in the "Affine Modes" section use sample-based prediction and do not consider this subblock-based motion compensation.From equation (3), we can see that the difference motion of each position is related to its position within the CU given affine parameters. Therefore, if we derive the motion of all samples within a subblock using equation (3) with respect to the center position of the subblock, then the samples belonging to the same subblock will have the same difference motion. For example, if the location of a sample is (i,j) within the CU, the center position of the subblock to which it belongs is evaluated as given by equation (24).

number

[0111]

[0121] Next, equation (3) gives (i,j) b Today J b Substituting ) into the equation results in equation (25).

number

[0112]

[0122] Using equation (25), in equation (6)

number

number

[0113]

[0123] In some embodiments, equation (26) is used to estimate the optimal affine parameters (a, b, c, d) using the least squares method. In such embodiments for motion estimation, the differential motion of samples belonging to one subblock is the same. Therefore, the final MV at the control point is more accurate with subblock-based motion compensation prediction compared to sample-based estimation methods using equation (7).

[0114]

[0124] In affine motion compensation, the position used to derive the subblock MV within the CU may not be the actual center position of the subblock. As shown in Figure 17, the affine CU is 8x4, and the subblock size for motion compensation is 4x4. The position used to derive the subblock MV can be calculated by equation (24), given the sample position (i,j). These positions are P0 and P1 of the left 4x4 subblock and the right 4x4 subblock, respectively. Based on the coordinates of P0 and P1, the MV is derived by equation (1) for a 4-parameter affine model, or by equations (8) and (9) for a 6-parameter affine model. However, using equation (24), P0 and P1 are not the centers of these two subblocks. MV0 and MV1 may not be accurate in the subblock motion compensation prediction. In one embodiment, we propose using equation (27) to calculate the position for subblock MV derivation.

number

[0115]

[0125] By equation (27), P0 is replaced by P0', and P0' is the center of the left 4x4 subblock. Thus, the corresponding MV0' is more accurate than MV0. In the affine motion estimation method described herein to improve the accuracy of affine motion estimation, equation (27) can be replaced by equation (24). Given the MV at the control point of the affine encoded CU, the MV of the subblock of the chroma component can reuse the MV of the luminance component or be derived individually using equation (27).

[0116] Affin MVD rounding

[0126] In some implementations of affine motion compensation, the subblock MV derived by the control point MV has 1 / 16 pixel precision, while the control point MV is rounded to 1 / 4 pixel precision. The control point MV is derived by adding the MVD to the MV predictor. The MVD is signaled with 1 / 4 pixel precision. The MV predictor is rounded to 1 / 4 pixel precision before being used to derive the control point MV. Due to the adaptive affine MVD precision, the MV predictor used to derive the control point MV of the current coded block may have higher precision than the MV precision of the current CU. In this case, the MV predictor is rounded to lower precision. Rounding results in a loss of information. In some embodiments proposed herein, the control point MV and MV predictor are kept at the highest precision, e.g., 1 / 16 pixel, while the MVD is rounded to a desired precision.

[0117]

[0127] In affine motion estimation, affine parameters can be estimated iteratively. For each iteration, the difference control point MV can be derived using the optical flow method as shown in equations (4) and (5). In the implementation in VTM, the control point MV at step k is updated by the following equation:

number

number

number

number

number

[0118]

[0128] In exemplary embodiments of the method provided herein, the control point MV in step k is updated by the following step: The upper left control point MV is updated according to equations (29) to (31).

number

number

[0119]

[0129] In equations (29) to (34),

number

[0120]

[0130] MVP i Because it has a pixel precision of 1 / 16,

number

[0121] Adaptive Affine MVD Coding

[0131] Affine MVDs with different accuracies can have different properties. Control point MVDs can have different physical meanings. For example, the absolute value of the MVD can be smaller on average for (1 / 8 pixel, 1 / 8 pixel, 1 / 8 pixel) or (1 / 16 pixel, 1 / 16 pixel, 1 / 16 pixel) accuracies compared to (1 / 4 pixel, 1 / 4 pixel, 1 / 4 pixel) accuracies. As described in the "MVD Coding" section above, the length of EG codes with different orders differs. Generally, for smaller EG orders, the length of EG codes with smaller values ​​is shorter, and for larger EG orders, the length of EG codes is longer. Some embodiments use adaptive EG orders for MVD coding that take into account the MVD accuracy and the meaning of its physical motion (e.g., rotation in different directions, zoom). In some embodiments, the upper left MVD (MVD 0x MVD 0y ) is MVD component MVD 0x and MVD 0y Since it is of translational motion, it has the same EG order as non-affine MVD coding. In the case of a 6-parameter affine model, the MVD component MVD 1y and MVD 2x This relates to rotational motion, and the MVD component MVD 1x and MVD 2y This relates to zoom operation. In the case of a 4-parameter affine model, the MVD component is MVD 1y This relates to rotational motion, and the MVD component MVD 1x This relates to the zoom function.

[0122]

[0132] In some embodiments, the MVD values ​​have different characteristics, so different MVD coding results in different EG coding orders. In some embodiments, the MVD is related to translational motion (MVD 0x MVD 0y In this case, the EG order is not signaled, and instead, such MVD may use the same EG order (e.g., 1) as non-affine MVD coding.

[0123]

[0133] In some embodiments, for the exponential Golomb codes used for different MVD components corresponding to non-translational motions such as the MVD components described in Table 9 for three MVD precisions, the EG degree is signaled. In the embodiment of Table 9, six EG degrees (EG degree [0] to EG degree [5]) are signaled in the bitstream. The degree range of EG is 0 to 3, and 2 bits are used for encoding. The MVD precision indicator indicates different MVD precisions. For example, the MVD precision indicator "0" represents the (1 / 4 pixel, 1 / 4 pixel, 1 / 4 pixel) precision, the MVD precision indicator "1" represents the (1 / 16 pixel, 1 / 16 pixel, 1 / 16 pixel) precision, and the MVD precision indicator "2" represents the (1 pixel, 1 pixel, 1 pixel) precision. These signaled EG degrees are for indicating the EG degrees used for the EG binarization of different MVD components with different MVD precisions. For example, EG degree [0] is for MVD components MVD 1y and MVD 2x in the state where the MVD precision indicator is "0" (i.e., the (1 / 4 pixel, 1 / 4 pixel, 1 / 4 pixel) precision set). In the case of a 4-parameter affine model, there is no need to encode MVD 2x and MVD 2y , and in Table 9 only MVD 1x and MVD 1y are encoded.

[0124]

Table 9

[0125]

[0134] The signaling of the EG degree can be performed, for example, in an image parameter set or a slice header. In an embodiment where the EG degree is signaled in the slice header, the encoder can select the EG degree based on the previously encoded image in the same temporal layer. After each inter-image is encoded, the encoder can compare the total number of bins using different EG codes of different degrees for all MVDs within its category. For example, all MVDs with an MVD precision of "0" 1y and MVD 2xThe encoder compares the total number of bins for EG order 0, EG order 1, EG order 2, and EG order 3, and selects the order with the smallest total number of bins. The selected order is then used for encoding the next image in the same temporal layer, and the selected order is also encoded in the slice header of the next image in the same temporal layer.

[0126] Further Embodiments

[0135] In some embodiments, a method is provided for decoding video from a bitstream. The method includes reading information from the bitstream for at least one block in the video that identifies one of several elements in a selected predetermined set of precisions, such that the identified element of the selected predetermined set of precisions indicates at least a selected first precision and a selected second precision; and decoding the block using an affine motion model, the affine motion model characterized by a first motion vector having at least a selected first precision and a second motion vector having a selected second precision. The method may include reading information from the bitstream that indicates the first and second motion vectors. The information that indicates the first and second motion vectors may include a first differential motion vector and a second differential motion vector.

[0127]

[0136] In some embodiments, information identifying one of several elements is read from the bitstream in block units.

[0128]

[0137] In some embodiments, a first motion vector is associated with a first control point of the block, and a second motion vector is associated with a second control point of the block.

[0129]

[0138] In some embodiments, each element of a selected predetermined set of precisions includes a first available precision and a second available precision. The second available precision may be greater than or equal to the first available precision.

[0130]

[0139] In some embodiments, information identifying a predetermined precision set selected from a plurality of available predetermined precision sets is read from the bitstream. In some such embodiments, the information identifying the selected predetermined precision set is signaled in an image parameter set, a sequence parameter set, or a slice header. Examples of predetermined position sets include: {(1 pixel, 1 / 4 pixel), (1 / 4 pixel, 1 / 4 pixel)}, {(1 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 8 pixel)}, {(1 pixel, 1 / 4 pixel), (1 / 4 pixel, 1 / 4 pixel), (1 / 8 pixel, 1 / 8 pixel)}, and {(1 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 8 pixel),(1 / 8 pixel, 1 / 8 pixel)}.

[0131]

[0140] In some embodiments, the affine motion model is further characterized by a third motion vector having a selected third precision, where identified elements of a predetermined set of precisions further indicate the selected third precision.

[0132]

[0141] In some embodiments, information identifying one of several elements is encoded in the bitstream using context-adaptive binary arithmetic coding.

[0133]

[0142] In some embodiments, a determination is made as to whether the size of a block is greater than a threshold size, and information identifying one of several elements is read from the bitstream for that block only if the block size is greater than the threshold size.

[0134]

[0143] In some embodiments, a selected predetermined set of precision is chosen based on the temporal layer of the image containing the blocks.

[0135]

[0144] In some embodiments, a predetermined set of precision is selected based on the shape of the block.

[0136]

[0145] In some embodiments, a method is provided for decoding video in a bitstream. The method includes reading from the bitstream (i) first information indicating a first precision from a first predetermined set of available precisions, and (ii) second information indicating a second precision from a second predetermined set of available precisions, for at least one block in the video, and decoding the block using an affine motion model, wherein the affine motion model is characterized by a first motion vector having at least a selected first precision and a second motion vector having a selected second precision, and signaling in the bitstream (i) first information indicating a first precision from a first predetermined set of available precisions, and (ii) second information indicating a second precision from a second predetermined set of available precisions. The first predetermined set and the second predetermined set may be different.

[0137]

[0146] In some embodiments, a first predetermined set is {1 pixel, 1 / 4 pixel, 1 / 8 pixel}, and a second predetermined set is {1 / 2 pixel, 1 / 4 pixel, 1 / 8 pixel}.

[0138]

[0147] In some embodiments, a first motion vector is associated with a first control point of the block, and a second motion vector is associated with a second control point of the block.

[0139]

[0148] In some embodiments, a method is provided for encoding video in a bitstream. The method comprises encoding at least one block in the video using an affine motion model, wherein the affine motion model is characterized by a first motion vector having at least a selected first precision and a second motion vector having a selected second precision, and signaling information in the bitstream that identifies one of a plurality of elements in a selected predetermined precision set, wherein the identified element of the selected predetermined precision set indicates at least a selected first precision and a selected second precision. The method may further include signaling information in the bitstream that indicates the first and second motion vectors. The information that indicates the first and second motion vectors may include a first difference motion vector and a second difference motion vector.

[0140]

[0149] In some embodiments, information identifying one of several elements is transmitted in block units.

[0141]

[0150] In some embodiments, a first motion vector is associated with a first control point of the block, and a second motion vector is associated with a second control point of the block.

[0142]

[0151] In some embodiments, each element of a selected predetermined set of precisions includes an available first precision and an available second precision. In some embodiments, the available second precision is greater than or equal to the available first precision.

[0143]

[0152] In some embodiments, the method includes signaling information in the bitstream that identifies a predetermined set of precision selected from a plurality of available sets of precision. The information identifying the selected set of precision may be signaled, for example, in an image parameter set, a sequence parameter set, or a slice header.

[0144]

[0153] Examples of predetermined position sets include the following: {(1 pixel, 1 / 4 pixel), (1 / 4 pixel, 1 / 4 pixel)}, {(1 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 8 pixel)}, {(1 pixel, 1 / 4 pixel), (1 / 4 pixel, 1 / 4 pixel), (1 / 8 pixel, 1 / 8 pixel)}, and {(1 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 4 pixel),(1 / 4 pixel, 1 / 8 pixel),(1 / 8 pixel, 1 / 8 pixel)}.

[0145]

[0154] In some embodiments, the affine motion model is further characterized by a third motion vector having a selected third precision, where identified elements of a predetermined set of precisions further indicate the selected third precision.

[0146]

[0155] In some embodiments, information identifying one of several elements is encoded in the bitstream using context-adaptive binary arithmetic coding.

[0147]

[0156] In some embodiments, the method includes determining whether the size of a block is greater than a threshold size, wherein information identifying one of a plurality of elements is signaled in the bitstream for that block only if the size of the block is greater than the threshold size.

[0148]

[0157] In some embodiments, a selected predetermined set of precision is chosen based on the temporal layer of the image containing the blocks.

[0149]

[0158] In some embodiments, a predetermined set of precision is selected based on the shape of the block.

[0150]

[0159] In some embodiments, a method is provided for encoding video in a bitstream. The method comprises encoding at least one block in the video using an affine motion model, wherein the affine motion model is characterized by a first motion vector having at least a selected first precision and a second motion vector having a selected second precision, and signaling in the bitstream (i) first information indicating a first precision from a first predetermined set of available precisions, and (ii) second information indicating a second precision from a second predetermined set of available precisions. The first predetermined set and the second predetermined set may be different.

[0151]

[0160] In some embodiments, a first predetermined set is {1 pixel, 1 / 4 pixel, 1 / 8 pixel}, and a second predetermined set is {1 / 2 pixel, 1 / 4 pixel, 1 / 8 pixel}.

[0152]

[0161] In some embodiments, a first motion vector is associated with a first control point of the block, and a second motion vector is associated with a second control point of the block.

[0153]

[0162] Some embodiments include a method for encoding video in a bitstream, the method comprising: determining a first rate-distortion cost for encoding at least one block in the video using a translational motion model; determining a second rate-distortion cost for encoding the block using an affine prediction model with a first set of affine model precisions; determining whether the second rate-distortion cost is less than the first rate-distortion cost; determining at least a third rate-distortion cost for encoding the block using an affine prediction model with a second set of affine model precisions, in response to the determination that the second rate-distortion cost is less than the first rate-distortion cost; and encoding the block in the bitstream using an encoding model associated with the lowest determined rate-distortion cost.

[0154]

[0163] In some embodiments, in response to a determination that a second rate-distortion cost is less than a first rate-distortion cost, a fourth rate-distortion cost for encoding a block using an affine prediction model with a fourth set of affine model accuracies is determined.

[0155]

[0164] In some embodiments, a method for encoding video in a bitstream is provided. The method includes, for at least one block in the video, determining affine parameters a, b, c, and d using the following equation I’ k (i,j) - I(i,j) = (g x (i,j) * i b + g y (i,j) * j b ) * c + (-g x (i,j) * j b + g y (i,j) * i b ) * d + g x (i,j) * a + g y (i,j) * b where I(i,j) is the original luminance signal, I’ (i,j) is the predicted luminance signal, g k (i,j) and g x (i,j) are the spatial gradients applied to I’ y (i,j), and k where S is a sub-block size greater than 1, and encoding a block in the bitstream using the determined affine parameters a, b, c, and d.

Number

[0157]

[0166] In some embodiments, the method is executed by an encoder, and identifying the differential motion vector includes determining a motion vector difference of the control point based on an initial motion vector, updating the differential motion vector based on the motion vector difference, rounding the differential motion vector to the second accuracy, and repeatedly adding the rounded differential motion vector to the motion vector predictor to generate an updated motion vector, a motion vector predictor, and an updated motion vector having the first accuracy.

[0158]

[0167] In some embodiments, the first accuracy is 1 / 16 pixel accuracy and the second accuracy is 1 / 4 pixel accuracy.

[0159]

[0168] In some embodiments, predicting the block by affine prediction is performed using two control points, each differential motion vector is identified for each control point, and each differential motion vector has the second accuracy.

[0160]

[0169] In some embodiments, block prediction is performed using affine prediction with three control points, where the respective differential motion vector is identified for each control point, and each respective differential motion vector has a second accuracy.

[0161]

[0170] In some embodiments, a method is provided for decoding video from a bitstream. The method includes determining the coding order for each of a plurality of differential motion vector (MVD) components for at least one block in the video, based at least partially on the information encoded in the bitstream; reading each of the MVD components from the bitstream using the determined coding order; and decoding the block using an affine motion model, wherein the affine motion model is at least partially characterized by the MVD components.

[0162]

[0171] In some embodiments, the method includes reading information from the bitstream that identifies the precision of each MVD component, wherein the coding order of the MVD component is determined based on a portion of the precision of each component. The MVD component may be coded using exponential Golomb coding, and the coding order may be the exponential Golomb coding order.

[0163]

[0172] Some embodiments include a method for decoding video from a bitstream. The method includes determining the coding order for each of a plurality of differential motion vector (MVD) components for at least one block in the video, wherein the coding order for each MVD component is determined on the basis of (i) the precision of the MVD component and (ii) whether the component relates to rotational motion or zoom motion; reading each of the MVD components from the bitstream using the respective determined coding orders; and decoding the block using an affine motion model, wherein the affine motion model is at least partially characterized by the MVD components.

[0164]

[0173] Some embodiments involve reading order information from a bitstream, where the order information is (i) 1 / 4 pixel precision and (ii) a first coding order associated with rotational motion, (i) 1 / 4 pixel precision and (ii) a second coding order associated with zoom motion, (i) 1 / 16 pixel precision and (ii) a third coding order associated with rotational motion, (i) 1 / 16 pixel precision and (ii) a fourth coding order associated with zoom motion, (i) pixel precision and (ii) a fifth coding order associated with rotational motion, (i) pixel precision and (ii) a sixth coding order associated with zoom motion, This further includes identifying the specifics. Each encoding order is performed using order information. This order information can be encoded, for example, in an image parameter set or slice header.

[0165]

[0174] In some embodiments, the MVD component is encoded using exponential Golomb coding, and the coding order is the exponential Golomb coding order.

[0166]

[0175] In some embodiments, a method is provided for encoding video in a bitstream, the method comprising: selecting order information for at least one block in the video, wherein the order information identifies the coding order of a difference motion vector (MVD) component based on (i) the precision of the MVD component and (ii) whether the component relates to rotational motion or zoom motion; encoding the order information into a bitstream; and encoding the block using an affine motion model, wherein the affine motion model is at least partially characterized by a plurality of MVD components, each of which is encoded into a bitstream using the coding order determined by the order information.

[0167]

[0176] In some embodiments, the order information is (i) 1 / 4 pixel precision and (ii) a first coding order associated with rotational motion, (i) 1 / 4 pixel precision and (ii) a second coding order associated with zoom motion, (i) 1 / 16 pixel precision and (ii) a third coding order associated with rotational motion, (i) 1 / 16 pixel precision and (ii) a fourth coding order associated with zoom motion, (i) pixel precision and (ii) a fifth coding order associated with rotational motion, (i) pixel precision and (ii) a sixth coding order associated with zoom motion, Identify. Determining the encoding order for each image can be done using order information. This order information can be encoded, for example, in an image parameter set or slice header.

[0168]

[0177] In some embodiments, the MVD component is encoded using exponential Golomb coding, and the coding order is the exponential Golomb coding order.

[0169]

[0178] Some embodiments include a non-temporary computer-readable storage medium for storing video encoded using any of the methods disclosed herein. Some embodiments also include a non-temporary computer-readable storage medium for storing instructions that operate to perform any of the methods disclosed herein.

[0170] Encoded bitstream structure

[0179] Figure 15 shows an example of an encoded bitstream structure. The encoded bitstream 1300 consists of several NAL (Network Abstraction Layer) units 1301. NAL units can contain encoded sample data such as encoded slices 1306, or high-level syntax metadata such as parameter set data, slice header data 1305, or supplemental additional information data 1307 (sometimes called SEI messages). A parameter set is a high-level syntax structure containing important syntax elements that can be applied to multiple bitstream layers (e.g., video parameter set 1302 (VPS)), or to an encoded video sequence within a single layer (e.g., sequence parameter set 1303 (SPS)), or to several encoded images within a single encoded video sequence (e.g., image parameter set 1304 (PPS)). Parameter sets can be transmitted with the encoded images of the video bitstream, or through other means (including out-of-band transmission using reliable channels, hard encoding, etc.). The slice header 1305 is also a high-level syntax structure that can contain relatively small or some image-related information relevant only to a specific slice or image type. The SEI message 1307 carries information that may not be required by the decoding process but may be used for various other purposes such as image output timing or display and / or loss detection and concealment.

[0171] Communication equipment and systems

[0180] Figure 16 shows an example of a communication system. The communication system 1400 may include an encoder 1402, a communication network 1404, and a decoder 1406. The encoder 1402 can communicate with the communication network 1404 via a connection 1408, which may be a wired or wireless connection. The encoder 1402 may be similar to the block-based video encoder in Figure 2A. The encoder 1402 may include a single-layer codec (e.g., Figure 2A) or a multi-layer codec. The decoder 1406 can communicate with the communication network 1404 via a connection 1410, which may be a wired or wireless connection. The decoder 1406 may be similar to the block-based video decoder in Figure 2B. The decoder 1406 may include a single-layer codec (e.g., Figure 2B) or a multi-layer codec.

[0172]

[0181] The encoder 1402 and / or decoder 1406 may be incorporated into a wide variety of wired communication devices and / or wireless transmit / receive units (WTRUs), including but not limited to digital televisions, wireless broadcast systems, network elements / terminals, servers such as content servers or web servers (e.g., hypertext transfer protocol (HTTP) servers), personal digital assistants (PDAs), laptop or desktop computers, tablet computers, digital cameras, digital recording devices, video game devices, video game consoles, cellular or satellite radiotelephones, digital media players, and / or similar devices.

[0173]

[0182] Communication network 1404 can be a suitable type of communication network. For example, communication network 1404 can be a plurality of access systems that provide content such as voice, data, video, messages, broadcasts, etc. to a plurality of wireless users. Communication network 1404 can enable a plurality of wireless users to access such content through sharing of system resources including wireless bandwidth. For example, communication network 1404 can utilize one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), and / or the like. Communication network 1404 can include a plurality of connected communication networks. Communication network 1404 can include one or more commercial private networks such as the Internet and / or a cellular network, a Wi-Fi hotspot, an Internet service provider (ISP) network, and / or the like.

[0174]

[0183] It should be noted that one or more of the various hardware elements in the embodiments described herein are referred to as “modules” that perform (i.e., do, execute, etc.) the various functions described herein in relation to their respective modules. Where used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that a person skilled in the art would deem appropriate for a given implementation. It should be noted that each module described may also include executable instructions for performing one or more functions described as being performed by the respective module, and these instructions may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or similar, and may be stored in any suitable one or more non-temporary computer-readable media, commonly referred to as RAM, ROM, etc.

[0175]

[0184] While the features and elements have been described above in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with other features and elements. In addition, the methods described herein can be implemented in computer programs, software, or firmware embedded in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as built-in hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital multipurpose disks (DVDs). A processor in conjunction with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method for video encoding, The process of repeatedly selecting a set of affine parameters for predicting the current block, wherein the selection is Using a candidate set of affine parameters, obtain a candidate prediction signal for the current block, and This is done by refining the candidate set of affine parameters, at least partially based on the gradient of the candidate prediction signal, the difference between the input block and the candidate prediction signal, and the respective positions associated with each sample of the current block. The current block includes multiple subblocks, each subblock includes multiple samples, and all samples within the same subblock use the same position to refine the candidate set of affine parameters, and are selected. Encoding the current block using the selected set of affine parameters, Methods that include...

2. The method according to claim 1, wherein each subblock is a 4x4 subblock.

3. Repeatedly selecting the aforementioned set of affine parameters Based on the initial set of affine parameters, the candidate predicted luminance signal I' ｋ To determine (i, j), This includes selecting an updated set of affine parameters a, b, c, and d based on the following formula, [Math 1] In the above equation, I(i,j) is the input luminance signal, and g x (i, j) is the horizontal gradient of the candidate predicted luminance signal, and g y (i, j) is the vertical gradient of the candidate predicted brightness signal, and (i, j) is the sample coordinate, and (i b ,j b ) is the coordinates of the center of the subblock containing the sample (i,j). The method according to claim 1.

4. The method according to claim 3, wherein the updated set of affine parameters a, b, c, and d is selected using the least squares method.

5. The subblock has a size S, [Math 2] The method according to claim 3.

6. The subblock has a size S, [Math 3] The method according to claim 3.

7. The method according to claim 1, wherein encoding the current block using a selected set of affine parameters includes encoding a control point motion vector based on the selected set of affine parameters.

8. Based on the control point motion vector, the last predicted signal for the current block is obtained, The residual is determined by subtracting the last predicted signal from the input block, The residual is encoded within the bitstream, The method according to claim 7, further comprising:

9. A video encoding device comprising one or more processors, wherein the one or more processors comprises at least, The process of repeatedly selecting a set of affine parameters for predicting the current block, wherein the selection is Using a candidate set of affine parameters, obtain a candidate prediction signal for the current block, and This is done by refining the candidate set of affine parameters, at least partially based on the gradient of the candidate prediction signal, the difference between the input block and the candidate prediction signal, and the respective positions associated with each sample of the current block. The current block includes multiple subblocks, each subblock includes multiple samples, and all samples within the same subblock use the same position to refine the candidate set of affine parameters, and are selected. Encoding the current block using the selected set of affine parameters, A device configured to perform the following actions.

10. The apparatus according to claim 9, wherein each subblock is a 4x4 subblock.

11. Repeatedly selecting the aforementioned set of affine parameters Based on the initial set of affine parameters, the candidate predicted luminance signal I' ｋ To determine (i, j), This includes selecting an updated set of affine parameters a, b, c, and d based on the following formula, [Math 4] In the above formula, I(i, j) is an input luminance signal, and g x (i, j) is the horizontal gradient of the candidate predicted luminance signal, and g y (i, j) is the vertical gradient of the candidate predicted luminance signal, ‘(i, j) is a sample coordinate, and (i b , j b ) is the coordinate of the center of the sub-block including the sample (i, j). The apparatus according to claim 9.

12. The apparatus according to claim 11, wherein the updated set of affine parameters a, b, c, and d is selected using the least squares method.

13. The subblock has a size S, [Math 5] The apparatus according to claim 11.

14. The subblock has a size S, [Math 6] The apparatus according to claim 11.

15. The apparatus according to claim 9, wherein encoding the current block using a selected set of affine parameters includes encoding a control point motion vector based on the selected set of affine parameters.

16. Based on the control point motion vector, the last predicted signal for the current block is obtained, The residual is determined by subtracting the last predicted signal from the input block, The residual is encoded within the bitstream, The apparatus according to claim 15, further configured to perform the following.

17. One or more processors, at least, The process of repeatedly selecting a set of affine parameters for predicting the current block, wherein the selection is Using a candidate set of affine parameters, obtain a candidate prediction signal for the current block, and This is done by refining the candidate set of affine parameters, at least partially based on the gradient of the candidate prediction signal, the difference between the input block and the candidate prediction signal, and the respective positions associated with each sample of the current block. The current block includes multiple subblocks, each subblock includes multiple samples, and all samples within the same subblock use the same position to refine the candidate set of affine parameters, and are selected. Encoding the current block using the selected set of affine parameters, A non-temporary computer-readable medium containing instructions for executing a command.

18. Repeatedly selecting the aforementioned set of affine parameters Based on the initial set of affine parameters, the candidate predicted luminance signal I' ｋ To determine (i, j), This includes selecting an updated set of affine parameters a, b, c, and d based on the following formula, [Number 7] In the above equation, I(i,j) is the input luminance signal, and g x (i, j) is the horizontal gradient of the candidate predicted luminance signal, and g y (i, j) is the vertical gradient of the candidate predicted brightness signal, and (i, j) is the sample coordinate, and (i b ,j b ) is the coordinates of the center of the subblock containing the sample (i,j). The computer-readable medium according to claim 17.

19. The computer-readable medium according to claim 18, wherein the updated set of affine parameters a, b, c, and d is selected using the least squares method.

20. The computer-readable medium according to claim 17, wherein encoding the current block using the selected set of affine parameters includes encoding control point motion vectors based on the selected set of affine parameters.

Images