Affine motion model derivation method

Abstract

Systems and methods for video coding using affine motion prediction are described. In an example method, a motion vector gradient is determined from respective motion vectors of a plurality of neighboring sub-blocks adjacent to a current block. Based on the motion vector gradient, an estimate of at least one affine parameter of the current block is determined. Based at least in part on the estimated affine parameter(s), an affine motion model is determined, and a prediction of the current block is generated using the affine motion model. The estimated parameter(s) can be used in the affine motion model itself. Alternatively, the estimated parameter(s) can be used in the prediction of the affine motion model. In some embodiments, only neighboring sub-blocks above and / or to the left of the current block are used to estimate the affine parameter(s).

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional application of U.S. Provisional Patent Application No. 62 / 814,125, entitled “Affine Motion Model Derivation Method,” filed on March 5, 2019, and claims the benefit of 35 U.S.C. §119(e), which is incorporated herein by reference in its entirety. Background Art

[0003] Video coding systems are widely used to compress digital video signals to reduce the storage requirements and / or transmission bandwidth of such signals. Among various types of video coding systems (e.g., block-based, wavelet-based, and object-based systems), block-based hybrid video coding systems are the most widely used and deployed today. Examples of block-based video coding systems include international video coding standards such as MPEG-1 / 2 / 4 Part 2, H.264 / MPEG-4 Part 10 AVC, VC-1, and the latest video coding standard called High Efficiency Video Coding (HEVC), which was developed by ITU-T / SG16 / Q.6 / VCEG and JCT-VC (Joint Collaboration on Video Coding) of ISO / IEC / MPEG.

[0004] In October 2017, ITU-T and ISO / IEC issued a Call for Proposals (CfP) on video compression with capabilities beyond HEVC. In April 2018, at the 10th JVET meeting, 22 CfP responses for the standard dynamic range category were received and evaluated, demonstrating a compression efficiency gain of about 40% over HEVC. Based on such evaluation results, the Joint Video Expert Team (JVET) initiated a new project to develop a new generation of video coding standard, referred to as Versatile Video Coding (VVC). In the same month, a reference software codebase, called VVC Test Model (VTM), was established to demonstrate a reference implementation of the VVC standard. For the initial VTM-1.0, most of the coding modules, including intra prediction, inter prediction, transform / inverse transform and quantization / dequantization, and in-loop filters, follow the existing HEVC design, except that a multi-type tree based block partitioning structure is used in VTM. Meanwhile, to facilitate the evaluation of new coding tools, another reference software library, called Benchmarking Set (BMS), is also generated. In the BMS codebase, a list of coding tools inherited from JEM, which provides higher coding efficiency and moderate implementation complexity, is included on top of VTM and used as a benchmark when similar coding techniques are evaluated during the VVC standardization process. Specifically, 9 JEM coding tools are integrated in BMS-1.0, including 65 angular intra prediction directions, modified coefficient coding, advanced multi-type transform (AMT) + 4x4 non-separable secondary transform (NSST), affine motion model, generalized adaptive loop filter (GALF), advanced temporal motion vector prediction (ATMVP), adaptive motion vector precision, decoder-side motion vector refinement (DMVR), and linear model (LM) chroma mode. SUMMARY

[0005] Embodiments described herein include methods used in video encoding and decoding (collectively, “coding”).

[0006] In some embodiments, a method includes determining, for at least one current block in a video, at least one motion vector gradient from respective motion vectors of a plurality of neighboring sub-blocks adjacent to the current block; determining, based on the at least one motion vector gradient, an estimate of at least one affine parameter for the current block; determining an affine motion model based at least in part on the determined estimate of the affine parameter; and generating a prediction of the current block using the affine motion model.

[0007] In some embodiments, determining the affine motion model includes determining a plurality of affine parameters of the affine motion model, and the determined estimate is used as at least one of the plurality of affine parameters.

[0008] In some embodiments, determining an affine motion model comprises: determining a plurality of affine parameters of an affine motion model predictor, wherein the determined estimate is used as at least one of the plurality of affine parameters of the affine motion model predictor; wherein the affine motion model is predicted by using the affine motion model predictor.

[0009] In some embodiments, determining an affine motion model comprises: determining an initial motion model using the determined estimate of the affine parameters; providing the initial motion model as an input to an optical flow based affine model refinement process; and generating the affine motion model using the affine model refinement process.

[0010] In some embodiments, generating the prediction of the current block comprises: determining a sub-block motion vector for at least one sub-block in the current block based on the affine motion model.

[0011] In some embodiments, the sub-block motion vector for at least one sub-block is determined according to:

[0012] and

[0013] wherein the determined estimate is used as at least one of the values a xx , a xy , a yx or a yy .

[0014] In some embodiments, the sub-block motion vector for at least one sub-block is determined according to:

[0015] and

[0016] wherein the determined estimate is used as at least one of the values a xx , a xy , a yx or a yy .

[0017] In some embodiments, the method further comprises, at the encoder: calculating a residual representing a difference between an input block and the prediction of the current block; and encoding the residual in a bitstream.

[0018] In some embodiments, the method further comprises, at the decoder: decoding a residual associated with the current block from a bitstream; and reconstructing the current block by adding the residual to the affine prediction.

[0019] In some embodiments, the affine parameters are a pqwhere p and q are each a direction selected from the group comprising x and y, and where the determined a pq is an average of q-directional gradients of p-components of motion vectors of the plurality of neighboring sub-blocks.

[0020] In further embodiments, encoder and decoder apparatuses are provided to perform the methods described herein. An encoder or decoder device can include a processor configured to perform the methods described herein. The apparatus can include a computer readable medium (e.g., non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, the computer readable medium (e.g., non-transitory medium) stores a video encoded using any of the methods described herein. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1A is a system diagram illustrating an example communications system in which one or more disclosed embodiments can be implemented.

[0022] Figure 1B is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that can be used within the communications system 100 of Figure 1A illustrated in FIG. 1.

[0023] Figure 2A is a functional block diagram of a block-based video encoder (e.g., an encoder for VVC).

[0024] Figure 2B is a functional block diagram of a block-based video decoder (e.g., a decoder for VVC).

[0025] Figures 3A-3B illustrates a four-parameter affine motion model, which includes an affine model Figure 3A and sub-block level motion derivation for affine blocks Figure 3B .

[0026] Figure 4 illustrates a six-parameter affine motion model. V0, V1, and V2 are control points, while (MVx, MVy) is a motion vector for a sub-block centered at position (x, y).

[0027] Figure 5A illustrates neighboring motion vectors used in RMVF motion parameter derivation.

[0028] Figure 5B illustrates reduced neighboring motion vector candidates for RMVF motion parameter derivation.

[0029] Figure 6 is a diagram for a current block (current CU) and a plurality of neighboring sub-blocks.

[0030] Figure 7 FIG. 1 is a diagram illustrating an example of a coded bitstream structure.

[0031] Figure 8 FIG. 2 is a diagram illustrating an example communication system.

[0032] Figures 9A-9B FIG. 3 is a flowchart illustrating video encoding (9A) and decoding (9B) methods performed in some embodiments.

[0033] Example network for implementation of embodiments

[0034] Figure 1A FIG. 1 is a schematic diagram illustrating an example communication system 100 in which one or more disclosed embodiments can be implemented. The communication system 100 can be a multiple access system for com municating with multiple wireless users; by sharing the available system resources (for example, time, frequency, and power). The communication system 100 can employ one or more channel access schemes, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0035] As Figure 1AAs shown, the communication system 100 can include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104, a CN 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. The WTRUs 102a, 102b, 102c, 102d can be any type of device configured to

[0036] The communication system 100 can also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b can be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106, the Internet 110, and / or the other networks 112. By way of example, the base stations 114a, 114b can be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a new radio (NR) Node B, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b can include any number of interconnected base stations and / or network elements.

[0037] The base stations 114a can be part of the RAN 104, which can also include other base stations and / or network equipment (not shown) such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base stations 114a and / or the base stations 114b can be configured to transmit and / or receive wireless signals within one or more carrier frequencies of the RAN 104. These frequencies can be in the licensed spectrum, the unlicensed spectrum, or a combination thereof. The cell can provide service to a particular geographical area that can be relatively fixed or can change over time, depending on implementation. The cell can further be divided into cell sectors. For example, the cell associated with a base station 114a can be divided into three sectors. Thus, in one embodiment, the base station 114a can include three transceivers, one for each sector of the cell. In an embodiment, the base station 114a can employ multiple-input multiple-output (MIMO) technology and can use multiple transceivers for each sector of the cell. For example, beamforming can be used to transmit and / or receive signals in a desired spatial direction.

[0038] The base stations 114a, 114b can communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over the air interface 116, which can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 can be established using any suitable radio access technology (RAT).

[0039] More specifically, as noted above, the communications system 100 can be a multiple access system and can employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104 and the WTRUs 102a, 102b, 102c can implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can establish the air interface 116 using wideband CDMA (WCDMA). WCDMA can include communication protocols such as High-Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA can include High-Speed Downlink (DL) Packet Access (HSDPA) and / or High-Speed UL Packet Access (HSUPA).

[0040] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c can implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which can establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-A Pro.

[0041] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c can implement a radio technology such as New Radio (NR) Radio Access, which can establish the air interface 116 using NR.

[0042] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c can implement multiple radio access technologies. For example, the base station 114a and WTRUs 102a, 102b, 102c can implement LTE wireless access and NR wireless access (e.g., using dual connectivity (DC) principles). Thus, the air interface utilized by WTRUs 102a, 102b, 102c can be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., an eNB and a gNB).

[0043] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c can implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 IX, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0044] Figure 1AThe base station 114b in FIG. 1A can be a wireless router, Home Node B, Home eNode B, or access point, for example, and can utilize any suitable RAT for facilitating wireless connectivity access in a localized area, such as a place of business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a roadway, and the like, in one embodiment, the base station 114b and the WTRUs 102c, 102d can implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d can implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d can utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or femtocell. As shown in FIG. 1A, the base station 114b can have a direct connection to the Internet 110. Thus, the base station 114b can not be required to access the Internet 110 via the CN 106. Figure 1A

[0045] The RAN 104 can be in communication with the CN 106, which can be any type of network configured to provide voice, data, applications, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data can have varying quality of service (QoS) requirements, such as differing throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 106 can provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and / or can perform high-level security functions such as user authentication. Although not shown in FIG. 1A, the CN 106 can be in direct or indirect communication with other those networks that employ the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which can employ a NR radio technology, the CN 106 can also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology. Figure 1A

[0046] ​​CN 106 can also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or the other networks 112. The PSTN 108 can include circuit-switched telephone networks that provide infrastructure for the provision of voice telephony. The Internet 110 can include a global system of interconnected computer networks and devices that use the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols to facilitate communications and exchange of data between devices. The networks 112 can include wired or wireless communications networks owned and / or operated by other service providers. For example, the networks 112 can include another CN that can employ a different RAT than the RAN 104.

[0047] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 can include multi-mode capabilities, e.g., the WTRUs 102a, 102b, 102c, 102d can include multiple transceivers for communicating with different wireless networks over different wireless links. For example, the WTRU 102a, 102b, 102c, 102d can include a transceiver Figure 1A The WTRU 102c shown in Figure 1A can be configured to communicate with the base station 114a using a cellular-based radio technology and the base station 114b using an IEEE 802 radio technology.

[0048] Figure 1B Figure IB is a system diagram illustrating an example WTRU 102. As shown in Figure 1B The WTRU 102 can include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138, among others. It will be appreciated that the WTRU 102 can include any sub-combination of the foregoing, consistent with embodiments.

[0049] The processor 118 can be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 can perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 can be coupled Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, however, it will be appreciated that the processor 118 and the transceiver 120 can be integrated together in an electronic package or chip.

[0050] The transmit / receive element 122 can be configured to transmit signals to, and receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 can be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 can be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 can be configured to transmit and / or receive both RF and light signals. It will be appreciated that the transmit / receive element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0051] Although the transmit / receive element 122 is depicted in the Figure 1B embodiment as a single element, the WTRU 102 can include any number of transmit / receive elements 122. More specifically, the WTRU 102 can employ MIMO technology. Thus, in one embodiment, the WTRU 102 can include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0052] The transceiver 120 can be configured to modulate the signals that are to be transmitted by the transmit / receive element 122 and to demodulate the signals that are received by the transmit / receive element 122. As noted above, the WTRU 102 can have multi-mode capabilities. Thus, the transceiver 120 can include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11, for example.

[0053] The processor 118 of the WTRU 102 can be coupled to, and can receive user input data from, the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 can also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 can access information from, and store information in, any suitable memory, such as the non-removable memory 130 and / or the removable memory 132. The non-removable memory 130 can include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 can include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 can access information from, and store information in, a memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0054] The processor 118 can receive power from the power source 134, and can be configured to distribute and / or control the power to the other components in the WTRU 102. The power source 134 can be any suitable device for powering the WTRU 102. For example, the power source 134 can include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0055] The processor 118 can also be coupled to the GPS chipset 136, which can be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 can receive location information over the air interface 116 from a base station (e.g., base stations 114a, 114b) and / or determine its location based on

[0056] The processor 118 may also be coupled to other peripheral devices 138, wherein the peripheral devices may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripheral devices 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, Modules, frequency modulation (FM) radio units, digital music players, media players, video game console modules, Internet browsers, virtual reality and / or augmented reality (VR / AR) devices, and activity trackers, etc. The peripheral devices 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors, geographic location sensors, altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors, etc.

[0057] The WTRU 102 may include a full-duplex radio in which the reception or transmission of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. A full-duplex radio may include an interference management unit that reduces and / or substantially eliminates self-interference via hardware (e.g., chokes) or signal processing by a processor (e.g., a separate processor (not shown) or by the processor 118). In an embodiment, the WTRU 102 may include a half-duplex radio that transmits and receives some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)).

[0058] Although Figures 1A-1B While the WTRU is described as a wireless terminal, it is appreciated that in certain representative embodiments, such a terminal may utilize a (eg, temporary or permanent) wired communication interface with a communication network.

[0059] In a representative embodiment, the other network 112 may be a WLAN.

[0060] In view of Figures 1A-1B As described above, one or more or all of the functions described herein may be performed by one or more emulation devices (not shown). These emulation devices may be one or more devices configured to emulate one or more or all of the functions described herein. For example, these emulation devices may be used to test other devices and / or emulate network and / or WTRU functions.

[0061] The one or more emulation devices can perform one or more or all of the functions while being implemented at least partially as part of the wired and / or wireless communication network. The one or more emulation devices can perform one or more or all of the functions while being temporarily implemented or deployed as part of the wired and / or wireless communication network. The emulation devices can directly couple to the other devices to perform the testing, and / or can use over-the-air wireless communication to perform the testing.

[0062] The one or more emulation devices can perform one or more or all of the functions while not being implemented or deployed as part of a wired and / or wireless communication network. For example, the emulation devices can be used in a test lab and / or a test scenario that is not deployed, such as a test, a wired and / or wireless communication network to implement tests on one or more components. The one or more emulation devices can be test equipment. The emulation devices can transmit and / or receive data using direct RF coupling and / or wireless communication via RF circuitry, for example, which can include one or more antennas. DETAILED DESCRIPTION

[0063] Block-based video coding

[0064] Similar to HEVC, VVC builds on a block-based hybrid video coding framework. Figure 2A A block diagram of a block-based hybrid video encoding system 200 is presented. Variations of the encoder 200 are contemplated, but for clarity, the encoder 200 is described below without describing all contemplated variations.

[0065] Before being encoded, a video sequence can undergo pre-encoding processing (204), for example, applying a color transform to the input color pictures (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a signal distribution that is more amenable to compression (e.g., using a histogram equalization of one of the color components). Metadata can be associated with the pre-processing and attached to the bitstream.

[0066] An input video signal 202 comprising pictures to be encoded is partitioned (206) and processed block-wise, e.g., in units of CUs. Different CUs can have different sizes. In VTM-1.0, a CU can be up to 128x128 pixels. However, unlike HEVC, which partitions blocks based on quad-tree only, in VTM-1.0, a coding tree unit (CTU) is divided into CUs to accommodate the varying local characteristics based on quad-tree / binary-tree / ternary-tree changes. In addition, the concept of multiple partition unit types in HEVC is removed, such that there is no longer a separation of CUs, prediction units PUs, and transform units TUs in VVC-1.0; instead, each CU always serves as the basic unit for both prediction and transform without further partitioning. In the multi-type tree structure, a CTU is first partitioned by a quad-tree structure. Then, each quad-tree leaf node can be further partitioned by binary and ternary tree structures. Different splitting types can be used, e.g., quad-partition, vertical binary partition, horizontal binary partition, vertical ternary partition, and horizontal ternary partition.

[0067] In Figure 2A In an encoder of the VVC-1.0, spatial prediction (208) and / or temporal prediction (210) can be performed. Spatial prediction (or “intra prediction”) uses pixels from already coded neighboring blocks in the same video picture / slice (which are referred to as reference samples) to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in video signals. Temporal prediction (also referred to as “inter prediction” or “motion compensated prediction”) uses reconstructed pixels from already coded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in video signals. The temporal prediction signal for a given CU can be signaled by indicating one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index can be additionally signaled, which is used to identify which reference picture from a reference picture store (212) the temporal prediction signal comes from.

[0068] A mode decision block (214) in the encoder selects the best prediction mode, e.g., based on rate-distortion optimization methods. This selection can be made after performing spatial and / or temporal prediction. The intra / inter decision can be indicated by, e.g., a prediction mode flag. The prediction block is subtracted from the current video block (216) to generate the prediction residual. The prediction residual is de-correlated using a transform (218) and quantized (220). (For some blocks, the encoder can bypass both the transform and quantization, in which case the residual can be coded directly without applying the transform or quantization processes). The quantized residual coefficients are inverse quantized (222) and inverse transformed (224) to form a reconstructed residual, which is then added back to the prediction block (226) to form the reconstructed signal for the CU. Further in-loop filtering, e.g., deblocking / SAO (Sample Adaptive Offset) filtering, can be applied (228) to the reconstructed CU to reduce coding artifacts before it is put into the reference picture store (212) and used to code future video blocks. To form the output video bitstream 230, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy coding unit (108) to be further compressed and packed to form the bitstream.

[0069] Figure 2B A block diagram of a block-based video decoder 250 is given. In the decoder 250, the bitstream is decoded by the decoder elements as follows. The video decoder 250 generally performs decoding in a manner reciprocal to the encoding as described in Figure 2A The encoder 200 generally also performs video decoding as part of encoding video data.

[0070] In particular, the input to the decoder includes a video bitstream 252, which can be generated by the video encoder 200. The video bitstream 252 is first unpacked and entropy decoded at an entropy decoding unit 254 to obtain transform coefficients, motion vectors, and other coded information. Picture partitioning information indicates how the pictures are partitioned. The decoder can therefore divide (256) the pictures according to the decoded picture partitioning information. The coded mode and prediction information are sent to a spatial prediction unit 258 (if intra coded) or a temporal prediction unit 260 (if inter coded) to form a prediction block. The residual transform coefficients are sent to an inverse quantization unit 262 and an inverse transform unit 264 to reconstruct a residual block. The prediction block and the residual block are then added together at 266 to generate a reconstructed block. The reconstructed block can further undergo in-loop filtering 268 before it is stored in a reference picture store 270 for use in predicting future video blocks.

[0071] The decoded picture 272 can further undergo post-decoding processing (274), e.g., inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or performing inverse remapping of the remapping process performed in the pre-encoding processing (204). The post-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream. The decoded, processed video can be sent to a display device 276. The display device 276 can be a device separate from the decoder 250, or the decoder 250 and the display device 276 can be components of the same device.

[0072] The various methods and other aspects described in this disclosure can be used to modify modules of the video encoder 200 or the decoder 250. Moreover, the systems and methods disclosed herein are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether preexisting or developed in the future) and extensions of any such standards and recommendations, including VVC and HEVC. The aspects described in this disclosure can be used individually or in combination, unless stated otherwise or technically precluded.

[0073] Affine mode

[0074] In HEVC, only translational motion model is applied to motion compensated prediction, while in real world, there are many kinds of motions: e.g. zooming-in / out, rotation, perspective motion and other irregular motions. In VTM-2.0, affine motion compensated prediction is applied. The affine motion model is either 4-parameter or 6-parameter. A first flag is signaled for each inter coded CU to indicate whether translational motion model or affine motion model is applied to inter prediction. If it is affine motion model, a second flag is signaled to indicate whether it is 4-parameter or 6-parameter model.

[0075] The affine motion model with four parameters can have the following parameters: two parameters for translational motion in horizontal and vertical directions, one parameter for zooming motion in both directions, and one parameter for rotation motion in both directions. The horizontal zooming parameter is equal to the vertical zooming parameter. The horizontal rotation parameter is equal to the vertical rotation parameter. The four-parameter affine motion model is coded in VTM using two motion vectors at two control point positions, which are defined at the top-left and top-right corners of the current CU. As shown in Figure 3A The affine motion field of the block is described by two control point motion vectors (V0, V1). Based on this control point motion, the motion field (v x , v y ) of the affine coded block can be described as

[0076]

[0077] where (v 0x, v 0y ) is the motion vector of the upper left control point, (v 1x , v 1y ) is the motion vector of the upper right control point, such as Figure 3A As shown, w is the width of the CU. In VTM-2.0, the motion field of the affine-coded CU is derived at the 4×4 block level; that is, for each 4×4 block in the current CU, (v x , v y ) and apply it to the corresponding 4×4 block, such as Figure 3B As shown in .

[0078] In some decoding techniques, the four parameters of the 4-parameter affine model are estimated iteratively. The MV pair at step k is represented as Denote the original luminance signal as I(i, j) and the predicted luminance signal as I' k (i, j). Spatial gradient g x (i, j) and g y (i, j) can be respectively obtained by performing the prediction signal I′ in the horizontal and vertical directions. k (i, j) is derived by applying a Sobel filter. The derivative of equation (1) can be expressed as:

[0079]

[0080] where (a, b) are the incremental translation parameters, and (c, d) are the incremental scaling and rotation parameters for step k. The incremental MV at the control point can be derived from its coordinates using equations (3) and (4). For example, (0, 0) and (w, 0) are the coordinates of the top left and top right control points, respectively.

[0081]

[0082]

[0083] Based on the optical flow equation, the relationship between brightness change, spatial gradient and temporal movement is formulated as:

[0084]

[0085] Substituting equation (2) and We get the equation for the parameters (a, b, c, d).

[0086]

[0087] Since all samples in the CU satisfy equation (6), the parameter set (a, b, c, d) can be solved using the least square method. In step (k+1), the motion vectors at the two control points can be solved using equations (3) and (4) and they are rounded to a certain precision (i.e., ¼ pixel). Using iteration, the MVs at the two control points can be refined until the MVs converge when the parameters (a, b, c, d) are all zeros or the number of iterations meets a pre-defined limit.

[0088] The affine motion model with six parameters can have the following parameters: two parameters for translational motion in horizontal and vertical directions, one parameter for scaling motion in horizontal direction, one parameter for rotation motion in horizontal direction, one parameter for scaling motion in vertical direction, and one parameter for rotation motion in vertical direction. The 6-parameter affine motion model can be coded using three MVs at three control points. As shown in Figure 4 The three control points for the 6-parameter affine coding CU can be defined at the top-left corner, top-right corner, and bottom-left corner of the CU. The motion at the top-left corner control point is related to translational motion, the motion at the top-right corner control point is related to rotation and scaling motion in horizontal direction, and the motion at the bottom-left corner control point is related to rotation and scaling motion in vertical direction. For the 6-parameter affine motion model, the rotation and scaling motion in horizontal direction can be different from those in vertical direction. The motion vector (v x , v y ) of each sub-block is derived using the three MVs at the control points as follows:

[0089]

[0090]

[0091] where (v 2x , v 2y ) is the motion vector of the bottom-left corner control point, (x, y) is the center position of the sub-block, and w and h are the width and height of the CU.

[0092] The six parameters of the 6-parameter affine model can be estimated in a similar way. Equation (2) is changed as follows.

[0093]

[0094] where (a, b) are the incremental translational parameters at step k, (c, d) are the incremental scaling and rotation parameters in horizontal direction at step k, and (e, f) are the incremental scaling and rotation parameters in vertical direction at step k. Equation (8) is changed accordingly.

[0095]

[0096] By considering all samples within the CU, the set of parameters (a, b, c, d, e, f) can be solved using least squares. The top-left control point's can be calculated with equation (3). The top-right control point's can be calculated with equation (11). The top-right control point's can be calculated with equation (12).

[0097]

[0098]

[0099] Regression-based motion vector field

[0100] To provide a fine granularity of the in-block motion vectors, a regression-based motion vector field (RMVF) tool is proposed in R. Ghaznavi-Youvalari, A. Aminlou, J. Lainema, “CE4-related: Merge mode with Regression based Motion Vector Field (RMVF),” JVET-L0171, Oct. 2018, Macau, China. This tool attempts to model the motion vector of each block at the sub-block level based on spatially neighboring motion vectors.

[0101] Figure 5A A neighboring 4x4 motion block used for the proposed RMVF method’s motion parameter derivation is shown. As seen, one row and one row of immediate neighboring motion vectors from each side of the block are used in the regression process, based on 4x4 sub-blocks (and their center positions).

[0102] To reduce the amount of neighboring motion information used for the RMVF parameter derivation, the approach of Figure 5B is used, where almost half of the neighboring 4x4 motion blocks are used for motion parameter derivation.

[0103] When collecting the motion information for the motion parameter derivation, the five regular regions (bottom-left, left, top-left, top, top-right) as shown in Figure 5A and 5B are used. The top-right and bottom-left reference motion regions are limited to only half of the respective width or height of the current block.

[0104] In the RMVF mode, the block’s motion is defined by a 6-parameter motion model. These parameters a xx , a xy , a yx, a yy , b x and b y are computed by solving a linear regression model in the sense of mean square error (MSE). The input of the regression model consists of the center position (x, y) of the available neighboring 4x4 sub-blocks and the motion vectors (mv x , mv y ) as defined above.

[0105] The motion vectors (MV subPU , MV subPU ) of the 8x8 sub-block with center position (X X_subPU , Y Y_subPU ) are then computed as:

[0106]

[0107] The motion vectors are computed for the 8x8 sub-blocks with respect to the center position of each sub-block. Therefore, in the RMVF mode, motion compensation with 8x8 sub-block accuracy is also applied.

[0108] In order to model the motion vector field efficiently, the RMVF tool is applied only in the case that at least one motion vector from at least three candidate regions is available.

[0109] In the design of JVET-L0171, a least mean square error method is employed to solve the linear regression problem. The complexity of this process is high. The embodiments described herein can reduce the complexity.

[0110] Example affine model parameter estimation techniques

[0111] As described above, the regression-based motion vector field method obtains the affine model parameters by solving a linear regression with the least mean square (LMS) error, where the input is the motion vectors from neighboring 4x4 sub-blocks. Due to the intensive multiplication, the LMS error method is computationally expensive. To reduce the computational complexity, the embodiments described herein operate to estimate the affine model parameters based on the neighboring sub-block MVs without the need for the LMS error method. Assuming the affine model defined in equation (13), the MV at position (x, y) can be computed by:

[0112]

[0113] Moreover, the affine parameters a xx , a xy , a yx , a yy related to the scaling and rotation in both directions can be obtained by:

[0114]

[0115] In some embodiments, the partial derivatives are approximated by the gradient of the neighboring motion vectors, and the estimated affine parameters a' xx , a' xy , a' yx or a' yy or can be derived according to the following equations:

[0116]

[0117] where MV T and MV L are the motion vectors of the 4x4 blocks from the above and left neighboring blocks of the current CU, respectively. N and M are the number of above and left neighbors. And g x and g y are the gradients in the horizontal and vertical directions, respectively. Figure 6 An example of a current CU 600 that is to be predicted using an affine motion model is shown. The values MV T for estimating the affine parameters can include the motion vectors of the 4x4 blocks 606, 608, 610, and 612 adjacent to the current CU 600 along its top edge. In some embodiments, the values MV T for estimating the affine parameters can further include additional blocks in the same row as blocks 606, 608, 610, and 612, such as, among others, additional blocks 602, 604, 614. The values MV L for estimating the affine parameters can include the motion vectors of the 4x4 blocks 616, 618, 620, and 622 adjacent to the current CU 600 along its left edge. In some embodiments, the values MV L for estimating the affine parameters can further include additional blocks in the same column as blocks 616, 618, 620, and 622, such as additional blocks 604 and 624.

[0118] In some cases, a coding unit can include multiple sub-blocks. In other examples, in accordance with the embodiments described herein, a coding unit itself can be considered a single sub-block, and the motion vector of that coding unit can be used as a sub-block motion vector.

[0119] The affine parameters a xx , a xy , a yx , and a yy may be expressed as a pq , where p represents the x or y direction and q represents the x or y direction. Consistent with equation (16), the estimate of each parameter a pq may be described as the average of the q-directional gradients of the p-components of the motion vectors of the multiple neighboring sub-blocks.

[0120] The gradient g can be estimated in different ways x and g y In one embodiment, g x is estimated using motion vectors from neighboring blocks above the current CU, and g y is estimated using motion vectors from neighboring blocks to the left of the current CU, as shown in the following equations:

[0121]

[0122] In some embodiments, the estimates of the parameters b x and b x are performed as follows:

[0123]

[0124] where, and are the MVs of each of the neighboring 4x4 blocks with motion information x , the average of MV y , x and y.

[0125] In other embodiments, the estimates of the parameters b x and b y are performed as follows:

[0126]

[0127] where MV TL is the top-left corner neighboring motion vector, and x TL and y TL are the positions of the top-left corner neighboring motion vector.

[0128] In some embodiments, it can be desirable to use a 4-parameter affine model instead of a 6-parameter affine model. In the 4-parameter affine model:

[0129]

[0130] In some embodiments, the 4-parameter affine model is derived by using only motion vectors from neighboring blocks above the current CU or using only motion vectors from neighboring blocks to the left of the current CU.

[0131] In some embodiments, the affine parameters are estimated using only motion vectors from neighboring blocks above the current CU, as follows:

[0132]

[0133]

[0134] a′ yy= a' xx

[0135] = a' xy = -a' yx

[0136] In other embodiments, only the motion vectors from the neighboring blocks on the left side of the current CU are used to estimate the affine parameters, specifically as follows:

[0137]

[0138]

[0139] = a' xx = -a' yy

[0140] = a' yx = -a' xy

[0141] In some embodiments, the parameters of the 2-parameter translation motion model are derived by assuming that the parameters a xx , a xy , a yx and a yy are zero. The parameters b x and b y may be estimated by averaging the neighboring MVs.

[0142] In another embodiment, the 6-parameter affine model, 4-parameter affine model and 2-parameter translation model derived from the above or left neighboring blocks are derived and one of them is selected. One approach is to select the model with the smallest model fitting error as the optimal model. The model fitting error can be computed by the sum of the distance between the MVs derived by the model candidates and the actual MVs of each sub-block. The distance can be measured by the weighted sum of absolute difference (SAD) or the weighted sum of squared difference (SSD). For example, given the model parameters (a xx , a xy , a yx , a yy , b x , b y ), the derived MV i (MVx' i , MVy' i ) of the i-th neighboring block centered at (x, y) can be computed by equation (14). The distance between the derived MV i and the MV i of the i-th neighboring block is computed as follows.

[0143]

[0144] where Wi is the weight of the ith neighboring block. It is measured by the distance between the neighboring block and the current CU. If the neighboring block is closer to the current CU, then the weight is larger. Compared to RMVF, the techniques described herein for estimating one or more parameters of an affine motion model can reduce the complexity of the derivation of the affine motion model.

[0145] Adaptive sub-block size for RMVF mode

[0146] Using the estimated affine model parameters, the motion vectors of the sub-blocks within the current CU can be derived according to equation (14). In some embodiments, the size of the sub-blocks can be adaptively determined according to the shape of the current CU. In one embodiment, the sub-blocks have the same aspect ratio as the current CU. The minimum horizontal size can be set to 4 or 8. For example, assume the ratio between the CU width and the CU height of the current CU is 2:1. Then, in this example, if the minimum horizontal size is equal to 4, the width of the sub-blocks is 8 and the height is 4. The minimum horizontal size can be determined by whether uni-prediction or bi-prediction is used in order to save memory access bandwidth. If the current CU is uni-predicted, the minimum horizontal size can be set to 4; if the current CU is bi-predicted, the minimum horizontal size can be set to 8.

[0147] Use of estimated affine motion parameters

[0148] The affine parameters estimated according to the methods described herein can be used in different ways in different embodiments. In some embodiments, a prediction of a current block is generated using an affine motion model defined by one or more of the estimated affine parameters a' xx , a' xy , a' yx , a' yy , b' x and / or b' y . For example, the prediction can be generated using estimates of the parameters a' xx , a' xy , a' yx , a' yy , b' x and / or b' y .

[0149] In some embodiments, the estimated parameters a' xx , a' xy , a' yx , a' yy , b' x and / or b' y are used as predictors for an affine motion model. For example, the parameters a' xx , a' xy , a' yx , a' yy , b' x and / or b' y may be estimated.One or more of the parameters d and one or more parameter differences d may be signaled in the bitstream xx ,da xy ,da yx ,da yy , db x and / or db y . An affine motion model can then be used to generate a prediction for the current block, where

[0150] a xx =a′ xx +da xx , a xy =a′ xy +da xy ,

[0151] a yx =a′ yx +da yx , a yy =a′ yy +da yy ,

[0152] b x =b′ x +db x , and b y =b′ y +db y .

[0153] The estimated parameter a′ xx , a′ xy , a′ yx , a′ yy , b′ x and / or b′ y In another example of an embodiment used as a predictor for an affine motion model, those parameters are used to generate predicted control point motion vectors, and a motion vector difference (MVD) value is signaled in the bitstream. The MVD value is added to the predicted control point motion vector, and the resulting control point motion vector defines the affine motion model used to generate the prediction for the current block.

[0154] The affine model estimated from the motion vectors of neighboring coded blocks can be used as a predictor for the affine model to be estimated for the current CU. Since the derivation of affine motion model parameters is significantly reduced, affine motion estimation can be improved without significantly increasing complexity. After affine model predictor selection, optical flow-based estimation or refinement can be applied.

[0155] Figure 9A and 9B An example method according to some embodiments is shown in FIG. In some embodiments, for example, an encoder performs Figure 9AThe illustrated method encodes a video in a bitstream. According to Figure 9A The illustrated method, for a current block in a video to be encoded, determines (902) at least one motion vector gradient from respective motion vectors of a plurality of neighboring sub-blocks neighboring the current block. Based on the motion vector gradient(s), determines (904) an estimate of at least one affine parameter of the current block. Based at least in part on the determined estimate of the affine parameter, determines (906) an affine motion model. Using the affine motion model, generates (908) a prediction for the current block. The functions illustrated at 902, 904, 906, 908 can be performed by, for example Figure 2A The motion prediction module 210. In embodiments where the method of Figure 9A The illustrated method, for a current block in a video to be encoded, determines (902) at least one motion vector gradient from respective motion vectors of a plurality of neighboring sub-blocks neighboring the current block. Based on the motion vector gradient(s), determines (904) an estimate of at least one affine parameter of the current block. Based at least in part on the determined estimate of the affine parameter, determines (906) an affine motion model. Using the affine motion model, generates (908) a prediction for the current block. The functions illustrated at 902, 904, 906, 908 can be performed by, for example

[0156] In some embodiments, the method illustrated as Figure 9B The illustrated method decodes a video from a bitstream. According to Figure 9B The illustrated method, for a current block in a video to be decoded, determines (914) at least one motion vector gradient from respective motion vectors of a plurality of neighboring sub-blocks neighboring the current block. Based on the motion vector gradient(s), determines (916) an estimate of at least one affine parameter of the current block. Based at least in part on the determined estimate of the affine parameter, determines (918) an affine motion model. Using the affine motion model, generates (920) a prediction for the current block. The functions illustrated at 914, 916, 918, 920 can be performed by, for example Figure 2B The motion compensation prediction module 260. In Figure 9B The illustrated method, for a current block in a video to be decoded, determines (914) at least one motion vector gradient from respective motion vectors of a plurality of neighboring sub-blocks neighboring the current block. Based on the motion vector gradient(s), determines (916) an estimate of at least one affine parameter of the current block. Based at least in part on the determined estimate of the affine parameter, determines (918) an affine motion model. Using the affine motion model, generates (920) a prediction for the current block. The functions illustrated at 914, 916, 918, 920 can be performed by, for example

[0157] As illustrated in FIG. 9, some example embodiments allow for encoding and decoding of video using estimated affine parameters. Without explicit signaling of the affine parameters, the number of bits used to encode the video can be reduced.

[0158] Coded bitstream structure

[0159] Figure 7is a diagram illustrating an example of a coded bitstream structure. The coded bitstream 1300 is composed of a plurality of NAL (network abstraction layer) units 1301. NAL units can contain coded sample data, such as a coded slice 1306, or high-level syntax metadata, such as parameter set data, slice header data 1305, or supplemental enhancement information data 1307 (which can be referred to as SEI messages). Parameter sets are high-level syntax structures containing basic syntax elements that can apply to multiple bitstream layers (e.g., video parameter set 1302 (VPS)), or to a coded video sequence within one layer (e.g., sequence parameter set 1303 (SPS)), or to several coded pictures within one coded video sequence (e.g., picture parameter set 1304 (PPS)). The parameter sets can be transmitted with the coded pictures of the video bitstream, or by other means, including out-of-band transmission using a reliable channel, hardcoding, etc. Slice headers 1305 are also high-level syntax structures that can contain some picture-related information that is relatively small or only relevant to certain slice or picture types. SEI messages 1307 carry information that can not be needed by the decoding process but can be useful for various other purposes, such as picture output timing or display, and loss detection and concealment.

[0160] Communication device and system

[0161] Figure 8 is a diagram illustrating an example of a communication system. The communication system 1400 can include an encoder 1402, a communication network 1404, and a decoder 1406. The encoder 1402 can communicate with the network 1404 via a connection 1408, which can be a wired connection or a wireless connection. The encoder 1402 can be similar to the block-based video encoder of Figure 2A . The encoder 1402 can include a single-layer codec (e.g., H.264) or a multi-layer codec. Figure 2A The decoder 1406 can communicate with the network 1404 via a connection 1410, which can be a wired connection or a wireless connection. The decoder 1406 can be similar to the block-based video decoder of Figure 2B . The decoder 1406 can include a single-layer codec (e.g., H.264) or a multi-layer codec. Figure 2B

[0162] ​The encoder 1402 and / or the decoder 1406 can be incorporated into a wide variety of wired and / or wireless communication devices, such as (but not limited to) a digital television, a wireless broadcast system, a network element / terminal, a server (e.g., a content or web server such as, for example, a hypertext transfer protocol (HTTP) server), a personal digital assistant (PDA), a laptop or desktop computer, a tablet computer, a digital camera, a digital recording device, a video gaming console, a cellular or satellite radio telephone, a digital media player, and / or the like.

[0163] The communication network 1404 can be a suitable type of communication network. For example, the communication network 1404 can be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communication network 1404 can enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communication network 1404 can employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), and / or the like. The communication network 1404 can comprise multiple interconnected communication networks. The communication network 1404 can include the Internet and / or one or more private commercial networks, such as cellular networks, WiFi hotspots, and / or Internet service provider (ISP) networks, etc.

[0164] Further embodiments

[0165] Systems and methods for video coding using affine motion prediction are described. In an example method of coding video in a bitstream, for a current video block, at least one motion vector gradient is determined from respective motion vectors of sub-blocks neighboring the current block. Based on the motion vector gradient, an estimate of at least one affine parameter for the current block is determined. In some embodiments, the current block is predicted with an affine motion model using the determined estimate of the affine parameter. In some embodiments, the estimated affine parameter is used to generate a first affine motion model, and the first affine motion model is used as a predictor for a second affine motion model that is ultimately used to predict the current block. In some embodiments, the estimated affine parameter is used as an input to an affine motion refinement process (e.g., based on optical flow). In some embodiments, only neighboring sub-blocks above the current block are used in estimating the affine parameter(s). In some embodiments, only neighboring sub-blocks to the left of the current block are used in estimating the affine parameter(s). In some embodiments, sub-blocks to the left of the current block are used to estimate one or more affine parameters, and sub-blocks above the current block are used to estimate one or more other affine parameters.

[0166] In some embodiments, a method for coding video in a bitstream is provided, the method including, for at least one current block of the video: determining at least one motion vector gradient from respective motion vectors of a plurality of neighboring sub-blocks neighboring the current block; and determining an estimate of at least one affine parameter of the current block based on the at least one motion vector gradient.

[0167] In some embodiments, the method further includes generating a prediction of the current block using an affine motion model using the determined estimate of the affine parameter.

[0168] In some embodiments, the method further includes: determining a first affine motion model using the determined estimate of the affine parameter; using the first affine motion model as a predictor for a second affine motion model; and generating a prediction of the current block using the second affine motion model.

[0169] In some embodiments, the method includes: determining a first affine motion model using the determined estimate of the affine parameter; providing the first affine motion model as an input to an affine model refinement process; generating a second affine motion model using the affine model refinement process; and generating a prediction of the current block using the second affine motion model. In some such embodiments, the affine model refinement process is based on optical flow.

[0170] In some embodiments, the neighboring sub-blocks include only sub-blocks above the current block. In some embodiments, the neighboring sub-blocks include only sub-blocks to the left of the current block.

[0171] In some embodiments, the affine parameter is a' xx and a' xx is estimated as an average of x-direction gradients of x-components of motion vectors of the plurality of neighboring sub-blocks. In some embodiments, the affine parameter is a' xy and a' xy is estimated as an average of y-direction gradients of x-components of motion vectors of the plurality of neighboring sub-blocks. In some embodiments, the affine parameter is a' yx and a' yx is estimated as an average of x-direction gradients of y-components of motion vectors of the plurality of neighboring sub-blocks. In some embodiments, the affine parameter is a' yy and a' yy is estimated as an average of y-direction gradients of y-components of motion vectors of the plurality of neighboring sub-blocks.

[0172] Some embodiments further include determining an affine parameter b' xand b' y :

[0173]

[0174]

[0175] wherein, and is a MV of a plurality of neighboring sub-blocks x , MV y , an average of x and y.

[0176] Some embodiments further include determining the affine parameter b' x and b' y :

[0177]

[0178]

[0179] wherein, V TL , x TL and y TL are a motion vector and a position of a top-left neighboring sub-block.

[0180] Some embodiments further include determining sub-block motion vectors for a plurality of sub-blocks in the current block.

[0181] Some embodiments include adaptively determining a size of a sub-block within the current block based on a size of the current block. In some such embodiments, the size of the sub-block within the current block is selected so that the sub-block has a same aspect ratio as the current block. In some embodiments, the size of the sub-block within the current block is selected to have a minimum horizontal size, wherein the minimum horizontal size is determined based on whether the current block is uni-predicted or bi-predicted.

[0182] In some embodiments, the method further includes, at the encoder: calculating a residual representing a difference between the affine prediction of the current block and the input block; and encoding the residual in the bitstream.

[0183] In some embodiments, the method further includes, at the decoder: decoding, from the bitstream, a residual representing a difference between the affine prediction of the current block and the input block; and reconstructing the current block by adding the residual to the affine prediction.

[0184] Some embodiments include a processor and a computer readable medium (e.g., a non-transitory medium) storing instructions operative to perform any of the methods described herein.

[0185] Some embodiments include a computer-readable medium (e.g., a non-transitory medium) storing video encoded using any of the methods described herein.

[0186] The present disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as having specificity, and are often described in a manner that can sound limiting at least to show individual characteristics. However, this is for purposes of clarity of description, and does not limit the disclosure or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide additional aspects. Moreover, these aspects can also be combined and interchanged with aspects described in earlier documents.

[0187] The aspects described and contemplated in the present disclosure can be implemented in many different forms. While some embodiments are specifically illustrated, other embodiments are contemplated, and the discussion of specific embodiments does not limit the breadth of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to communicating generated or encoded bitstreams. These and other aspects can be implemented as methods, apparatuses, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having bitstreams generated according to any of the described methods stored thereon.

[0188] In the present disclosure, the terms “reconstruction” and “decoding” are used interchangeably, the terms “pixel” and “sample” are used interchangeably, and the terms “image,” “picture,” and “frame” are used interchangeably. Generally, but not necessarily, the term “reconstruction” is used on the encoder side, while “decoding” is used on the decoder side.

[0189] Various methods are described herein, and each of the methods includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions can be modified or combined. Additionally, terms such as “first,” “second,” etc. can be used in various embodiments to modify an element, component, step, action, etc., such as, for example, “first decoding” and “second decoding.” Unless specifically required, the use of these terms does not limit the order of the modified operations. Thus, in this example, the first decoding does not need to be performed before the second decoding, and can occur, for example, before, during, or in a time period overlapping the second decoding.

[0190] For example, various numerical values can be used in the present disclosure. The specific values are for example purposes, and the described aspects are not limited to these specific values.

[0191] The embodiments described herein can be implemented by computer software executable by a processor or other hardware, or a combination thereof. As non-limiting examples, embodiments can be implemented by one or more integrated circuits. The processor(s) can be of any type suitable to the technical environment and can include one or more of a microprocessor, a general purpose computer, a special purpose computer, and a processor based on a multi-core architecture.

[0192] Various implementations relate to decoding. As used in this disclosure, “decoding” can include, for example, all or part of the processes performed on a received encoded sequence in order to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or instead include processes performed by decoders of various implementations described in this disclosure, such as extracting a picture from a tiled (packaged) picture, determining an upsampling filter to use and then upsampling the picture, and flipping the picture back to its intended orientation.

[0193] As further examples, in one embodiment “decoding” refers only to entropy decoding, in another embodiment “decoding” refers only to differential decoding, and in another embodiment “decoding” refers to a combination of entropy decoding and differential decoding. Based on the specific context of the description, it will be clear whether the phrase “decoding process” is intended to refer specifically to a subset of operations or more generally to a broader decoding process.

[0194] Various implementations relate to encoding. In a similar manner as discussed above with respect to “decoding,” as used in this disclosure “encoding” can include, for example, all or part of the processes performed on an input video sequence in order to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as partitioning, differential encoding, transform, quantization, and entropy encoding. In various embodiments, such processes also or instead include processes performed by encoders of various implementations described in this disclosure.

[0195] As further examples, in one embodiment “encoding” refers only to entropy encoding, in another embodiment “encoding” refers only to differential encoding, and in another embodiment “encoding” refers to a combination of differential encoding and entropy encoding. Based on the specific context of the description, it will be clear whether the phrase “encoding process” is intended to refer specifically to a subset of operations or more generally to a broader encoding process.

[0196] When an apparatus is presented in flow diagram form, it is to be understood that each box of the flow diagram also provides a block diagram of corresponding apparatus. Similarly, when an apparatus is presented in block diagram form, it is to be understood that it also provides a flow diagram of corresponding method / process.

[0197] Various embodiments relate to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often given a constraint on computational complexity. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solve the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, which includes all considered mode or coding parameter values, followed by a full evaluation of their coding cost and associated distortion of the reconstructed signal after coding and decoding. Faster approaches can also be used to save encoding complexity, in particular based on predicting or approximating the distortion from the prediction or prediction residual signal instead of the reconstructed signal. A mix of these two approaches can also be used, for example by using the approximated distortion only for some of the possible encoding options, and the full distortion for the other encoding options. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a full evaluation of both the coding cost and the associated distortion.

[0198] Implementations and aspects described herein can be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, an apparatus such as a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0199] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation” and other variations thereof means that a particular feature, structure, characteristic, and so forth being described in connection with an embodiment is included in at least one embodiment. Thus, the appearance of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation” or related phrases in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0200] In addition, the disclosure can relate to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0201] Moreover, the present disclosure can relate to“accessing” a variety of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (e.g., from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0202] Additionally, the present disclosure can relate to“receiving” a variety of information. As with“accessing,” receiving is intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information or retrieving the information (e.g., from memory). Moreover, receiving is typically involved, in one way or another, during operations such as storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0203] It will be understood that, when, for example,“A / B,”“A and / or B,” and“at least one of A and B” are used, any of these alternatives are intended to be encompassed by use of the following“ / ,”“and / or,” and“at least one of.” As a further example, in the case of“A, B, and / or C” and“at least one of A, B, and C,” such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This can be extended, for example, to any number of listed options.

[0204] Furthermore, as used herein, the word "signal" refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a particular one of a plurality of parameters for region-based filter parameter selection for de-artifact filtering. As such, in one embodiment, the same parameter is used at the encoder side and the decoder side. Thus, for example, the encoder can send (explicit signaling) the particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter along with the other parameters, then signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding the transmission of any actual functionality, bit savings are achieved in various embodiments. It should be appreciated that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. Although the foregoing relates to the verb form of the word "signal", the word "signal" can also be used as a noun herein.

[0205] Implementations can produce various signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal can be formatted to carry the bitstream of an described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of spectrum) or a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0206] Note that various hardware elements of one or more described embodiments are referred to as "modules" that perform (i.e., implement, execute, etc.) various functions described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable for a given implementation. Each described module can also include instructions executable by the hardware to perform one or more functions described as being performed by the respective module, and note that these instructions can take the form of, or include, hardware (i.e., hardwired) instructions, firmware instructions, and / or software instructions, and can be stored in any suitable non-transitory computer- readable medium or media, such as is commonly referred to as RAM, ROM, etc.

[0207] Although features and elements are described above in particular combinations, each feature or element can be used alone without the other features and elements or in any combination with the other features and elements. In addition, the methods described herein can be implemented in a computer program, software, or firmware incorporated in a computer- readable medium for execution by a computer or processor. Examples of computer- readable media include but are not limited to read only memory (ROM); random access memory (RAM); register storage; cache memory; semiconductor memory devices; magnetic media such as internal hard disks and removable disks; magneto-optical media; and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software can be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A video encoding or decoding method, comprising: For at least one current block in a video, determining at least one motion vector gradient from corresponding motion vectors of a plurality of neighboring sub-blocks adjacent to the current block; Based on the motion vectors of the plurality of adjacent sub-blocks Quantity The average value of the directional gradient determines at least one affine parameter of the current block of which and Each is from including and The direction selected in the group; determining an affine motion model based at least in part on the determined estimates of the affine parameters; as well as Generating a prediction of the current block using the affine motion model includes: determining a sub-block motion vector of at least one sub-block in the current block based on the affine motion model; The sub-block motion vector of at least one sub-block is determined according to: ； The estimated value is used as , , or At least one of, or the determined estimate is used as the value , , or A predictor of at least one of .

2. The method of claim 1 , wherein determining the affine motion model comprises: A plurality of affine parameters of the affine motion model is determined, and wherein the determined estimate is used as at least one of the plurality of affine parameters.

3. The method of claim 1 , wherein determining the affine motion model comprises: determining a plurality of affine parameters for an affine motion model predictor, wherein the determined estimate is used as at least one of the plurality of affine parameters for the affine motion model predictor; The affine motion model is predicted using the affine motion model predictor.

4. The method of claim 1 , wherein determining the affine motion model comprises: determining an initial motion model using the determined estimates of the affine parameters; providing the initial motion model as input to an optical flow-based affine model refinement process; as well as Using the affine model refinement process, the affine motion model is generated.

5. The method according to any one of claims 1 to 4, further comprising: At the encoder, calculating a residual representing a difference between an input block and said prediction of said current block; as well as The residual is encoded in a bitstream.

6. The method according to any one of claims 1 to 4, further comprising: At the decoder, decoding a residual associated with the current block from a bitstream; as well as The current block is reconstructed by adding the residual to the affine prediction.

7. A video encoding or decoding apparatus, comprising: A processor configured to perform at least: For at least one current block in a video, determining at least one motion vector gradient from corresponding motion vectors of a plurality of neighboring sub-blocks adjacent to the current block; Based on the motion vectors of the plurality of adjacent sub-blocks Quantity The average value of the directional gradient determines at least one affine parameter of the current block of which and Each is from including and The direction selected in the group; determining an affine motion model based at least in part on the determined estimates of the affine parameters; as well as Generating a prediction of the current block using the affine motion model includes: determining a sub-block motion vector of at least one sub-block in the current block based on the affine motion model; The sub-block motion vector of at least one sub-block is determined according to: ； The estimated value is used as , , or At least one of, or the determined estimate is used as the value , , or A predictor of at least one of .

8. The apparatus of claim 7, wherein determining the affine motion model comprises: A plurality of affine parameters are determined, and wherein the determined estimate is used as at least one of the plurality of affine parameters.

9. The apparatus according to any one of claims 7-8, wherein the processor is further configured to perform encoding using a method comprising the steps of: calculating a residual representing a difference between an input block and said prediction of said current block; and The residual is encoded in a bitstream.

10. The apparatus according to any one of claims 7-8, wherein the processor is further configured to perform decoding using a method comprising the steps of: decoding a residual associated with the current block from a bitstream; and The current block is reconstructed by adding the residual to the affine prediction.

11. A computer-readable medium comprising instructions for causing one or more processors to execute the method of any one of claims 1-4.

Images