Method and apparatus for reducing coding latency for decoder-side motion refinement - Patents.com

Abstract

To provide video coding systems and methods for reducing coding latency introduced by DMVR.SOLUTION: Two non-refined motion vectors are identified for coding of a first block of samples (e.g., a first coding unit) using bi-prediction. One or both of the non-refined motion vectors are used to predict motion information for a second block of samples (e.g., a second coding unit). The two non-refined motion vectors are refined using DMVR, and the refined motion vectors are used to generate a prediction signal of the first block of samples. Such embodiments allow the second block of samples to be coded substantially in parallel with the first block without waiting for completion of DMVR on the first block.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a patent application from and claims the benefit of U.S. Provisional Patent Application No. 62 / 690,507, entitled “Methods and Apparatus for Reducing the Coding Latency of Decoder-Side Motion Refinement,” filed June 27, 2018, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Video coding systems are widely used to compress digital video signals to reduce the storage requirements and / or transmission bandwidth of such signals. Various types of video coding systems are the most widely used and developed, such as block-based systems, wavelet-based systems, and object-based systems, and more recently, block-based hybrid video coding systems. Examples of block-based video coding systems include international video coding standards such as MPEG1 / 2 / 4 part2, H.264 / MPEG-4 part10 AVC, VC-1, and the latest video coding standard called High Efficiency video Coding (HEVC) developed by ITU-T / SG16 / Q.6 / VCEG and ISO / IEC / MPEG's Joint Collaborative Team on Video Coding (JCT-VC).

[0003] The first version of the HEVC standard was finalized in October 2013, and provides roughly 50% bitrate savings or equivalent perceptual quality compared to the previous generation video coding standard H.264 / MPEG AVC. Although the HEVC standard provides significant coding improvements over its predecessors, there is evidence that better coding efficiency than HEVC can be achieved with additional coding tools. Based on that, both VCEG and MPEG have started work on pursuing new coding techniques for subsequent video coding standardization. The Joint Video Exploration Team (JVET) was formed by ITU-TVECG and ISO / IECMPEG in October 2015 to start a significant research of advanced techniques that can enable significant enhancement of coding efficiency. A reference software called the Joint Exploration Model (JVET) is maintained by JVET by integrating several additional coding tools on top of the HEVC Test Model (HM).

[0004] In October 2017, a Joint Request for Proposals (CfP) for video compression with capabilities exceeding HEVC was issued by ITU-T and ISO / IEC. In April 2018, 23 CfP responses have been received and evaluated at the 10th JVET meeting, proving that it will obtain approximately 40% compression efficiency over HEVC. Based on such evaluation results, JVET has started a new project to develop a new generation of video coding standard called Versatile Video Coding (VVC). In the same month, a reference software code base called VVC Test Model (VTM) has been established to prove the reference implementation of the VVC standard. For the initial VTM-1.0, most of the coding modules, including intra prediction, inter prediction, transform / inverse transform, and quantization / inverse quantization, as well as in-loop filters, comply with the existing HEVC design, except that a multi-type tree-based block partitioning structure is used in VTM. Meanwhile, another reference software base called Benchmark Set (BMS) has also been generated to facilitate the evaluation of new coding tools. In the BMS code base, a list of coding tools inherited from JEM, which results in higher coding efficiency and moderate implementation complexity, is included on top of VTM and is used as a benchmark when evaluating similar coding techniques during the VVC standardization process. The JEM coding tools integrated in BMS-1.0 include 65 angular intra prediction directions, modified coefficient coding, evolutionary multiple transform (AMT) + 4 × 4 non-separable quadratic transform (NSST), affine motion model, generalized adaptive loop filter (GALF), advanced temporal motion vector prediction (ATMVP), adaptive motion vector precision, decoder-side motion vector refinement (DMVR), and LM chroma mode. Summary of the Invention

[0005] Some embodiments include a method for use in video encoding and decoding (collectively "coding"). Some embodiments of a block-based video coding method include refining a first non-refined motion vector and a second non-refined motion vector in a first block to generate a first refined motion vector and a second refined motion vector, predicting motion information of a second block using one or both of the first non-refined motion vector and the second non-refined motion vector, where the second block is a spatial neighbor of the first block, and predicting the first block bi-predictively using the first refined motion vector and the second refined motion vector.

[0006] In an embodiment of the video coding method, a first unrefined motion vector and a second unrefined motion vector associated with a first block are identified. Motion information of a second block adjacent to the first block is predicted using one or both of the first unrefined motion vector and the second unrefined motion vector. The first unrefined motion vector and the second unrefined motion vector are refined, for example, using a decoder-side motion vector refinement (DMVR). The refined motion vector is used to generate a first refined motion vector and a second refined motion vector that can be used for bi-prediction of the first block. Using the unrefined motion vector(s) to predict the motion information of the second block is performed using one or more techniques such as advanced motion vector prediction (AMVP), temporal motion vector prediction (TMVP), advanced temporal motion vector prediction (TMVP), and using the unrefined motion vector(s) as spatial merge candidates. In the case of spatial prediction, the second block may be a spatial neighbor of the first block, and in the case of temporal prediction, the second block may be a collocated block of a subsequently coded picture. In some embodiments, the deblocking filter strength for the first block is determined based at least in part on the first unrefined motion vector and the second unrefined motion vector.

[0007] In another embodiment of the video coding method, a first unrefined motion vector and a second unrefined motion vector associated with a first block are identified. The first unrefined motion vector and the second unrefined motion vector are refined, for example, using DMVR, to generate a first refined motion vector and a second refined motion vector. Motion information of a second block is predicted using either spatial motion prediction or temporal motion prediction, where (i) if spatial motion prediction is used, one or both of the first unrefined motion vector and the second unrefined motion vector are used to predict the motion information, and (ii) if temporal motion prediction is used, one or both of the first refined motion vector and the second refined motion vector are used to predict the motion information.

[0008] In another embodiment of the video coding method, at least one predictor is selected for predicting motion information of a current block, the selection being made from a set of available predictors, the available predictors including (i) at least one unrefined motion vector from a spatial neighboring block of the current block and (ii) at least one refined motion vector from a co-located block of the current block.

[0009] In another embodiment of the video coding method, at least two non-overlapping regions in a slice are determined. A first unrefined motion vector and a second unrefined motion vector associated with a first block in the first region are identified. The first unrefined motion vector and the second unrefined motion vector are refined to generate a first refined motion vector and a second refined motion vector. In response to determining that motion information of a second block adjacent to the first block is predicted using the motion information of the first block, the motion information of the second block is predicted using one or both of the first unrefined motion vector and the second unrefined motion vector (i) if the first block is not on a lower or right boundary of the first region, and (ii) if the first block is on a lower or right boundary of the first region, the motion information of the second block is predicted using one or both of the first refined motion vector and the second refined motion vector.

[0010] In another embodiment of the video coding method, at least two non-overlapping regions in a slice are determined. A first unrefined motion vector and a second unrefined motion vector associated with a first block in the first region are identified. The first unrefined motion vector and the second unrefined motion vector are refined to generate a first refined motion vector and a second refined motion vector. In response to determining that motion information of a second block adjacent to the first block is predicted using the motion information of the first block, the motion information of the second block is predicted using (i) one or both of the first unrefined motion vector and the second unrefined motion vector if the second block is within the first region, and (ii) one or both of the first refined motion vector and the second refined motion vector if the second block is not within the first region.

[0011] In another embodiment of the video coding method, at least two non-overlapping regions in a slice are determined. A first unrefined motion vector and a second unrefined motion vector associated with a first block in the first region are identified. The first unrefined motion vector and the second unrefined motion vector are refined to generate a first refined motion vector and a second refined motion vector. Motion information of the second block is predicted using either spatial motion prediction or temporal motion prediction, and (i) if the first block is not on a lower or right boundary of the first region and if spatial motion prediction is used, one or both of the first unrefined motion vector and the second unrefined motion vector are used to predict the motion information, and (ii) if the first block is on a lower or right boundary of the first region and if temporal motion prediction is used, one or both of the first refined motion vector and the second refined motion vector are used to predict the motion information.

[0012] In another embodiment of the video coding method, at least two non-overlapping regions are defined in a slice, a set of available predictors for predicting motion information of a current block in a first region is determined, and the set of available predictors is constrained not to include motion information of any block in a second region different from the first region.

[0013] Some embodiments relate to a method for refining a motion vector. In one embodiment, a first unrefined motion vector and a second unrefined motion vector for a current block are determined. (0) is generated using the first unrefined motion vector, and the second prediction I (1) is generated using the second unrefined motion vector.

[0014]

number

[0015] An optical flow model is used to determine the first unrefined motion vector and the second unrefined motion vector. The first unrefined motion vector and the second unrefined motion vector are refined using motion refinement to generate a first refined motion vector and a second refined motion vector. The current block is bi-predictively predicted using the first refined motion vector and the second refined motion vector.

[0016] In another embodiment of the video coding method, a first unrefined motion vector and a second unrefined motion vector for a current block are determined. (0) is generated using the first unrefined motion vector, and the second prediction I (1) is generated using the second unrefined motion vector.

[0017]

number

[0018] is determined,

[0019]

number

[0020] where θ is the set of coordinates of all samples in the current block,

[0021]

number

[0022] The first unrefined motion vector and the second unrefined motion vector are refined using motion refinement to generate a first refined motion vector and a second refined motion vector. The current block is bi-predictively predicted using the first refined motion vector and the second refined motion vector.

[0023] In another embodiment of the video coding method, a first motion vector and a second motion vector for a current block are determined. The first motion vector and the second motion vector are: (a) A first prediction P using a first motion vector 0 and using the second motion vector to generate a second prediction P 1 and (b) First prediction P 0 and the second prediction P 1 By averaging tmp and (c) Template signal P tmp A first motion refinement (Δx, Δy) for the first motion vector based on * 0 and the second motion refinement (Δx, Δy) for the second motion vector * using an optical flow model to determine 1; (d) First motion refinement (Δx, Δy) * Refine the first motion vector using 0 and the second motion refinement (Δx, Δy) * refining a second motion vector using The iterative refinement is performed by performing steps including:

[0024] Further embodiments include encoder and decoder (collectively "codec") systems configured to perform the methods described herein. Such systems may include a processor and a non-transitory computer storage medium storing instructions that, when executed on the processor, are operable to perform the methods described herein. Additional embodiments include a non-transitory computer-readable medium storing video encoded using the methods described herein. [Brief description of the drawings]

[0025] [Figure 1A] FIG. 1 is a system diagram illustrating an example communication system in which one or more disclosed embodiments may be implemented. [Figure 1B] 1 is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that may be used within the communications system illustrated in FIG. 1A, in accordance with an embodiment. [Diagram 2] FIG. 1 is a functional block diagram of a block-based video encoder, such as an encoder used for VVC. [Figure 3A] 1 shows block partitions and quaternary partitions in a multi-type tree structure. [Figure 3B] 1 shows a block partition, a vertical binary partition, in a multi-type tree structure. [Figure 3C] 1 shows a block partition in a multi-type tree structure, a horizontal binary partition. [Figure 3D] 1 shows a block partition, a vertical ternary partition, in a multi-type tree structure. [Figure 3E] 1 shows block partitions and horizontal ternary partitions in a multi-type tree structure. [Figure 4] FIG. 2 is a functional block diagram of a block-based video decoder, such as a decoder used for VVC. [Diagram 5] 1 illustrates an example of spatial motion vector prediction. [Figure 6] 1 illustrates an embodiment of temporal motion vector prediction (TMVP). [Figure 7] 1 illustrates an embodiment of Advanced Temporal Motion Vector Prediction (ATMVP). [Figure 8A] 1 illustrates an embodiment of decoder-side motion vector refinement (DMVR). [Figure 8B] 1 illustrates an embodiment of decoder-side motion vector refinement (DMVR). [Figure 9] Illustrates parallel decoding for VTM-1.0. [Figure 10] 4 illustrates the decoding latency introduced by DMVR. [Figure 11] 1 illustrates an embodiment in which refined MVs from a DMVR are used only to generate a bi-predictive signal. [Figure 12] 1 illustrates an embodiment in which refined MVs from DMVR are used for temporal motion prediction and deblocking, and unrefined MVs are used for spatial motion prediction. [Figure 13] 1 illustrates an embodiment in which refined MVs from the DMVR are used for temporal motion prediction and unrefined MVs are used for spatial motion prediction and deblocking. [Figure 14] 1 illustrates parallel decoding after applying a latency elimination method to a DMVR, according to some embodiments. [Figure 15] 1 illustrates an embodiment that uses unrefined MVs for DMVR blocks inside a picture segment for spatial motion prediction and deblocking. [Figure 16]1 illustrates an embodiment in which a current picture is divided into multiple segments and coding latency is reduced for blocks within each segment. [Figure 17] 1 illustrates an embodiment in which a current picture is divided into multiple segments and coding latency is reduced for blocks from different segments. [Figure 18] 1 is a flowchart of a motion refinement process using optical flow, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Example network for implementation of embodiment 1A illustrates an example communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communication system 100 may enable the multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication system 100 may utilize one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tailed Unique Word Discrete Fourier Transform Spread OFDM (ZT UW DTS-S-OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, and Filter Bank Multi-Carrier (FBMC).

[0027] 1A, the communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104 / 113, a CN 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, although it will be recognized that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a “station” and / or “STA,” may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearables, head mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain situations), consumer electronics devices, and devices operating on commercial and / or industrial wireless networks, etc. Any of the WTRUs 102a, 102b, 102c, 102d may be referred to interchangeably as a UE.

[0028] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a NodeB, an eNodeB, a home NodeB, a home eNodeB, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. Although the base stations 114a, 114b are each depicted as a single element, it will be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0029] The base station 114a may be part of the RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be in the licensed spectrum, the unlicensed spectrum, or a combination of the licensed and unlicensed spectrum. A cell may provide coverage for wireless services in a particular geographic area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, one for each sector of the cell. In an embodiment, the base station 114a may utilize multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in desired spatial directions.

[0030] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0031] More specifically, as described above, the communication system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, and SC-FDMA. For example, the base station 114a and the WTRUs 102a, 102b, 102c in the RAN 104 / 113 may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 116 using Wideband CDMA (WCDMA). WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​Uplink (UL) Packet Access (HSUPA).

[0032] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE), and / or LTE Advanced (LTE-A), and / or LTE Advanced Pro (LTE-A Pro).

[0033] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may establish the air interface 116 using New Radio (NR) and may implement a radio technology such as NR radio access.

[0034] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement both LTE and NR radio access, e.g., using a dual connectivity (DC) principle. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions transmitted to / from multiple types of base stations (e.g., eNBs and gNBs).

[0035] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a wireless technology such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), and GSM EDGE (GERAN).

[0036] The base station 114b in FIG. 1A may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a localized area, such as an office, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., used by drones), and a roadway. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a wireless technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a wireless technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or femtocell. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not need to access the Internet 110 via the CN 106 / 115.

[0037] The RAN 104 / 113 may communicate with the CN 106 / 115, which may be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have various Quality of Service (QoS) requirements, such as different throughput requirements, delay requirements, error resilience requirements, reliability requirements, data throughput requirements, and mobility requirements. The CN 106 / 115 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, and / or perform high-level security functions, such as user authentication. Although not shown in FIG. 1A, it will be appreciated that the RAN 104 / 113 and / or the CN 106 / 115 may communicate directly or indirectly with other RANs that utilize the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which may utilize NR radio technology, the CN 106 / 115 may also communicate with another RAN (not shown) that utilizes GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0038] The CN 106 / 115 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communications protocols, such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) in the TCP / IP Internet protocol suite. The networks 112 may include wired and / or wireless communications networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs that may utilize the same RAT as the RAN 104 / 113 or a different RAT.

[0039] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with a base station 114a, which may employ a cellular-based wireless technology, and with a base station 114b, which may utilize IEEE 802 wireless technology.

[0040] 1B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a Global Positioning System (GPS) chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any subcombination of the above elements while remaining consistent with an embodiment.

[0041] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in conjunction with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although FIG. 1B depicts the processor 118 and the transceiver 120 as separate components, it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0042] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It will be appreciated that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0043] 1B, the transmit / receive element 122 is depicted as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may utilize MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0044] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive element 122 and demodulate signals received by the transmit / receive element 122. As mentioned above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as, for example, NR and IEEE 802.11.

[0045] The processor 118 of the WTRU 102 may be coupled to and may receive user input data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit). The processor 118 may output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may obtain information from and store data in any type of suitable memory, such as a non-removable memory 130 and / or a removable memory 132. The non-removable memory 130 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as located on a server or home computer (not shown).

[0046] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control power to other components within the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0047] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or instead of information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 116 from a base station (e.g., base stations 114a, 114b) and / or may determine its location based on the timing of signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may obtain location information using any suitable location determination method while remaining consistent with an embodiment.

[0048] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photos and / or videos), a Universal Serial Bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0049] The WTRU 102 may include a full-duplex radio where transmission and reception of some or all of the signals associated with a particular subframe (e.g., for both the UL (e.g., for transmission) and the downlink (e.g., for reception)) may be parallel and / or simultaneous. The full-duplex radio may include an interference management unit 139 to reduce and / or substantially eliminate self-interference via hardware (e.g., a choke) or via signal processing via a processor (e.g., a separate processor (not shown) or processor 118). In an embodiment, the WTRU 102 may include a half-duplex radio for transmission and reception of some or all of the signals (e.g., associated with a particular subframe for either the UL (e.g., for transmission) or the downlink (e.g., for reception).

[0050] Although in FIGS. 1A-1B the WTRU is described as a wireless terminal, it is contemplated that in certain representative embodiments such a terminal may use a wired communications interface (e.g., temporarily or permanently) with the communications network.

[0051] In an exemplary embodiment, the other network 112 may be a WLAN.

[0052] 1A-1B and the corresponding description, one or more or all of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more or all of the functions described herein. For example, an emulation device may be used to test other devices and / or to simulate network and / or WTRU functions.

[0053] The emulation device may be designed to perform one or more tests of other devices in a laboratory environment and / or in an operator network environment. For example, the one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices in the communication network. The one or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for testing purposes and / or may use over-the-air wireless communication to perform the tests.

[0054] The emulation device or devices may perform one or more functions, including all functions, without being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation device may be utilized in a test lab and / or in a test scenario in a non-deployed (e.g., test) wired and / or wireless communication network to perform tests of one or more components. The emulation device or devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (which may include, for example, one or more antennas) may be used by the emulation device to transmit and / or receive data.

[0055] Block-Based Video Coding Like HEVC, VVC is built on a block-based hybrid video coding framework. Figure 2 is a functional block diagram of an embodiment of a block-based hybrid video coding system. An input video signal 103 is processed block by block. A block may be referred to as a coding unit (CU). In VTM-1.0, a CU may be up to 128x128 pixels. However, compared to HEVC, which partitions blocks based only on quadtrees, in VTM-1.0, coding tree units (CTUs) may be divided into CUs based on quad / bin / ternary trees to fit varying characteristics. In addition, the concept of multiple partition unit types in HEVC may be eliminated, so that in VVC, the separation of CUs, prediction units (PUs), and transform units (TUs) is not used, and instead, each CU may be used as a basic unit for both prediction and transformation without further partitioning. In the multi-type tree structure, the CTUs are first partitioned by a quadtree structure. Then, each quadtree leaf node may be further partitioned by a binary tree and a ternary tree structure. As shown in FIGS. 3A-3E, there may be five branching types: quaternary partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal ternary partitioning, and vertical ternary partitioning.

[0056] In FIG. 2, spatial prediction (161) and / or temporal prediction (163) may be performed. Spatial prediction (or "intra prediction") uses pixels from samples of already coded neighboring blocks (called reference samples) in the same video picture / slice to predict the current video block. Spatial prediction reduces spatial redundancy inherent in the video signal. Temporal prediction (called "inter prediction" or "motion compensated prediction") uses unconstructed pixels from already coded video pictures to predict the current video block. Temporal prediction reduces temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. Also, if multiple reference pictures are supported, a reference picture index is additionally transmitted, which is used to identify which reference picture in the reference picture store (165) the temporal prediction signal comes from.

[0057] After spatial prediction and / or temporal prediction, a mode decision block (181) in the encoder selects the best prediction mode, for example based on a rate-distortion optimization method. The prediction block is then subtracted from the current video block (117), and the prediction residual is de-correlated using a transform (105) and quantized (107). The quantized residual coefficients are inverse quantized (111) and inverse transformed (113) to form a reconstructed residual, which is then added back to the prediction block (127) to form a reconstructed signal for the CU. Additionally, in-loop filtering, such as a deblocking filter, may be applied to the reconstructed CU before placing it in the reference picture store (165), and may be used to code subsequent video blocks. To form the output video bitstream 121, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy coding unit (109) to be further compressed and packed to form the bitstream.

[0058] 4 is a functional block diagram of a block-based video decoder. A video bitstream 202 is unpacked and entropy decoded in an entropy decoding unit 208. The coding mode and prediction information are sent to either a spatial prediction unit 260 (if intra-coded) or a temporal prediction unit 262 (if inter-coded) to form a prediction block. The residual transform coefficients are sent to an inverse quantization unit 210 and an inverse transform unit 212 to reconstruct the residual block. The prediction block and the residual block are then added together at 226, and the reconstructed block further passes through in-loop filtering before being stored in a reference picture store 264. The reconstructed video in the reference picture store is then used to predict subsequent video blocks and sent to drive a display device.

[0059] As mentioned before, BMS-1.0 follows the same encoding / decoding workflow of VTM-1.0, as shown in Figures 2 and 4. However, some coding modules, especially those associated with temporal prediction, are further extended and enhanced. In the following, some inter-tools included in BMS-1.0 or previous JEM are briefly described.

[0060] Motion Vector Prediction Like HEVC, to reduce the overhead of signaling motion information, both VTM and BMS include two modes of coding the motion information of each CU, namely merge mode and non-merge mode. In merge mode, the motion information of the current CU is directly derived from spatial and temporal neighboring blocks, and a competition-based scheme is applied to select the best neighboring block among all available candidates, and correspondingly, only the index of the best candidate is transmitted to reconstruct the motion information of the CU at the decoder. When an inter-coded PU is coded in non-merge mode, the MV is coded differently using an MV predictor derived from the evolved motion vector prediction (AMVP) technique. Like merge mode, AMVP derives the MV predictor from spatial and temporal neighboring candidates. Then, the difference between the MV predictor and the actual MV, and the index of the predictor are transmitted to the decoder.

[0061] FIG. 5 shows an example for spatial MV prediction. In a current picture (CurrPic) to be coded, a square CurrCU is the current CU and has the best matching block (CurrRefCU) in a reference picture (CurrRefPic). The MV of CurrCU, i.e., MV2, is to be predicted. The spatial neighbors of the current CU are the upper neighbor, left neighbor, upper left neighbor, lower left neighbor, and upper right neighbor of the current CU. In FIG. 5, the neighboring CUs are indicated as the upper neighbor, NeighbCU. Both the reference picture (NeighbRefPic) and MV (MV1) of NeighbCU are known because NeighbCU was coded before CurrCU.

[0062] Figure 6 shows an example for temporal MV prediction (TMVP). Four pictures (ColRefPic, CurrRefPic, ColPic, CurrPic) are shown in Figure 6. In the current picture (CurrPic) to be coded, the square CurrCU is the current CU and has the best matching block (CurrRefCU) in the reference picture (CurrRefPic). The MV of CurrCU, i.e., MV2, is to be predicted. The temporal neighbor of the current CU is specified as the collocated CU (ColCU) in the neighboring picture (ColPic). Both the reference picture (ColRefPic) and MV (MV1) of ColCU are known because ColPic was coded before CurrPic.

[0063] For spatial motion vector prediction and temporal motion vector prediction, assuming that time and space are restricted, the MV between different blocks is treated as translational with uniform speed. In the embodiment of Figures 5 and 6, the time distance between CurrPic and CurrRefPic is TB, and the time distance between CurrPic and NeighbRefPic in Figure 5 or between ColPic and ColRefPic in Figure 6 is TD. The scaled MV predictor may be calculated as Equation (1).

[0064]

number

[0065] In VTM-1.0, each merge block has at most one set of motion parameters (one motion vector and one reference picture index) for each prediction direction L0 and L1. In contrast, an additional merge candidate based on evolved temporal motion vector prediction (ATMVP) is included in BMS-1.0 to enable derivation of motion information at the sub-block level. Using such a mode, temporal motion vector prediction is improved by allowing a CU to derive multiple MVs for sub-blocks within the CU. Generally, ATMVP derives motion information of a current CU in two steps, as shown in FIG. 7. The first step is to identify the corresponding block of the current block (called a co-located block) in a temporal reference picture. The selected temporal reference picture is called a co-located picture. The second step is to divide the current block into sub-blocks and derive the motion information of each sub-block from the corresponding small block in the co-located picture.

[0066] In the first step, the co-located block and the co-located picture are identified by the motion information of the spatial neighboring blocks of the current block. In the current design, the first available candidate in the merge candidate list is considered. FIG. 7 illustrates this process. In particular, in the example of FIG. 7, block A is identified as the first available merge candidate of the current block based on the scanning order of the merge candidate list. Then, its reference index together with the corresponding motion vector (MVA) of block A is used to identify the co-located picture and the co-located block. The position of the co-located block in the co-located picture is determined by adding the motion vector (MVA) of block A to the coordinates of the current block.

[0067] In the second step, for each sub-block in the current block, the motion information of its corresponding small block in the co-located block (indicated by a small arrow in FIG. 7) is used to derive the motion information of the sub-block. In particular, after the motion information of each small block in the co-located block is identified, it is converted into the motion vector and reference index of the corresponding sub-block in the current block in the same manner as TMVP.

[0068] Decoder-side Motion Vector Refinement (DMVR) For the merge mode in VTM, when the selected merge candidate is bi-predicted, the prediction signal of the current CU is formed by averaging two prediction blocks using two MVs associated with the candidate's reference lists L0 and L1. However, the motion information of the merge candidate (derived from either spatial neighbors or spatial neighbors of the current CU) may not be accurate enough to represent the true motion of the current CU, thus worsening the efficiency of inter prediction. To further improve the coding performance of the merge mode, a decoder-side motion vector refinement (DMVR) method is applied in BMS-1.0 to refine the MV of the merge mode. In particular, when the selected merge candidate is bi-predicted, a bi-predictive template is first generated as the average of two prediction signals based on the MVs from each of the reference lists L0 and L1. Then, a block-matching based motion refinement is performed locally around the initial MV using the bi-predictive template as a target, as described below.

[0069] FIG. 8A shows a motion refinement process applied in DMVR. Generally, DMVR refines the MV of a merge candidate by the following two steps. As shown in FIG. 8A, in the first step, a bi-predictive template is generated by averaging two prediction blocks using the initial MVs (i.e., MV0 and MV1) in L0 and L1 of the merge candidate. Then, for each reference list (i.e., L0 or L1), a block-matching based motion search is performed in a local area around the initial MV. For each MV, i.e., for MV0 or MV1 of the corresponding reference list around the initial MV in that list, a cost value (e.g., sum of absolute difference (SAD)) between the bi-predictive template and the corresponding prediction block using that motion vector is measured. For each of the two prediction directions, the MV that minimizes the template in that prediction direction is considered as the final MV in the reference list of the merge candidate. In the current BMS-1.0, for each prediction direction, eight neighboring MVs (with one integer sample offset) surrounding the initial MV are considered during the motion refinement process. Finally, two refined MVs (MV0' and MV1' shown in FIG. 8A) are used to generate the final bi-predictive signal of the current CU. In addition, in conventional DMVR, to further improve coding efficiency, the refined MVs of a DMVR block are used to predict the motion information of its spatial and temporal neighboring blocks (e.g., based on spatial AMVP, spatial merge candidates, TMVP, and ATMVP), as well as to calculate the boundary strength value of the deblocking filter applied to the current CU.FIG. 8B is a flowchart of an example of a DMVR process, in which "spatial AMVP" and "spatial merge candidate" refer to spatial MV prediction processes for spatially neighboring CUs in the current picture that are coded after the current CU per the CU coding order, "TMVP" and "ATMVP" refer to temporal MV prediction processes for later CUs in subsequent pictures (pictures that are coded after the current picture based on the picture coding order), and "deblocking" refers to a deblocking filtering process of both the current block and its spatial neighboring blocks.

[0070] In the method shown in Figure 8B, a bi-predictive template is generated at 802. Motion refinement is performed on the L0 motion vector at 804, and motion refinement is performed on the L1 motion vector at 806. A final bi-predictive template is generated at 808 using the refined L0 and L1 motion vectors. In the method of Figure 8B, the refined motion vectors are used to predict the motion of a subsequently coded block. For example, the refined motion vectors are used for spatial AMVP (810), TMVP (814), and ATMVP (816). The refined motion vectors are also used as spatial merge candidates (812) and to calculate boundary strength values ​​of a deblocking filter applied to the current CU (818).

[0071] Bidirectional Optical Flow Bi-prediction in VTM / BMS-1.0 is the combination of two temporal prediction blocks obtained from an already reconstructed reference picture using averaging. However, due to the limitations of block-based motion compensation, there may be small residual motions that can be obtained between two prediction blocks, thus reducing the efficiency of motion-compensated prediction. To address this issue, bi-directional optical flow (BIO) is used in JEM to compensate such motions sample-by-sample within a block. In particular, BIO is a sample-by-sample motion refinement that is performed on top of block-based motion-compensated prediction when bi-prediction is used. The derivation of refined motion vectors for each sample in a block is based on a classical optical flow model. I (k) (x,y) is the sample value in the matrix (x,y) of the prediction block derived from the reference picture list k (k=0,1), and ∂I (k) (x,y) / ∂x and ∂I (k) Let (x,y) / ∂y be the horizontal and vertical gradients of a sample. The bi-predictive signal modified by BIO is obtained as Equation (2).

[0072]

number

[0073] τ0 and τ1 are the I (0) and I (1) , and the temporal distance between the reference pictures Ref0 and Ref1 associated with the sample position (x,y). x ,v y ) is calculated by minimizing the difference Δ between the values ​​of the samples after motion refinement compensation, as shown as equation (3).

[0074]

number

[0075] In addition, to provide regularity in the derived motion refinements, it is assumed that the motion refinements are consistent within a local surrounding area centered at (x,y), and thus, as in equation (4), we can calculate the motion refinement (v x ,v y ) is derived.

[0076]

number

[0077] Unlike DMVR, the motion refinement derived by BIO (v x ,v y It should be noted that the bi-predictive signal I is only applied to enhance the bi-predictive signal, and is not applied to modify the motion information of the current CU. In other words, the MVs used to predict the MVs of spatial and temporal neighboring blocks and to determine the deblocking boundary strength of the current CU are still the original MVs (i.e., the block-based motion compensation signal I before BIO is applied). (0) (x,y) and I (1) The MV used to generate (x,y).

[0078] DMVR coding latency Like HEVC and its predecessors, VTM-1.0 employs motion compensated prediction (MCP) to efficiently reduce temporal redundancy between pictures, thus achieving high inter-coding efficiency. There is no dependency between MCPs of spatially neighboring CUs because the MV used to generate the prediction signal of one CU is either signaled in the bitstream or inherited from its spatial / temporal neighbors. As a result, the MCP processing of all inter blocks in the same picture / slice is independent of each other. Thus, for VTM-1.0 and HEVC, the decoding processing of multiple inter blocks can be performed in parallel, e.g., they may be assigned to different threads to exploit the parallelism.

[0079] As described above, the DMVR tool is applied in BMS-1.0. To avoid generating extra signaling overhead, the motion refinement is derived using two prediction signals associated with the original L0 and L1 MVs of the CU. Thus, when the motion information of a CU is predicted from one of its spatial neighbors coded by the DMVR (e.g., by AMVP and merge mode), its decoding process waits until the MV of the neighboring block is completely reconstructed by the DMVR. This significantly complicates the pipeline design, especially at the decoder side, and therefore leads to a significant increase in complexity for the hardware implementation.

[0080] To illustrate the coding latency caused by DMVR, Figures 9 and 10 show an example comparing the decoding process of VTM-1.0 and BMS-1.0. For ease of explanation, a case is described in which there are four CUs with equal block sizes, all four CUs are coded by DMVR, each of which is decoded by a separate decoding thread, and the decoding complexity of each individual decoding module (e.g., MCP, DMVR, inverse quantization, and inverse transform) is assumed to be the same for the four CUs. As shown in Figure 9, because four CUs can be decoded in parallel, the total decoding time of VTM-1.0 is shorter than the decoding time of one CU, i.e., T MCP +T de-quant +T inv-trans Due to the dependency introduced by DMVR, for the decoding process of BMS-1.0 (shown in FIG. 10), the decoding of each individual coding block cannot be invoked until the DMVR of its spatial neighboring blocks is completely finished. Thus, the total decoding time of four CUs for BMS-1.0 is T total =4×(T MCP +T DMVR )+T de-quant +T inv-trans As can be seen, the use of predicted samples to refine the motion information by DMVR introduces dependencies among neighboring inter blocks and therefore significantly increases the latency for both the encoding and decoding processes.

[0081] Overview of Latency Reduction Methods A method for eliminating or reducing the encoding / decoding latency of DMVR while retaining its key coding performance is proposed in the present disclosure. In particular, various embodiments of the disclosure include one or more of the following aspects:

[0082] Unlike in the current DMVR method in BMS-1.0, where the refined DMVR motion of one block is always used to predict the motion of its spatial / temporal neighboring blocks and derive the deblocking filter structure, in some embodiments, it is proposed to fully or partially use the unrefined MV of the DMVR block (the MV used to generate the original bi-predictive signal) for MV prediction and deblocking processing. Assuming that the original MV can be directly obtained from parsing and motion vector reconstruction (motion vector predictor plus parsed motion vector difference) without DMVR, there is no dependency between neighboring blocks and the decoding processing of multiple inter-CUs can be performed in parallel.

[0083] This may result in some coding performance degradation, since the unrefined MV may be less accurate than the refined MV. To reduce such loss, in some embodiments, it is proposed to split the picture / slice into multiple regions. Moreover, in some embodiments, additional constraints are proposed so that the decoding of multiple CUs within the same region or the decoding of multiple CUs from different regions can be performed independently.

[0084] In some embodiments, an optical flow based motion derivation method is proposed to replace the block matching based motion search for calculating the motion refinement of each DMVR CU. Compared with the block matching based method that performs the motion search in a small local window, some embodiments directly calculate the motion refinement based on spatial sample derivatives and temporal sample derivatives. This can not only reduce the computational complexity, but also increase the motion refinement accuracy because the derived refined motion value is not limited to the search window.

[0085] Using Unrefined Motion Vectors for DMVR Latency Reduction As pointed out above, using the refined MV of one DMVR block as the MV predictor of its neighboring blocks is inappropriate for parallel encoding / decoding for a true CODEC design, because the encoding / decoding of the neighboring blocks is not performed until the refined MV of the current block is fully reconstructed through DMVR. Based on such analysis, this section proposes a method to eliminate the coding latency caused by DMVR. In some embodiments, the core design of DMVR (e.g., block matching based motion refinement) remains the same as the existing design. However, the MV of the DMVR block used to perform MV prediction (e.g., AMVP, merge, TMVP, and ATMVP) and deblocking is modified so that the dependency between neighboring blocks caused by DMVR can be eliminated.

[0086] Using Unrefined Motion Vectors for Spatial and Temporal Motion Estimation - Patent application In some embodiments, it is proposed to always perform MV prediction and deblocking using unrefined motion of DMVR blocks instead of using refined motion. Figure 11 shows the modified DMVR process after such a method is applied. As shown in Figure 11, instead of using refined MV, unrefined MV (original MV before DMVR) is used to derive MV predictor and determine boundary strength of deblocking filter. Only refined MV is used to generate final bi-predictive signal of block. Such an embodiment may be used to eliminate DMVR encoding / decoding latency because there is no dependency between refined MV of current block and decoding of its neighboring blocks.

[0087] Using Unrefined Motion Vectors for Spatial Motion Estimation - Patent application In the embodiment of FIG. 11, the unrefined MV of the DMVR block is used to derive the temporal motion predictor for the co-located block in the subsequent picture through TMVP and ATMVP, and to calculate the boundary strength for the deblocking filter between the current block and its spatial neighbors. This may lead to some coding performance loss, because the unrefined MV may be less accurate than the refined MV. On the other hand, the temporal motion prediction (TMVP and ATMVP) uses the MV of the previously decoded picture (particularly the co-located picture) to predict the MV in the current picture. Therefore, before performing the temporal motion prediction for the current picture, the refined MV of the DMVR CU in the co-located picture is already reconstructed. The similar situation is also applicable to the deblocking filter process, and because the deblocking filter is applied to the reconstructed reconstructed samples, it can be activated only after the samples of the current block are fully reconstructed through MC (including DMVR), inverse quantization, and inverse transform. Therefore, the refined MV is already available before the deblocking is applied to the DMVR block.

[0088] In the method shown in FIG. 11, an unrefined motion vector for a first block is identified at 1100. The unrefined motion vector may be signaled for the first block using any of a variety of available MV signaling techniques. The unrefined motion vector is used to generate a bi-predictive template at 1102. Motion refinement is performed on the L0 motion vector at 1104, and motion refinement is performed on the L1 motion vector at 1106. The refined L0 and L1 motion vectors are used to generate a final bi-predictive template for the first block at 1108. In the method of FIG. 11, the unrefined motion vector is used to predict the motion of a subsequently coded block (e.g., a second block). For example, the unrefined motion vector is used for spatial AMVP (1110), TMVP (1114), and ATMVP (1116). The unrefined motion vectors are also used as spatial merging candidates (1112) and to compute boundary strength values ​​for the deblocking filter (1118).

[0089] In another embodiment, to address these problems and achieve better coding performance, it is proposed to use different MVs (unrefined MV and refined MV) of DMVR blocks for spatial motion prediction, temporal motion prediction, and deblocking filter. In particular, in this embodiment, only unrefined MVs are used to derive MV predictors for spatial motion prediction (e.g., spatial AMVP and spatial merge candidates), and refined MVs are used not only to derive the final prediction of a block, but also to generate MV predictors for temporal motion prediction (TMVP and ATMVP) and calculate the boundary strength parameters of the deblocking filter. Figure 12 shows the DMVR process according to this second embodiment.

[0090] In the method shown in FIG. 12, an unrefined motion vector for a first block is identified at 1200. The unrefined motion vector may have been signaled for the first block using any of a variety of available MV signaling techniques. The unrefined motion vector is used to generate a bi-predictive template at 1202. Motion refinement is performed on the L0 motion vector at 1204, and motion refinement is performed on the L1 motion vector at 1206. A final bi-prediction for the first block is generated at 1208 using the refined L0 and L1 motion vectors. In the method of FIG. 12, the unrefined motion vector is used to predict the motion of a subsequently coded block (e.g., a second block) in the same picture as the first block. For example, the unrefined motion vector is used for spatial AMVP (1110) and as a spatial merge candidate (1212). The refined motion vector is used to predict the motion of a subsequently coded block (e.g., a third block) in another picture, for example using TMVP (1214) or ATMVP (1216). The refined motion vector is also used to calculate a boundary strength value for the deblocking filter (1218).

[0091] Using Unrefined Motion Vectors for Spatial Motion Estimation and Deblocking - Patent application In the example of FIG. 12, different MVs of DMVR blocks are used for spatial motion prediction and deblocking filters. Meanwhile, unlike the MVs used for temporal motion prediction (stored in external memory), the MVs used for spatial motion prediction and deblocking are often stored using on-chip memory for practical CODEC designs to increase data access speed. Therefore, some implementations of the method of FIG. 12 require two different on-chip memories to store both unrefined and refined MVs for each DMVR block. This may double the line buffer size used to cache MVs, which may be undesirable for hardware implementations. In order to keep the total on-chip memory size of MV storage the same as in VTM-1.0, in a further embodiment, it is proposed to use unrefined MVs of DMVR blocks for deblocking processing. FIG. 13 shows an example of DMVR processing according to this embodiment. In particular, as in the method in FIG. 12, in addition to generating the final bi-predictive signal, the refined DMVR MVs are also used to generate temporal motion predictors through TMVP and ATMVP. However, in the embodiment of FIG. 13, the unrefined MVs are used not only to derive the spatial motion predictors (spatial AMVP and spatial merge), but also to determine the boundary strength for the deblocking filter of the current block.

[0092] In the method shown in FIG. 13, an unrefined motion vector for a first block is identified at 1300. The unrefined motion vector may have been signaled for the first block using any of a variety of available MV signaling techniques. The unrefined motion vector is used to generate a bi-predictive template at 1302. Motion refinement is performed on the L0 motion vector at 1304, and motion refinement is performed on the L1 motion vector at 1306. A final bi-prediction for the first block is generated at 1308 using the refined L0 and L1 motion vectors. In the method of FIG. 13, the unrefined motion vector is used to predict the motion of a subsequently coded block (e.g., a second block) in the same picture as the first block. For example, the unrefined motion vector is used for spatial AMVP (1310) and as a spatial merge candidate (1312). The refined motion vector is used to predict the motion of a subsequently coded block (e.g., a third block) in another picture, for example using TMVP (1314) or ATMVP (1318).

[0093] The embodiments of Figures 11 to 13 can reduce or eliminate the encoding / decoding latency caused by DMVR, assuming that the dependency of the decoding of one block on the reconstruction of the refined MV of its spatially neighboring DMVR blocks does not exist in those embodiments. Based on the same example in Figure 10, Figure 14 shows an example of a parallel decoding process when one of the methods of Figures 11 to 13 is applied. As shown in Figure 14, there is no decoding latency between neighboring blocks, because the decoding of multiple DMVR blocks can be performed in parallel. Correspondingly, the total decoding time can be equal to the decoding of one block, which is T MCP +T DMVR +T de-quant +T inv-trans It may be expressed as:

[0094] A segment-based approach to DMVR latency reduction. As pointed out above, one source of encoding / decoding latency for DMVR is the dependency between the reconstruction of a DMVR block's refined MV and the decoding of its neighboring blocks, incurred by spatial motion prediction (e.g., spatial AMVP and spatial merge modes). Although methods such as those of Figures 11 to 13 can eliminate or reduce the coding latency of DMVR, this reduced latency may come at the expense of reduced coding efficiency due to the use of less accurate unrefined MVs for spatial motion prediction. Meanwhile, as shown in Figure 10, the worst-case encoding / decoding latency incurred by DMVR is directly related to the maximum number of consecutive blocks coded by the DMVR mode. To address these issues, in some embodiments, a region-based method is used to reduce the encoding / decoding latency and the coding loss incurred by using unrefined MVs for spatial motion prediction.

[0095] In particular, in some embodiments, a picture is divided into multiple non-overlapping segments, and the unrefined MV of each DMVR block in a segment is used as a predictor to predict the MV of its neighboring blocks in the same segment. However, when a DMVR block is located on the right or bottom boundary of a segment, its unrefined MV is not used, and instead, the refined MV of the block is used as a predictor to predict the MV of blocks from neighboring segments for better spatial motion prediction efficiency.

[0096] FIG. 15 shows an example of DMVR processing according to an embodiment, and FIG. 16 shows an example where blank blocks represent DMVR blocks using unrefined MVs for spatial motion prediction, spatial merging, and deblocking, and patterned blocks represent DMVR blocks using refined MVs for spatial motion prediction, spatial merging, and deblocking. In the example of FIG. 16, the encoding / decoding of different inter blocks within the same segment may be performed independently of each other, and the decoding of blocks from different segments is still dependent. For example, because blocks on the left boundary of segment #2 can use the refined MVs of adjacent DMVR blocks in segment #1 as spatial MV predictors, their decoding processes cannot start until the DMVRs of those adjacent blocks in segment #1 are fully performed. In addition, as shown in FIG. 15, similar to the method in FIG. 13, the same MV of one DMVR block is used for spatial motion prediction and deblocking filters to avoid increasing the on-chip memory for storing MVs. In another embodiment, it is proposed to always use refined MVs for deblocking processing.

[0097] In the method shown in Figure 15, an unrefined motion vector for a first block is identified at 1502. The unrefined motion vector may be signaled for the first block using any of a variety of available MV signaling techniques. At 1504, the unrefined motion vector is used to generate a bi-predictive template. At 1506, motion refinement is performed on the L0 motion vector, and at 1508, motion refinement is performed on the L1 motion vector. At 1510, a final bi-prediction for the first block is generated using the refined L0 and L1 motion vectors.

[0098] At 1512, a determination is made as to whether the first block is located on a right segment boundary or a bottom segment boundary. If the first block is not located on a right segment boundary or a bottom segment boundary, then the unrefined motion vector is used to predict the motion of a subsequently coded block (e.g., a second block) in the same picture as the first block. For example, the unrefined motion vector is used for the spatial AMVP (1514) and as a spatial merge candidate (1516). The unrefined motion vector is also used to calculate a boundary strength value of the deblocking filter (1518). On the other hand, if the first block is located on a right segment boundary or a bottom segment boundary, then the refined motion vector is used (e.g., by the AMVP 1514 and the spatial merge candidate 1516) to predict the motion of a subsequently coded block (e.g., a second block) in the same picture as the first block, and the refined motion vector is also used to calculate a boundary strength value of the deblocking filter (1518). Regardless of the outcome of the decision in 1512, the refined motion vector is used to predict the motion of a subsequently coded block (e.g., a third block) in another picture, for example using TMVP (1520) or ATMVP (1522).

[0099] In the embodiment of FIG. 16, only refined MVs are enabled for spatial motion prediction of blocks that are on the left / top boundary of segments within one picture. However, depending on the segment size, the overall percentage of blocks for which refined MVs can be applied for spatial motion prediction may be relatively small. The result may still be a non-negligible performance degradation for spatial motion prediction. To further improve performance, some embodiments propose to allow refined MVs of DMVR blocks within one segment to predict MVs of neighboring blocks within the same segment. However, as a result, decoding of multiple blocks within one segment cannot be done in parallel. To improve encoding / decoding parallelism, the method also proposes to prohibit the current block from using MVs (either unrefined MVs or refined MVs) of neighboring blocks that are from another segment as predictors for spatial motion prediction (e.g., spatial AMVP and spatial merge). In particular, with such a method, if a neighboring block is from a different segment to the current block, it is treated as unavailable for spatial motion vector prediction.

[0100] One such embodiment is shown in Figure 17. In Figure 17, blank blocks represent CUs that are allowed to use neighboring MVs for spatial motion prediction (neighboring MVs are refined MVs if the neighboring block is a DMVR block, or unrefined MVs otherwise), and patterned blocks represent CUs that are prevented from using MVs of their neighboring blocks from different segments for spatial motion prediction. The embodiment according to Figure 17 allows parallelized decoding of inter blocks that are not within one segment but span segments.

[0101] Generally, DMVR is only enabled for bidirectionally predicted CUs with both forward and backward prediction signals. In particular, DMVR requires the use of two reference pictures, one with a smaller picture order count (POC) and the other with a larger POC than that of the current picture. In contrast, low latency (LD) pictures are predicted from reference pictures, both of which precede the current picture in display order and have POCs of all reference pictures in L0 and L1 smaller than that of the current picture. Therefore, DMVR cannot be applied to LD pictures, and the coding latency caused by DMVR does not exist in LD pictures. Based on such analysis, in some embodiments, it is proposed to only apply the above DMVR parallelism constraint (disabling spatial motion prediction across segment boundaries) to non-LD pictures when DMVR is applied. For LD pictures, the constraint is not applied and it is still allowed to predict the MV of the current block based on the MV of its spatial neighbors from another segment. In a further embodiment, the encoder / decoder determines whether the constraint applies based on checking the POC of all reference pictures in L0 and L1 without additional signaling. In another embodiment, it is proposed to add a picture / slice-level flag to indicate whether the DMVR parallelism constraint applies to the current picture / slice.

[0102] In some embodiments, the number of segments and the position of each segment inside a picture / slice are selected by the encoder and signaled to the decoder. Signaling may be performed similarly for other parallelism tools in HEVC and JEM (e.g., slice, tile, and wavefront parallel processing (WPP)). Different choices may lead to different tradeoffs between coding performance and encoding / decoding parallelism. In one embodiment, it is proposed to set the size of each segment equal to the size of one CTU. In terms of signaling, syntax elements may be added at the sequence level and / or picture level. For example, the number of CTUs in each segment may be signaled in the sequence parameter set (SPS) and / or picture parameter set (PPS), or may be signaled in the slice header. Other variations of syntax elements may be used, for example, the number of CTU rows in each picture / slice may be used, the number of segments in each picture / slice may be used, among other alternatives.

[0103] Example of a motion refinement method Additional embodiments described herein serve to replace block-matching motion search for computing DMVR motion refinements. Compared to block-matching based methods that perform motion search within small local windows, example embodiments directly compute motion refinements based on spatial and temporal sample derivatives. Such embodiments can reduce computational complexity and increase refinement accuracy because the derived refined motion values ​​are not restricted to the search window.

[0104] Motion refinement using block-level BIO As discussed above, BIO is used in JEM to provide sample-by-sample motion refinement on top of block-based motion compensated prediction when a block is bi-predicted. Based on the current design, BIO only enhances the motion compensated prediction samples as a result of refinement, without updating the MVs stored in the MV buffer and used for spatial and temporal motion prediction and deblocking filters. This means that, contrary to current DMVR, BIO does not introduce any encoding / decoding latency between neighboring blocks. However, in the current BIO design, motion refinement is derived in small units (e.g., 4×4). This incurs non-negligible computational complexity, especially on the decoder side. This is undesirable for hardware CODEC implementations. Therefore, to address the latency of DMVR and maintain acceptable coding complexity, in some embodiments, it is proposed to use block-based BIO to compute local motion refinement for video blocks coded by DMVR. In particular, in the proposed embodiment, the core design of BIO (e.g., calculation of gradients and refined motion vectors) is kept the same as in the existing design of calculating motion refinement. However, to reduce complexity, the amount of motion refinement is derived based on the CU level, and a single value is aggregated for all samples within the CU and used to calculate a single motion refinement, and all samples within the current CU share the same motion refinement. Based on the same notation used above for BIO, an example of the proposed block-level BIO motion refinement is derived as equation (5).

[0105]

number

[0106] θ is the set of coordinates of the sample in the current CU, and Δ(x,y) is the optical flow error metric as shown in equation (3) above.

[0107] As indicated above, the motivation of BIO is to improve the accuracy of predicted samples based on local gradient information at each sample position inside the current block. For a large video block containing many samples, it is possible that the local gradients at different sample positions may exhibit highly variable characteristics. In such cases, the block-based BIO derivation described above may not yield reliable motion refinement for the current block, thus leading to coding performance loss. Based on such consideration, in some embodiments, it is proposed to only enable CU-based BIO motion derivation for a DMVR block when its block size is small (e.g., below one given threshold). Otherwise, the CU-based BIO motion derivation is disabled, and instead, the existing block matching-based motion refinement (the proposed DMVR latency elimination / reduction method described above is applied in some embodiments) is used to derive local motion refinement for the current block.

[0108] Motion refinement using optical flow As mentioned above, BIO ensures that the derived L0 and L1 motion refinements at each sample position are symmetric around the current picture, i.e.

[0109]

number

[0110] and

[0111]

number

[0112] estimating local motion refinements based on the assumption that

[0113]

number

[0114] and

[0115]

number

[0116] are the horizontal and vertical motion refinements associated with prediction lists L0 and L1. However, such assumptions may not hold for blocks coded by DMVR. For example, in existing DMVRs (as shown in FIG. 8A), two separate block-matching-based motion searches are performed for L0 and L1, so that the MVs that minimize the template costs of L0 and L1 prediction signals can be different. Due to such symmetric motion constraints, the motion refinements derived by BIO may not always be accurate to enhance prediction quality for DMVR (and may even degrade prediction quality at times).

[0117] In some embodiments, an improved motion derivation method is used to calculate the motion refinement for the DMVR. The classical optical flow model states that the brightness of a picture remains constant over time, as expressed as equation (6).

[0118]

number

[0119] x and y represent spatial coordinates and t represents time. The right-hand side of equation (6) may be expanded by a Taylor expansion in (x, y, t). The equation for optical flow then becomes, to first order, equation (7).

[0120]

number

[0121] Using the camera capture time as the basic time unit (e.g., setting dt=1), equation (7) may be discretized by changing the optical flow function from the continuous domain to the discrete domain. Assuming that I(x,y) is the sample value captured from the camera, then equation (7) becomes equation (8).

[0122]

number

[0123] In various embodiments, one or more error metrics may be defined based on the degree to which the expression to the left in equation (9) is not equal to zero. Motion refinement may be employed to substantially minimize the error metric.

[0124] In some embodiments, it is proposed to use a discretized optical flow model to estimate local motion refinements in L0 and L1. In particular, a bi-predictive template is generated by averaging two predicted blocks using the initial L0 and L1 MVs of a merge candidate. However, instead of performing block matching motion search in a local region, the optical flow model in equation (8) is used in some proposed embodiments to directly derive a refined MV for each reference list L0 / L1, as expressed in equation (9).

[0125]

number

[0126] P 0 and P 1is the predicted signal generated using the original MV for each of the reference lists L0 and L1, and P tmp is the bi-predictive template signal,

[0127]

number

[0128] and

[0129]

number

[0130] can be calculated based on different gradient filters, e.g., the Sobel filter, or the 2D separable gradient filter used by BIO (described in J. Chen, E. Alshina, GJ Sullivan, JR Ohm, J. Boyce, “Algorithm description of joint exploration test model 6”, JVET-G1001, Jul. 2017, Torino, Italy). 0 and P 1 The horizontal / vertical gradient of the

[0131]

number

[0132] ,

[0133]

number

[0134] , and P tmp -P k The predicted signal P 0 or P 1represents a set of equations for each sample in the two unknown parameters Δx k and Δy k Therefore, the overdetermined problem can be solved by minimizing the sum of squared errors in equation (9) as equation (10).

[0135]

number

[0136]

number

[0137] is the time difference between the L0 / L1 predicted signal and the bi-predictive template signal, and θ is a set of coordinates in the coding block. By solving the linear minimum mean square error (LLMSE) problem in equation (10), (Δx, Δy) * k We can obtain an analytical expression for

[0138]

number

[0139] Based on equation (11), in some embodiments, to improve the accuracy of the derived MV, such methods perform motion refinement (i.e., (Δx, Δy) * k Such an embodiment may use the original L0 and L1 MVs of the current block to generate an initial bi-predictive template signal, and select the corresponding delta motion (Δx, Δy) based on Equation (11). * k The refined MV can then be used, together with new L0 and L1 prediction samples, as a motion to generate a bi-predictive template sample, which can then be refined by a local refinement (Δx, Δy)* k This process may be repeated until MV is no longer updated or a maximum number of iterations is reached. One embodiment of such a process is summarized by the following steps, as shown in FIG.

[0140] At 1802, a counter l is initialized to l=0. At 1804, the initial L0 and L1 predicted signals

[0141]

number

[0142] and

[0143]

number

[0144] , as well as the initial bi-predictive template signal

[0145]

number

[0146] is the original MV of the block

[0147]

number

[0148] and

[0149]

number

[0150] The local L0 and L1 motion refinement based on equation (11) in 1806 and 1808 is performed using

[0151]

number

[0152] and

[0153]

number

[0154] , and Block's MV,

[0155]

number

[0156] and

[0157]

number

[0158] will be updated as.

[0159]

number

[0160] and

[0161]

number

[0162] is zero (determined in 1810), or l=l max If so (as determined in 1812), then the final bi-prediction may be generated using the refined motion vector in 1814. Otherwise, in 1816, the counter l is incremented and the process is repeated to obtain MV

[0163]

number

[0164] and

[0165]

number

[0166] Using the L0 and L1 prediction signals

[0167]

number

[0168] and

[0169]

number

[0170] and bi-predictive template signal

[0171]

number

[0172] is updated (at 1806, 1808).

[0173] FIG. 18 shows an example of DMVR processing using an example optical-flow-based motion derivation method for calculating motion refinements of DMVR blocks. As shown in FIG. 18, optical MVs of one DMVR block are identified by iteratively modifying original MVs based on an optical flow model. Such a method can provide good motion estimation accuracy, but may incur significant complexity increase. To reduce the derivation complexity, in one embodiment of the disclosure, it is proposed to apply only one iteration for deriving motion refinements using the proposed motion derivation method, for example, only to apply the processes shown in 1804 to 1808 to derive modified MVs of DMVR blocks.

[0174] The optical flow-based motion derivation model can be more efficient for small CUs than for large CUs due to the high consistency among the characteristics of samples inside small blocks. In some embodiments, it is proposed to enable the proposed optical flow-based motion derivation for a DMVR block when its block size is small (e.g., below a given threshold). Otherwise, the existing block matching-based motion refinement is used to derive the local motion refinement for the current block (e.g., according to the proposed DMVR latency elimination / reduction method described herein).

[0175] It will be noted that one or more of the various hardware elements of the described embodiments are referred to as "modules" that perform various functions described herein in association with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) recognized by those skilled in the art for a given implementation. It will be noted that each described module may also include executable instructions to perform one or more functions described as being performed by the respective module, which may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, and / or software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium, commonly referred to as RAM, ROM, etc.

[0176] Although the features and elements have been described above in certain combinations, one skilled in the art will recognize that each feature or element may be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a TRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. refining a first unrefined motion vector and a second unrefined motion vector to generate a first refined motion vector and a second refined motion vector in a first block; bi-predictively predicting the first block using the first refined motion vector and the second refined motion vector; determining a deblocking boundary filter strength based at least in part on the first unrefined motion vector and the second unrefined motion vector; 2. A video decoding method comprising:

2. 2. The method of claim 1, further comprising: extracting motion information for a second block using one or both of the first unrefined motion vector and the second unrefined motion vector, the second block being a spatial neighbor of the first block.

3. The method of claim 1 , further comprising applying a filter to a boundary of the first block using the determined deblocking boundary filter strength.

4. 4. The method of claim 1, wherein refining the first unrefined motion vector and the second unrefined motion vector is performed using decoder-side motion vector refinement (DMVR).

5. 4. The method of claim 1, wherein refining the first unrefined motion vector and the second unrefined motion vector comprises selecting the first refined motion vector and the second refined motion vector to substantially minimize an error metric.

6. 6. The method of claim 5, wherein the error metric is a template cost, and refining the first unrefined motion vector and the second unrefined motion vector comprises selecting the first refined motion vector and the second refined motion vector to substantially minimize the template cost with respect to a template signal generated by bi-prediction using the first unrefined motion vector and the second unrefined motion vector.

7. Extracting motion information for a second block includes adding a motion vector difference to at least one of the first unrefined motion vector and the second unrefined motion vector to generate at least one reconstructed motion vector; generating an inter prediction of the second block using at least one reconstructed motion vector; The method of claim 2 , further comprising:

8. refining a first unrefined motion vector and a second unrefined motion vector to generate a first refined motion vector and a second refined motion vector in a first block; bi-predictively predicting the first block using the first refined motion vector and the second refined motion vector; determining a deblocking boundary filter strength based at least in part on the first unrefined motion vector and the second unrefined motion vector; A video decoding device comprising one or more processors configured to execute at least the above.

9. 9. The apparatus of claim 8, further configured to: extract motion information of a second block using one or both of the first unrefined motion vector and the second unrefined motion vector, the second block being a spatial neighbor of the first block.

10. The apparatus of claim 8 , further configured to: apply a filter to a boundary of the first block using the determined deblocking boundary filter strength.

11. 11. The apparatus of claim 8, wherein refining the first unrefined motion vector and the second unrefined motion vector is performed using decoder-side motion vector refinement (DMVR).

12. 11. The apparatus of claim 8, wherein refining the first unrefined motion vector and the second unrefined motion vector comprises selecting the first refined motion vector and the second refined motion vector to substantially minimize an error metric.

13. 11. The apparatus of claim 8, further configured to extract motion information of a third block using at least one of the first refined motion vector and the second refined motion vector, the third block and the first block being co-located blocks in different pictures.

14. 10. The apparatus of claim 9, wherein extracting motion information for the second block includes using at least one of the first unrefined motion vector and the second unrefined motion vector as a spatial merging candidate.

15. refining a first unrefined motion vector and a second unrefined motion vector to generate a first refined motion vector and a second refined motion vector in a first block; bi-predictively predicting the first block using the first refined motion vector and the second refined motion vector; determining a deblocking boundary filter strength based at least in part on the first unrefined motion vector and the second unrefined motion vector; A video encoding method comprising:

16. 16. The method of claim 15, further comprising: extracting motion information for a second block using one or both of the first unrefined motion vector and the second unrefined motion vector, the second block being a spatial neighbor of the first block.

17. The method of claim 15 , further comprising applying a filter to a boundary of the first block using the determined deblocking boundary filter strength.

18. 18. The method of claim 15, wherein refining the first unrefined motion vector and the second unrefined motion vector is performed using decoder-side motion vector refinement (DMVR).

19. 18. The method of claim 15, further comprising: extracting motion information of a third block using at least one of the first refined motion vector and the second refined motion vector, the third block and the first block being co-located blocks in different pictures.

20. refining a first unrefined motion vector and a second unrefined motion vector to generate a first refined motion vector and a second refined motion vector in a first block; bi-predictively predicting the first block using the first refined motion vector and the second refined motion vector; determining a deblocking boundary filter strength based at least in part on the first unrefined motion vector and the second unrefined motion vector; 1. A video encoding device comprising one or more processors configured to execute at least the steps of:

21. 21. The apparatus of claim 20, further configured to: extract motion information for a second block using one or both of the first unrefined motion vector and the second unrefined motion vector, the second block being a spatial neighbor of the first block.

22. The apparatus of claim 20 , further configured to: apply a filter to a boundary of the first block using the determined deblocking boundary filter strength.

23. 23. The apparatus of claim 20, wherein refining the first unrefined motion vector and the second unrefined motion vector is performed using decoder-side motion vector refinement (DMVR).

24. 23. The apparatus of claim 20, further configured to extract motion information of a third block using at least one of the first refined motion vector and the second refined motion vector, the third block and the first block being co-located blocks in different pictures.

Images