Method and apparatus for reducing coding latency for decoder-side motion correction

Abstract

The invention relates to a method and apparatus for reducing coding latency for decoder-side motion correction. Embodiments of video coding systems and methods for reducing coding latency resulting from decoder-side motion vector correction (DMVR) are described. In one example, two unmodified motion vectors are identified for coding of a sampled first block (e.g., a first coding unit) using bidirectional prediction. One or both of the unmodified motion vectors are used to predict motion information of a sampled second block (e.g., a second coding unit). The two uncorrected motion vectors are corrected using DMVR, and the corrected motion vectors are used to generate a prediction signal for the sampled first block. Such embodiments allow the sampled second block to be coded substantially in parallel with the first block without waiting for DMVR completion on the first block. In further embodiments, optical flow-based techniques are described for motion vector correction.

Description

This application is a divisional application of Chinese Patent Application No. 201980044209.9, filed on June 20, 2019, entitled “Method and Device for Reducing Decoding Waiting Time of Motion Correction on the Decoder Side”, and the entire contents of the parent application are incorporated herein by reference. CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a formal application of U.S. Provisional Patent Application Serial No. 62 / 690,507 (filed on June 27, 2018) entitled “Method and Apparatus for Reducing Decoding Latency for Decoder-Side Motion Correction” and claims the benefit of that application under 35 U.S.C. §119(e), the entire contents of which are incorporated herein by reference. Background Art

[0002] Video coding systems are widely used to compress digital video signals to reduce the storage requirements and / or transmission bandwidth of these signals. Among the various types of video coding systems, such as block-based, wavelet-based, and object-based systems, block-based hybrid video coding systems are currently the most widely used and deployed. Examples of block-based video coding systems include international video coding standards such as MPEG1 / 2 / 4 Part 2, H.264 / MPEG-4 Part 10 AVC, VC-1, and the most recent video coding standard called High Efficiency Video Coding (HEVC), which was developed by ITU-T / SG16 / Q.6 / VCEG and JCT-VC (Joint Video Coding Task Force) of ISO / IEC / MPEG.

[0003] The first version of the HEVC standard was completed in October 2013 and provides approximately 50% bitrate savings or equivalent perceived quality compared to the previous generation video coding standard H.264 / MPEG AVC. Although the HEVC standard provides significant coding improvements over its predecessor, there is evidence that superior coding efficiency can be achieved using additional coding tools than HEVC. Based on this, VCEG and MPEG started the development of new coding technologies for future video coding standardization. The Joint Video Development Team (JVET), established in October 2015 by ITU-T VECG and ISO / IECMPEG, started important research on advanced technologies that can achieve substantial enhancements in coding efficiency. JVET maintains reference software called the Joint Development Model (JEM) by integrating some additional coding tools on top of the HEVC Test Model (HM).

[0004] In October 2017, ITU-T and ISO / IEC issued a joint call for proposals (CfP) for video compression beyond HEVC capabilities. In April 2018, 23 responses were received and evaluated at the tenth JVET meeting, demonstrating compression efficiency gains of approximately 40% over HEVC. Based on this evaluation result, JVET launched a new program to develop a new generation of video coding standards called Versatile Video Coding (VCC). In the same month, a reference software code library called VVC Test Model (VTM) was established to certify the VVC standard reference implementation. For the initial VTM-1.0, according to the existing HEVC design, most coding modules include intra-frame prediction, inter-frame prediction, transform / inverse transform and quantization / dequantization, and in-loop filters, unless a multi-type tree-based block partitioning structure is used in VTM. At the same time, in order to facilitate the evaluation of new coding tools, another reference software library called Benchmark Set (BMS) was also generated. In the BMS code base, a series of decoding tools inherited from JEM that provide higher decoding efficiency and moderate implementation complexity are also included on the basis of VTM, and are used as a benchmark when evaluating similar decoding technologies during the VVC standardization process. The JEM decoding tools integrated in BMS-1.0 include 65 angle intra-frame prediction directions, modified coefficient decoding, advanced multiple transform (AMT) + 4x4 non-separable secondary transform (NSST), affine motion model, generalized adaptive loop filter (GALF), advanced temporal motion vector prediction (ATMVP), adaptive motion vector prediction, decoder-side motion vector correction (DMVR) and LM chroma mode. Summary of the invention

[0005] Some embodiments include methods used in video encoding and decoding (collectively referred to as "decoding"). In some embodiments of the block-based video decoding method, the method includes: in a first block, refining a first uncorrected motion vector and a second uncorrected motion vector to generate a first corrected motion vector and a second corrected motion vector; using one or both of the first uncorrected motion vector and the second uncorrected motion vector to predict motion information of a second block, the second block being a spatial neighbor of the first block; and predicting the first block by bidirectional prediction using the first corrected motion vector and the second corrected motion vector.

[0006] In an example of a video decoding method, a first uncorrected motion vector and a second uncorrected motion vector associated with a first block are identified. One or both of the first uncorrected motion vector and the second uncorrected motion vector are used to predict motion information of a second block adjacent to the first block. For example, the first uncorrected motion vector and the second uncorrected motion vector are corrected using decoder-side motion vector correction (DMVR). The corrected motion vector is used to generate a first corrected motion vector and a second corrected motion vector, which can be used for bidirectional prediction of the first block. Predicting motion information of the second block using one or more uncorrected motion vectors can be performed using one or more techniques and using one or more uncorrected motion vectors as spatial merge candidates, the technique being, for example, spatial advanced motion vector prediction (AMVP), temporal motion vector prediction (TMVP), advanced temporal motion vector prediction (TMVP). In the case of spatial prediction, the second block may be a spatial neighbor of the first block; in the case of temporal prediction, the second block may be a co-located block in a subsequently decoded picture. In some embodiments, a deblocking filter strength of the first block is determined based at least in part on the first uncorrected motion vector and the second uncorrected motion vector.

[0007] In another example of a video decoding method, a first uncorrected motion vector and a second uncorrected motion vector associated with a first block are identified. The first uncorrected motion vector and the second uncorrected motion vector are corrected, for example using DMVR, to generate a first corrected motion vector and a second corrected motion vector. Motion information of the second block is predicted using spatial motion prediction or temporal motion prediction, wherein (i) if spatial motion prediction is used, one or both of the first uncorrected motion vector and the second uncorrected motion vector are used to predict the motion information, and (ii) if temporal motion prediction is used, one or both of the first corrected motion vector and the second corrected motion vector are used to predict the motion information.

[0008] In another example of a video decoding method, at least one predictor is selected for predicting motion information of a current block. The at least one predictor is selected from a set of available predictors, wherein the available predictors include (i) at least one uncorrected motion vector from a spatially neighboring block of the current block and (ii) at least one corrected motion vector from a co-located block of the current block.

[0009] In another example of a video decoding method, at least two non-overlapping regions in a slice are determined. A first uncorrected motion vector and a second uncorrected motion vector associated with a first block are identified in a first region. The first uncorrected motion vector and the second uncorrected motion vector are corrected to generate a first corrected motion vector and a second corrected motion vector. In response to determining that motion information of a second block adjacent to the first block is predicted using motion information of the first block, the motion information of the second block is predicted using: (i) one or both of the first uncorrected motion vector and the second uncorrected motion vector if the first block is not at a bottom edge or a right edge of the first region, and (ii) one or both of the first corrected motion vector and the second corrected motion vector if the first block is at a bottom edge or a right edge of the first region.

[0010] In another example of a video decoding method, at least two non-overlapping regions in a slice are determined. A first uncorrected motion vector and a second uncorrected motion vector associated with a first block are identified in a first region. The first uncorrected motion vector and the second uncorrected motion vector are corrected to generate a first corrected motion vector and a second corrected motion vector. In response to determining that motion information of a second block adjacent to the first block is predicted using motion information of the first block, the motion information of the second block is predicted using: (i) one or both of the first uncorrected motion vector and the second uncorrected motion vector if the second block is in the first region, and (ii) one or both of the first corrected motion vector and the second corrected motion vector if the second block is not in the first region.

[0011] In another example of a video coding method, at least two non-overlapping regions in a slice are determined. A first uncorrected motion vector and a second uncorrected motion vector associated with a first block are identified in a first region. The first uncorrected motion vector and the second uncorrected motion vector are corrected to generate a first corrected motion vector and a second corrected motion vector. Motion information of the second block is predicted using spatial motion prediction or temporal motion prediction, wherein (i) if the first block is not at a bottom edge or a right edge of the first region and if spatial motion prediction is used, the motion information is predicted using one or both of the first uncorrected motion vector and the second uncorrected motion vector, and (ii) if the first block is at a bottom edge or a right edge of the first region or if temporal motion prediction is used, the motion information is predicted using one or both of the first corrected motion vector and the second corrected motion vector.

[0012] In another example of the video coding method, at least two non-overlapping regions are defined in a slice. A set of available predictors is determined for predicting motion information of a current block in a first region, wherein the set of available predictors is restricted to not including motion information of any block in a second region different from the first region.

[0013] Some embodiments relate to a method for correcting motion vectors. In one example, a first uncorrected motion vector and a second uncorrected motion vector are determined for a current block. A first prediction vector is generated using the first uncorrected motion vector. (0) , and generating a second prediction I using the second uncorrected motion vector (1) The optical flow model is used to determine the motion correction of the current block The first uncorrected motion vector and the second uncorrected motion vector are corrected using motion correction to generate a first corrected motion vector and a second corrected motion vector. The current block is predicted by bi-directional prediction using the first corrected motion vector and the second corrected motion vector.

[0014] In another example of a video coding method, a first uncorrected motion vector and a second uncorrected motion vector are determined for a current block. A first prediction vector is generated using the first uncorrected motion vector. (0) , and generating a second prediction I using the second uncorrected motion vector (1) . Determine the motion correction for the current block in where θ is the coordinate set of all samples in the current block, and The first uncorrected motion vector and the second uncorrected motion vector are corrected using motion correction to generate a first corrected motion vector and a second corrected motion vector. The current block is predicted by bi-directional prediction using the first corrected motion vector and the second corrected motion vector.

[0015] In another example of the video decoding method, a first motion vector and a second motion vector are determined for a current block. The first motion vector and the second motion vector are modified by iteratively performing steps, the steps comprising: (a) Generate a first prediction P using the first motion vector 0 , and generating a second prediction P using the second motion vector 1 ; (b) By the first prediction P 0 and the second prediction P 1 Averaging is performed to generate a bidirectional prediction template signal P 模板 ; (c) Determine a first motion correction (Δx, Δy) for the first motion vector using an optical flow model based on the template signal P template * 0 and the second motion correction (Δx, Δy) for the second motion vector * 1; and (d) Use the first motion correction (Δx, Δy) * 0 to correct the first motion vector and use the second motion correction (Δx, Δy) * 1 to correct the second motion vector.

[0016] Additional embodiments include encoder and decoder (collectively referred to as "codec") systems configured to perform the methods described herein. The system may include a processor and a non-transitory computer storage medium storing instructions that, when executed on the processor, are operable to perform the methods described herein. Additional embodiments include a non-transitory computer readable medium storing a video encoded using the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A is a system diagram illustrating an example communication system in which one or more disclosed embodiments may be implemented;

[0018] Figure 1B It is shown that according to the embodiment, Figure 1A A system diagram of an example wireless transmit / receive unit (WTRU) for use within a communication system is shown;

[0019] Figure 2 is a functional block diagram of a block-based video encoder, such as an encoder used for VCC;

[0020] Figures 3A-3E Block partitioning showing multiple types of tree structures: Quarter partitioning ( Figure 3A ); vertical bisection ( Figure 3B ); horizontal dichotomy ( Figure 3C ); vertical third division ( Figure 3D ); horizontal third division ( Figure 3E );

[0021] Figure 4 is a functional block diagram of a block-based video decoder, such as a decoder for VVC;

[0022] Figure 5 An example of spatial motion vector prediction is shown;

[0023] Figure 6 An example of temporal motion vector prediction (TMVP) is shown;

[0024] Figure 7An example of Advanced Temporal Motion Vector Prediction (ATMVP) is shown;

[0025] Figures 8A-8B An example of decoder-side motion vector correction (DMVR) is shown;

[0026] Fig. 9 Parallel decoding for VTM-1.0 is shown;

[0027] Fig.10 shows the decoding latency caused by DMVR;

[0028] Fig.11 An embodiment is shown in which only the modified MV from the DMVR is used to generate a bidirectional prediction signal;

[0029] Fig.12 An embodiment is shown in which the modified MV from DMVR is used for temporal motion prediction and deblocking and the unmodified MV is used for spatial motion prediction;

[0030] Fig.13 An implementation of using modified MV from DMVR for temporal motion prediction and unmodified MV for spatial motion prediction and deblocking is shown;

[0031] Fig.14 shows parallel decoding after applying a latency elimination method for DMVR according to some embodiments;

[0032] Fig.15 An embodiment of using the unmodified MV of the DMVR block inside a picture segment for spatial motion prediction and deblocking is shown;

[0033] Fig.16 An embodiment in which a current picture is divided into a plurality of segments and encoding latency is reduced for blocks within each segment is shown;

[0034] Fig.17 An embodiment in which a current picture is divided into multiple fragments and encoding latency is reduced for blocks from different fragments is shown;

[0035] Fig.18 is a flow chart of a motion correction process using optical flow according to some embodiments. Example Network of Detailed Description

[0036] Figure 1A1 is a diagram showing an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content by sharing system resources including wireless bandwidth. For example, the communication system 100 may use one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), zero tail unique word DFT spread OFDM (ZT UW DTS-sOFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, and filter bank multi-carrier (FBMC), etc.

[0037] like Figure 1A As shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, Internet 110, and other networks 112, although it should be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network components. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, any of the WTRUs 102a, 102b, 102c, 102d may be referred to as a “station” and / or “STA”, which may be configured to transmit and / or receive wireless signals, and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular phone, a personal digital assistant (PDA), a smart phone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable device, a head-mounted display (HMD), a vehicle, a drone, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices working in an industrial and / or automated process chain environment), consumer electronic devices, and devices working on a commercial and / or industrial wireless network, etc. Any of the WTRUs 102a, 102b, 102c, 102d may be interchangeably referred to as a UE.

[0038] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to facilitate access to one or more communication networks (e.g., the CN 106 / 115, the Internet 110, and / or other networks 112) by wirelessly interfacing with at least one of the WTRUs 102a, 102b, 102c, 102d. For example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNode B, a Home Node B, a Home eNode B, a gNB, an NR Node B, a site controller, an access point (AP), a wireless router, and the like. Although the base stations 114a, 114b are each described as a single component, it should be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network components.

[0039] The base station 114a may be part of the RAN 104 / 113, and the RAN may also include other base stations and / or network components (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, and the like. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies called cells (not shown). These frequencies may be in a licensed spectrum, an unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage for a specific geographic area that is relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, a cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, that is, each transceiver corresponds to a sector of the cell. In an embodiment, the base station 114a may use multiple-input multiple-output (MIMO) technology and may use multiple transceivers for each sector of the cell. For example, by using beamforming, signals may be transmitted and / or received in a desired spatial direction.

[0040] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0041] More specifically, as described above, the communication system 100 may be a multiple access system and may use one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, and SC-FDMA, among others. For example, the base station 114a in the RAN 104 / 113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may use Wideband CDMA (WCDMA) to establish the air interface 116. WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0042] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-APro).

[0043] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR radio access, which may establish the air interface 116 using new radio (NR).

[0044] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may jointly implement LTE radio access and NR radio access (e.g., using dual connectivity (DC) principles). Thus, the air interface used by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0045] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), and GSM EDGE (GERAN), among others.

[0046] Figure 1A The base station 114b in the figure can be a wireless router, a Home NodeB, a Home eNodeB, or an access point, and can use any appropriate RAT to facilitate wireless connectivity in a local area, such as a business place, a residence, a vehicle, a campus, an industrial facility, an air corridor (such as for use by drones), a road, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d can establish a wireless local area network (WLAN) by implementing a radio technology such as IEEE 802.11. In an embodiment, the base station 114b and the WTRUs 102c, 102d can establish a wireless personal area network (WPAN) by implementing a radio technology such as IEEE 802.15. In another embodiment, the base station 114b and the WTRUs 102c, 102d can establish a picocell or a femtocell by using a cellular-based RAT (such as WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). Figure 1A As shown, the base station 114b may be directly connected to the Internet 110. Thus, the base station 114b does not need to access the Internet 110 via the CN 106 / 115.

[0047] The RAN 104 / 113 may be in communication with the CN 106 / 115, which may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have different quality of service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 / 115 may provide call control, billing services, mobile location-based services, prepaid calls, Internet connectivity, video distribution, etc., and / or may perform advanced security functions such as user authentication. Although in Figure 1AAlthough not shown, it will be appreciated that the RAN 104 / 113 and / or the CN 106 / 115 may be in direct or indirect communication with other RANs that employ the same RAT or a different RAT as the RAN 104 / 113. For example, in addition to being connected to the RAN 104 / 113 employing NR radio technology, the CN 106 / 115 may also be in communication with another RAN (not shown) employing GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0048] The CN 106 / 115 may also act as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer network devices that use common communication protocols, such as TCP, User Datagram Protocol (UDP), and / or IP in the Transmission Control Protocol / Internet Protocol (TCP / IP) Internet protocol suite. The networks 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, where the one or more RANs may use the same RAT or a different RAT as the RAN 104 / 113.

[0049] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). Figure 1A The illustrated WTRU 102c may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0050] Figure 1B is a system diagram showing an example WTRU 102. Figure 1B As shown, the WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keyboard 126, a display / touchpad 128, non-removable memory 130, a removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and other peripherals 138. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0051] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal decoding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, however it should be appreciated that the processor 118 and the transceiver 120 may also be integrated together in an electronic package or chip.

[0052] The transmit / receive component 122 may be configured to transmit or receive signals to or from a base station (e.g., base station 114a) via the air interface 116. For example, in one embodiment, the transmit / receive component 122 may be an antenna configured to transmit and / or receive RF signals. As an example, in an embodiment, the transmit / receive component 122 may be a transmitter / detector configured to transmit and / or receive IR, UV, or visible light signals. In yet another embodiment, the transmit / receive component 122 may be configured to transmit and / or receive RF and light signals. It should be appreciated that the transmit / receive component 122 may be configured to transmit and / or receive any combination of wireless signals.

[0053] Although in Figure 1B 102 as a single component, but the WTRU 102 may include any number of transmit / receive components 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive components 122 (e.g., multiple antennas) for transmitting and receiving radio signals over the air interface 116.

[0054] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive component 122 and to demodulate signals received by the transmit / receive component 122. As described above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers that allow the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0055] The processor 118 of the WTRU 102 may be coupled to the speaker / microphone 124, the keyboard 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), and may receive user input data from these components. The processor 118 may also output user data to the speaker / microphone 124, the keyboard 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from and store data in any suitable memory such as the non-removable memory 130 and / or the removable memory 132. The non-removable memory 130 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from and store data in memories that are not physically located in the WTRU 102, such as, for example, a server or a home computer (not shown).

[0056] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control power for use by other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (Ni-Cd), nickel-zinc (Ni-Zn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0057] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) related to the current location of the WTRU 102. In addition to or in lieu of the information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) via the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information via any suitable positioning method while remaining consistent with an embodiment.

[0058] The processor 118 may also be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game console module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, and an activity tracker, etc. The peripheral device 138 may include one or more sensors, which may be one or more of the following: a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0059] The WTRU 102 may include a full-duplex radio in which reception and transmission of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous for the radio. A full-duplex radio may include an interface management unit that reduces and / or substantially eliminates self-interference by means of hardware (e.g., chokes) or by signal processing by a processor (e.g., a separate processor (not shown) or by the processor 118). In an embodiment, the WTRU 102 may include a half-duplex radio that transmits and receives some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)).

[0060] Although in Figure 1A-1B The WTRU is described as a wireless terminal, however it should be appreciated that in certain typical embodiments, such a terminal may use a (eg, temporary or permanent) wired communication interface with a communication network.

[0061] In a typical implementation, the other network 112 may be a WLAN.

[0062] Given that Figure 1A-1B As well as the corresponding description, one or more or all of the functions described herein may be performed by one or more simulation devices (not shown). These simulation devices may be one or more devices configured to simulate one or more or all of the functions described herein. For example, these simulation devices may be used to test other devices and / or simulate network and / or WTRU functions.

[0063] The simulation device can be designed to implement one or more tests about other devices in a laboratory environment and / or an operator network environment. For example, the one or more simulation devices can perform one or more or all functions while being implemented and / or deployed as part of a wired and / or wireless communication network in whole or in part to test other devices inside the communication network. The one or more simulation devices can perform one or more or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The simulation device can be directly coupled to other devices to perform the test, and / or can use over-the-air wireless communication to perform the test.

[0064] The one or more simulation devices can perform one or more functions (including all functions) while not being implemented / deployed as part of a wired and / or wireless communication network. For example, the simulation device can be used in a test lab and / or a test scenario of a wired and / or wireless communication network that is not deployed (e.g., tested) to implement tests on one or more components. The one or more simulation devices can be test equipment. The simulation device can transmit and / or receive data using direct RF coupling and / or wireless communication with the aid of RF circuits (as an example, the circuits can include one or more antennas). DETAILED DESCRIPTION Block-based video decoding

[0065] Like HEVC, VVC is built on a block-based hybrid video coding framework. Figure 2 is a functional block diagram of an example of a block-based hybrid video coding system. The input video signal 103 is processed block by block. The block may be referred to as a coding unit (CU). In VTM-1.0, a CU can be up to 128x128 pixels. However, compared to HEVC which is based only on quadtree partitioning of blocks, in VTM-1.0, a coding tree unit (CTU) may be divided into multiple CUs based on a quadtree / binary / tri-tree to adapt to changing local characteristics. In addition, the concept of multiple partition unit types in HEVC may be removed, whereby the separation of CU, prediction unit (PU), and transform unit (TU) is not used in VVC; instead, each CU may be used as a basic unit for prediction and transformation without further partitioning. In the multi-type tree structure, the CTU is first divided by a quadtree structure. Then, each quadtree leaf node can be further divided by a binary and tri-tree structure. As Figures 3A-3E As shown, there are five types of divisions: quarter division, horizontal half division, vertical half division, horizontal third division, and vertical third division.

[0066] exist Figure 2In the video processing of a video block, spatial prediction (161) and / or temporal prediction (163) may be performed. Spatial prediction (or "intra-frame prediction") uses pixels from samples (called reference samples) of already decoded neighboring blocks in the same video picture / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also known as "inter-frame prediction" or "motion compensated prediction") uses reconstructed pixels from already decoded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU is typically sent via one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index is also sent, which is used to identify which reference picture in the reference picture repository (165) the temporal prediction signal comes from.

[0067] After spatial and / or temporal prediction, a mode decision block (181) in the encoder selects the best prediction mode, for example based on a rate-distortion optimization method. The prediction block (117) is then subtracted from the current video block, and the prediction residual is decorrelated using a transform (105) and a quantization (107). The quantized residual coefficients are inverse quantized (111) and inverse transformed (113) to form a reconstructed residual, which is then added back to the prediction block (127) to form a reconstructed signal for the CU. In addition, an in-loop filter, such as a deblocking filter, may be applied (167) to the reconstructed CU before it is placed in a reference picture repository (165) and used to encode future video blocks. To form an output video bitstream 121, the decoding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy decoding unit (109) to be further compressed and packaged to form a bitstream.

[0068] Figure 4 226 is a functional block diagram of a block-based video decoder. The video bitstream 202 is unpacked and entropy decoded at the entropy decoding unit 208. The decoding mode and prediction information are sent to the spatial prediction unit 260 (if intra-coded) or the temporal prediction unit 262 (if inter-coded) to form a prediction block. The residual transform coefficients are sent to the inverse quantization unit 210 and the inverse transform unit 212 to reconstruct the residual block. At 226, the prediction block and the residual block are then added together. The reconstructed block is in-loop filtered before it is stored in the reference picture store 264. The reconstructed video in the reference picture store is then sent out to drive the display device and used to predict future video blocks.

[0069] As mentioned before, BMS-1.0 follows Figure 2 and Figure 4The same encoding / decoding workflow of VTM-1.0 shown in FIG. However, some decoding modules, especially those associated with temporal prediction, are further extended and enhanced. In the following description, some tools included in BMS-1.0 or the previous JEM are briefly described. Motion Vector Prediction

[0070] Like HEVC, in order to reduce the overhead of signaling motion information, both VTM and BMS include two modes to decode the motion information of each CU, called merge mode and non-merge mode. In merge mode, the motion information of the current CU is obtained directly from the spatial and temporal neighboring blocks, and a competition-based scheme is applied to select the best neighboring block from all available candidates; accordingly, only the index of the best candidate is sent for reconstructing the motion information of the CU in the decoder. If an inter-coded PU is decoded in non-merge mode, the MV is decoded differently using the MV predictor obtained from the Advanced Motion Vector Prediction (AMVP) technique. As in merge mode, AMVP obtains the MV predictor from spatial and temporal neighboring candidates. Then, the difference between the MV predictor and the actual MV, as well as the index of the predictor, are transmitted to the decoder.

[0071] Figure 5 An example of spatial MV prediction is shown. In the current picture (CurrPic) to be decoded, the square CurrCU is the current CU, which has the best matching block CurrRefCU in the reference picture (CurrRefPic). The MV of CurrCU (i.e., MV2) is to be predicted. The spatial neighbors of the current CU can be the neighboring CUs above, to the left, to the upper left, to the lower left, and to the upper right of the current CU. Figure 5 The neighboring CU is shown as the upper neighbor NeighbCU. The reference picture (NeighbRefPic) and MV (MV1) of NeighbCU are both known because NeighbCU has been decoded before CurrCU.

[0072] Figure 6 An example of temporal MV prediction (TMVP) is shown. Figure 6Four pictures (ColRefPic, CurrRefPic, ColPic, CurrPic) are shown in FIG. In the current picture (CurrPic) to be decoded, the square CurrCU is the current CU, which has the best matching block (CurrRefCU) in the reference picture (CurrRefPic). The MV of CurrCU (i.e., MV2) is to be predicted. The temporal neighbor of the current CU is designated as the co-located CU (ColCU) in the neighboring picture (ColPic). Both the reference picture (ColRefPic) and the MV (MV1) of ColCU are known because ColPic has been decoded before CurrPic.

[0073] For spatial and temporal motion vector prediction, given limited time and space, the MV between different blocks is regarded as a translation of uniform speed. Figure 5 and 6 In the example, the time distance between CurrPic and CurrRefPic is TB, and Figure 5 The time distance between CurrPic and NeighbRefPic or Figure 6 The temporal distance between ColPic and ColRefPic is TD. The scaled MV predictor can be calculated as

[0074] In VTM-1.0, each merge block has at most one motion parameter set (one motion vector and one reference picture index) for each prediction direction L0 and L1. In contrast, in BMS-1.0, additional merge candidates based on Advanced Temporal Motion Vector Prediction (ATMVP) are included to enable motion information at the sub-block level. Using this mode, temporal motion vector prediction can be improved by allowing a CU to obtain multiple MVs for sub-blocks in the CU. In general, if Figure 7 As shown in Figure 1, ATMVP obtains the motion information of the current CU in two steps. The first step is to identify the corresponding block of the current block in the temporal reference picture (called the co-located block). The selected temporal reference picture is called the co-located picture. The second step is to divide the current block into multiple sub-blocks and obtain the motion information of each sub-block from the corresponding small block in the co-located picture.

[0075] In the first step, the co-located blocks and co-located pictures are determined by the motion information of the spatial neighboring blocks of the current block. In the current design, the first available candidate in the merge candidate list is considered. Figure 7 This process is shown in FIG. Figure 7In the example of , block A is identified as the first available merge candidate for the current block based on the scan order of the merge candidate list. Then, the corresponding motion vector (MVA) of block A and its reference index are used to identify the co-located picture and the co-located block. The position of the co-located block in the co-located picture is determined by adding the motion vector (MVA) of block A to the coordinates of the current block.

[0076] In the second step, for each sub-block in the current block, its corresponding small block in the same block (such as Figure 7 The motion information of the sub-block (indicated by the small arrow in the figure) is used to obtain the motion information of the sub-block. Specifically, after the motion information of each small block in the co-located block is identified, the motion information is converted into a motion vector and a reference index of the corresponding sub-block in the current block in the same way as in TMVP. Decoder-side motion vector correction (DMVR)

[0077] For the merge mode in VTM, when the selected merge candidate is bidirectionally predicted, the prediction signal of the current CU is formed by averaging the two prediction blocks using the two MVs associated with the candidate's reference lists L0 and L1. However, the motion information of the merge candidate (which is obtained from the spatial or temporal neighbors of the current CU) may not be accurate enough to represent the actual motion of the current CU and may therefore jeopardize the inter-frame prediction efficiency. In order to further improve the decoding performance of the merge mode, the decoder-side motion vector correction (DMVR) method is applied in BMS-1.0 to correct the MV of the merge mode. Specifically, when the selected merge candidate is bidirectionally predicted, a bidirectional prediction template is first generated as an average of two prediction signals based on the MVs from reference lists L0 and L1, respectively. Then, block matching-based motion correction is performed locally around the initial MV using the bidirectional prediction template as a target, as explained below.

[0078] Fig. 8A The motion correction process applied in DMVR is shown in FIG. Generally speaking, DMVR corrects the MV of the merge candidate through the following two steps. Fig. 8AAs shown, in the first step, a bidirectional prediction template is generated by averaging two prediction blocks of the initial MV (i.e., MV0 and MV1) in L0 and L1 using the merge candidate. Then, for each reference list (i.e., L0 or L1), a block matching-based motion search is performed in a local area around the initial MV. For each MV, i.e., MV0 or MV1 of the corresponding reference list in the list surrounding the initial MV, the cost value (e.g., the sum of absolute differences (SAD)) between the bidirectional prediction template and the corresponding prediction block using the motion vector is measured. For each of the two prediction directions, the MV that minimizes the template cost in the prediction direction is considered to be the final MV in the reference list of the merge candidate. In the current BMS-1.0, for each prediction direction, eight neighboring MVs (with an integer sample offset) surrounding the initial MV are considered during the motion correction process. Finally, the two corrected MVs (such as Fig. 8A MV0' and MV1' shown in are used to generate the final bidirectional prediction signal of the current CU. In addition, in traditional DMVR, in order to further improve the decoding efficiency, the modified MV of the DMVR block is used to predict the motion information of its spatial and temporal neighboring blocks (for example, based on spatial AMVP, spatial merge candidate, TMVP and ATMVP), and calculate the boundary strength value of the deblocking filter applied to the current CU. Figure 8B : is a flowchart of an example of a DMVR process, in which "spatial AMVP" and "spatial merge candidate" relate to the spatial MV prediction process of spatially neighboring CUs that are in the current picture and decoded after the current CU in the CU decoding order; "TMVP" and "ATMVP" relate to the temporal MV prediction process of future CUs in subsequent pictures (pictures that are decoded after the current picture based on the picture decoding order); and "deblocking" relates to the deblocking filtering process of the current block and its spatially neighboring blocks.

[0079] exist Figure 8B In the method shown, at 802, a bidirectional prediction template is generated. At 804, motion correction is performed on the L0 motion vector, and at 806, motion correction is performed on the L1 motion vector. At 808, a final bidirectional prediction is generated using the corrected L0 and L1 motion vectors. Figure 8B In the method, the modified motion vector is used to predict the motion of the block to be decoded later. For example, the modified motion vector is used for spatial AMVP (810), TMVP (814) and ATMVP (816). The modified motion vector is also used as a spatial merge candidate (812) and calculates the boundary strength value of the deblocking filter applied to the current CU (818). Bidirectional optical flow

[0080] The bidirectional in VTM / BMS-1.0 is a combination of two temporal prediction blocks obtained from a reference picture that has been reconstructed using an average. However, due to the limitations of block-based motion compensation, a small residual motion can be observed between the two prediction blocks, thereby reducing the efficiency of motion compensated prediction. To solve this problem, bidirectional optical flow (BIO) was used in JEM in the past to compensate for this motion of each sample within the block. Specifically, BIO is a sample-based motion correction performed in addition to block-based motion compensated prediction when using bidirectional. The correction of the motion vector of each sample in a block is obtained based on a typical optical flow model. Assume I (k) (x, y) is the sample value at coordinate (x, y) of the prediction block obtained from reference picture list k (k=0, 1), and and are the horizontal and vertical gradients of the samples. The bidirectional signal modified by BIO is obtained as: Where τ0 and τ1 are related to I (0) and I (1) The temporal distance of the associated reference pictures Ref0 and Ref1 to the current picture. Furthermore, the motion correction (v) at the sample position (x, y) is calculated by minimizing the difference Δ between the values ​​of the samples after motion correction compensation. x ,v y ), as shown in

[0081] Furthermore, to provide regularity to the resulting motion correction, the motion correction is assumed to be consistent within a local region centered at (x, y); therefore, (v x ,v y ) is as follows

[0082] It should be mentioned that, unlike DMVR, the motion correction (v x ,v y ) is only applied to enhance the bidirectional signal but does not modify the motion information of the current CU. In other words, the MV used to predict the MVs of spatial and temporal neighboring blocks and to determine the deblocking boundary strength of the current CU is still the original MV (i.e., the block-based motion compensation signal I used to generate the CU before applying BIO). (0) (x,y) and I (1) MV of (x,y)). DMVR decoding waiting time

[0083] Like HEVC and its predecessors, VTM-1.0 uses motion compensated prediction (MCP) to effectively reduce temporal redundancy between pictures, thereby achieving high inter coding efficiency. Since the MV used to generate the prediction signal for one CU is signaled in the bitstream or inherited from its spatial / temporal neighbors, there is no dependency between the MCPs of spatially adjacent CUs. Therefore, the MCP processes of all inter blocks in the same picture / slice are independent of each other. Therefore, for VTM-1.0 and for HEVC, the decoding process of multiple inter blocks can be done in parallel, for example, they are assigned to different threads to exploit parallelism.

[0084] As mentioned above, the DMVR tool is applied in BMS-1.0. In order to avoid incurring additional signaling overhead, motion correction is obtained using two prediction signals associated with the original L0 and L1 MVs of the CU. Therefore, when the motion information of a CU is predicted from one of its spatial neighbors (e.g., through AMVP and merge mode) decoded by DMVR, its decoding process waits until DMVR fully reconstructs the MV of the neighboring block. This will obviously complicate the pipeline design, especially on the decoder side, resulting in a significant increase in the complexity of hardware implementation.

[0085] To illustrate the decoding latency caused by DMVR, Fig. 9 and Fig.10 An example comparing the decoding process of VTM-1.0 and BMS-1.0 is shown. For ease of explanation, a case is described where there are four CUs of equal block size and all four CUs are decoded by DMVR, each decoded by a separate decoding thread; the decoding complexity of each individual decoding module (e.g., MCP, DMVR, dequantization, and inverse transform) is considered to be the same for the four CUs. Fig. 9 As shown in Figure 2, since the four CUs can be decoded in parallel, the total decoding time of VTM-1.0 is equal to the decoding time of one CU, that is, T MCP +T 解量化 +T 逆变换 Due to the dependency brought by DMVR, the decoding process of BMS-1.0 (such as Fig.10 For BMS-1.0, the decoding of each individual coded block cannot be called until the DMVR of its spatial neighboring blocks is completely completed. Therefore, the total decoding time of four CUs is equal to T 总 =4*(T MCP +T DMVR )+T 解量化 +T 逆变换 It can be seen that DMVR uses prediction sampling to correct motion information, which brings dependencies between adjacent blocks, thereby significantly increasing the waiting time of the encoding and decoding process. Overview of Wait Time Reduction Methods

[0086] The method proposed in the present disclosure is used to eliminate or reduce the encoding / decoding latency of DMVR while maintaining its main decoding performance. Specifically, various embodiments of the present disclosure include one or more of the following aspects.

[0087] Unlike the current DMVR method in BMS-1.0, where the modified DMVR motion of a block is always used to predict the motion of its spatial / temporal neighboring blocks and obtain the deblocking filter strength, in some embodiments it is proposed to fully or partially use the unmodified MV of the DMVR block (the MV used to generate the original bidirectional prediction signal) for the MV prediction and deblocking process. Given that the original MV can be directly obtained from parsing and motion vector reconstruction (motion vector prediction period plus parsing motion vector difference) without DMVR, there is no dependency between neighboring blocks, and the decoding process of multiple inter-CUs can be completed in parallel.

[0088] Since the unmodified MV is not as accurate as the modified MV, this may lead to some decoding performance degradation. In order to reduce this loss, in some embodiments, it is proposed to divide the picture / slice into multiple regions. In addition, in some embodiments, additional restrictions are proposed, so that the decoding of multiple CUs in the same region or multiple CUs from different regions can be performed independently.

[0089] In some embodiments, an optical flow-based motion derivation method is proposed to replace the block matching-based motion search to calculate the motion correction for each DMVR CU. Compared with the block-based method that performs motion search in a small local window, some embodiments directly calculate the motion correction based on spatial and temporal sampling. This can not only reduce the computational complexity but also increase the motion correction accuracy because the value of the derived corrected motion is not limited to the search window. Use of uncorrected motion vectors for DMVR latency reduction

[0090] As described above, using the modified MV of a DMVR block as the MV predictor of its neighboring blocks is not friendly to parallel encoding / decoding designed for practical codecs, because the encoding / decoding of the neighboring blocks is not performed until the modified MV of the current block is fully reconstructed by DMVR. Based on this analysis, a method for removing the decoding latency caused by DMVR is proposed in this section. In some embodiments, the core design of DMVR (e.g., motion correction based on block matching) remains the same as the existing design. However, the MV of the DMVR block used to perform MV prediction (e.g., AMVP, merging, TMVP, and ATMVP) and deblocking is modified so that the dependency between neighboring blocks caused by DMVR can be removed. Use of uncorrected motion vectors for spatial and temporal motion prediction

[0091] In some embodiments, instead of using corrected motion, MV prediction and deblocking are performed always using uncorrected motion of the DMVR block. Fig.11 The modified DMVR process after using this method is shown. Fig.11 As shown, instead of using the modified MV, the unmodified MV (the original MV before DMVR) is used to obtain the MV predictor and determine the boundary strength of the deblocking filter. The modified MV is only used to generate the final bidirectional prediction signal of the block. Since the dependency between the modified MV of the current block and the decoding of its neighboring blocks does not exist, this implementation can be used to remove the encoding / decoding latency of DMVR. Use of uncorrected motion vectors for spatial motion prediction

[0092] exist Fig.11 In the example of , the uncorrected MV of the DMVR block is used to obtain the temporal motion predictor of the co-located block in the future picture through TMVP and ATMVP, and to calculate the boundary strength of the deblocking filter between the current block and its spatial neighbors. Since the uncorrected MV can be less accurate than the corrected MV, this may result in some decoding performance loss. On the other hand, temporal motion prediction (TMVP and ATMVP) uses previously decoded pictures (specifically, co-located pictures) to predict the MV in the current picture. Therefore, before performing temporal motion prediction of the current picture, the corrected MV of the DMVR CU in the co-located picture has been reconstructed. A similar situation applies to the deblocking filter process: since the deblocking filter is applied to the reconstructed samples, it can only be called after the samples of the current block are completely reconstructed through MC (including DMVR), dequantization and inverse transformation. Therefore, the corrected MV is already available before deblocking is applied to the DMVR block.

[0093] exist Fig.11 In the illustrated method, at 1100, an uncorrected motion vector for a first block is identified. The uncorrected motion vector may have been signaled for the first block using any of a variety of available MV signaling techniques. At 1102, the uncorrected motion vector is used to generate a bi-directional prediction template. At 1104, motion correction is performed on the L0 motion vector, and at 1106, motion correction is performed on the L1 motion vector. At 1108, a final bi-directional prediction for the first block is generated using the corrected L0 and L1 motion vectors. Fig.11 In the method, the uncorrected motion vector is used to predict the motion of a subsequently decoded block (e.g., a second block). For example, the uncorrected motion vector is used for spatial AMVP (1110), TMVP (1114), and ATMVP (1116). The uncorrected motion vector is also used as a spatial merge candidate (1112) and to calculate the boundary strength value of the deblocking filter (1118).

[0094] In another embodiment, in order to solve these problems and achieve better decoding performance, it is proposed to use different MVs (unmodified MVs and modified MVs) of the DMVR block for spatial motion prediction, temporal motion prediction, and deblocking filter. Specifically, in this embodiment, the unmodified MV is only used to obtain the MV predictor (e.g., spatial AMVP and spatial merge candidate) for spatial motion prediction, while the modified MV is used to not only obtain the final prediction of the block but also generate the MV predictor (TMVP and ATMVP) for temporal motion prediction and calculate the boundary strength parameter of the deblocking filter. Fig.12 The DMVR process according to this second embodiment is shown.

[0095] exist Fig.12 In the illustrated method, at 1200, an uncorrected motion vector for a first block is identified. The uncorrected motion vector has been signaled for the first block using any of a variety of available MV signaling techniques. At 1202, the uncorrected motion vector is used to generate a bi-directional prediction template. At 1204, motion correction is performed for the L0 motion vector, and at 1206, motion correction is performed for the L1 motion vector. At 1208, a final bi-directional prediction for the first block is generated using the corrected L0 and L1 motion vectors. Fig.12 In the method, the uncorrected motion vector is used to predict the motion of a subsequently decoded block (e.g., a second block) in the same picture as the first block. For example, the uncorrected motion vector is used for spatial AMVP (1110) and as a spatial merge candidate (1212). The corrected motion vector is used to predict the motion of a subsequently decoded block (e.g., a third block) in other pictures, for example using TMVP (1214) or ATMVP (1216). The corrected motion vector is also used to calculate a boundary strength value for a deblocking filter (1218). Use of uncorrected motion vectors for spatial motion prediction and deblocking

[0096] exist Fig.12 In the example of , different MVs of the DMVR block are used for spatial motion prediction and deblocking filter. On the other hand, unlike the MVs for temporal motion prediction (which are stored in external memory), the MVs for spatial motion prediction and deblocking are usually stored using on-chip memory designed for actual codecs to increase data access speed. Therefore, Fig.12 Some implementations of the method require two different on-chip memories to store the uncorrected and corrected MVs for each DMVR block. This would double the line buffer size used to cache the MVs, which is undesirable for hardware implementations. In order to keep the total on-chip memory size for MV storage the same as that of VTM-1.0, it is proposed in a further embodiment to use the uncorrected MVs of the DMVR blocks for the deblocking process. Fig.13An example of the DMVR process according to this embodiment is shown. Fig.12 In the same way, the modified DMVR MV is used to generate the temporal motion predictor via TMVP and ATMVP in addition to generating the final bidirectional prediction signal. Fig.13 In an embodiment of the present invention, the unmodified MV is used not only to obtain the spatial motion predictor (spatial AMVP and spatial merging) but also to determine the boundary strength of the deblocking filter of the current block.

[0097] exist Fig.13 In the illustrated method, at 1300, an uncorrected motion vector for a first block is identified. The uncorrected motion vector may be signaled for the first block using any of a variety of available MV signaling techniques. At 1302, the uncorrected motion vector is used to generate a bi-directional prediction template. At 1304, motion correction is performed for the L0 motion vector, and at 1306, motion correction is performed for the L1 motion vector. At 1308, a final bi-directional prediction for the first block is generated using the corrected L0 and L1 motion vectors. Fig.13 In the method, the uncorrected motion vector is used to predict the motion of a subsequently decoded block (e.g., a second block) in the same picture as the first block. For example, the uncorrected motion vector is used for spatial AMVP (1310) and as a spatial merge candidate (1312). The uncorrected motion vector is also used to calculate a boundary strength value (1314) for a deblocking filter. The corrected motion vector is used to predict the motion of a subsequently decoded block (e.g., a third block) in other pictures, for example, using TMVP (1316) or ATMVP (1318).

[0098] Figure 11-13 Implementations of can reduce or eliminate the encoding / decoding latency caused by DMVR, since in these implementations there is no dependency of decoding of one block on reconstruction of the modified MV of its neighboring DMVR blocks. Fig.10 In the same example, Fig.14 Shown in the application Figure 11-13 One example of a parallel decoding process is Fig.14 As shown, since the decoding of multiple DMVR blocks can be performed in parallel, there is no decoding latency between adjacent blocks. Accordingly, the total decoding time can be equal to the decoding of one block, which can be expressed as T MCP +T DMVR +T 解量化 +T 逆变换 . A fragment-based approach to DMVR latency reduction

[0099] As mentioned above, one reason for the encoding / decoding latency of DMVR is the dependency between the reconstruction of the modified MV of a DMVR block and the decoding of its neighboring blocks, which is caused by spatial motion prediction (e.g., spatial AMVP and spatial merge mode). Figure 11-13 The method can remove or reduce the decoding latency of DMVR, but this reduced latency comes at the expense of decoding efficiency due to the use of less accurate uncorrected MV for spatial motion prediction. Fig.10 As shown, the worst-case encoding / decoding latency caused by DMVR is directly related to the maximum number of consecutive blocks decoded by the DMVR mode. To address these issues, in some embodiments, a region-based approach is used to reduce the encoding / decoding latency while reducing the decoding loss caused by using uncorrected MVs for spatial motion prediction.

[0100] Specifically, in some embodiments, a picture is divided into multiple non-overlapping segments, and the uncorrected MV of each DMVR block in a segment is used as a predictor to predict the MV of its neighboring blocks in the same segment. However, when a DMVR block is located at the right or bottom boundary of a segment, its uncorrected MV will not be used; instead, the corrected MV of the block is used as a predictor to predict the MV of blocks from neighboring segments for better spatial motion prediction efficiency.

[0101] Fig.15 shows an example of a DMVR process according to one embodiment, and Fig.16 An example is shown in which blank blocks represent DMVR blocks that use uncorrected MVs for spatial motion prediction, spatial merging, and deblocking, and patterned blocks represent DMVR blocks that use corrected MVs for spatial motion prediction, spatial merging, and deblocking. Fig.16 In the example of , encoding / decoding of different inter-blocks within the same segment can be performed independently of each other, while decoding of blocks from different segments is still dependent. For example, since the blocks on the left boundary of segment #2 can use the modified MVs of the neighboring DMVR blocks in segment #1 as spatial MV predictors, their decoding process will not start until the DMVR of these neighboring blocks in segment #1 is fully completed. In addition, as Fig.15 As shown, Fig.13 Similar to the method of , the same MV of a DMVR block is used for spatial motion prediction and deblocking filter to avoid increasing the on-chip memory for storing MVs. In another embodiment, it is proposed to always use the modified MV for the deblocking process.

[0102] exist Fig.15In the illustrated method, at 1502, an uncorrected motion vector for a first block is determined. The uncorrected motion vector has been signaled for the first block using any of a variety of available MV signaling techniques. At 1504, the uncorrected motion vector is used to generate a bidirectional prediction template. At 1506, motion correction is performed for the L0 motion vector, and at 1508, motion correction is performed for the L1 motion vector. At 1510, a final bidirectional prediction for the first block is generated using the corrected L0 and L1 motion vectors.

[0103] At 1512, it is determined whether the first block is located at a right or bottom segment boundary. If the first block is not located at a right or bottom segment boundary, the uncorrected motion vector is used to predict the motion of a subsequently coded block (e.g., a second block) within the same picture as the first block. For example, the uncorrected motion vector is used for spatial AMVP (1514) and used as a spatial merge candidate (1516). The uncorrected motion vector is also used to calculate a boundary strength value for a deblocking filter (1518). On the other hand, if the first block is located at a right or bottom segment boundary, the corrected motion vector is used to predict the motion of a subsequently coded block (e.g., a second block) within the same picture as the first block (e.g., using AMVP 1514 and spatial merge candidate 1516), and the corrected motion vector is also used to calculate a boundary strength value for a deblocking filter (1518). Regardless of the result of the determination at 1512, the corrected motion vector is used to predict the motion of a subsequently coded block (e.g., a third block) in other pictures, for example, using TMVP (1520) or ATMVP (1522).

[0104] exist Fig.16 In an embodiment, the modified MV is enabled only for spatial motion prediction of blocks located at the left / top boundary of a segment within a picture. However, depending on the segment size, the total percentage of blocks to which the modified MV can be applied for spatial motion prediction is relatively small. The result may still be a non-negligible degradation in spatial motion prediction performance. In order to further improve performance, it is proposed in some embodiments to allow the modified MV of a DMVR block within a segment to predict the MV of a neighboring block within the same segment. However, the result is that the decoding of multiple blocks within a segment cannot be performed in parallel. In order to improve encoding / decoding parallelism, in this method, it is also proposed to prohibit the current block from using the MV (unmodified MV or modified MV) of a neighboring block from another segment as a predictor for spatial motion prediction (such as spatial AMVP and spatial merging). Specifically, in this way, if the neighboring block is from a different segment than the current block, it will be regarded as unavailable for spatial motion vector prediction.

[0105] exist Fig.17 Such an implementation is shown in FIG. Fig.17In , a blank block represents a CU that is allowed to use the neighboring MV (if the neighboring block is a DMVR block, the neighboring MV can be a modified MV, otherwise it is an unmodified MV) for spatial motion prediction; a patterned block represents a CU that is prohibited from using the MV of its neighboring blocks from different fragments for spatial motion prediction. Fig.17 The implementation method allows parallel decoding of blocks between slices, but does not allow parallel decoding within a slice.

[0106] In general, DMVR is enabled only for bidirectionally predicted CUs with forward and backward prediction signals. Specifically, DMVR requires the use of two reference pictures: one with a smaller picture order count (POC) and the other with a POC larger than the POC of the current picture. In contrast, a low-delay (LD) picture is predicted from a reference picture that is earlier in the display order than the current picture, and the POCs of all reference pictures in L0 and L1 are smaller than the POC of the current picture. Therefore, DMVR cannot be applied to LD pictures, and the decoding latency caused by DMVR does not exist in LD pictures. Based on this analysis, in some embodiments, when DMVR is applied, it is proposed to apply the above-mentioned DMVR parallel restriction (disabling spatial motion prediction across slice boundaries) only to non-LD pictures. For LD pictures, no restriction is applied, and it is still allowed to predict the MV of the current block based on the MV of the spatial neighbors of the current block from another slice. In a further embodiment, the encoder / decoder determines whether to apply the restriction based on checking the POCs of all reference pictures in L0 and L1 without additional signaling. In another embodiment, it is proposed to add a picture / slice level flag to indicate whether the DMVR parallel restriction is applied to the current picture / slice.

[0107] In some embodiments, the encoder selects the number of fragments within a picture / slice and the position of each fragment and sends it to the decoder with a signal. The signaling can be performed similarly to other parallel tools in HEVC and JEM (e.g., slices, tiles, and wavefront parallel processing (WPP)). Different choices may result in different trade-offs between decoding performance and encoding / decoding parallelism. In one embodiment, it is proposed to set the size of each fragment to be equal to the size of a CTU. In terms of signaling, syntax elements can be added at the sequence and / or picture level. For example, the number of CTUs in each fragment can be signaled in a sequence parameter set (SPS) and / or a picture parameter set (PPS), or can be signaled in a slice header. Other changes in syntax elements can be used, such as the number of CTU rows can be used, or the number of fragments in each picture / slice can be used, as well as other alternatives. Example of motion correction method

[0108] Additional embodiments described herein operate to replace the block matching motion search used to compute DMVR motion corrections. Compared to block matching based methods that perform motion searches in small local windows, example embodiments directly compute motion corrections based on spatial and temporal sample derivatives. Such embodiments reduce computational complexity and can increase motion correction accuracy because the resulting value of the corrected motion is not limited to the search window. Motion correction using block-level BIO

[0109] As mentioned above, BIO was previously used in JEM to provide sample-based motion correction in addition to block-based motion compensation prediction when bidirectionally predicting blocks. Based on the current design, BIO only enhances the motion step prediction samples as the correction result, without updating the MVs stored in the MV buffer and used for spatial and temporal motion prediction and deblocking filters. This means that, contrary to the current DMVR, BIO does not bring any encoding / decoding time between adjacent blocks. However, in the current BIO design, motion correction is obtained in the smallest unit (e.g., 4x4). This may lead to non-negligible computational complexity, especially on the decoder side. This is undesirable for hardware codec implementation. Therefore, in order to solve the latency of DMVR while maintaining acceptable decoding complexity, it is proposed in some embodiments to use block-based BIO to calculate local motion correction of video blocks for DMVR decoding. Specifically, in the proposed embodiment, the BIO core design (e.g., gradient and corrected motion vector calculation) remains the same as the existing design for calculating motion correction. However, in order to reduce complexity, the motion correction amount is obtained based on the CU level, a single value is aggregated for all samples within the CU and used to calculate a single motion correction; and all samples within the current CU will share the same motion correction. Based on the same comments above regarding BIO usage, an example of proposed block-level BIO motion correction follows: where θ is the coordinate of the sample within the current CU, and where Δ(x,y) is the optical flow error metric proposed in Equation 3 above.

[0110] As indicated above, the motivation of BIO is to improve the accuracy of prediction samples based on local gradient information at each sample position within the current block. For large video blocks containing many samples, it is possible that the local gradients at different sample positions may exhibit very varying characteristics. In this case, the above block-based BIO derivation may not be able to provide reliable motion correction for the current block, thereby resulting in decoding performance loss. Based on this consideration, in some embodiments, it is proposed to enable CU-based BIO motion derivation only when the block size of the DMVR block is small (e.g., not greater than a given threshold). Otherwise, CU-based BIO motion derivation is disabled; instead, the existing block matching-based motion correction (using the above-mentioned proposed DMVR latency removal / reduction method applied in some embodiments) will be used to derive local motion correction for the current block. Motion correction using optical flow

[0111] As mentioned above, BIO estimates local motion corrections based on the assumption that the L0 and L1 motion corrections derived at each sampling position are symmetric about the current picture, i.e., as well as in and are the horizontal and vertical motion corrections associated with prediction lists L0 and L1. However, this assumption may not be true for DMVR-decoded blocks. For example, in existing DMVR (e.g. Fig. 8A As shown in FIG. 2 , two separate block-matching-based motion searches are performed for L0 and L1, so that the MVs that minimize the template costs for the L0 and L1 prediction signals can be different. Due to this symmetrical motion restriction, the motion correction derived by BIO may not always be accurate enough to enhance the prediction quality for DMVR (and may even degrade the prediction quality at times).

[0112] In some embodiments, an improved motion derivation method is used to calculate motion correction for DMVR. The classical optical flow model that describes the brightness of an image remains constant over time is expressed as follows, E(x,y,t)=E(x+dx,y+dy,t+dt) (6) where x and y represent spatial coordinates and t represents time. The right side of the equation can be expanded by a Taylor series with respect to (x, y, t). After that, the optical flow equation becomes first order,

[0113] Using the camera capture time as the basic time unit (e.g., setting dt = 1), equation (7) can be discretized by changing the optical flow function from the continuous domain to the discrete domain. Let I(x, y) be the sample value captured from the camera, then equation (7) becomes

[0114] In various embodiments, one or more error metrics may be defined based on the degree to which the left-hand side expression in equation (9) is not equal to zero. Motion correction may be used to substantially minimize the error metric.

[0115] In some embodiments, it is proposed to use discrete optical flow models to estimate local motion corrections in L0 and L1. Specifically, a bidirectional prediction template is generated by averaging the two prediction blocks of the initial L0 and L1 MVs using the merged candidate. However, instead of performing a block matching motion search in the local area, the optical flow model in equation (8) is used in some proposed embodiments to directly derive the corrected MV for each reference list L0 / L1, expressed as follows: Among them, P 0 and P 1 is the prediction signal generated using the original MV for reference lists L0 and L1, respectively, 模板 is a bidirectional prediction template signal; and is the prediction signal P 0 and P 1 The horizontal / vertical gradients of can be calculated based on different gradient filters, such as the Sobel filter or the 2D separable gradient filter used by BIO, as described in: J. Chen, E. Alshina, G. J. Sullivan, J. R. Ohm, J. Boyce, "Algorithm description of joint exploration test model 6", JVET-G1001, July 2017, Turin, Italy. Equation (9) represents a set of equations: One equation for the predicted signal P 0 or P 1 For each sample in , a separate and P 模板 -P k . With two unknown parameters Δx k and Δy k , the problem of multi-factor decision can be solved by minimizing the sum of squared differences of equation 9, as follows in is the time difference between the L0 / L1 prediction signal and the bidirectional prediction template signal, and θ is the coordinate set within the decoding block. By solving the linear minimum mean square error (LLMSE) problem in equation (10), we can obtain (Δx, Δy)* k The analytical formula is

[0116] Based on equation (11), in some embodiments, in order to improve the accuracy of the derived MV, such a method can recursively select the motion correction (ie, (Δx, Δy) * k ). Such an implementation may operate in the following manner: using the original L0 and L1 MVs of the current block to generate an initial bidirectional prediction template signal and calculating the corresponding incremental motion (Δx, Δy) based on equation (11) * k ; The modified MV is then used as the motion for generating new L0 and L1 prediction samples and bidirectional prediction template samples, which are then used to update the local correction (Δx, Δy) * k The process can be repeated until the MV is not updated or the maximum number of iterations is reached. An example of such a process is summarized as follows, Fig.18 shown.

[0117] At 1802, a counter l is initialized to l=0. At 1804, the original MV of the block is used and Generate initial L0 and L1 prediction signals and and the initial bidirectional prediction template signal At 1806 and 1808, local L0 and L1 motion corrections based on equation (11) and and the MV of the block is updated to and

[0118] if and is zero (determined at 1810) or if l=l max (determined at 1812), then at 1814, the final bidirectional prediction is generated using the modified motion vector. Otherwise, at 1816, counter l is incremented and the process iterates, using MV and Update (at 1806, 1808) the L0 and L1 prediction signals and And the bidirectional prediction template signal

[0119] Fig.18An example of a DMVR process for calculating motion corrections for DMVR blocks using an example optical flow-based motion derivation method is shown. Fig.18 As shown, the optical MV of a DMVR block is identified by iteratively modifying the original MV based on the optical flow model. Although such a method can provide good motion estimation accuracy, it also brings a lot of increased complexity. In order to reduce the complexity of derivation, in one embodiment of the disclosure, it is proposed to apply only one iteration to derive motion correction using the proposed motion derivation method, for example, only apply the process shown in 1804 to 1808 to derive the modified MV of the DMVR block.

[0120] Due to the high consistency between sampling characteristics within small blocks, the optical flow based motion derivation model is more efficient for small CUs than for large CUs. In some embodiments, it is proposed to implement optical flow based motion derivation for DMVR blocks when the block size of the DMVR block is small (e.g., not larger than a given threshold). Otherwise, the existing block matching based motion correction will be used to derive the local motion correction for the current block (e.g., together with the proposed DMVR latency removal / reduction method described herein).

[0121] Note that the various hardware elements of one or more embodiments described are referred to as "modules", which implement (e.g., run, execute, etc.) the various functions described in this application associated with the respective modules. The modules used here include hardware that a person skilled in the art considers suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more storage devices). Each described module may also include executable instructions for performing one or more functions described when the respective module is executed, and note that these instructions may take the form of or include hardware (i.e., hard-wired) instructions, firmware instructions, and / or software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium or media, such as generally referred to as RAM, ROM, etc.

[0122] Although the above features and elements are described in specific combinations, it is understood by those skilled in the art that each feature or element can be used alone or in any combination with other features and elements. In addition, the methods described in the present application can be implemented in a computer program, software and / or firmware incorporated in a computer-readable medium to be executed by a computer and / or a processor. Examples of computer-readable media include, but are not limited to, electronic signals (transmitted via wired and / or wireless connections) and / or computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memories (ROMs), random access memories (RAMs), registers, buffer memories, semiconductor memory devices, magnetic media (for example, but not limited to, internal hard disks and removable disks), magneto-optical media, and / or optical media (for example, CD-ROM disks and / or digital versatile disks (DVDs)). A processor associated with the software can be used to implement a radio frequency transceiver for a WTRU, UE, base station, RNC, and / or any host computer.

Claims

1. A video decoding method, include: identifying at least one uncorrected motion vector associated with a first block in a picture; modifying the at least one uncorrected motion vector to produce at least one modified motion vector associated with the first block; For a second block in the picture that is adjacent to the first block, determining whether the second block is in a different segment of the picture than the first block; predicting motion information of the second block, wherein the prediction is based on the at least one uncorrected motion vector or a determination of the at least one corrected motion vector is based on the determination of whether the second block is in a different segment of the picture than the first block; as well as Based on the predicted motion information, the second block is predicted.

2. The method according to claim 1, further comprising: include: Based on the at least one modified motion vector, the first block is predicted. 3 . The method of claim 1 , wherein in response to determining that the second block is in a different segment of the picture than the first block, using the at least one revised motion vector to predict the motion information of the second block. 4 . The method of claim 1 , wherein in response to determining that the second block is in a same fragment of the picture as the first block, using the at least one uncorrected motion vector to predict the motion information of the second block.

5. The method of claim 1, wherein the modifying is performed using decoder side motion vector modification (DMVR).

6. A video decoding device comprising one or more processors configured to at least perform the following operations: identifying at least one uncorrected motion vector associated with a first block in a picture; modifying the at least one uncorrected motion vector to produce at least one modified motion vector associated with the first block; For a second block in the picture that is adjacent to the first block, determining whether the second block is in a different segment of the picture than the first block; predicting motion information of the second block, wherein the prediction is based on the at least one uncorrected motion vector or a determination of the at least one corrected motion vector is based on the determination of whether the second block is in a different segment of the picture than the first block; as well as Based on the predicted motion information, the second block is predicted.

7. The device of claim 6, further configured to predict the first block based on the at least one revised motion vector. 8 . The apparatus of claim 6 , wherein in response to determining that the second block is in a different segment of the picture than the first block, the motion information of the second block is predicted using the at least one revised motion vector.

9. The apparatus of claim 6, wherein in response to determining that the second block is in a same fragment of the picture as the first block, the motion information of the second block is predicted using the at least one uncorrected motion vector.

10. The apparatus of claim 6, wherein the second block is predicted by using bi-directional prediction.

11. A video encoding method, include: identifying at least one uncorrected motion vector associated with a first block in a picture; modifying the at least one uncorrected motion vector to produce at least one modified motion vector associated with the first block; For a second block in the picture that is adjacent to the first block, determining whether the second block is in a different segment of the picture than the first block; predicting motion information of the second block, wherein the prediction is based on the at least one uncorrected motion vector or a determination of the at least one corrected motion vector is based on the determination of whether the second block is in a different segment of the picture than the first block; as well as Based on the predicted motion information, the second block is predicted.

12. The method according to claim 11, further comprising: include: Based on the at least one modified motion vector, the first block is predicted. 13 . The method of claim 11 , wherein in response to determining that the second block is in a different segment of the picture than the first block, predicting the motion information of the second block using the at least one revised motion vector.

14. The method of claim 11, wherein in response to determining that the second block is in a same fragment of the picture as the first block, predicting the motion information of the second block using the at least one uncorrected motion vector.

15. The method according to claim 11, further comprising: include: Information indicating the arrangement of slices in the picture is signaled in the bitstream.

16. A video encoding apparatus comprising one or more processors configured to at least perform the following operations: identifying at least one uncorrected motion vector associated with a first block in a picture; modifying the at least one uncorrected motion vector to produce at least one modified motion vector associated with the first block; For a second block in the picture that is adjacent to the first block, determining whether the second block is in a different segment of the picture than the first block; predicting motion information of the second block, wherein the prediction is based on the at least one uncorrected motion vector or a determination of the at least one corrected motion vector is based on the determination of whether the second block is in a different segment of the picture than the first block; as well as Based on the predicted motion information, the second block is predicted.

17. The device of claim 16, further configured to predict the first block based on the at least one revised motion vector.

18. The apparatus of claim 16, wherein in response to determining that the second block is in a different segment of the picture than the first block, the motion information of the second block is predicted using the at least one revised motion vector.

19. The device of claim 16, wherein in response to determining that the second block is in a same fragment of the picture as the first block, the motion information of the second block is predicted using the at least one uncorrected motion vector.

20. The device according to claim 16, further comprising: include: An index associated with the second block is signaled in a bitstream, the index indicating the motion information for predicting the second block based on the first block.

Images