Complexity reduction and bit-width control for bidirectional optical flow
By employing simplified filters and reduced bit width parameters, the method addresses high complexity and bit width issues in bidirectional optical flow, enhancing the efficiency and cost-effectiveness of video coding systems.
Patent Information
- Application Number
- JP2024153291
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-01-07
- Filing Date
- 2024-09-05
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2039-09-17
AI Technical Summary
Existing video coding systems, particularly those using bidirectional optical flow (BIO) in advanced video coding standards like VVC, face challenges with high computational complexity and large bit widths, which increase implementation costs and resource demands.
Implement a method to reduce computational complexity and bit width by using simplified interpolation filters, reduced bit width parameters, and selective application of BIO for certain coding units, along with gradient and sample padding techniques to minimize the number of expanded samples.
The proposed method reduces the computational complexity and bit width requirements of BIO, making it more efficient and cost-effective for hardware implementation while maintaining coding efficiency.
Smart Images

Figure 2024166257000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application claiming the benefit under 35 U.S.C. §119(e) of U.S. Provisional Application No. 62 / 734,763 (filed September 21, 2018), U.S. Provisional Application No. 62 / 738,655 (filed September 28, 2018), and U.S. Provisional Application No. 62 / 789,331 (filed January 7, 2019), all of which are entitled “Complexity Reduction and Bit-Width Control for Bi-Directional Optical Flow,” and are hereby incorporated by reference in their entireties. [Background technology]
[0002] Video coding systems are widely used to compress digital video signals to reduce the storage requirements and / or transmission bandwidth of such signals. Among various types of video coding systems, such as block-based, wavelet-based, and object-based systems, block-based hybrid video coding systems are currently the most widely used and deployed. Examples of block-based video coding systems include international video coding standards such as MPEG1 / 2 / 4 Part 2, H.264 / MPEG-4 Part 10 AVC, VC-1, and High Efficiency Video Coding (HEVC) developed by ITU-T / SG16 / Q.6 / VCEG and ISO / IEC / MPEG's Joint Collaborative Team on Video Coding (JCT-VC).
[0003] The first version of the HEVC standard was finalized in October 2013, which provides approximately 50% bitrate savings or equivalent perceptual quality compared to the previous generation video coding standard H.264 / MPEG AVC. While the HEVC standard provides significant coding improvements over its predecessors, there is evidence that better coding efficiency for HEVC can be achieved with additional coding tools. Based on this, both VCEG and MPEG have started exploration work on new coding techniques for future video coding standardization. In October 2015, the Joint Video Exploration Team (JVET) was formed by ITU-T VECG and ISO / IEC MPEG to initiate significant research into advanced techniques that may enable substantial improvements in coding efficiency. By integrating several additional coding tools on the HEVC test model (HM), a reference software called the joint exploration model (JEM) was maintained by JVET.
[0004] In October 2017, Non-Patent Document 1 was published by ITU-T and ISO / IEC. In April 2018, 23 CfP responses were received and evaluated at the 10th JVET meeting, which showed a compression efficiency improvement of about 40% over HEVC. Based on such evaluation results, JVET launched a new project to develop a new generation video coding standard named Versatile Video Coding (VVC). In the same month, one reference software code base called VVC Test Model (VTM) was established to show the reference implementation of the VVC standard. Meanwhile, another reference software base called Benchmark Set (BMS) was also generated to facilitate the evaluation of new coding tools. In the BMS code base, a list of additional coding tools that provide higher coding efficiency and moderate implementation complexity is included on the VTM and used as a benchmark in evaluating similar coding technologies during the VVC standardization process. Specifically, there are five JEM coding tools integrated into BMS-2.0, including 4x4 non-separable secondary transform (NSST), generalized bi-prediction (GBi), bi-directional optical flow (BIO), decoder-side motion vector refinement (DMVR), and current picture referencing (CPR). [Prior art documents] [Non-patent literature]
[0005] [Non-Patent Document 1] joint call for proposals (CfP) on video compression with capability beyond HEVC, Oct. 2017 [Non-Patent Document 2] S. Jeong et al., “CE4 Ultimate motion vector expression (Test 4.5.4)”, JVET-L0054, Oct. 2018 [Non-Patent Document 3] S. Esenlik et al., “Simplified DMVR for inclusion in VVC”, JVET-L0670, Oct. 2018 [Non-Patent Document 4] M.-S. Chiang et al., “CE10.1.1: Multi-hypothesis prediction for improving AMVP mode, skip or merge mode, and intra mode”, JVET-L0100, Oct. 2018 [Non-Patent Document 5] M. Winken et al., “CE10-related: Multi-Hypothesis Inter Prediction with simplified AMVP process”, JVET-L0679, Oct. 2018 Summary of the Invention
[0006]
[0004] Embodiments described herein include methods for use in video encoding and decoding (collectively "coding"). In some embodiments, a video coding method is provided that includes, for at least one current block in a video coded using bidirectional optical flow, calculating a first prediction signal array I from a first reference picture. (0) First horizontal gradient array based on (i,j)
[0007]
number
[0008] and calculating a second prediction signal array I from a second reference picture. (1)A second horizontal gradient array based on (i,j)
[0009]
number
[0010] and performing a right bit shift on the sum of (i) the first horizontal gradient array and (ii) the second horizontal gradient array to generate a reduced-bit-width horizontal intermediate parameter array, Ψ x (i,j), and calculating at least a horizontal motion refinement v based at least in part on the reduced bitwidth horizontal intermediate parameter array. x and calculating at least the horizontal motion refinement v x and generating a prediction of the current block using bidirectional optical flow using
[0011] In some embodiments, the method includes: (0) (i,j) and the second predicted signal array I (1) calculating a signal difference parameter array θ(i,j) by a method comprising: calculating the difference between (i) the signal difference parameter array θ(i,j) and (ii) the horizontal gradient intermediate parameter array Ψ x and calculating a signal horizontal gradient correlation parameter S3 by summing the components of the element-wise multiplication with (i,j), x The step of calculating the horizontal motion refinement v x The method includes the step of obtaining:
[0012] In such an embodiment, the step of calculating the signal difference parameter array θ(i,j) includes: (0) (i,j) and the second predicted signal array I (1)Before calculating the difference with (i,j), the first predicted signal array I (0) (i,j) and the second predicted signal array I (1) (i,j) and performing a right bit shift on each of them.
[0013] In some embodiments, the method includes: (0) First vertical gradient array based on (i,j)
[0014]
number
[0015] and calculating a second prediction signal array I from a second reference picture. (1) A second vertical gradient array based on (i,j)
[0016]
number
[0017] and performing a right bit shift on a sum of (i) the first vertical gradient array and (ii) the second vertical gradient array to generate a reduced bit width vertical intermediate parameter array Ψ y (i,j) and the reduced bitwidth horizontal intermediate parameter array Ψ x (i,j) and the reduced bit-width vertical intermediate parameter array Ψ y Based at least in part on (i,j), a vertical motion refinement v y and calculating the horizontal motion refinement v x and vertical motion refinement v y It is generated using
[0018] Some such embodiments include: (i) a horizontal intermediate parameter array Ψ x (i,j) and (ii) the vertical intermediate parameter array Ψ y(i,j) ∈ ... y The steps of computing the horizontal motion refinement v x and (ii) a cross-gradient correlation parameter S2.
[0019] In some such embodiments, (i) horizontal motion refinement v x and (ii) the step of determining the product of the cross-gradient correlation parameter S2 is performed by dividing the cross-gradient correlation parameter S2 by the most significant bit MSB parameter portion S 2,m and the least significant bit (LSB) of the parameter part S 2,s and (i) horizontal motion refinement v x and (ii) the MSB parameter part S 2,m (i) determining a horizontal motion refinement v x and (ii) the LSB parameter part S 2,S the step of determining an LSB product of the first input and the second input, performing a left bit shift of the MSB product to generate a bit-shifted MSB product, and adding the LSB product and the bit-shifted MSB product.
[0020] In some embodiments, generating a prediction for the current block using bidirectional optical flow comprises, for each sample in the current block:
[0021]
number
[0022] (v) Horizontal movement refinement x , and (vi) vertical motion refinement v y and for each sample in the current block, calculating a bidirectional optical flow sample offset b based on at least the first predicted signal array I (0) (i,j), the second predicted signal array I (1)(i,j), and the sum of the bidirectional optical flow sample offsets b.
[0023] In some embodiments, a gradient array
[0024]
number
[0025] The step of calculating each of the predicted signal array I (0) (i,j), I (1) Pad the samples outside (i,j) with their nearest boundary samples inside the predicted signal array.
[0026] In some embodiments, the step of calculating at least some values of the signal difference parameter array θ(i,j) comprises calculating a predicted signal array I (0) (i,j), I (1) padding the samples outside (i,j) with their respective nearest boundary samples inside the predicted signal array. In some embodiments, the horizontal intermediate parameter array Ψ x The step of calculating at least some of the values of (i,j) includes the step of:
[0027]
number
[0028] with their respective nearest boundary samples inside the horizontal gradient array.
[0029] In some embodiments, the vertical intermediate parameter array Ψ y The step of calculating at least some of the values of (i,j) includes the step of:
[0030]
number
[0031] with their respective nearest boundary samples inside the vertical gradient array.
[0032] In some embodiments, the signal horizontal gradient correlation parameter S3 and the cross gradient correlation parameter S2 are calculated for each sub-block in the current block.
[0033] The embodiments described herein may be performed by an encoder or by a decoder to generate a prediction of a video block.
[0034] In some embodiments, for at least one current block in a video coded using bidirectional optical flow, a first gradient component (e.g., ∂I (0) / ∂x or ∂I (0) A second gradient component (e.g., ∂I (1) / ∂x or ∂I (1) The first gradient component and the second gradient component are summed and a downward bit shift is performed on the resulting sum to obtain a reduced bit-width correlation parameter (e.g., Ψ x or Ψ y ) is generated. A BIO motion refinement is calculated based at least in part on the reduced bitwidth correlation parameters. The calculated motion refinement is used to predict blocks using bidirectional optical flow.
[0035] In some embodiments, for at least one current block in a video coded using bidirectional optical flow, a second prediction signal based on a second reference picture (e.g., I (1) ) to obtain a first prediction signal based on a first reference picture (e.g., I (0)A reduced bit-width correlation parameter (e.g., θ) is generated by subtracting θ from θ and performing a downward bit shift of the resulting difference. A BIO motion refinement is calculated based at least in part on the reduced bit-width correlation parameter. The calculated motion refinement is used to predict blocks using bidirectional optical flow.
[0036] In some embodiments, for at least one current block in a video coded using bidirectional optical flow, a first prediction (e.g., I (0) A first predicted signal of a reduced bit width is generated by performing a downward bit shift on the first predicted signal. (1) A reduced bit-width second prediction signal is generated by performing a downward bit shift on the reduced bit-width first prediction signal. A reduced bit-width correlation parameter (e.g., θ) is generated by subtracting the reduced bit-width first prediction signal from the reduced bit-width second prediction signal. A BIO motion refinement is calculated based at least in part on the reduced bit-width correlation parameter, and the calculated motion refinement is used to predict the block using bidirectional optical flow.
[0037] In some embodiments, for at least one current block in a video coded using bidirectional optical flow, a first gradient component of a reduced bit width is calculated based on a first prediction signal of a reduced bit width from a first reference picture. A second gradient component of a reduced bit width is calculated based on a second prediction signal of a reduced bit width from a second reference picture. The first reduced bit width gradient component and the second reduced bit width gradient component are summed to generate a reduced bit width correlation parameter. A motion refinement is calculated based at least in part on the reduced bit width correlation parameter, and the block is predicted using bidirectional optical flow using the calculated motion refinement.
[0038] In some embodiments, for at least one current block in a video coded using bidirectional optical flow, a first motion-compensated prediction signal and a second motion-compensated prediction signal for samples in the current block are generated, and the first motion-compensated prediction signal and the second motion-compensated prediction signal for samples in the current block are generated using a first interpolation filter having a first number of taps. The first motion-compensated prediction signal and the second motion-compensated prediction signal are also generated for samples in an extended region around the current block, and the first motion-compensated prediction signal and the second motion-compensated prediction signal for samples outside the current block are generated using a second interpolation filter having a second number of taps that is less than the first number of taps. A motion refinement is calculated based at least in part on the first motion-compensated prediction signal and the second motion-compensated prediction signal, and the block is predicted using bidirectional optical flow using the calculated motion refinement.
[0039] In some embodiments, for at least one current block in a video coded using bidirectional optical flow, a first motion compensated prediction signal and a second motion compensated prediction signal are generated, where the first motion compensated prediction signal and the second motion compensated prediction signal for samples in the current block are generated using a horizontal interpolation filter having a first number of taps and a vertical interpolation filter having a second number of taps that is less than the first number of taps. A motion refinement is calculated based at least in part on the first motion compensated prediction signal and the second motion compensated prediction signal, and the block is predicted using the bidirectional optical flow using the calculated motion refinement.
[0040] In some embodiments, a first motion compensated prediction signal and a second motion compensated prediction signal are generated for at least one current block in a video coded using bidirectional optical flow. The first motion compensated prediction signal and the second motion compensated prediction signal for samples in the current block are generated using a horizontal interpolation filter having a first number of taps and a vertical interpolation filter having a second number of taps. The horizontal and vertical filters are applied in a predetermined order, with a filter applied earlier in the order having a greater number of taps than a filter applied later in the order. A motion refinement is calculated based at least in part on the first motion compensated prediction signal and the second motion compensated prediction signal, and the block is predicted using the bidirectional optical flow using the calculated motion refinement.
[0041] In some embodiments, a method for coding a video including a plurality of coding units is provided. For the plurality of coding units in a video coded using bi-prediction, bi-directional optical flow is disabled for at least coding units having a height equal to or less than a threshold height (e.g., BIO may be disabled for coding units of height 4). For bi-predicted coding units with bi-directional optical flow disabled, bi-prediction without bi-directional optical flow is performed. For bi-predicted coding units where bi-directional optical flow is not disabled (e.g., for at least one of the bi-predicted coding units where bi-directional optical flow is not disabled), bi-prediction with bi-directional optical flow is performed.
[0042] In some embodiments, for at least one current block in a video coded using bidirectional optical flow, a first motion compensation prediction signal and a second motion compensation prediction signal for samples in the current block are generated. The first and second values relate to samples in an extended region around the current block, the extended region not including samples more than one row or column away from the current block. A motion refinement is calculated based at least in part on the first and second motion compensation prediction signals and the first and second values for samples in the extended region. A block is predicted using bidirectional optical flow using the calculated motion refinement.
[0043] In some embodiments, a method for coding a video including a plurality of coding units is provided. For a plurality of coding units in a video coded using bi-prediction, bi-directional optical flow is disabled for at least coding units predicted using a sub-block-level inter prediction technique (e.g., advanced temporal motion vector prediction and affine prediction, etc.). For bi-predicted coding units for which bi-directional optical flow is disabled, bi-prediction without bi-directional optical flow is performed. For bi-predicted coding units for which bi-directional optical flow is not disabled (e.g., for at least one of the bi-predicted coding units for which bi-directional optical flow is not disabled), bi-prediction with bi-directional optical flow is performed.
[0044] In some embodiments, for at least one current block in a video coded using bidirectional optical flow, a first motion compensated prediction signal and a second motion compensated prediction signal are generated for samples in the current block. The first motion compensated prediction signal and the second motion compensated prediction signal for samples in the current block are generated using a horizontal interpolation filter having a first number of taps and a vertical interpolation filter having a first number of taps. The first motion compensated prediction signal and the second motion compensated prediction signal are also generated for samples in an extended region around the current block, and the first motion compensated prediction signal and the second motion compensated prediction signal for samples outside the current block are generated using a horizontal interpolation filter having a first number of taps and a vertical interpolation filter having a second number of taps that is less than the first number of taps. A motion refinement is calculated based at least in part on the first motion compensated prediction signal and the second motion compensated prediction signal. A block is predicted using bidirectional optical flow using the calculated motion refinement.
[0045] In additional embodiments, encoder and decoder systems are provided for performing the methods described herein. The encoder or decoder systems may include a processor and a non-transitory computer-readable medium that stores instructions for performing the methods described herein. Additional embodiments may include a non-transitory computer-readable storage medium that stores video encoded using the methods described herein. [Brief description of the drawings]
[0046] [Figure 1A] FIG. 1 is a system diagram illustrating an example communication system in which one or more disclosed embodiments may be implemented. [Figure 1B] 1A is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that may be used within the communications system illustrated in FIG. 1A, according to an embodiment. [Figure 2A]FIG. 1 is a functional block diagram of a block-based video encoder, such as the encoder used for VVC. [Figure 2B] FIG. 2 is a functional block diagram of a block-based video decoder, such as the decoder used in VVC. [Figure 3A] FIG. 13 is a diagram showing four-partitioning, which is block partitioning in a multi-type tree structure. [Figure 3B] FIG. 13 is a diagram showing vertical bipartitioning, which is block partitioning in a multi-type tree structure. [Figure 3C] FIG. 13 is a diagram showing horizontal bipartitioning, which is block partitioning in a multi-type tree structure. [Figure 3D] FIG. 13 is a diagram showing vertical 3-partitioning, which is block partitioning in a multi-type tree structure. [Figure 3E] FIG. 13 is a diagram showing horizontal 3-partitioning, which is block partitioning in a multi-type tree structure. [Figure 4] Schematic diagram of prediction using Bidirectional Optical Flow (BIO). [Diagram 5] FIG. 1 illustrates a method for generating enhanced samples for a BIO using a simplified filter according to some embodiments. [Figure 6] FIG. 1 illustrates a method for generating enhanced samples for a BIO using a simplified filter according to some embodiments. [Figure 7] A diagram illustrating sample and gradient padding to reduce the number of interpolated samples in an extended region of one BIO coding unit (CU) in accordance with some embodiments. [Figure 8] FIG. 2 illustrates an example of a coded bitstream structure. [Figure 9] FIG. 1 illustrates an exemplary communication system. [Figure 10] FIG. 1 illustrates the use of integer samples as extended samples for BIO derivation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0047] Exemplary Network for Implementation of the Embodiments 1A illustrates an example communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcasts, etc., to multiple wireless users. The communication system 100 enables the multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication system 100 may utilize one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tailed Unique Word DFT Spread OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multi-Carrier (FBMC), etc.
[0048] 1A, the communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104 / 113, a CN 106 / 115, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, although it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, the WTRUs 102a, 102b, 102c, 102d may all be referred to as “stations” and / or “STAs” and may be configured to transmit and / or receive wireless signals, and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, IoT devices, watches or other wearables, head mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain contexts), consumer electronics, devices operating on commercial and / or industrial wireless networks, etc. The WTRUs 102a, 102b, 102c, and 102d may all be referred to interchangeably as UEs.
[0049] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106 / 115, the Internet 110, and / or the network 112. For example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNode B, a home Node B, a home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, etc. Although the base stations 114a, 114b are each shown as a single element, it will be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.
[0050] The base station 114a may be part of the RAN 104 / 113, which may include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for wireless services for a particular geographic area, which may be relatively fixed or may change over time. A cell may be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, one for each sector of the cell. In an embodiment, the base station 114a may utilize MIMO technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.
[0051] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).
[0052] More specifically, as noted above, the communication system 100 may be a multiple access system and may utilize one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, and SC-FDMA. For example, the base stations 114a and the WTRUs 102a, 102b, 102c in the RAN 104 / 113 may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115 / 116 / 117 using Wideband CDMA (WCDMA). WCDMA may include communication protocols such as High Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed Downlink (DL) Packet Access (HSDPA) and / or High Speed UL Packet Access (HSUPA).
[0053] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE Advanced (LTE-A) and / or LTE Advanced Pro (LTE-A Pro).
[0054] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as New Radio (NR) radio access that may establish the air interface 116 using NR.
[0055] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement LTE and NR radio access together, e.g., using a dual connectivity (DC) principle. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).
[0056] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a wireless technology such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), and GSM EDGE (GERAN).
[0057] The base station 114b in FIG. 1A may be, for example, a wireless router, a home Node B, a home eNode B, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a local area, such as a workplace, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., used by drones), a roadway, etc. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or femtocell. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the CN 106 / 115.
[0058] The RAN 104 / 113 may communicate with the CN 106 / 115, which may be any type of network configured to provide voice, data, application, and / or VoIP services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have various quality of service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, and mobility requirements. The CN 106 / 115 may provide call control, billing services, mobile location services, prepaid calling, Internet connectivity, video distribution, and / or perform high level security functions, such as user authentication. Although not shown in FIG. 1A, it will be understood that the RAN 104 / 113 and / or the CN 106 / 115 may communicate directly or indirectly with other RANs that utilize the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which may utilize NR radio technology, the CN 106 / 115 may also communicate with another RAN (not shown) that utilizes GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.
[0059] The CN 106 / 115 may also act as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as TCP, UDP, and / or IP in the TCP / IP Internet protocol suite. The networks 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs that may utilize the same RAT as the RAN 104 / 113 or a different RAT.
[0060] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with a base station 114a that can employ a cellular-based wireless technology and a base station 114b that can employ an IEEE 802 wireless technology.
[0061] 1B is a system diagram of an example WTRU 102. As shown in FIG. 1B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a GPS chipset 136, and / or other peripherals 138, etc. It will be understood that the WTRU 102 may include any sub-combination of the above elements while remaining consistent with an embodiment.
[0062] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, an ASIC, an FPGA circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although FIG. 1B depicts the processor 118 and the transceiver 120 as separate components, it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.
[0063] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It will be appreciated that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.
[0064] 1B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may utilize MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.
[0065] The transceiver 120 may be configured to modulate signals transmitted by the transmit / receive element 122 and demodulate signals received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as, for example, NR and IEEE 802.11.
[0066] The processor 118 of the WTRU 102 may be coupled to and receive user input data from the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. The processor 118 may access information in and store data in any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include RAM, ROM, a hard disk, or any other type of memory storage device. The removable memory 132 may include a SIM card, a memory stick, an SD memory card, and the like. In other embodiments, the processor 118 may access information in and store data in memory that is not physically located on the WTRU 102, such as on a server or home computer (not shown).
[0067] The processor 118 may be configured to receive power from the power source 134 and to distribute and / or control the power to other components within the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.
[0068] The processor 118 may be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding a current location of the WTRU 102. In addition to or in place of information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may obtain location information by way of any suitable location determination method while remaining consistent with an embodiment.
[0069] The processor 118 may be further coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a USB port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.
[0070] The WTRU 102 may include a full-duplex radio, where transmission and reception of some or all of the signals (e.g., associated with a particular subframe for both the UL (e.g., for transmission) and the downlink (e.g., for reception)) may be parallel and / or simultaneous. The full-duplex radio may include an interference management unit to reduce or substantially eliminate self-interference via signal processing by hardware (e.g., a choke) or a processor (e.g., a separate processor (not shown) or the processor 118). In an embodiment, the WTRU 102 may include a half-duplex radio for transmission and reception of some or all of the signals (e.g., associated with a particular subframe for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).
[0071] Although the WTRU is described in Figures 1A-1B as a wireless terminal, it is contemplated that in certain representative embodiments such a terminal may use a wired communications interface (e.g., temporarily or permanently) with the communications network.
[0072] In an exemplary embodiment, the other network 112 may be a WLAN.
[0073] 1A-1B and corresponding description, one or more or all of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more or all of the functions described herein. For example, an emulation device may be used to test other devices and / or to simulate network and / or WTRU functions.
[0074] The emulation device may be designed to implement one or more tests of other devices in a lab environment and / or an operator network environment. For example, the one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices in the communication network. The one or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for testing and / or may perform testing using wireless wireless communication.
[0075] The emulation device or devices may perform one or more functions, inclusive, without being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation device may be utilized in a test scenario in a test lab and / or in an undeployed (e.g., test) wired and / or wireless communication network to implement testing of one or more components. The emulation device or devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (which may include, for example, one or more antennas) may be used to transmit and / or receive data by the emulation device.
[0076] Detailed Description Block-Based Video Coding Like HEVC, VVC is built on a block-based hybrid video coding framework. Figure 2A shows a block diagram of a block-based hybrid video coding system. The input video signal 103 is processed block by block (called coding unit (CU)). In VTM-1.0, a CU can be up to 128x128 pixels. However, unlike HEVC, which partitions blocks based on quadtrees only, in VTM-1.0, coding tree units (CTUs) are partitioned into CUs based on quadtrees / binary trees / ternary trees to adapt to various local characteristics. Furthermore, the concept of multiple partition unit types in HEVC has been removed, and the separation of CUs, prediction units (PUs), and transform units (TUs) no longer exists in VVC. Instead, each CU is always used as a basic unit for both prediction and transformation without further partitioning. In the multi-type tree structure, one CTU is first partitioned by a quadtree structure. Each quadtree leaf node can be further partitioned with binary and ternary tree structures. As shown in FIGS. 3A to 3E, there are five division types: four-partition, two-horizontal partition, two-vertical partition, three-horizontal partition, and three-vertical partition.
[0077] As shown in FIG. 2A, spatial prediction (161) and / or temporal prediction (163) may be performed. Spatial prediction (or "intra prediction") predicts the current video block using pixels from samples (called reference samples) of already coded neighboring blocks in the same video picture / slice. Spatial prediction reduces spatial redundancy inherent in video signals. Temporal prediction (also called "inter prediction" or "motion compensated prediction") predicts the current video block using pixels reconstructed from already coded video pictures. Temporal prediction reduces temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal references. If multiple reference pictures are supported, a reference picture index is additionally transmitted and is used to identify which reference picture in the reference picture store (165) the temporal prediction signal comes from. After spatial prediction and / or temporal prediction, a mode decision block (181) in the encoder selects the best prediction mode, for example based on a rate-distortion optimization method. The prediction block is then subtracted from the current video block (117), and the prediction residual is decorrelated using a transform (105) and quantization (107). The quantized residual coefficients are inverse quantized (111) and inverse transformed (113) to form a reconstructed residual, which is fed back to the prediction block (127) to form a reconstructed signal for the CU. In-loop filtering, such as a deblocking filter, sample adaptive offset (SAO), and adaptive in-loop filter (ALF), may be applied to the reconstructed CU before it is put into the reference picture store (165) and used to code future video blocks (167). To form the output video bitstream 121, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy coding unit (109) for further compression and packing to form the bitstream.
[0078] FIG. 2B shows a functional block diagram of a block-based video decoder. The video bitstream 202 is first unpacked and entropy decoded in an entropy decoding unit 208. The coding mode and prediction information are sent to either a spatial prediction unit 260 (if intra-coded) or a temporal prediction unit 262 (if inter-coded) to form a prediction block. The residual transform coefficients are sent to an inverse quantization unit 210 and an inverse transform unit 212 to reconstruct a residual block. The prediction block and the residual block are added together at 226. The reconstructed block may further pass through in-loop filtering before being stored in a reference picture store 264. The reconstructed video in the reference picture store is sent to drive a display device and is also used to predict future video blocks.
[0079] As mentioned above, BMS-2.0 follows the same VTM-2.0 encoding / decoding workflow as shown in Figures 2A and 2B. However, some coding modules, especially those associated with temporal prediction, are further enhanced to improve coding efficiency. This disclosure is directed to reducing the computational complexity and solving the large bit-width problem associated with the existing BIO tool of BMS-2.0. In the following, the main design aspects of the BIO tool are introduced, and then a more detailed analysis of the computational complexity and bit-width of the existing BIO implementation is provided.
[0080] Bi-predictive prediction based on optical flow model Traditional bi-prediction in video coding is a simple combination of two temporal prediction blocks obtained from already reconstructed reference pictures. However, due to the constraints of block-based motion compensation (MC), there may be residual small motion observable between the samples of the two prediction blocks, which may reduce the efficiency of the motion-compensated prediction. To solve this problem, bi-directional optical flow (BIO) is applied in BMS-2.0 to reduce the impact of such motion on all samples inside a block. Specifically, BIO is a sample-by-sample motion refinement that is performed in addition to block-based motion-compensated prediction when bi-prediction is used. In the current BIO design, the derivation of refined motion vectors for each sample in one block is based on the classical optical flow model. I (k) Let (x,y) be the sample value at coordinates (x,y) of the prediction block obtained from reference picture list k (k=0,1). (k) (x,y) / ∂x and ∂I (k) (x,y) / ∂y are the horizontal and vertical gradients of the samples. Given an optical flow model, the motion refinement (v x ,v y )teeth,
[0081]
number
[0082] It can be derived by:
[0083] In Figure 4, (MV x0 ,MV y0 ) and (MV x1 ,MV y1 ) in Block I (0) and I (1) We further denote the block-level motion vectors used to generate the motion refinement (v x ,v y )teeth,
[0084]
number
[0085] As shown in Fig. 4, the difference Δ between the values of the samples after motion refinement compensation (A and B in Fig. 4) is calculated by minimizing the difference Δ between the values of the samples after motion refinement compensation.
[0086] Furthermore, to ensure the regularity of the derived motion refinement, the motion refinement is assumed to be consistent with respect to samples within one small unit (i.e., a 4 × 4 block). In BMS-2.0, (v x ,v y ) is
[0087]
number
[0088] The Δ is derived by minimizing Δ in a 6×6 window Ω around each 4×4 block, as in
[0089] To solve the optimization problem specified in equation (3), BIO uses an incremental method that optimizes refinement by moving first horizontally and then vertically. This results in
[0090]
number
[0091] where:
[0092]
number
[0093] is the floor function that outputs the maximum value below the input, and th BIO is the motion refinement threshold to prevent error propagation due to coding noise and irregular local motion, which is 2 18-BDis equal to. The operator (?:) is a ternary conditional operator, and an expression of the form (a? b : c) is evaluated to b if the value of a is true, and otherwise it is evaluated to c. The function clip3(a, b, c) returns a if c < a, returns c if a ≤ c ≤ b, and returns b if b < c. The values of S1, S2, S3, S5, and S6 are further
[0094] [Number]
[0095] calculated as, where
[0096] [Number]
[0097] is.
[0098] In BMS-2.0, the BIO gradients in both the horizontal and vertical directions at (6) are directly obtained by calculating the difference between two adjacent samples (either horizontally or vertically depending on the direction of the derived gradient) at one sample position of each L0 / L1 prediction block, and are obtained, for example, as follows.
[0099] [Number]
[0100] In Equation (5), L is the bit-depth increase of the internal BIO process to maintain data accuracy, and is set to 5 in BMS-2.0. Further, to avoid division by a smaller value, the adjustment parameters r and m in Equation (4) are r = 500·4 BD-8 m = 700·4 BD-8 (8) Based on the motion refinement derived by equation (4), the final bi-predictive signal of the current CU may be calculated by interpolating L0 / L1 prediction samples along the motion trajectory based on optical flow equation (1), as specified below:
[0101]
number
[0102] where b is the bidirectional optical flow sample offset, shift is the right shift applied to combine the L0 and L1 predicted signals for bi-prediction, and may be set equal to 15-BD; offset is the bit depth offset, which may be set to 1 << (14-BD)+2 · (1 << 13), and rnd( ·) is the rounding function that rounds the input value to the nearest integer value.
[0103] BIO Bitwidth Analysis As with the aforementioned standard HEVC, for a bi-predicted CU in VVC, when MV refers to a fractional sample position, the L0 / L1 predicted signal, i.e., I (0) (x,y) and I (1) (x,y) are generated with intermediate precision (i.e., 16 bits) to maintain the precision of the subsequent averaging operation. Furthermore, if any of the two MVs is an integer, the precision of the corresponding predicted sample (taken directly from the reference picture) is increased to intermediate precision before averaging is applied. Given a bi-predictive signal at intermediate bit depth and assuming the input video is 10 bits, Table 1 summarizes the bit-widths of the intermediate parameters required at each stage of the BIO process as presented in the section "Bi-predictive Prediction Based on Optical Flow Model".
[0104] [Table 1]
[0105] As can be seen from Table 1, the maximum bit width of the entire BIO process is the vertical motion refinement v y where S6 (42 bits) is V x (9 bits) and S2 (33 bits). Therefore, the maximum bit width of the existing BIO design is equal to 42+1=43 bits. Furthermore, the multiplication (i.e., v x S2) takes S2 as input, so the 33-bit multiplier is v y Therefore, a direct implementation of the current BIO in BMS-2.0 requires a 33-bit multiplier and has a maximum bitwidth of 43 bits for the intermediate parameters.
[0106] Computational Complexity Analysis of BIO In this section, a computational complexity analysis is performed for the existing BIO design. Specifically, the number of operations (e.g., multiplications and additions) used to generate the final motion compensated prediction with BIO applied is calculated according to the current BIO implementation in BMS-2.0. Furthermore, to facilitate the following discussion, we assume that the size of the current CU predicted by BIO is equal to W×H, where W is the width and H is the height of the CU.
[0107] Generating L0 and L1 prediction samples As shown in Equation (3), the local motion refinement (v x ,v y To derive v , both the required sample value and the gradient value are calculated for all samples in a 6×6 surrounding window around the sample. Thus, the local motion refinement (v x ,v yTo derive L0 and L1 prediction samples, the gradients of (W+2)×(H+2) samples are used by the BIO. Furthermore, as shown in equation (7), both horizontal and vertical gradients are obtained by directly calculating the difference between two adjacent samples. Thus, to calculate the (W+2)×(H+2) gradient value, the total number of prediction samples in both L0 and L1 prediction directions is equal to (W+4)×(H+4). Since the current motion compensation is based on a 2D separable finite impulse response (FIR) 8-tap filter, the number of both multiplications and additions used to generate the L0 and L1 prediction samples is equal to ((W+4)×(H+4+7)×8+(W+4)×(H+4)×8)×2.
[0108] Gradient calculation As shown in equation (7), only one addition is required per sample since the gradients are directly calculated from the two adjacent predicted samples. Considering that both horizontal and vertical gradients are derived over an extended region of (W+2)x(H+2) in both L0 and L1, the total number of additions required to derive the gradients is equal to ((W+2)×(H+2))×2×2.
[0109] Correlation Parameter Calculation As shown in equations (5) and (6), there are five correlation parameters (i.e., S1, S2, S3, S5, and S6) calculated by BIO for every sample in the extended region (W+2)×(H). Furthermore, there are five multiplications and three additions used to calculate the five parameters at each sample position. Thus, the total number of multiplications and additions to calculate the correlation parameters is equal to ((W+2)×(H+2))×5 and ((W+2)×(H+2))×3, respectively.
[0110] Sum As mentioned above, BIO movement refinement (v x ,v y) is derived separately for each 4x4 block in the current CU. To derive the motion refinement for each 4x4 block, the sum of five correlation parameters within a 6x6 surrounding region is calculated. Thus, at this stage, the summation of the five correlation parameters uses a total of (W / 4)x(H / 4)x6x6x5 additions.
[0111] Motion refinement derivation As shown in Equation (4), the local motion refinement (v x ,v y ), there are two additions to add the tuning parameter r to S1 and S3. Furthermore, v y Therefore, to derive the motion refinement for all 4×4 blocks in a CU, the number of multiplications and additions used is equal to (W / 4)×(H / 4) and (W / 4)×(H / 4)×3, respectively.
[0112] Generating a bi-predictive signal Given the derived motion refinement, two more multiplications and six additions are used to derive the final predicted sample value at each sample location, as shown in equation (9). At this stage, a total of W×H×2 multiplications and W×H×6 additions are performed.
[0113] Problems Addressed by Some Embodiments As mentioned above, BIO can improve the efficiency of prediction in bi-prediction by improving both the granularity and accuracy of the motion vectors used in the motion compensation stage. Although BIO can effectively improve coding performance, the complexity of practical hardware implementation increases significantly. In this disclosure, the following complexity problems exist in the current BIO design of BMS-2.0 are identified.
[0114] Large intermediate bit width and large multiplier for BIO Similar to the HEVC standard, when the MV points to a fractional sample position in the reference picture, a 2D separable FIR filter is applied in the motion compensation stage to interpolate the predicted samples of the prediction block. Specifically, first, one interpolation filter is applied horizontally to derive intermediate samples according to the horizontal fractional component of the MV, and then another interpolation filter is applied vertically on the above horizontal fractional samples according to the vertical fractional component of the MV. Assuming that the input is a 10-bit video (i.e., BD=10), Table 2 shows the bit-width measurement of the motion compensation prediction process in VTM / BMS-2.0, assuming that the horizontal and vertical MVs point to half-sample positions corresponding to the worst-case bit-width of the interpolated samples from the motion compensation process. Specifically, in the first stage, the values of the input reference samples associated with the positive and negative filter coefficients are set to the maximum input value (i.e., 2 BD The worst case bit-width of the intermediate data after the first interpolation process (horizontal interpolation) is calculated by setting the input value of the second interpolation to the worst possible value output from the first interpolation (i.e., -1) and the minimum input value (i.e., 0). The worst case bit-width of the second interpolation process (vertical interpolation) is then obtained by setting the input value of the second interpolation to the worst possible value output from the first interpolation.
[0115] [Table 2]
[0116] As can be seen from Table 2, the maximum bit width of the motion compensation interpolation exists in the vertical interpolation process, where the input data is 15 bits and the filter coefficients are 7-bit signed values, therefore the bit width of the output data from the vertical interpolation is 22 bits. Furthermore, when the input data to the vertical interpolation process is 15 bits, a 15-bit multiplier is sufficient for generating intermediate fractional sample values in the motion compensation stage.
[0117] However, as analyzed above, the existing BIO design requires 33-bit multipliers and has 43-bit intermediate parameters to maintain the accuracy of intermediate data. Compared with Table 2, both figures are much larger than those of normal motion compensation interpolation. In practice, such a large bit-width increase (especially the required multiplier bit-width increase) can be very expensive for both hardware and software, increasing the implementation cost of BIO.
[0118] High computational complexity of BIO Based on the above complexity analysis, Tables 3 and 4 show the number of multiplications and additions that need to be done per sample for different CU sizes according to the current BIO and compare them with the complexity statistics of a normal 4×4 bi-predicted CU, which corresponds to the worst-case computational complexity in VTM / BMS-2.0. For a 4×4 bi-predicted CU, given the length of the interpolation filter (e.g., 8), the total number of multiplications and additions is equal to (4×(4+7)×8+4×4×8)×2=960 (i.e., 60 per sample) and (4×(4+7)×8+4×4×8)×2+4×4×2=992 (i.e., 62 per sample).
[0119] [Table 3]
[0120] [Table 4]
[0121] As shown in Tables 3 and 4, the computational complexity shows a significant increase compared to the worst-case complexity of regular bi-prediction by enabling the existing BIO in BMS-2.0. The peak complexity increase occurs for 4x4 bi-predictive CUs, where the number of multiplications and additions with BIO enabled is 329% and 350% of the worst-case bi-prediction case.
[0122] Overview of Exemplary Embodiments In order to solve at least some of the problems mentioned above, in this section, methods are proposed to reduce the complexity of BIO-based motion compensation prediction while maintaining coding gain. First, to reduce implementation cost, this disclosure proposes a bit-width control method to reduce the internal bit-width used in hardware BIO implementation. In some proposed methods, BIO-enabled motion compensation prediction may be implemented using 15-bit multipliers and intermediate values within 32 bits.
[0123] Second, a method is proposed to reduce the computational complexity of BIO by using simplified filters and reducing the number of extended prediction samples used in BIO motion refinement.
[0124] Furthermore, in some embodiments, it is proposed to disable the BIO operation of CU size, which brings about a significant increase in computational complexity compared to normal bi-prediction. Based on the combination of these complexity reductions, the worst-case computational complexity (e.g., the number of multiplications and additions) of motion compensation prediction when BIO is enabled can be reduced to approximately the same level as the worst-case complexity of normal bi-prediction.
[0125] Exemplary BIO Bitwidth Control Method As pointed out above, the current implementation of BIO in BMS-2.0 uses 33-bit multipliers and 43-bit bit-width of intermediate parameters, which is much larger than that of HEVC's motion compensation interpolation implementation. Therefore, it is very costly to implement BIO in hardware and software. In this section, a bit-width control method is proposed to reduce the bit-width required for BIO. In an exemplary method, the horizontal intermediate parameter array Ψ in Equation (6) is reduced to reduce the overall bit-width of the intermediate parameters, as shown below. x (i,j), the vertical intermediate parameter array Ψ y (i,j), and one or more of the signal difference parameter arrays θ(i,j) are first a Bit and n b Bit-shifted downward:
[0126]
number
[0127] Furthermore, to further reduce the bit width, the original L-bit internal bit depth increase can be removed. With such a modification, the horizontal gradient correlation parameter (S1), the cross-gradient correlation parameter (S2), the signal horizontal gradient correlation parameter (S3), the vertical gradient correlation parameter (S5), and the signal vertical gradient correlation parameter (S6) can be implemented as follows by calculating with the equations in (5):
[0128]
number
[0129] Different numbers of right shifts (i.e., n a and n b ) is Ψ x (i,j), Ψ y Taking into account that the motion vectors S1, S2, S3, S5, and S6 are applied to the vectors θ(i,j), θ(i,j), and θ(i,j), the values of S1, S2, S3, S5, and S6 are downscaled by different factors, thereby reducing the derived motion refinement (v x ,v y ) may vary in width. Thus, an additional left shift may be introduced in equation (4) to provide a precise range of widths for the derived motion refinement. Specifically, in the exemplary method, the horizontal motion refinement v x and vertical motion refinement v y can be derived as follows:
[0130]
number
[0131] Note that unlike equation (4), in this embodiment, the adjustment parameters r and m are not applied. x ,v yIn order to reduce the dynamic range of , in this embodiment, the th BIO =2 18-BD A motion refinement threshold th' that is small compared to BIO =2 13-BD In equation (12), the product v x S2 is an input with a bit width greater than 16 bits, so v y To avoid this, the value of the cross-gradient correlation parameter S2 may be divided into two parts, the first part S 2,S is the lowest n S2 The second part S 2,m It is proposed that S2 includes other bits. Based on this, the value S2 can be expressed as follows:
[0132]
number
[0133] Then, substituting equation (13) into equation (12), we obtain the vertical motion refinement v y The calculation of is as follows:
[0134]
number
[0135]
number
[0136] Assuming that the input video is 10 bits, Table 5 summarizes the bit-widths of the intermediate parameters when the example bit-width control method is applied for BIO. As shown in Table 5, using the proposed example bit-width control method, the internal bit-width of the entire BIO process does not exceed 32 bits. The multiplication with the worst possible input is v in Eq. (14). x S 2,m where the input S 2,m is 15 bits, and the input v x is 4 bits. Therefore, one 15-bit multiplier is sufficient when the exemplary method is applied for BIO.
[0137] [Table 5-1]
[0138] [Table 5-2]
[0139] Finally, in equation (10), the L0 and L1 prediction samples I (0) (i,j) and I (1) The BIO parameter θ(i,j) is calculated by applying a right shift to the difference between (i,j). (0) (i,j) and I (1)Both (i,j) values are 16-bit and their difference can be one 17-bit value. Such a design may not be well suited for SIMD-based software implementation. For example, a 128-bit SIMD register can only process 4 samples in parallel. Therefore, in another example, it is proposed to first apply a right shift before calculating the difference when calculating the signal difference parameter array θ(i,j), as follows: θ(i,j)=(I (1) (i,j)≫n b )-(I (0) (i,j)≫n b ) (16)
[0140] In such an embodiment, since the input value of each operation is 16 bits or less, more samples can be processed in parallel. For example, by using equation (16), eight samples can be processed simultaneously by one 128-bit SIMD register. In some embodiments, a similar method is applied to the gradient calculation of equation (7), where a 4-bit right shift is applied before calculating the difference between the L0 predicted sample and the L1 predicted sample to maximize the payload of each SIMD calculation. Specifically, by doing so, the gradient value can be calculated as follows:
[0141]
number
[0142] Exemplary Methods for Reducing BIO Computational Complexity As shown above, the existing BIO design in BMS-2.0 results in a large complexity increase (e.g., the number of multiplications and additions) compared to the worst-case computational complexity of normal bi-prediction. In the following, a method is proposed to reduce the worst-case computational complexity of BIO.
[0143] BIO complexity reduction by generating enhanced samples using simplified filters As mentioned above, assuming that the current CU is W×H, the gradients of samples in the extended region (W+2)×(H+2) are calculated to derive the motion refinement for all 4×4 blocks in the CU. In the existing BIO design, the same interpolation filter (8-tap filter) used for motion compensation is used to generate these extended samples. As shown in Tables 3 and 4, the complexity due to the interpolation of samples in the extended region is the complexity bottleneck of the BIO. Therefore, in order to reduce the BIO complexity, instead of using the 8-tap interpolation filter, it is proposed to use a simplified interpolation filter with a shorter tap length for the generation of samples in the extended surrounding region of the BIO CU. On the other hand, the generation of the extended samples requires access to more reference samples from the reference picture, which may increase the memory bandwidth of the BIO. To avoid memory bandwidth growth, reference sample padding, which is used in the current BIO of BMS-2.0, can be applied, in which reference samples outside the normal reference region of the normal motion compensation of a CU (i.e., (W+7)×(H+7)) are padded by the nearest boundary samples of the normal reference region. To calculate the size of the padded reference samples, assuming that the length of the simplified filter used to generate the extended samples is N, the number of padded reference samples M along each of the top, bottom, left, right, and right boundaries of the normal reference region is equal to:
[0144]
number
[0145] As shown in equation (18), the reference samples on the boundary of the normal reference region are padded with two rows or two columns in each direction by using an 8-tap filter to generate the extended predicted samples. FIG. 5 illustrates the use of simplified filter and reference sample padding to generate samples in the extended region by BIO. As in FIG. 5, the simplified filter is only used to generate predicted samples in the extended region. For positions in the region of the current CU, those predicted samples are generated by applying the default 8-tap interpolation to maintain BIO coding efficiency. In particular, as an embodiment of the present disclosure, it is proposed to use a bilinear interpolation filter (i.e., a 2-tap filter) to generate the extended samples, thereby further reducing the number of operations used for BIO. FIG. 6 illustrates the case of using a bilinear filter to interpolate the extended samples for BIO. The predicted samples in the CU are interpolated by the default 8-tap filter. As shown in FIG. 6, due to the shortened filter length, the bilinear filter does not need to access additional reference samples outside the normal reference region to interpolate the required samples in the extended region. Therefore, in this case, padding of reference samples can be avoided, which can further reduce the complexity of BIO operations.
[0146] Furthermore, based on the comparison of the complexity statistics for different CU sizes in Tables 3 and 4, it can be seen that the complexity increase is larger for CU sizes with lower height. For example, 8×4 CU and 4×8 CU contain the same number of samples, but they show different complexity increase percentages. Specifically, for 8×4 CU, the number of multiplications and additions increases by 149% and 172%, respectively, after enabling BIO, while for 4×8 CU, the corresponding complexity increase is 126% and 149%, respectively. Such complexity difference is brought about by the fact that in the current motion compensation design, a horizontal interpolation filter is applied first, and then a vertical interpolation filter is applied. When the applied MV refers to a fractional position in the vertical direction, more intermediate samples are generated from the horizontal interpolation and used as input for the vertical interpolation. Therefore, the complexity effect of generating more reference samples in the expanded region is relatively more significant for CU sizes with lower height.
[0147] In some embodiments, it is proposed to disable certain CU sizes with small heights to reduce the worst-case BIO complexity. Besides the above method of only disabling certain CU sizes, another way to solve the increased number of operations in the vertical interpolation process is to simplify the interpolation filter used for vertical interpolation. In the current design, the same 8-tap interpolation filter is applied in both horizontal and vertical directions. To reduce the complexity, in some embodiments, it is proposed to use different interpolation filters in the horizontal and vertical interpolation filters when BIO is enabled, and the filter size applied to the second filter process (e.g., vertical interpolation) is smaller than the filter size applied to the first filter process (e.g., horizontal interpolation). For example, a 4-tap chroma interpolation filter may be used to replace the current 8-tap interpolation filter for vertical interpolation. By doing so, the complexity can be reduced by about half for the generation of predicted samples in the extended region. Within the CU, samples may be generated using an 8-tap filter for vertical interpolation. To further reduce the complexity, an even smaller size of the interpolation filter, for example, a bilinear filter, may be used.
[0148] In one specific example, referred to herein as Option 1, it is proposed to use a bilinear filter to generate sample values in an extended region of the BIO to reduce the worst-case BIO complexity, and to disable the BIO completely for CUs with height 4 (i.e., 4×4, 8×4, 16×4, 32×4, 64×4, and 128×4) as well as 4×8 CUs. Tables 6 and 7 show the number of multiplications and additions used to perform per sample for various CU sizes by Option 1 and compare them with the worst-case number for normal bi-prediction. In Tables 6 and 7, the highlighted rows represent CU sizes for which the BIO is disabled. In these rows, the operations associated with the corresponding BIO are set to 0, and the respective complexity is the same as normal bi-prediction for CUs of the same size. As can be seen, in Option 1, the computational complexity peak occurs for 8×8 BIO CUs, where the number of multiplications and additions is 110% and 136% of the worst-case complexity of normal bi-prediction.
[0149] [Table 6]
[0150] [Table 7]
[0151] Reducing BIO complexity by reducing the size of the expanded regions As shown in FIG. 5 and FIG. 6, the above-mentioned BIO complexity reduction method still operates to interpolate two additional rows / columns of predicted samples around each boundary of the current CU. Although the simplified filter is used to reduce the number of operations, some complexity increase still occurs due to the number of samples that need to be interpolated. To further reduce the BIO complexity, in some embodiments, a method is proposed to reduce the number of extended samples from two rows / columns to a single row / column at each CU boundary. Specifically, instead of using (W+4)×(H+4) samples by the current BIO, in some embodiments, only (W+2)×(H+2) samples are used for further complexity reduction. However, as shown in equation (7), the gradient calculation of each sample uses the sample values of both the left and right neighboring elements (for horizontal gradients) or the upper and lower neighboring elements (for vertical gradients). Therefore, by reducing the expanded region size to (W+2)×(H+2), the method can only calculate the gradient values of samples within the CU, and therefore the existing BIO motion refinement cannot be performed directly on the 4×4 blocks located at the four corners of the CU region. To address this issue, in some embodiments, the gradients of sample positions outside the CU (i.e.,
[0152]
number
[0153] ) and sample values (i.e., I (k) A method is applied in which both the sample values and gradients (x,y) are set equal to their nearest neighbors in the CU. Figure 7 illustrates such a padding process for both sample values and gradients. In Figure 7, the dark blocks represent predicted samples in the CU and the white blocks represent predicted samples in the expanded region.
[0154] In the example shown in the figure, additional prediction samples of only a single row / column are generated in the extended region so that the gradients of all samples in the CU region (i.e., the dark blocks in Fig. 7) can be derived accurately. However, for the subblocks at the four corners of the CU (e.g., the subblocks enclosed in the thick black squares in Fig. 7), they use the gradient information of some samples in the extended region (e.g., the white blocks in Fig. 7) that are missing, since their BIO motion refinement is derived from the local region around the subblock (e.g., the region enclosed in the dashed black square in Fig. 7). To solve this problem, their missing gradients are padded by duplicating the gradient values of the nearest boundary samples in the CU region, as indicated by the arrows in Fig. 7. Furthermore, if only the gradients are padded, a problem may arise in that the gradients and sample values used for sample positions in the extended region are shifted, i.e., the sample values are their true sample values, while the gradients are the gradients of their neighboring samples in the CU. This may reduce the accuracy of the derived BIO motion refinement. Therefore, to avoid such inconsistencies, both the sample values and the gradients of the samples in the extended region are padded during the BIO derivation process.
[0155] To achieve even greater complexity reduction, in some embodiments, the proposed padding method is combined with the method of using a simplified interpolation filter and the method of disabling BIO for certain CU sizes mentioned above. In one specific example, called option 2, it is proposed to reduce the extended sample area to (W+2)×(H+2) by using padded samples and gradients for BIO derivation and applying a bilinear filter to generate extended samples in one additional row / column around the CU boundary. Furthermore, BIO is made inapplicable for CUs with height 4 (i.e., 4×4, 8×4, 16×4, 32×4, 64×4, and 128×4) as well as 4×8 CUs. Tables 8 and 9 show the corresponding number of multiplications and additions used per sample for various CU sizes after applying such a method and compare them with the worst-case number for normal bi-prediction. Similar to Tables 6 and 7, the highlighted rows represent CU sizes for which BIO is disabled. As seen with option 2, the number of multiplications and additions is 103% and 129% of the worst case complexity of regular bi-prediction.
[0156] [Table 8]
[0157] [Table 9]
[0158] In another embodiment, it is proposed to still use the default 8-tap filter to interpolate the predicted samples in the extended region of the BIO CU. However, to reduce the BIO complexity, the size of the extended region is reduced from (W+4)×(H+4) to (W+2)×(H+2), i.e., there is one additional row / column at each of the top, left, bottom, and right boundaries of the CU. As described in FIG. 7, both the sample values and gradients of the samples in the extended region are padded during the BIO derivation process to calculate the missing gradients and avoid misalignment between the predicted samples and the gradients. Furthermore, similar to options 1 and 2, certain block sizes (e.g., all CUs with height equal to 4, and CUs with sizes of 4×8, 4×16, 8×8, 16×8) can be disabled.
[0159] In another embodiment, it is proposed to discard all prediction samples in the extended area of the BIO CU, so that the BIO process is only concerned with the interpolation of prediction samples in the current CU area. By doing so, the corresponding BIO operation of generating prediction samples is the same as that of normal bi-prediction. However, due to the reduced number of interpolated samples, the gradients of boundary samples on the current CU cannot be derived by the normal BIO process. In such a case, it is proposed to pad the gradient values of the intra prediction samples of the CU to be the gradients of samples at the CU boundary.
[0160] In another embodiment of the present disclosure, referred to herein as Option 3, it is proposed to shrink the extended sample area to (W+2)×(H+2) by using padded samples and gradients for BIO derivation, and apply the same 8-tap interpolation used in normal motion compensation to generate extended samples in one additional row / column around the CU boundary. Furthermore, in Option 3, BIO is disabled for CUs with height 4 (i.e., 4×4, 8×4, 16×4, 32×4, 64×4, and 128×4) as well as 4×8 CUs.
[0161] For the above-mentioned methods, the size of the extended region is reduced from (W+4)×(H+4) to (W+2)×(H+2), but the methods may still operate to interpolate one additional row / column around the boundary of the BIO CU. As shown in Tables 8 and 9, such methods may still result in a non-negligible complexity increase in the overall BIO complexity. To further reduce the BIO computational complexity, some embodiments propose to directly use reference samples located at integer sample positions (without interpolation) and taken directly from the reference picture as samples in the extended region, and use them to derive gradient values of boundary samples of the current CU. Figure 10 shows an embodiment in which integer reference samples are used as extended samples for BIO derivation. As shown in Figure 10, samples in the CU region (shaded blocks) are generated by applying a default 8-tap interpolation filter. However, for samples in the extended region (non-shaded blocks), instead of using an interpolation filter (e.g., a bilinear filter or an 8-tap interpolation filter), their sample values are directly set equal to the corresponding sample values at integer sample positions in the reference picture. By doing so, all operations introduced by the interpolation of the extended samples can be avoided, thereby providing a significant reduction in the complexity of the BIO. In another embodiment, instead of using integer reference samples, it is proposed to directly set the samples in the extended region equal to the nearest neighbor samples at the CU boundary.
[0162] Since only a single row / column of additional prediction samples is used for BIO derivation in the above method, in some embodiments, a padding method as shown in FIG. 7 may be applied to pad both sample values and gradients of samples on CU boundaries to the extended region during the BIO derivation process. In such embodiments, BIO may be disabled for certain CU sizes to reduce worst-case BIO complexity. For example, in some embodiments, BIO may be disabled for CUs with height 4 (i.e., 4×4, 8×4, 16×4, 32×4, 64×4, and 128×4) as well as 4×8 CUs.
[0163] Disable BIO for CUs predicted by sub-block mode In HEVC, each prediction unit has at most one MV for the prediction direction. In contrast, the current VTM / BMS-2.0 includes two sub-block level inter prediction techniques, including advanced temporal motion vector prediction (ATMVP) and affine prediction. In those coding modes, a video block is further divided into multiple small sub-blocks, and motion information for each sub-block is derived separately. The motion information for each sub-block is used to generate a prediction signal for the block in the motion compensation stage. On the other hand, the current BIO in BMS-2.0 can provide motion refinement at the 4×4 sub-block level in addition to the CU-level motion compensation prediction. Due to the fine granularity of the motion field of sub-block coded CUs, the additional coding benefit obtained from the refined motion by BIO may be very limited. In some embodiments, BIO is disabled for CUs coded by sub-block mode.
[0164] Disable BIO for CUs using pre-determined prediction modes In VVC, some inter bi-prediction modes are based on the assumption that motion is linear and that motion vectors of list 0 and list 1 are symmetric. These modes include Merge with MVD mode (MMVD) described in Non-Patent Document 2 and decoder-side MV derivation with bilateral matching described in Non-Patent Document 3. Because these modes use symmetric motion to generate predictions, applying BIO to these predictions may not be efficient. To reduce complexity, in some embodiments, BIO is disabled for coding units predicted using symmetric modes such as MMVD or decoder-side MV derivation with bilateral matching.
[0165] Multiple hypothesis prediction for intra modes is described in Non-Patent Document 4. Multiple hypothesis prediction for intra modes combines one intra prediction and one inter merge indexed prediction. Since one prediction is obtained from intra prediction, in some embodiments, BIO is disabled for coding units predicted using this combined inter and intra multiple hypothesis prediction.
[0166] Multi-hypothesis inter prediction is described in Non-Patent Document 5. In multi-hypothesis inter prediction, up to two additional MVs are signaled for one inter merge coding CU. There are up to four MVs for one CU, namely two MVs from explicit signaling and two MVs from merge candidates indicated by the merge index. Those multiple inter predictions are combined with a weighted average. In this case, the prediction may be good enough. To reduce complexity, in some embodiments, BIO may be disabled for coding units predicted using this multi-hypothesis inter prediction mode.
[0167] Coded bitstream structure FIG. 8 is a diagram illustrating an example of a coded bitstream structure. The coded bitstream 1300 includes several NAL (Network Abstraction Layer) units 1301. The NAL units may include coded sample data, such as coded slice data 1306, or high-level syntax metadata, such as parameter set data, slice headers 1305, or supplemental enhancement information data 1307 (sometimes referred to as SEI messages). A parameter set is a high-level syntax structure that includes basic syntax elements that may apply to multiple bitstream layers (e.g., video parameter set 1302 (VPS)), or to a coded video sequence within one layer (e.g., sequence parameter set 1303 (SPS)), or to several coded pictures within one coded video sequence (e.g., picture parameter set 1304 (PPS)). The parameter sets may be transmitted together with the coded pictures of the video bitstream, or may be transmitted via other means (including out-of-band transmission using a reliable channel, hard coding, etc.). The slice header 1305 is a high-level syntax structure that may be relatively small or contain some picture-related information that is relevant only to a particular slice or picture type. The SEI message 1307 carries information that may not be required by the decoding process, but may be used for various other purposes, such as picture output timing or display, and loss detection and concealment.
[0168] Communication Devices and Systems FIG. 9 illustrates an example of a communication system. The communication system 1400 may include an encoder 1402, a communication network 1404, and a decoder 1406. The encoder 1402 may communicate with the communication network 1404 via a connection 1408, which may be a wired or wireless connection. The encoder 1402 may be similar to the block-based video encoder of FIG. 2A. The encoder 1402 may include a single-layer codec (e.g., FIG. 2A) or a multi-layer codec. The decoder 1406 may communicate with the communication network 1404 via a connection 1410, which may be a wired or wireless connection. The decoder 1406 may be similar to the block-based video decoder of FIG. 2B. The decoder 1406 may include a single-layer codec (e.g., FIG. 2B) or a multi-layer codec.
[0169] The encoder 1402 and / or decoder 1406 may be incorporated into a wide variety of wired communication devices and / or wireless transmit / receive units (WTRUs), including, but not limited to, digital televisions, wireless broadcast systems, network elements / terminals, servers such as content servers or web servers (e.g., HyperText Transfer Protocol (HTTP) servers), personal digital assistants (PDAs), laptop or desktop computers, tablet computers, digital cameras, digital recording devices, video game devices, video game consoles, cellular or satellite radio telephones, digital media players, and the like.
[0170] The communication network 1404 may be any suitable type of communication network. For example, the communication network 1404 may be a multiple access system providing content, such as voice, data, video, messaging, broadcasts, etc., to multiple wireless users. The communication network 1404 enables multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication network 1404 may utilize one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), and / or Single Carrier FDMA (SC-FDMA). The communication network 1404 may include multiple connected communication networks. The communication network 1404 may include the Internet and / or one or more private commercial networks, such as cellular networks, WiFi hotspots, and / or Internet Service Provider (ISP) networks.
[0171] Encoder and decoder systems and methods In some embodiments, a method for encoding or decoding a video is provided, which includes, for at least one current block in a video coded using bidirectional optical flow, calculating a first gradient component based on a first prediction signal from a first reference picture, calculating a second gradient component based on a second prediction signal from a second reference picture, summing the first gradient component and the second gradient component and performing a downward bit shift of the resulting sum to generate a reduced bitwidth correlation parameter, calculating a motion refinement based at least in part on the reduced bitwidth correlation parameter, and predicting the block using bidirectional optical flow using the calculated motion refinement.
[0172] In some embodiments, the first gradient component is ∂I (0) / ∂x, and the second gradient component is ∂I (1) / ∂x, and the reduced bitwidth correlation parameter is:
[0173]
number
[0174] In some embodiments, the first gradient component is ∂I (0) / ∂y, and the second gradient component is ∂I (1) / ∂y, and the reduced bitwidth correlation parameter is:
[0175]
number
[0176] In some embodiments, a method for encoding or decoding a video is provided, the method including: for at least one current block in a video coded using bidirectional optical flow, generating a reduced bit-width correlation parameter by subtracting a first prediction signal based on a first reference picture from a second prediction signal based on a second reference picture and performing a downward bit-shift of the resulting difference, calculating a motion refinement based at least in part on the reduced bit-width correlation parameter, and predicting the block using the bidirectional optical flow using the calculated motion refinement.
[0177] In some embodiments, the first predicted signal is I (0) and the second predicted signal is I (1) and the reduced bit width correlation parameter is θ(i,j)=(I (1) (i,j)-I (0) (i,j))≫n b It is.
[0178] In some embodiments, a method for encoding or decoding video is provided, the method comprising: for at least one current block in a video coded using bidirectional optical flow:
[0179]
number
[0180] and computing a horizontal motion refinement as
[0181]
number
[0182] and calculating a vertical motion refinement as and predicting blocks using bidirectional optical flow using the calculated horizontal and vertical motion refinement. S1=Σ (i,j)∈Ω Psi x (i,j)·Ψ x (i,j), S2=Σ (i,j)∈Ω Psi x (i,j)·Ψ y (i,j), S3=Σ (i,j)∈Ω θ(i,j) Ψ x (i,j), S5=Σ (i,j)∈Ω Psi y (i,j)·Ψ y (i,j), and S6=Σ (i,j)∈Ω θ(i,j) Ψ y (i,j) It is.
[0183] In some embodiments, a method for encoding or decoding a video is provided. The method includes, for at least one current block in a video coded using bidirectional optical flow, generating a first predicted signal of a reduced bit width by performing a downward bit shift on a first predicted signal from a first reference picture, generating a second predicted signal of a reduced bit width by performing a downward bit shift on a second predicted signal from a second reference picture, generating a reduced bit width correlation parameter by subtracting the first predicted signal of the reduced bit width from the second predicted signal of the reduced bit width, calculating a motion refinement based at least in part on the reduced bit width correlation parameter, and predicting a block using the calculated motion refinement with bidirectional optical flow. In some such embodiments, the reduced bit width correlation parameter is θ(i,j)=(I (1) (i,j)≫n b )-(I (0) (i,j)≫n b ) It is.
[0184] In some embodiments, a method for coding video is provided, which includes, for at least one current block in a video coded using bidirectional optical flow, calculating a first gradient component of a reduced bit width based on a first prediction signal of a reduced bit width from a first reference picture, calculating a second gradient component of a reduced bit width based on a second prediction signal of a reduced bit width from a second reference picture, summing the first reduced bit width gradient component and the second reduced bit width gradient component to generate a reduced bit width correlation parameter, calculating a motion refinement based at least in part on the reduced bit width correlation parameter, and predicting the block using bidirectional optical flow using the calculated motion refinement.
[0185] In some such embodiments, the first gradient component of the reduced bit width is ∂I (0) / ∂x, and the second gradient component of the reduced bit width is ∂I (1) / ∂x, and the reduced bitwidth correlation parameter is:
[0186]
number
[0187] In some embodiments, the first gradient component of the reduced bit width is ∂I (0) / ∂y, and the second gradient component of the reduced bit width is ∂I (1) / ∂y, and the reduced bitwidth correlation parameter is:
[0188]
number
[0189] In some embodiments, the step of calculating the first gradient component of the reduced bitwidth based on the first prediction signal of the reduced bitwidth from the first reference picture includes:
[0190]
number
[0191] The step of calculating the second gradient component of the reduced bit width based on the second prediction signal of the reduced bit width from the second reference picture includes:
[0192]
number
[0193] The method includes the step of calculating:
[0194] In some embodiments, the step of calculating the first gradient component of the reduced bitwidth based on the first prediction signal of the reduced bitwidth from the first reference picture includes:
[0195]
number
[0196] Calculating The step of calculating a second gradient component of a reduced bit width based on a second prediction signal of a reduced bit width from a second reference picture includes:
[0197]
number
[0198] The method includes the step of calculating:
[0199] In some embodiments, a method of coding video is provided, the method including: for at least one current block in a video coded using bidirectional optical flow, generating a first motion compensated prediction signal and a second motion compensated prediction signal for samples in the current block, where the first motion compensated prediction signal and the second motion compensated prediction signal for samples in the current block are generated using a first interpolation filter having a first number of taps; generating a first motion compensated prediction signal and a second motion compensated prediction signal for samples in an extended region around the current block, where the first motion compensated prediction signal and the second motion compensated prediction signal for samples outside the current block are generated using a second interpolation filter having a second number of taps that is less than the first number of taps; calculating a motion refinement based at least in part on the first motion compensated prediction signal and the second motion compensated prediction signal; and predicting the block using bidirectional optical flow using the calculated motion refinement.
[0200] In some embodiments, the first interpolation filter is an 8-tap filter and the second interpolation filter is a 2-tap filter. In some embodiments, the second interpolation filter is a bilinear interpolation filter.
[0201] In some embodiments, a method for coding video is provided, the method including: generating a first motion compensated prediction signal and a second motion compensated prediction signal for at least one current block in a video coded using bidirectional optical flow, where the first motion compensated prediction signal and the second motion compensated prediction signal for samples in the current block are generated using a horizontal interpolation filter having a first number of taps and a vertical interpolation filter having a second number of taps that is less than the first number of taps; calculating a motion refinement based at least in part on the first motion compensated prediction signal and the second motion compensated prediction signal; and predicting the block using bidirectional optical flow using the calculated motion refinement.
[0202] In some embodiments, a method of coding video is provided, the method including generating a first motion compensated prediction signal and a second motion compensated prediction signal for at least one current block in a video coded using bidirectional optical flow, where the first motion compensated prediction signal and the second motion compensated prediction signal for samples in the current block are generated using a horizontal interpolation filter having a first number of taps and a vertical interpolation filter having a second number of taps, the horizontal filter and the vertical filter being applied in a predetermined order, with a filter applied earlier in the order having a greater number of taps than a filter applied later in the order; calculating a motion refinement based at least in part on the first motion compensated prediction signal and the second motion compensated prediction signal; and predicting the block using bidirectional optical flow using the calculated motion refinement.
[0203] In some embodiments, a method of coding a video including a plurality of coding units is provided, the method including, for a plurality of coding units in a video coded using bi-prediction, disabling bi-directional optical flow for coding units having at least a height of 4, performing bi-prediction without bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is disabled, and performing bi-prediction with bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is not disabled. In some such embodiments, bi-directional optical flow is further disabled for coding units having a height of 8 and a width of 4.
[0204] In some embodiments, a method of coding a video including a plurality of coding units is provided, the method including, for a plurality of coding units in a video coded using bi-prediction, disabling bi-directional optical flow for coding units having a height less than or equal to a threshold height, performing bi-prediction without bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is disabled, and performing bi-prediction with bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is not disabled.
[0205] In some embodiments, a method of coding video is provided, the method including: for at least one current block in a video coded using bidirectional optical flow, generating a first motion compensated prediction signal and a second motion compensated prediction signal for samples in the current block, generating first and second values for samples in an extended region around the current block, the extended region not including samples more than one row or column away from the current block, calculating a motion refinement based at least in part on the first and second motion compensated prediction signals and the first and second values for samples in the extended region, and predicting a block using bidirectional optical flow using the calculated motion refinement.
[0206] In some such embodiments, generating first values for the samples in the expanded region includes setting each first sample value in the expanded region equal to a first predicted sample value of its respective nearest neighbor in the current block. In some embodiments, generating second values for the samples in the expanded region includes setting each second sample value in the expanded region equal to a second predicted sample value of its respective nearest neighbor in the current block.
[0207] Some embodiments further include generating first and second gradient values at samples in an extended region around the current block, where generating first gradient values at samples in the extended region includes setting each first gradient value in the extended region equal to a gradient value calculated at its respective nearest neighbor in the current block using the first prediction signal, and generating second gradient values at samples in the extended region includes setting each second gradient value in the extended region equal to a gradient value calculated at its respective nearest neighbor in the current block using the second prediction signal.
[0208] In some embodiments, a method of coding a video including a plurality of coding units is provided, the method including disabling bi-directional optical flow for at least coding units predicted using a sub-block level inter prediction technique for a plurality of coding units in a video coded using bi-prediction, performing bi-prediction without bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is disabled, and performing bi-prediction with bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is not disabled.
[0209] In some such embodiments, bi-prediction is disabled for at least coding units predicted using advanced temporal motion vector prediction (ATMVP).
[0210] In some embodiments, bi-prediction is disabled for at least coding units predicted using affine prediction.
[0211] In some embodiments, a method for coding a video including a plurality of coding units is provided, the method including disabling bi-directional optical flow for coding units having at least a height of 4 for a plurality of coding units in a video coded using bi-prediction, performing bi-prediction without bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is disabled, and performing bi-prediction with bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is not disabled, the performing bi-prediction with bi-directional optical flow for each current coding unit includes generating a first motion compensated prediction signal and a second motion compensated prediction signal for samples in the current coding unit, the first motion compensated prediction signal and the second motion compensated prediction signal for samples in the current block. the first and second motion compensated prediction signals for samples in an extended region around the current coding unit are generated using a first interpolation filter having a first number of taps; generating the first and second motion compensated prediction signals for samples in an extended region around the current coding unit, where the first and second motion compensated prediction signals for samples outside the current coding unit are generated using a second interpolation filter having a second number of taps that is less than the first number of taps; calculating a motion refinement based at least in part on the first and second motion compensated prediction signals; and predicting the current coding unit using bidirectional optical flow using the calculated motion refinement.
[0212] In some such embodiments, the first interpolation filter is an 8-tap filter and the second interpolation filter is a 2-tap filter. In some embodiments, the second interpolation filter is a bilinear interpolation filter.
[0213] In some embodiments, bidirectional optical flow is further disabled for coding units having a height of eight and a width of four.
[0214] In some embodiments, a method for coding a video that includes a plurality of coding units is provided. The method includes, for a plurality of coding units in a video coded using bi-prediction, disabling bi-directional optical flow for coding units having at least height 4; performing bi-prediction without bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is disabled; and performing bi-prediction using bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is not disabled, where performing bi-prediction using bi-directional optical flow for each current coding unit includes generating a first motion compensated prediction signal and a second motion compensated prediction signal for samples in the current coding unit; generating first and second values for samples in an extended region around the current coding unit, the extended region not including samples multiple rows or columns away from the current coding unit; calculating a motion refinement based at least in part on the first and second motion compensated prediction signals and the first and second values for samples in the extended region; and predicting the current coding unit using the bi-directional optical flow using the calculated motion refinement.
[0215] In some embodiments, generating first values for samples in the extended region includes setting each first sample value in the extended region equal to a first predicted sample value of its respective nearest neighbor in the current coding unit.
[0216] In some embodiments, generating second values for the samples in the extended region includes setting each second sample value in the extended region equal to a second predicted sample value of its respective nearest neighbor in the current coding unit.
[0217] Some embodiments further include generating first and second gradient values at samples in an extended region around the current coding unit, where generating the first gradient values at samples in the extended region includes setting each first gradient value in the extended region equal to a gradient value calculated at its respective nearest neighbor in the current coding unit using the first prediction signal, and generating the second gradient values at samples in the extended region includes setting each second gradient value in the extended region equal to a gradient value calculated at its respective nearest neighbor in the current coding unit using the second prediction signal.
[0218] In some such embodiments, bidirectional optical flow is further disabled for coding units having a height of eight and a width of four.
[0219] In some embodiments, a method of coding video is provided, the method including: for at least one current block in a video coded using bidirectional optical flow, generating a first motion compensated prediction signal and a second motion compensated prediction signal for samples in the current block, where the first motion compensated prediction signal and the second motion compensated prediction signal for samples in the current block are generated using a horizontal interpolation filter having a first number of taps and a vertical interpolation filter having the first number of taps; generating a first motion compensated prediction signal and a second motion compensated prediction signal for samples in an extended region around the current block, where the first motion compensated prediction signal and the second motion compensated prediction signal for samples outside the current block are generated using a horizontal interpolation filter having the first number of taps and a vertical interpolation filter having a second number of taps that is less than the first number of taps; calculating a motion refinement based at least in part on the first motion compensated prediction signal and the second motion compensated prediction signal; and predicting the block using bidirectional optical flow using the calculated motion refinement.
[0220] In some embodiments, a method for coding a video including a plurality of coding units is provided, the method including, for a plurality of coding units in a video coded using bi-prediction, disabling bi-directional optical flow for at least coding units predicted using a symmetric prediction mode, performing bi-prediction without bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is disabled, and performing bi-prediction with bi-directional optical flow for bi-predicted coding units for which bi-directional optical flow is not disabled.
[0221] In some embodiments, bi-prediction is disabled for coding units predicted using at least merge mode with MVD (MMVD). In some embodiments, bi-prediction is disabled for coding units predicted using decoder-side MV derivation with bilateral matching.
[0222] In some embodiments, a method of coding a video including a plurality of coding units is provided, the method including, for a plurality of coding units in a video coded using bi-prediction, disabling bi-directional optical flow for coding units predicted using at least multiple hypothesis prediction for intra-mode, performing bi-prediction without bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is disabled, and performing bi-prediction with bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is not disabled.
[0223] In some embodiments, a method is provided for coding a video including a plurality of coding units, the method including disabling bi-directional optical flow for at least coding units predicted using multiple hypothesis inter prediction for a plurality of coding units in a video coded using bi-prediction, performing bi-prediction without bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is disabled, and performing bi-prediction with bi-directional optical flow for the bi-predicted coding units for which bi-directional optical flow is not disabled.
[0224] It should be noted that the various hardware elements of one or more of the described embodiments are referred to as "modules" that perform (i.e., perform, execute, etc.) various functions described herein with respect to the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more ASICs, one or more FPGAs, one or more memory devices) deemed appropriate for a given implementation by one of ordinary skill in the art. It should also be noted that each described module may include executable instructions to perform one or more functions described as being performed by the respective module, which may take or include the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may be stored in one or more suitable non-transitory computer-readable mediums, such as those commonly referred to as RAM, ROM, etc.
[0225] Although the features and elements are described above in certain combinations, one skilled in the art will understand that each feature or element may be used alone or in any combination with other features and elements. The methods described herein may be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, ROM, RAM, registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and DVDs. A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.
Claims
1. Obtaining a first prediction signal array from a first reference picture; performing a right bit-shift on two samples of the first predicted signal array and determining a difference between the two right bit-shifted samples of the first predicted signal array to obtain a first array of first gradient components; Obtaining a second prediction signal array from a second reference picture; performing a right bit shift on two samples of the second predicted signal array and determining a difference between two right bit shifted samples of the second predicted signal array to obtain a second array of first gradient components; obtaining an intermediate parameter array of a first component, the intermediate parameter array including determining a sum of a first array of the first gradient components and a second array of the first gradient components; obtaining a motion refinement for at least a first component based at least in part on the intermediate parameter array for the first component; generating a prediction of a current block in the video using bidirectional optical flow using motion refinement of at least the first component; 2. A video decoding method comprising:
2. 2. The method of claim 1, wherein obtaining the intermediate parameter array of the first components comprises performing a right bit shift on a sum of the first array of the first gradient components and the second array of the first gradient components.
3. 3. The method of claim 1 or claim 2, further comprising: obtaining a signal difference parameter array, the signal difference parameter array including performing a right bit shift on each of the first prediction signal array and the second prediction signal array before obtaining a difference between the first prediction signal array and the second prediction signal array, and the motion refinement of the first component is based at least in part on the signal difference parameter array.
4. The first component intermediate parameter array is a horizontal intermediate parameter array, and the first component motion refinement is a horizontal motion refinement, and the method comprises: obtaining a signal horizontal gradient correlation parameter by summing components of an element-wise multiplication of the signal difference parameter array and the horizontal mean parameter array; The method of claim 3 , wherein obtaining the horizontal motion refinement comprises bit-shifting the signal horizontal gradient correlation parameter to obtain the horizontal motion refinement.
5. the first gradient component is a horizontal gradient; for at least some horizontal gradients of the first array having coordinates (i,j), two samples of the first predicted signal array have coordinates (i+1,j) and (i-1,j); for at least some horizontal gradients of the second array having coordinates (i,j), two samples of the second predicted signal array have coordinates (i+1,j) and (i-1,j); The method of claim 4.
6. performing a right bit shift on two samples of the first predicted signal array and determining a difference between the two right bit shifted samples of the first predicted signal array to obtain a first array of second gradient components; performing a right bit shift on two samples of the second predicted signal array and determining a difference between two right bit shifted samples of the second predicted signal array to obtain a second array of second gradient components; obtaining an intermediate parameter array of a second component, the intermediate parameter array including determining a sum of a first array of the second gradient components and a second array of the second gradient components; obtaining a motion refinement for a second component based at least in part on the intermediate parameter array for the second component; Further comprising: The method of claim 3 , wherein the prediction of the current block further uses motion refinement of the second component.
7. the second gradient component is a vertical gradient; For at least some vertical gradients of the first array having coordinates (i,j), two samples of the first predicted signal array have coordinates (i,j+1) and (i,j-1); For at least some vertical gradients of the second array having coordinates (i,j), two samples of the second predicted signal array have coordinates (i,j+1) and (i,j-1). The method of claim 6.
8. Obtaining a first prediction signal array from a first reference picture; performing a right bit-shift on two samples of the first predicted signal array and determining a difference between the two right bit-shifted samples of the first predicted signal array to obtain a first array of first gradient components; Obtaining a second prediction signal array from a second reference picture; performing a right bit shift on two samples of the second predicted signal array and determining a difference between two right bit shifted samples of the second predicted signal array to obtain a second array of first gradient components; obtaining an intermediate parameter array of a first component, the intermediate parameter array including determining a sum of a first array of the first gradient components and a second array of the first gradient components; obtaining a motion refinement for at least a first component based at least in part on the intermediate parameter array for the first component; generating a prediction of a current block in the video using bidirectional optical flow using motion refinement of at least the first component; 23. A video decoder apparatus comprising: one or more processors configured to execute:
9. 9. The apparatus of claim 8, wherein obtaining the intermediate parameter array of the first components comprises performing a right bit shift on a sum of the first array of first gradient components and the second array of first gradient components.
10. 10. The apparatus of claim 8 or claim 9, further comprising: obtaining a signal difference parameter array, the signal difference parameter array including performing a right bit shift on each of the first prediction signal array and the second prediction signal array before obtaining a difference between the first prediction signal array and the second prediction signal array, and wherein the motion refinement of the first component is based at least in part on the signal difference parameter array.
11. the first component intermediate parameter array is a horizontal intermediate parameter array, and the first component motion refinement is a horizontal motion refinement, and the one or more processors: obtaining a signal horizontal gradient correlation parameter by summing components of an element-wise multiplication of the signal difference parameter array and the horizontal mean parameter array; The apparatus of claim 10 , wherein obtaining the horizontal motion refinement is further configured to perform bit-shifting the signal horizontal gradient correlation parameter to obtain the horizontal motion refinement.
12. the first gradient component is a horizontal gradient; for at least some horizontal gradients of the first array having coordinates (i,j), two samples of the first predicted signal array have coordinates (i+1,j) and (i-1,j); for at least some horizontal gradients of the second array having coordinates (i,j), two samples of the second predicted signal array have coordinates (i+1,j) and (i-1,j); 12. The apparatus of claim 11.
13. Obtaining a first prediction signal array from a first reference picture; performing a right bit-shift on two samples of the first predicted signal array and determining a difference between the two right bit-shifted samples of the first predicted signal array to obtain a first array of first gradient components; Obtaining a second prediction signal array from a second reference picture; performing a right bit shift on two samples of the second predicted signal array and determining a difference between two right bit shifted samples of the second predicted signal array to obtain a second array of first gradient components; obtaining an intermediate parameter array of a first component, the intermediate parameter array including determining a sum of a first array of the first gradient components and a second array of the first gradient components; obtaining a motion refinement for at least a first component based at least in part on the intermediate parameter array for the first component; generating a prediction of a current block in the video using bidirectional optical flow using motion refinement of at least the first component; A video encoding method comprising:
14. 14. The method of claim 13, wherein obtaining the intermediate parameter array of the first component comprises performing a right bit shift on a sum of the first array of the first gradient components and the second array of the first gradient components.
15. 15. The method of claim 13 or 14, further comprising: obtaining a signal difference parameter array, the method comprising: performing a right bit shift on each of the first prediction signal array and the second prediction signal array before obtaining a difference between the first prediction signal array and the second prediction signal array; and wherein the motion refinement of the first component is based at least in part on the signal difference parameter array.
16. The first component intermediate parameter array is a horizontal intermediate parameter array, and the first component motion refinement is a horizontal motion refinement, and the method comprises: obtaining a signal horizontal gradient correlation parameter by summing components of an element-wise multiplication of the signal difference parameter array and the horizontal mean parameter array; The method of claim 15 , wherein obtaining the horizontal motion refinement comprises bit-shifting the signal horizontal gradient correlation parameter to obtain the horizontal motion refinement.
17. Obtaining a first prediction signal array from a first reference picture; performing a right bit-shift on two samples of the first predicted signal array and determining a difference between the two right bit-shifted samples of the first predicted signal array to obtain a first array of first gradient components; Obtaining a second prediction signal array from a second reference picture; performing a right bit shift on two samples of the second predicted signal array and determining a difference between two right bit shifted samples of the second predicted signal array to obtain a second array of first gradient components; obtaining an intermediate parameter array of a first component, the intermediate parameter array including determining a sum of a first array of the first gradient components and a second array of the first gradient components; obtaining a motion refinement for at least a first component based at least in part on the intermediate parameter array for the first component; generating a prediction of a current block in the video using bidirectional optical flow using motion refinement of at least the first component; 23. A video encoder apparatus comprising: one or more processors configured to execute:
18. 20. The apparatus of claim 17, wherein obtaining the intermediate parameter array of the first component comprises performing a right bit shift on a sum of the first array of first gradient components and the second array of first gradient components.
19. 19. The apparatus of claim 17 or 18, further comprising: obtaining a signal difference parameter array, the signal difference parameter array including performing a right bit shift on each of the first prediction signal array and the second prediction signal array before obtaining a difference between the first prediction signal array and the second prediction signal array, and wherein a motion refinement of the first component is based at least in part on the signal difference parameter array.
20. the first gradient component is a vertical gradient; For at least some vertical gradients of the first array having coordinates (i,j), two samples of the first predicted signal array have coordinates (i,j+1) and (i,j-1); For at least some vertical gradients of the second array having coordinates (i,j), two samples of the second predicted signal array have coordinates (i,j+1) and (i,j-1).
20. The apparatus of claim 19.
Citation Information
Patent Citations
Complexity Reduction and Bitwidth Control for Bidirectional Optical Flow
JP7311589B2
Complexity Reduction and Bitwidth Control for Bidirectional Optical Flow
JP7553659B2
Spatial prediction method, image decoding method, and image coding method
US20130028530A1
Spatial prediction method, image decoding method, and image encoding method
WO2011129084A1
Method and apparatus of motion refinement based on BI-directional optical flow for video coding
WO2018166357A1
