Extended amvp merge motion representation

Abstract

Video coding can perform using extended AMVP merge motion representation for low delay blocks and / or pictures. A video coding device can determine that a first picture order count (POC) value associated with a first reference picture associated with an adaptive motion vector prediction (AMVP) portion of a current coding unit (CU) is different from a second POC value of a second reference picture associated with a merge portion of a merge list candidate. The video coding device can generate a merge list using the merge list candidate based on the determination. The video coding device can obtain motion information associated with the current coding unit (CU) based on the merge list.

Description

[0001] Cross-reference to related applications This application claims the benefit of European Provisional Application No. 23306686.9, filed on 2 October 2023, the contents of which are incorporated herein by reference. Background Technology

[0002] Video codec systems can be used to compress digital video signals, for example, to reduce the storage and / or transmission bandwidth required for such signals. Video codec systems can include, for example, block-based, wavelet-based, and / or object-based systems. Summary of the Invention

[0003] Systems, methods, and means for performing video encoding and decoding using extended adaptive motion vector prediction (AMVP) merging motion representations are disclosed. AMVP merging modes are scalable to a richer set of motion representations in inter-frame codec blocks. Affine AMVP merging motion representations can be enabled for low-latency blocks, slices, and / or pictures, for example, similar to support for low-latency non-affine AMVP merging modes. AMVP merging motion representation modes can be extended to merge affine merging motion representation modes, which can include bi-prediction for a given prediction unit (PU), thereby mixing affine and non-affine merging modes. A PU corresponding to a bi-prediction block can include, for example, a merging candidate in one inter-frame direction associated with a list of reference pictures (L0 or L1), and an affine merging candidate in the opposite inter-frame direction associated with, for example, another list of reference pictures (L1 or L0).

[0004] In the example, signaling can be used to merge affine merge flags (e.g., at the PU level) to indicate the use of merged affine merge motion representation modes for a given block. For example, motion vector encoding / decoding can be implemented using high-level control. Motion data encoding / decoding systems can be activated / deactivated (e.g., canonically) via (e.g., dedicated) Sequence Parameter Set (SPS) signaling flags, Picture Parameter Set (PPS) signaling flags, picture header syntax elements, slice header syntax elements, subpicture level syntax elements, and / or codec tree unit (CTU) level syntax elements.

[0005] In the example, a video codec device including a processor can be configured to implement a method that may include one or more of the following: determining that a first picture sequence count (POC) value associated with a first reference picture associated with a merge portion of an AMVP merge candidate differs from a second POC value associated with a second reference picture associated with a merge portion of the merge candidate; generating a merge list using the merge list candidate; and obtaining motion information associated with the current codec unit (CU) based on the merge list. The current CU may be in an affine AMVP merge prediction mode. The merge list candidate may be a low-latency merge list candidate. The method may include skipping a bilateral matching-based reordering of low-latency merge list candidates in the merge list. The method may include performing a bilateral matching-based reordering of non-low-latency merge list candidates in the merge list. Attached Figure Description

[0006] Figure 1A This is a system diagram illustrating an example communication system in which one or more of the disclosed embodiments may be implemented.

[0007] Figure 1B This illustrates that, according to an embodiment, it is possible to Figure 1A The system diagram shown is of an example wireless transmit / receive unit (WTRU) used in the communication system.

[0008] Figure 1C This illustrates that, according to an embodiment, it is possible to Figure 1A The system diagram shows an example radio access network (RAN) and an example core network (CN) used in the communication system shown.

[0009] Figure 1D This illustrates that, according to an embodiment, it is possible to Figure 1A The system diagram shows another example RAN and another example CN used in the communication system shown.

[0010] Figure 2 An example video encoder is shown.

[0011] Figure 3 An example video decoder is shown.

[0012] Figure 4 Examples of systems in which various aspects and examples can be implemented are shown.

[0013] Figure 5 An example is shown representing the structure of the codec tree unit, codec unit, and prediction unit for a compressed image.

[0014] Figure 6 Examples of codec tree units, prediction units, and transform units in video encoding and decoding are shown.

[0015] Figure 7 An example of dividing a codec unit into prediction units is shown.

[0016] Figure 8 An example of CTU partitioning of the codec tree is shown.

[0017] Figure 9 An example of the splitting patterns supported in multi-type tree splitting is shown.

[0018] Figure 10 An example of signaling for inter-frame prediction information is shown.

[0019] Figure 11 An example of MVP candidate list construction in AMVP mode is shown.

[0020] Figure 12 Examples of adjacent spatial positions A0, A1 (left), B0, B1, B2 (top) of the current block and the corresponding blocks (H and C) of the TMVP are shown.

[0021] Figure 13 An example of the location of the spatial and temporal motion vector predictors used in the merged mode is shown.

[0022] Figure 14 An example of constructing a list of candidates for merging motion vector predictions is shown.

[0023] Figure 15 An example of constructing a list of candidates for merging motion vector predictions is shown.

[0024] Figure 16 An example of motion representation categories based on overall blocks and sub-blocks is shown.

[0025] Figures 17A-17B An example of constructing a list of non-sub-block merging candidates is shown.

[0026] Figure 18 An example of the allowed motion vector difference (MVD) is shown.

[0027] Figure 19 An example representation of CU motion data in GPM mode is shown.

[0028] Figure 20 An example of GPM separation grouped at the same angle is shown.

[0029] Figure 21 An example of mixing between two prediction partitions performed in GPM is shown.

[0030] Figure 22 An example of an affine motion model based on control points is shown.

[0031] Figure 23 An example of an affine motion field representation based on 4×4 sub-blocks is shown.

[0032] Figure 24 An example of a location that inherits the predicted value of affine motion is shown.

[0033] Figure 25 An example of control point motion vector inheritance is shown.

[0034] Figure 26 An example of locating candidate positions for constructing an affine merging pattern is shown.

[0035] Figure 27 An example is shown that can be used to derive spatially adjacent and non-nearby block locations for spatial merge candidates.

[0036] Figure 28 An example of template matching is shown in the search area around the initial MV.

[0037] Figure 29 An example of bidirectional matching is shown.

[0038] Figure 30 An example of AMVP merging enabled for dual-prediction blocks with two reference images, one from the past and one from the future, is shown.

[0039] Figure 31 An example of motion data parsing in the AMVP merging mode is shown.

[0040] Figure 32 An example of parsing AMVP merge pattern information is shown.

[0041] Figure 33 An example of motion data reconstruction and export is shown in the BM-AMVP merge mode.

[0042] Figure 34 An example of the motion analysis process in the merging affine merging mode is shown. Detailed Implementation

[0043] A more detailed understanding can be obtained by referring to the following description, which is given by way of example in conjunction with the accompanying drawings.

[0044] Figure 1AThis diagram illustrates an example communication system 100 in which one or more of the disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, and broadcasting to multiple wireless users. The communication system 100 enables multiple wireless users to access this content by sharing system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero Tail Unique Word DFT Spread Spectrum OFDM (ZT UWDTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multicarrier (FBMC), etc.

[0045] like Figure 1A As shown, the communication system 100 may include wireless transceiver units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, Public Switched Telephone Network (PSTN) 108, Internet 110, and other networks 112. However, it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, WTRUs 102a, 102b, 102c, and 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. Any of WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0046] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a and 114b may be any type of device configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks (e.g., CN 106 / 115, Internet 110, and / or other networks 112). For example, base stations 114a and 114b may be base transceiver stations (BTS), node Bs, eNodeBs, home node Bs, home eNodeBs, gNBs, NRNodeBs, site controllers, access points (APs), wireless routers, etc. Although base stations 114a and 114b are each described as a single element, it should be understood that base stations 114a and 114b may include any number of interconnected base station and / or network elements.

[0047] Base station 114a may be part of RAN 104 / 113, and may also include other base stations and / or network elements (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage of a specific geographic area, which may be relatively fixed or may change over time. A cell may also be divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver per sector of the cell. In one embodiment, base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0048] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116. Air interface 116 can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Any suitable radio access technology (RAT) can be used to establish air interface 116.

[0049] More specifically, as described above, the communication system 100 can be a multiple access system and can employ one or more access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN 104 / 113 can implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can establish air interfaces 115 / 116 / 117 using Wideband CDMA (WCDMA). WCDMA can include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or evolved HSPA (HSPA+). HSPA can include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0050] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which can use Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro) to establish air interface 116.

[0051] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as NR radio access, which can establish an air interface 116 using a new radio (NR).

[0052] In one embodiment, base station 114a and WTRUs 102a, 102b, and 102c can implement multiple radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can jointly implement LTE radio access and NR radio access, for example, using the dual connectivity (DC) principle. Therefore, the air interface used by WTRUs 102a, 102b, and 102c can be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0053] In other embodiments, base station 114a and WTRUs 102a, 102b, and 102c can implement radio technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate GSM Evolution (EDGE), and GSM EDGE (GERAN).

[0054] For example, Figure 1A Base station 114b can be a wireless router, home node B, home eNodeB, or access point, and can utilize any suitable RAT to facilitate wireless connectivity in a local area, such as commercial locations, homes, vehicles, campuses, industrial facilities, air corridors (e.g., for drone use), roads, etc. In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d can utilize cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish picocells or femtocells. Figure 1A As shown, base station 114b can be directly connected to Internet 110. Therefore, it is not required that base station 114b access Internet 110 via CN 106 / 115.

[0055] RAN 104 / 113 can communicate with CN 106 / 115, which can be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRUs 102a, 102b, 102c, and 102d. Data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. CN 106 / 115 can provide call control, billing services, location-based services, prepaid calling, internet connectivity, video distribution, and / or perform advanced security functions such as user authentication. Although in Figure 1AAlthough not shown, it should be understood that RAN 104 / 113 and / or CN 106 / 115 can communicate directly or indirectly with other RANs using the same RAT as RAN 104 / 113 or a different RAT. For example, in addition to connecting to RAN 104 / 113, which may utilize NR radio technology, CN 106 / 115 can also communicate with another RAN (not shown) using GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0056] CN 106 / 115 can also serve as a gateway for WTRU 102a, 102b, 102c, 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN 104 / 113 or a different RAT.

[0057] Some or all of the WTRUs 102a, 102b, 102c, and 102d in the communication system 100 may include multi-mode capabilities (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example... Figure 1A The WTRU 102c shown can be configured to communicate with a base station 114a that can use cellular-based radio technology and with a base station 114b that can use IEEE 802 radio technology.

[0058] Figure 1B This is a system diagram illustrating example WTRU 102. (See diagram below.) Figure 1B As shown, WTRU 102 may include a processor 118, a transceiver 120, a transmitting / receiving element 122, a speaker / microphone 124, a keyboard 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a Global Positioning System (GPS) chipset 136, and / or other peripheral devices 138, etc. It should be understood that WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with the embodiments.

[0059] Processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. Processor 118 may perform signal encoding / decoding, data processing, power control, input / output processing, and / or any other functions that enable WTRU 102 to operate in a wireless environment. Processor 118 may be coupled to transceiver 120, and transceiver 120 may be coupled to transmitting / receiving element 122. Although Figure 1B While processor 118 and transceiver 120 are described as separate components, it should be understood that processor 118 and transceiver 120 may be integrated together in an electronic package or chip.

[0060] Transmitting / receiving element 122 can be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via air interface 116. For example, in one embodiment, transmitting / receiving element 122 can be an antenna configured to transmit and / or receive RF signals. In one embodiment, transmitting / receiving element 122 can be, for example, a transmitter / detector configured to transmit and / or receive IR, UV, or visible light signals. In yet another embodiment, transmitting / receiving element 122 can be configured to transmit and / or receive both RF and optical signals. It should be understood that transmitting / receiving element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0061] Although the transmitting / receiving element 122 is in Figure 1B While described as a single element, WTRU 102 may include any number of transmitting / receiving elements 122. More specifically, WTRU 102 may employ MIMO technology. Thus, in one embodiment, WTRU 102 may include two or more transmitting / receiving elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals via air interface 116.

[0062] Transceiver 120 can be configured to modulate signals to be transmitted by transmitting / receiving element 122 and demodulate signals received by transmitting / receiving element 122. As described above, WTRU 102 can have multi-mode capability. Therefore, for example, transceiver 120 may include multiple transceivers to enable WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0063] The processor 118 of WTRU 102 can be coupled to a speaker / microphone 124, a keyboard 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit) and can receive user input data therefrom. The processor 118 can also output user data to the speaker / microphone 124, keyboard 126, and / or display / touchpad 128. Furthermore, the processor 118 can access and store information from any type of suitable memory (e.g., non-removable memory 130 and / or removable memory 132). Non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital storage (SD) card, etc. In other embodiments, the processor 118 can access and store information from memory that is not physically located on WTRU 102 (e.g., on a server or home computer (not shown)).

[0064] The processor 118 can receive power from the power supply 134 and can be configured to distribute and / or control power to other components in the WTRU 102. The power supply 134 can be any suitable device for powering the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0065] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to, or instead of, information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via air interface 116, and / or determine its location based on the timing of signals received from two or more nearby base stations. It should be understood that the WTRU 102 may acquire location information using any suitable location determination method while remaining consistent with the embodiments.

[0066] The processor 118 may be further coupled to other peripheral devices 138, which may include one or more software and / or hardware modules providing additional features, functions, and / or wired or wireless connectivity. For example, peripheral devices 138 may include accelerometers, electronic compasses, satellite transceivers, digital cameras (for photos and / or video), Universal Serial Bus (USB) ports, vibration devices, television transceivers, hands-free headsets, Bluetooth® modules, FM radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, etc. Peripheral devices 138 may include one or more sensors, which may be gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors; geolocation sensors; altimeters, light sensors, touch sensors, magnetometers, barometers, attitude sensors, biosensors, and / or humidity sensors.

[0067] WTRU 102 may include a full-duplex radio for which the transmission and reception of some or all signals (e.g., associated with a specific subframe of both UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference via hardware (e.g., a choke) or via signal processing (e.g., a separate processor (not shown) or via processor 118). In one embodiment, WTRU 102 may include a half-duplex radio for which the transmission and reception of some or all signals (e.g., associated with a specific subframe of either UL (e.g., for transmission) or downlink (e.g., for reception) may be concurrent and / or simultaneous.

[0068] Figure 1C This is a system diagram illustrating RAN 104 and CN 106 according to one embodiment. As described above, RAN 104 can communicate with WTRUs 102a, 102b, and 102c via air interface 116 using E-UTRA radio technology. RAN 104 can also communicate with CN 106.

[0069] RAN 104 may include eNode-B 160a, 160b, 160c; however, it should be understood that RAN 104 may include any number of eNode-Bs while remaining consistent with the embodiments. eNode-B 160a, 160b, 160c may each include one or more transceivers for communicating with WTRU 102a, 102b, 102c via air interface 116. In one embodiment, eNode-B 160a, 160b, 160c may implement MIMO technology. Therefore, for example, eNode-B 160a may use multiple antennas to transmit and / or receive radio signals from WTRU 102a.

[0070] Each of the eNode-B 160a, 160b, and 160c can be associated with a specific cell (not shown) and can be configured to handle radio resource management decisions, handover decisions, and user scheduling in the UL and / or DL, etc. Figure 1C As shown, eNode-B 160a, 160b, and 160c can communicate with each other via the X2 interface.

[0071] Figure 1C The CN 106 shown may include a Mobility Management Entity (MME) 162, a Serving Gateway (SGW) 164, and a Packet Data Network (PDN) Gateway (or PGW) 166. While each of the foregoing elements is described as part of CN 106, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0072] The MME 162 can connect to each eNode-B 162a, 162b, 162c in RAN 104 via the S1 interface and can be used as a control node. For example, the MME 162 can be responsible for authenticating users of WTRUs 102a, 102b, 102c, bearer activation / deactivation, selecting a specific serving gateway during the initial attachment of WTRUs 102a, 102b, 102c, etc. The MME 162 can provide control plane functions for handover between RAN 104 and other RANs (not shown) employing other radio technologies (such as GSM and / or WCDMA).

[0073] The SGW 164 can connect to each eNode B 160a, 160b, or 160c in RAN 104 via the S1 interface. The SGW 164 can typically route and forward user data packets to / from WTRUs 102a, 102b, or 102c. The SGW 164 can perform other functions, such as anchoring the user plane during inter-eNode B handover; triggering paging when DL data is available for WTRUs 102a, 102b, or 102c; and managing and storing the context of WTRUs 102a, 102b, or 102c.

[0074] The SGW 164 can connect to the PGW 166, which can provide WTRU 102a, 102b, and 102c with access to packet-switched networks such as Internet 110, to facilitate communication between WTRU 102a, 102b, 102c and IP-enabled devices.

[0075] CN 106 can facilitate communication with other networks. For example, CN 106 can provide WTRU 102a, 102b, and 102c with access to a circuit-switched network such as PSTN 108 to facilitate communication between WTRU 102a, 102b, and 102c and traditional landline communication equipment. For example, CN 106 may include, or be able to communicate with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that serves as an interface between CN 106 and PSTN 108. Furthermore, CN 106 can provide WTRU 102a, 102b, and 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers.

[0076] Despite WTRU in Figure 1A-1D While described as a wireless terminal, it is conceivable that, in some representative embodiments, such a terminal may use (e.g., temporarily or permanently) a wired communication interface with a communication network.

[0077] In a representative embodiment, the other network 112 may be a WLAN.

[0078] A WLAN in Infrastructure Basic Services Set (BSS) mode can have an access point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP can access or peer into a distributed system (DS) or another type of wired / wireless network that carries traffic into and / or out of the BSS. Traffic originating outside the BSS destined for a STA can be delivered to the AP via it. Traffic originating from a STA destined for a destination outside the BSS can be sent to the AP for delivery to the appropriate destination. For example, traffic between STAs within the BSS can be sent via the AP, where the source STA can send traffic to the AP, and the AP can deliver traffic to the destination STA. Traffic between STAs within the BSS can be considered and / or referred to as peering traffic. Peering traffic can be sent between source and destination STAs (e.g., directly between them) using Direct Link Establishment (DLS). In some representative embodiments, the DLS can use 802.11e DLS or 802.11z Tunneled DLS (TDLS). A WLAN using the Standalone BSS (IBSS) mode may not have an access point (AP), and STAs within the IBSS or using the IBSS (e.g., all STAs) can communicate directly with each other. The IBSS communication mode is sometimes referred to as the "self-organizing" communication mode in this document.

[0079] When operating in 802.11ac infrastructure mode or a similar mode, the AP can transmit beacons on a fixed channel (e.g., the primary channel). The primary channel can be of a fixed width (e.g., a 20 MHz bandwidth) or dynamically set via signaling. The primary channel can be the working channel of the BSS and can be used by the STA to establish a connection with the AP. In some representative embodiments, Carrier Sense Multiple Access with Collision Avoidance (CSMA / CA) can be implemented, for example in an 802.11 system. For CSMA / CA, each STA, including the AP, can sense the primary channel. If a particular STA senses / detects and / or determines that the primary channel is busy, that particular STA can back off. A single STA (e.g., only one station) can transmit at any given time within a given BSS.

[0080] High-throughput (HT) STAs can communicate using a 40MHz wide channel, for example, by combining a primary 20MHz channel with adjacent or non-adjacent 20MHz channels.

[0081] Very High Throughput (VHT) STAs can support channels with widths of 20MHz, 40MHz, 80MHz, and / or 160MHz. 40MHz and / or 80MHz channels can be formed by combining consecutive 20MHz channels. A 160MHz channel can be formed by combining eight consecutive 20MHz channels, or by combining two non-consecutive 80MHz channels, which can be referred to as an 80+80 configuration. For the 80+80 configuration, after channel encoding, the data passes through a segment parser, which splits the data into two streams. Each stream can be processed separately using Inverse Fast Fourier Transform (IFFT) and time-domain processing. These streams can be mapped onto the two 80MHz channels, and the data can be transmitted by the transmitting STA. At the receiver of the receiving STA, the operation of the 80+80 configuration described above can be reversed, and the combined data can be sent to the Media Access Control (MAC).

[0082] 802.11af and 802.11ah support operating modes below 1 GHz. The channel operating bandwidth and carrier in 802.11af and 802.11ah are reduced compared to those used in 802.11n and 802.11ac. 802.11af supports 5 MHz, 10 MHz, and 20 MHz bandwidths in the TV Blank (TVWS) spectrum, while 802.11ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz using non-TVWS. According to a representative embodiment, 802.11ah can support metering-type control / machine-type communications, such as MTC devices in macro coverage areas. MTC devices may have certain capabilities, such as limited capabilities, including support (e.g., only support) certain and / or limited bandwidths. MTC devices may include batteries with a battery life exceeding a threshold (e.g., to maintain very long battery life).

[0083] WLAN systems that can support multiple channels and channel bandwidths (e.g., 802.11n, 802.11ac, 802.11af, and 802.11ah) include a channel that can be designated as the primary channel. The bandwidth of the primary channel can be equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel can be set and / or limited by one of the STAs operating in the BSS that supports the minimum bandwidth operating mode. In the 802.11ah example, for STAs that support (e.g., only support) the 1MHz mode (e.g., MTC type devices), the primary channel can be 1MHz wide, even if the AP and other STAs in the BSS support 2MHz, 4MHz, 8MHz, 16MHz, and / or other channel bandwidth operating modes. Carrier Sense and / or Network Allocation Vector (NAV) settings can depend on the status of the primary channel. If the primary channel is busy, for example due to STAs (only supporting the 1MHz operating mode) sending to the AP, the entire available band can be considered busy, even if most of the band remains idle and may be available.

[0084] In the United States, the available frequency band for 802.11ah is from 902MHz to 928MHz. In South Korea, the available frequency band is from 917.5MHz to 923.5MHz. In Japan, the available frequency band is from 916.5MHz to 927.5MHz. The total available bandwidth for 802.11ah is 6MHz to 26MHz, depending on the country code.

[0085] Figure 1D This is a system diagram illustrating RAN 113 and CN 115 according to one embodiment. As described above, RAN 113 can communicate with WTRUs 102a, 102b, and 102c via air interface 116 using NR radio technology. RAN 113 can also communicate with CN 115.

[0086] RAN 113 may include gNBs 180a, 180b, and 180c; however, it should be understood that RAN 113 may include any number of gNBs while remaining consistent with the embodiments. gNBs 180a, 180b, and 180c may each include one or more transceivers for communicating with WTRUs 102a, 102b, and 102c via air interface 116. In one embodiment, gNBs 180a, 180b, and 180c may implement MIMO technology. For example, gNBs 180a and 180b may utilize beamforming to transmit signals to and / or receive signals from gNBs 180a, 180b, and 180c. Therefore, for example, gNB 180a may use multiple antennas to transmit and / or receive radio signals from WTRU 102a. In one embodiment, gNBs 180a, 180b, and 180c can implement carrier aggregation technology. For example, gNB 180a can transmit multiple component carriers (not shown) to WTRU 102a. A subset of these component carriers may be on unlicensed spectrum, while the remaining component carriers may be on licensed spectrum. In one embodiment, gNBs 180a, 180b, and 180c can implement Coordinated Multipoint (CoMP) technology. For example, WTRU 102a can receive coordinated transmissions from gNBs 180a and 180b (and / or gNB 180c).

[0087] WTRUs 102a, 102b, and 102c can communicate with gNBs 180a, 180b, and 180c using transmissions associated with scalable digitization. For example, OFDM symbol spacing and / or OFDM subcarrier spacing can vary for different transmissions, different cells, and / or different portions of the radio transmission spectrum. WTRUs 102a, 102b, and 102c can communicate with gNBs 180a, 180b, and 180c using subframes or transmission time intervals (TTIs) of various or scalable lengths (e.g., containing different numbers of OFDM symbols and / or continuously varying lengths of absolute time).

[0088] gNBs 180a, 180b, and 180c can be configured to communicate with WTRUs 102a, 102b, and 102c in standalone and / or non-standalone configurations. In standalone configuration, WTRUs 102a, 102b, and 102c can communicate with gNBs 180a, 180b, and 180c without simultaneously accessing other RANs (e.g., eNode-Bs 160a, 160b, and 160c). In standalone configuration, WTRUs 102a, 102b, and 102c can utilize one or more gNBs 180a, 180b, and 180c as mobility anchors. In standalone configuration, WTRUs 102a, 102b, and 102c can communicate with gNBs 180a, 180b, and 180c using signals in unlicensed frequency bands. In a non-standalone configuration, WTRUs 102a, 102b, and 102c can communicate / connect with gNBs 180a, 180b, and 180c, while also communicating / connecting with another RAN such as eNode-Bs 160a, 160b, and 160c. For example, WTRUs 102a, 102b, and 102c can implement DC principles to communicate substantially simultaneously with one or more gNBs 180a, 180b, and 180c, as well as one or more eNode-Bs 160a, 160b, and 160c. In a non-standalone configuration, eNode-Bs 160a, 160b, and 160c can be used as mobility anchors for WTRUs 102a, 102b, and 102c, and gNBs 180a, 180b, and 180c can provide additional coverage and / or throughput for serving WTRUs 102a, 102b, and 102c.

[0089] Each of gNBs 180a, 180b, and 180c can be associated with a specific cell (not shown) and can be configured to handle radio resource management decisions, handover decisions, user scheduling in UL and / or DL, network slicing support, dual connectivity, interoperability between NR and E-UTRA, routing user plane data to User Plane Functions (UPF) 184a and 184b, and routing control plane information to Access and Mobility Management Functions (AMF) 182a and 182b, etc. Figure 1D As shown, gNB 180a, 180b, and 180c can communicate with each other via the Xn interface.

[0090] Figure 1DThe CN 115 shown may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one Session Management Function (SMF) 183a, 183b, and possibly a Data Network (DN) 185a, 185b. Although each of the foregoing elements is described as part of the CN 115, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0091] AMF 182a and 182b can connect to one or more gNBs 180a, 180b, and 180c in RAN 113 via the N2 interface and can be used as control nodes. For example, AMF 182a and 182b can be responsible for authenticating users of WTRU 102a, 102b, and 102c, supporting network slicing (e.g., handling different PDU sessions with different requirements), selecting specific SMF 183a and 183b, managing registration areas, terminating NAS signaling, mobility management, etc. AMF 182a and 182b can use network slicing to customize CN support for WTRU 102a, 102b, and 102c based on the service type used by WTRU 102a, 102b, and 102c. For example, different network slices can be established for different use cases (e.g., services relying on Ultra Reliable Low Latency (URLLC) access, services relying on Enhanced Massive Mobile Broadband (eMBB) access, services for Machine Type Communication (MTC) access, etc.). AMF 162 can provide control plane functions for handover between RAN 113 and other RANs (not shown) that employ other radio technologies, such as LTE, LTE-A, LTE-A Pro and / or non-3GPP access technologies, such as WiFi.

[0092] SMFs 183a and 183b can connect to AMFs 182a and 182b in CN 115 via the N11 interface. SMFs 183a and 183b can also connect to UPFs 184a and 184b in CN 115 via the N4 interface. SMFs 183a and 183b can select and control UPFs 184a and 184b, and configure the routing of services through UPFs 184a and 184b. SMFs 183a and 183b can perform other functions, such as managing and allocating UE IP addresses, managing PDU sessions, controlling policy enforcement and QoS, and providing downlink data notifications. PDU session types can be IP-based, non-IP-based, Ethernet-based, etc.

[0093] UPF 184a and 184b can be connected to one or more gNBs 180a, 180b, and 180c in RAN 113 via the N3 interface. This interface can provide WTRU 102a, 102b, and 102c with access to a packet-switched network (e.g., Internet 110) to facilitate communication between WTRU 102a, 102b, 102c and IP-enabled devices. UPF 184 and 184b can perform other functions such as routing and forwarding packets, enforcing user plane policies, supporting multi-destination PDU sessions, handling user plane QoS, buffering downlink packets, and providing mobility anchoring.

[0094] CN 115 can facilitate communication with other networks. For example, CN 115 may include, or be able to communicate with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) serving as an interface between CN 115 and PSTN 108. Furthermore, CN 115 can provide WTRUs 102a, 102b, and 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers. In one embodiment, WTRUs 102a, 102b, and 102c can be connected to local data networks (DNs) 185a and 185b via UPFs 184a and 184b through their N3 interfaces and the N6 interface between UPFs 184a and 184b and DNs 185a and 185b.

[0095] Given Figure 1A-1D as well as Figure 1A-1D As described in the corresponding descriptions herein, one or all of the functions described for one or more of the WTRU 102a-d, base station 114a-b, eNode-B 160a-c, MME 162, SGW 164, PGW 166, gNB 180a-c, AMF 182a-b, UPF 184a-b, SMF183a-b, DN 185a-b, and / or any other device described herein (one or more) may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more of the functions described herein. For example, an emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0096] Simulation devices can be designed to perform one or more tests on other devices in laboratory and / or carrier network environments. For example, one or more simulation devices can perform one or more or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. One or more simulation devices can perform one or more or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. Simulation devices can be directly coupled to another device for testing and / or performing tests using over-the-air wireless communication.

[0097] One or more emulation devices may perform one or more functions, including all functions, rather than being implemented / deployed as part of a wired and / or wireless communication network. For example, emulation devices may be used in test scenarios outside of deployment (e.g., testing) wired and / or wireless communication networks and / or test laboratories to implement testing of one or more components. One or more emulation devices may be test equipment. Emulation devices may transmit and / or receive data using direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas).

[0098] This application describes various aspects, including tools, features, examples, models, methods, etc. Many of these aspects are described in detail, and often in a manner that may sound restrictive, at least to illustrate individual characteristics. However, this is for the purpose of clarity and does not limit the application or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, aspects can also be combined and interchanged with those described in earlier applications.

[0099] The aspects described and envisioned in this application can be implemented in many different forms. Figure 5-34 Some examples can be provided, but other examples can also be considered. Figure 5-34 The discussion does not limit the breadth of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting the generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having bitstreams generated according to any of the described methods stored thereon.

[0100] In this application, the terms “reconstruction” and “decoding” are used interchangeably, the terms “pixel” and “sample” are used interchangeably, and the terms “image”, “picture” and “frame” are used interchangeably.

[0101] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first," "second," etc., can be used in various examples to modify elements, components, steps, operations, etc., such as "first decoding" and "second decoding," for example. Unless specifically required, the use of these terms does not imply a sequence of modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or in a time period overlapping with the second decoding.

[0102] The various methods and other aspects described in this application can be used to modify, for example... Figure 2 and Figure 3 The illustrated video encoder 200 and decoder 300 modules are, for example, decoding modules. Furthermore, the subject matter disclosed herein can be applied to, for example, any type, format, or version of video encoding (whether described in standards or recommendations, whether pre-existing or future-developed, and any extensions to such standards and recommendations). Unless otherwise stated or technically excluded, the aspects described in this application may be used individually or in combination.

[0103] Various numerical values ​​are used in the examples described in this application, such as chroma format (e.g., 4:2:0, 4:2:2, 4:4:4), block / CTU size (e.g., 32×32, 64×64, 128×128, or 256×256 pixels), partition size, number of sub-blocks, number of patterns, number of candidates, number of references, sub-block size, number of parameters, flag values, multiplier values, constant values, etc. These and other specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0104] Figure 2 This is a diagram illustrating an example video encoder (e.g., a block-based hybrid video encoder). Variations of the example encoder 200 are envisioned, but for clarity, encoder 200 is described below without describing all anticipated variations.

[0105] Before being encoded, the video sequence may undergo pre-coding (201), such as applying color transformations to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of the input image components to obtain a more resilient signal distribution to compression (e.g., using histogram equalization with one of the color components). Metadata may be associated with pre-processing and appended to the bitstream.

[0106] In encoder 200, the image is encoded by encoder elements as described below. The image to be encoded is segmented (202) and processed in units, for example, codec units (CUs). Each unit is encoded using, for example, intra-frame or inter-frame modes. When a unit is encoded in intra-frame mode, it performs intra-frame prediction (260). In inter-frame mode, motion estimation (275) and compensation (270) are performed. The encoder determines (205) which of the intra-frame or inter-frame modes to use to encode the unit and indicates the intra-frame / inter-frame decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (210) the prediction block from the original image block.

[0107] The predicted residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with the motion vector and other syntax elements (e.g., image segmentation information), are entropy encoded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass the transform and quantization, i.e., the residual is directly encoded without applying the transform or quantization process.

[0108] The encoder decodes the coded blocks to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inversely transformed (250) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (255) to reconstruct the image blocks. An in-loop filter (265) is applied to the reconstructed image to perform, for example, deblocking / SAO (Sample Adaptive Offset) / ALF (Adaptive Loop Filtering) filtering, thereby reducing coding artifacts. The filtered image is stored in a reference image buffer (280).

[0109] Figure 3 This is a diagram illustrating an example video decoder. In the example decoder 300, the bitstream is decoded by decoder elements, as described below. The video decoder 300 typically performs the same operations as... Figure 2 The encoding process described herein is the opposite of the decoding process. Encoder 200 typically also performs video decoding as part of the encoded video data.

[0110] Specifically, the input to the decoder includes a video bitstream, which can be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, prediction modes, motion vectors, and other encoded information. Image segmentation information indicates how the image is segmented. Therefore, the decoder can segment (335) the image based on the decoded image segmentation information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (355) to reconstruct image blocks. The prediction blocks can be obtained (370) from intra-frame prediction (360) or motion-compensated prediction (i.e., inter-frame prediction) (375). An in-loop filter (365) is applied to reconstruct the image. The filtered image is stored at a reference image buffer (380). In some examples (e.g., for a given image), the contents of the reference image buffer 380 on the decoder 300 side can be the same as the contents of the reference image buffer 280 on the encoder 200 side (e.g., for the same image).

[0111] The decoded image may undergo further post-decoding processing (385), such as inverse color transformation (e.g., a conversion from YCbCr4:2:0 to RGB4:4:4) or inverse remapping, which is the inverse of the remapping process performed in the pre-encoding process (201). Post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream. In the example, the decoded image (e.g., after applying an in-loop filter (365), and / or after post-decoding processing (385), if post-decoding processing is used) may be sent to a display device for presentation to the user.

[0112] Figure 4 This is a diagram illustrating an example of a system in which the various aspects and examples described herein can be implemented. System 400 can be implemented as a device including the various components described below and configured to perform one or more aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 400 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one example, the processing and encoder / decoder elements of system 400 are distributed across multiple ICs and / or discrete components. In various examples, system 400 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various examples, system 400 is configured to implement one or more aspects described herein.

[0113] System 400 includes at least one processor 410 configured to execute instructions loaded thereon for implementing various aspects described herein, such as those described herein. Processor 410 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 400 includes at least one memory 420 (e.g., a volatile memory device and / or a non-volatile memory device). System 400 includes a storage device 440 which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 440 may include internal storage devices, additional storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0114] System 400 includes an encoder / decoder module 430 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 430 may include its own processor and memory. The encoder / decoder module 430 represents a module that can be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both encoding and decoding modules. Furthermore, the encoder / decoder module 430 may be implemented as a separate element of system 400, or it may be included within processor 410 as a combination of hardware and software known to those skilled in the art.

[0115] Program code to be loaded onto processor 410 or encoder / decoder 430 to execute the various aspects described herein may be stored in storage device 440 and subsequently loaded onto memory 420 for execution by processor 410. According to various examples, one or more of processor 410, memory 420, storage device 440, and encoder / decoder module 430 may store one or more various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0116] In some examples, the memory within processor 410 and / or encoder / decoder module 430 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other examples, external memory (e.g., the processing device could be processor 410 or encoder / decoder module 430) is used for one or more of these functions. External memory could be memory 420 and / or storage device 440, such as volatile memory and / or non-volatile flash memory. In several examples, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one example, fast external volatile memory, such as RAM, is used as working memory for video encoding and decoding operations.

[0117] As shown in block 445, inputs can be provided to the components of system 400 through various input devices. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcasting company, (ii) component (COMP) input terminals (or a set of COMP input terminals), (iii) universal serial bus (USB) input terminals, and / or (iv) high-definition multimedia interface (HDMI) input terminals. Figure 4 Other examples not shown include composite video.

[0118] In various examples, the input devices of block 445 have associated respective input processing elements, as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting a signal band to a band), (ii) down-converting the selected signal, (iii) further band-limiting to a narrower band to select, for example, a signal band, which in some examples may be referred to as a channel, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and / or (vi) demultiplexing to select a desired data packet stream. The RF section of various examples includes one or more elements performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner performing various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box example, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to the desired frequency band. Various examples rearrange the order of the aforementioned (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as, for example, inserting amplifiers and analog-to-digital converters. In various examples, the RF section includes an antenna.

[0119] USB and / or HDMI terminals may include their respective interface processors for connecting system 400 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as needed, for example, within a separate input processing IC or within processor 410. Similarly, as needed, various aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within processor 410. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 410 and encoder / decoder 430, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0120] Various components of system 400 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data therebetween using a suitable connection arrangement 425 (e.g., internal buses known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0121] System 400 includes a communication interface 450, which is capable of communicating with other devices via a communication channel 460. The communication interface 450 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 460. The communication interface 450 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 460 may be implemented, for example, in a wired and / or wireless medium.

[0122] In various examples, data is streamed or otherwise provided to system 400 using a wireless network such as Wi-Fi (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)). In these examples, the Wi-Fi signal is received via a communication channel 460 and a communication interface 450 adapted for Wi-Fi communication. The communication channel 460 in these examples is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other examples use a set-top box to provide streaming data to system 400, with the set-top box transmitting data via an HDMI connection to input block 445. Still other examples use an RF connection to input block 445 to provide streaming data to system 400. As mentioned above, various examples provide data in a non-streaming manner. Furthermore, various examples use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth® networks.

[0123] System 400 can provide output signals to various output devices, including a display 475, a speaker 485, and other peripheral devices 495. Various examples of the display 475 include one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a flexible display, and / or a foldable display. The display 475 can be used in televisions, tablets, laptops, cellular phones (mobile phones), or other devices. The display 475 can also be integrated with other components (e.g., as in a smartphone) or standalone (e.g., an external monitor for a laptop). In various examples, other peripheral devices 495 include one or more of a standalone digital video disc (or digital multifunction disc) (DVD, for both terms), a disc player, a stereo system, and / or a lighting system. Various examples use one or more peripheral devices 495 that provide functionality based on the output of system 400. For example, a disc player performs the function of playing the output of system 400.

[0124] In various examples, signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols capable of enabling device-to-device control with or without user intervention is used to transmit control signals between system 400 and display 475, speaker 485, or other peripheral devices 495. Output devices can be communicatively coupled to system 400 via dedicated connections through their respective interfaces 470, 480, and 490. Alternatively, output devices can be connected to system 400 via communication interface 450 using communication channel 460. Display 475 and speaker 485 can be integrated into a single unit with other components of system 400 in electronic devices such as televisions. In various examples, display interface 470 includes display drivers, such as, for example, a timing controller (TCon) chip.

[0125] For example, if the RF section of input 445 is part of a standalone set-top box, then display 475 and speaker 485 can alternatively be separated from one or more other components. In various examples where display 475 and speaker 485 are external components, the output signal can be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0126] These examples can be implemented by computer software implemented by processor 410, or by hardware, or by a combination of hardware and software. As a non-limiting example, these examples can be implemented by one or more integrated circuits. Memory 420 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as, as a non-limiting example, optical storage devices, magnetic storage devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 410 can be of any type suitable for the technical environment and can include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0127] Various implementations involve decoding. As used in this application, "decoding" can include, for example, all or part of a process performed on a received encoded sequence to produce a final output suitable for display. In various examples, these processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various examples, these processes also include, or alternatively include, processes performed by a decoder of the various implementations described in this application, such as determining a first picture sequence count (POC) value associated with a first reference picture associated with a merge portion of an AMVP candidate merge list, which differs from a second POC value associated with a second reference picture associated with a merge portion of a merge list candidate; generating a merge list using the merge list candidates; obtaining motion information associated with the current codec unit (CU) based on the merge list; skipping bidirectional matching-based reordering of low-latency merge list candidates in the merge list; performing bidirectional matching-based reordering of non-low-latency merge list candidates in the merge list; etc.

[0128] As further examples, in one example, "decoding" refers only to entropy decoding; in another example, "decoding" refers only to differential decoding; and in yet another example, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to refer to a broader decoding process will be clear based on the specific context of the description and is considered well understood by those skilled in the art.

[0129] Various implementations involve encoding. Similar to the discussion of “decoding” above, “encoding” as used in this application can include, for example, all or part of a process performed on an input video sequence to generate an encoded bitstream. In various examples, these processes also include, or alternatively include, processes performed by encoders of various implementations described in this application, such as determining a first picture sequence count (POC) value associated with a first reference picture associated with a merge portion of an AMVP candidate merge list, which differs from a second POC value associated with a second reference picture associated with a merge portion of a merge list candidate; generating a merge list using merge list candidates; obtaining motion information associated with the current codec unit (CU) based on the merge list; skipping bidirectional matching-based reordering of low-latency merge list candidates in the merge list; performing bidirectional matching-based reordering of non-low-latency merge list candidates in the merge list; etc.

[0130] As further examples, in one example, "encoding" refers only to entropy encoding; in another example, "encoding" refers only to differential encoding; and in yet another example, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to specifically refer to a subset of operations or to refer to a broader encoding process will be clear based on the specific context of the description and is considered well understood by those skilled in the art.

[0131] Note that the syntax elements used in this article (such as affine_flag, SPLIT_BT_VER, SPLIT_BT_HOR, SPLIT_TT_VER, SPLIT_TT_HOR, Nb_Cand, Max_Cand, PCIIP, Wmerge, Pmerge, Wintra, Pintra, MVk, MV1x, MV1y, MV0x, MV0y, CPMVk, MV0_pass1, deltaMV, amvpMergeModeFlag[L0], amvpMergeModeFlag[L1], MvpIdx[L0], refListMerge, refListAMVP, etc.) are descriptive terms. Therefore, they do not preclude the use of other syntax element names.

[0132] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0133] The implementations and aspects described herein can be implemented, for example, in methods or processes, apparatuses, software programs, data streams, or signals. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the features in question can be implemented in other forms (e.g., apparatuses or programs). Apparatuses can be implemented, for example, in suitable hardware, software, and firmware. Methods can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

[0134] References to “an example” or “an example” or “an implementation” or “an implementation” and their variations mean that a particular feature, structure, characteristic, etc., described in connection with the example is included in at least one example. Therefore, the appearance of the phrase “in an example” or “in the example” or “in an implementation” or “in the implementation”, and any other variations appearing in various places throughout this application, do not necessarily refer to the same example.

[0135] Furthermore, this application may relate to "determining" various information pieces. Determining information may include, for example, one or more of estimated information, calculated information, predicted information, or information retrieved from memory. Obtaining may include receiving, retrieving, constructing, generating, and / or determining.

[0136] Furthermore, this application may relate to "accessing" various information fragments. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0137] Furthermore, this application may relate to "receiving" various pieces of information. Like "access," receiving is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) one or more. Moreover, "receiving" is generally referred to in one way or another during operations such as, for example, storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0138] It should be understood that any use of " / ", "and / or", and "at least one" (e.g., in the cases of "A / B", "A and / or B", and "at least one of A and B") is intended to include selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to as many items as listed, as will be apparent to those skilled in the art and related fields.

[0139] Furthermore, as used herein, the word “signal” specifically refers to instructing the corresponding decoder to do something. Encoder signals can include, for example, residual signals, metadata, encoding / decoding or motion representation patterns, segmentation patterns, reference images, motion vector prediction values ​​(MVP), motion vector differences (MVD), indices, candidate lists, weights, types of affine patterns, etc. Thus, in one example, the same parameters are used on both the encoder and decoder sides. Therefore, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters and others, signaling can be used without sending them (implicit signaling) to simply allow the decoder to know and select specific parameters. Bit savings are achieved in various examples by avoiding the transmission of any actual functionality. It should be understood that signaling can be accomplished in many ways. For example, in various examples, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the verb form of the word “signaling” has been used above, the word “signaling” can also be used as a noun in this article.

[0140] It will be apparent to those skilled in the art that implementations can generate various signals that are formatted to carry, for example, information that can be stored or transmitted. The information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described example. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via a variety of different wired or wireless links. Signals may be stored on, or accessed or received from, a processor-readable medium.

[0141] This document describes numerous examples. Features of the examples may be provided individually or in any combination across various claim classes and types. Furthermore, examples may include one or more of the features, devices, or aspects described herein (individually or in any combination across various claim classes and types). For example, features described herein may be implemented in a bitstream or signal that includes information generated as described herein. This information may allow a decoder to decode the bitstream, and an encoder, bitstream, and / or decoder may be implemented according to any of the embodiments described. For example, features described herein may be implemented by creating and / or transmitting and / or receiving and / or decoding a bitstream or signal. For example, features described herein may implement a method, process, apparatus, medium storing instructions, medium storing data, or signal. For example, features described herein may be implemented by a TV, set-top box, cellular phone, tablet computer, or other electronic device performing decoding. The TV, set-top box, cellular phone, tablet computer, or other electronic device may display (e.g., using a monitor, screen, or other type of display) a resulting image (e.g., an image reconstructed from the residual of a video bitstream). The TV, set-top box, cellular phone, tablet computer, or other electronic device may receive a signal including an encoded image and perform decoding.

[0142] Systems, methods, and means for performing video encoding and decoding using extended adaptive motion vector prediction (AMVP) merging motion representations are disclosed. AMVP merging modes are scalable to a richer set of motion representations in inter-frame codec blocks. Affine AMVP merging motion representations can be enabled for low-latency blocks, slices, and / or pictures, for example, similar to support for low-latency non-affine AMVP merging modes. AMVP merging motion representation modes can be extended to merge affine merging motion representation modes, which can include bi-prediction for a given prediction unit (PU), thereby mixing affine and non-affine merging modes. A PU corresponding to a bi-prediction block can include, for example, a merging candidate in one inter-frame direction associated with a list of reference pictures (L0 or L1), and an affine merging candidate in the opposite inter-frame direction associated with, for example, another list of reference pictures (L1 or L0).

[0143] In the example, signaling merge affine merge flags (e.g., at the PU level) can be used to indicate the use of merge affine merge motion representation modes for a given block. For example, motion vector encoding / decoding can be implemented using high-level control. Motion data encoding / decoding systems can be (e.g., canonically) activated / deactivated, for example, through (e.g., dedicated) Sequence Parameter Set (SPS) signaling flags, Picture Parameter Set (PPS) signaling flags, picture header syntax elements, slice header syntax elements, subpicture level syntax elements, and / or codec tree unit (CTU) level syntax elements.

[0144] In the example, a video codec device including a processor may be configured to implement a method that may include one or more of the following: determining that a first picture sequence count (POC) value associated with a first reference picture associated with a merge portion of an AMVP merge candidate differs from a second POC value associated with a second reference picture associated with a merge portion of the merge candidate; generating a merge list using the merge list candidate; and obtaining motion information associated with the current codec unit (CU) based on the merge list. The current CU may be in an affine AMVP merge prediction mode. The merge list candidate may be a low-latency merge list candidate. The method may include skipping a bidirectional matching-based reordering of low-latency merge list candidates in the merge list. The method may include performing a bidirectional matching-based reordering of non-low-latency merge list candidates in the merge list.

[0145] Inter-frame prediction information can be represented in compressed video, for example, using different types of video compression schemes. Block-based video codecs can associate motion information with blocks encoded and decoded in inter-frame mode (e.g., each block). A block structure can be used in a video codec scheme to represent compressed images. Motion representations can be assigned to inter-frame blocks.

[0146] Video compression systems can divide images into codec tree units (CTUs). The size of a CTU can be, for example, 64×64, 128×128, or 256×256 pixels. Each CTU can be represented by a codec tree within the compression domain. For example, a quadtree partition (QT) of CTUs can exist, where each leaf can be called a codec unit (CU), such as... Figure 5 and Figure 6 The example is shown in the image.

[0147] Figure 5 Examples of codec tree units, codec units, and prediction unit structures representing compressed images (e.g., images compressed using a video codec scheme) are shown.

[0148] Intra-frame or inter-frame prediction parameters (e.g., prediction information) can be given to (e.g., each) CU. CUs can be spatially partitioned into one or more prediction units (PUs). Prediction information can be assigned to (e.g., each) PU. For example, intra-frame or inter-frame encoding / decoding modes can be assigned at the CU level.

[0149] Figure 6 Examples of codec tree units, prediction units, and transform units in video encoding and decoding are shown.

[0150] Figure 7 The image shows examples of partition types that exist in video encoding and decoding. For example... Figure 7As shown, partition types may include square partitions (e.g., 2N×2N and N×N) that can be used (e.g., only partition types) in both intra-frame and inter-frame CUs, symmetrical non-square partitions (e.g., 2N×N, N×2N) that can be used (e.g., only) in inter-frame CUs, and / or asymmetrical partitions that can be used (e.g., only) in inter-frame CUs.

[0151] Figure 7 An example of dividing a codec unit into prediction units is shown.

[0152] Block structures can be used in video codec schemes. Block structures can be used to represent compressed images. Images can be divided into square CTUs (e.g., in various video codec schemes). For example, the size of a CTU can be 32×32, 64×64, or 128×128. The CTU division of an image can (e.g., therefore) form a regular grid, where the top and left boundaries can spatially coincide with the top and left boundaries of the image.

[0153] like Figure 8 As shown in the example, each CTU can be separated into codec units according to the codec tree. The codec tree can consist of multiple (e.g., two) stages. The CTU can be partitioned (e.g., firstly) by a quadtree (e.g., or a quadtree / QT). For example, quadtree partitioning can divide the codec tree node corresponding to a square image block into four (4) nodes corresponding to four (4) equal-sized sub-blocks, as shown in the example. Figure 8 As shown by the solid line.

[0154] A tetrahedral leaf can (e.g., then) be split by a multi-type tree (MTT) (e.g., further), which can involve four (4) types of splitting (e.g., or splitting patterns), such as Figure 9 Examples are shown in the diagram. Separation types can be vertical and horizontal binary separation (BT) patterns, such as SPLIT_BT_VER and SPLIT_BT_HOR, and vertical and horizontal ternary separation (TT) patterns, SPLIT_TT_VER and SPLIT_TT_HOR. Binary separation can divide a block into two sub-blocks based on the separation orientation, the size of which can be half the size of the parent block. Ternary separation can divide a block into three (3) sub-blocks, where, in the considered separation orientation, the sizes can be 1 / 4, 1 / 2, and 1 / 4 of the parent block, respectively.

[0155] Figure 8 An example of CTU partitioning is shown. Figure 8 An example of CTU partitioning based on a codec tree according to a video codec scheme is shown.

[0156] Figure 9An example of the splitting patterns supported in multi-type tree splitting is shown.

[0157] For example, in cases where joint encoding and decoding are shared by luminance and chrominance components, the leaves of the CTU's codec tree can be codec units.

[0158] Intra-picture video coding can use separate codec trees. A separate codec tree can be used on one side for the luma component and on the other side for the chroma component. The luma component portion of a CTU can be referred to as a luma codec tree block. A luma codec tree block (CTB) can (e.g., then) be associated with a codec tree, where codec tree leaves can be associated with luma codec blocks. Intra-picture video coding can use separate luma / chroma codec trees and three (3) component pictures, where two chroma CTBs can share the same codec tree.

[0159] In some types of video codec schemes, the unit sizes (CU, PU, ​​and TU) may be equal. For example, codec units may not typically be divided into PUs or TUs unless in one or more (e.g., specific) codec modes.

[0160] In video encoding and decoding schemes, motion vectors can be used to represent inter-frame prediction information. For example, an encoding and decoding unit encoding and decoding in inter-frame mode may employ one or more (e.g., several) motion vectors, respectively assigned to each PU in the CU, as explained herein. The encoding and decoding of motion information can be performed, for example, according to an adaptive motion vector prediction (AMVP) mode and / or a merging mode. Figure 10 The image shows an example of encoding and decoding as inter-frame prediction information. For example... Figure 10 As shown, for example, there can be three (3) (e.g., main) modes for encoding and decoding inter-frame prediction parameters: skip-merge mode, non-skip-merge mode, and AMVP mode. For example, skip and merge modes can be signaled using two (2) dedicated flags. For example, AMVP mode can be enabled if / when both flags are false.

[0161] Figure 10 An example of signaling for inter-frame prediction information (e.g., based on a video codec scheme) is shown.

[0162] Table 1 provides a summary of examples of the differences between AMVP, merge, and skip modes for inter-frame codec CUs: Table 1 – Examples of differences between AMVP, merge, and skip modes in Inter-Frame Codec (CU) .

[0163] Merging indices allows for the export of prediction types (e.g., P or B images), reference image list indices, and / or associated motion vectors.

[0164] like Figure 10 As shown (e.g., in the AMVP mode of a video codec scheme), the reference images (e.g., up to two reference images) of the PUs considered for time prediction can (e.g., explicitly) be signaled, for example, having a motion vector associated with each reference image and each PU.

[0165] Motion vectors can be predicted and decoded. The encoder can select a motion vector prediction value (MVP) for (e.g., each) a reference image, and a signaling motion vector difference (MVD) relative to (e.g., each) the selected MVP. The motion data reconstructed on the decoder side can include the sum of the MVPs for a given PU and their associated MVDs.

[0166] An MVP can be selected from an AMVP candidate list, which may include (e.g., two) elements from each reference image. The index of the selected MVP can be signaled in the bitstream. For example, it can be based on... Figure 11 The example workflow shown is used to construct the MVP candidate list.

[0167] For example, if the inter-frame codec block exists in the corresponding spatial location, then candidate MVPs can be derived from the left adjacent positions A0 and A1, such as... Figure 12 As shown in the example, an MVP can be derived from the top adjacent block, and (e.g., then) a temporal MVP can be derived from a reference image at spatial location H (e.g., if available) or location C (e.g., if otherwise / unavailable). For example, the derived MVP can be scaled based on the temporal distance between the reference image associated with the MVP and the current reference image being considered.

[0168] Redundancy checks can be performed between exported spatial MVPs. For example, duplicate exported MVPs can be discarded.

[0169] The final AMVP candidate list can include (e.g., the first two) derived MVP candidates. For example, if fewer than two (2) MVP candidates are obtained, the AMVP candidate list can be completed with zero motion vectors. Figure 11 An example of MVP candidate list construction in the AMVP mode of a video codec mechanism is shown. Figure 12 Examples of adjacent spatial positions A0, A1 (left), B0, B1, B2 (top) of the current block and the corresponding blocks (H and C) of the TMVP are shown.

[0170] Merging modes can be implemented in video codec schemes. For example... Figure 10As shown in the example, motion information encoding / decoding according to the merging mode can be performed in two modes, such as skip mode and merge mode. The decoder can (e.g., be able to) retrieve the PU's motion information based on (e.g., in both modes) a trust order (e.g., a single) field (e.g., the merge index). The merge index indicates which motion vector prediction value (MVP) in the list of merged motion information prediction values ​​can be used to derive the motion information of the current PU. The list of motion information prediction values ​​can be referred to as the merge list or the merge candidate list. Candidate motion information prediction values ​​can be referred to as merge candidates.

[0171] Merging modes can be implemented in a video codec scheme. Merging modes may include deriving inter-frame prediction information (e.g., also referred to as motion information) for a given prediction unit from selected motion information prediction value candidates. The motion information considered may include (e.g., all) inter-frame prediction parameters of the PU, which may include one or more of, for example, unidirectional or bidirectional temporal prediction type; (e.g., reference picture index within a list of reference pictures for each); and / or motion vectors.

[0172] The merge candidate list can (e.g., systematically) consist of multiple (e.g., 5) merge candidates. Examples illustrate how the merge list can be constructed, for example, on the encoder side and the decoder side. One or more spatial locations (e.g., up to 5 spatial locations) can be considered to retrieve potential candidates. Spatial locations can be accessed in order, such as, for example, the following order: (1) left (A1); (2) top (B1); (3) top right (B0); (4) bottom left (A0); and (5) top left (B2).

[0173] Symbols A0, A1, B0, B1, and B2 can represent Figure 13 The example illustrates the spatial location. Spatial candidates can be selected, which may include relevant motion information and may differ from one another. For example, a temporal prediction (e.g., TMVP) can be selected by considering temporal motion information at location H. For example, if the reference image under consideration is unavailable, the "center" could be a candidate at location H. For example, a trimming process (e.g., as shown in the example) can be performed. Figure 14 (as shown in the example) to eliminate redundant candidates from the selected spatial and temporal candidate sets.

[0174] like Figure 14As shown, for example in the case of slice B, another type of candidate (e.g., a combined candidate type) can be pushed onto the merge list (e.g., if the merge list is not full). Combined candidate types can be formed, for example by forming a candidate consisting of motion information associated with a (e.g., one) list of reference images (L0) from a candidate already existing in the merge list (e.g., one), and motion associated with another list of reference images (L1) from another candidate already existing in the merge list.

[0175] For example, if the merge list is still not full (e.g., has five (5) elements), the zero motion vector can be pushed to the back of the merge list until it is full.

[0176] Figure 13 An example is shown illustrating the location of spatial and temporal motion vector predictions used in the merging mode of a video codec scheme. For example... Figure 13 As shown, spatial merging candidates are shown on the left, while temporal merging candidates are shown on the right. Figure 14 An example of constructing a candidate list of merged motion vector prediction values ​​for a video codec scheme is shown.

[0177] Figure 15 The diagram illustrates an example of the process (e.g., the overall process) for constructing a merge list (e.g., in a video codec scheme). Figure 15 An example of constructing a list of candidates for merging motion vector prediction values ​​is shown (e.g., in a video codec scheme).

[0178] Inter-frame prediction information can be represented and encoded / decoded in video codecs, for example, within a video codec scheme. For instance, motion data representation might be richer in one video codec scheme than another. Motion data representations can be categorized (e.g., two main categories), such as global block-based motion representations and sub-block-based motion representations, as... Figure 16 As shown in the example. One or more (e.g., two) modes for encoding and decoding motion information can be used in (e.g., each) categories. Modes for encoding and decoding information can include, for example, merge / skip and AMVP.

[0179] Figure 16 An example of motion representation categories based on overall blocks and sub-blocks is shown.

[0180] Holistic block-based motion representation can include assigning groups (e.g., a set) of motion information to inter-frame blocks. A set of motion information can consist of one or two motion vectors and an associated reference image. The block's motion information can be represented in the form of a holistic block (e.g., a single) motion vector.

[0181] Sub-block-based motion coding modes divide a block into sub-blocks (e.g., 4×4 or 8×8 luma sample sub-blocks). A separate set of motion information can be assigned to (e.g., each) a sub-block.

[0182] Motion representation and encoding / decoding based on whole blocks can be implemented in video encoding / decoding schemes.

[0183] The AMVP mode based on whole blocks can be implemented in certain video codec schemes. For example, the AMVP mode can (e.g., explicitly) signal as motion vectors of the MV difference of a selected MVP relative to a given reference picture, and as reference picture indices that identify the reference picture associated with the motion vectors of the codec.

[0184] The AMVP motion vector prediction (MVP) candidate list can include multiple (e.g., two) elements. One or more of the following elements can be used to construct the AMVP MVP candidate list on the encoder and decoder sides: multiple (e.g., up to four (4)) spatial candidates; (e.g., up to one (1)) temporal MVP candidates; (up to four (4)) history-based motion vector prediction (HMVP) candidates; and / or zero motion vectors, for example, if two (2) MVP candidates are required in the final list.

[0185] HMVP can use previously encoded MVs as the MVs associated with neighboring or non-neighboring blocks relative to the current block. An HMVP candidate table can be maintained on both the encoder and decoder sides. The HMVP table can be updated dynamically, for example, as a first-in-first-out (FIFO) buffer for MVPs. In the example, there may be up to five candidates in the HMVP table. For example, after encoding and decoding an inter-prediction block that is not in sub-block mode (e.g., including affine mode) or geometric segmentation mode (GPM), the table can be updated by appending the associated motion information to the end of the table as a new HMVP candidate. FIFO rules can be applied to manage the table. For example, redundant candidates in the HMVP table can be removed instead of the first candidate. For example, the table can be reset at (e.g., each) CTU row, for example, to enable parallel processing.

[0186] The global block-based AMVP motion codec mode can employ one or more of the following (e.g., in addition to the AMVP candidate list): Symmetric Motion Vector Difference (SMVD); Adaptive Motion Vector Resolution (AMVR); and / or Bi-Prediction (BCW) with codec unit weights.

[0187] SMVD can include setting the MVD associated with reference picture list 1 (L1) for a given block to be equal to the opposite of the MVD associated with reference picture list 0 (L0). The reference pictures used in SMVD mode can be derived by the decoder, for example, using predefined rules. SMVD can reduce the rate cost of encoding and decoding MVD information. SMVD can be selected at the block level.

[0188] AMVR can allow / enable signaling MVD at quarter-pixel, half-pixel, integer-pixel, or 4-pixel luminance sample resolutions, which allows / enables bit saving in the encoding and decoding of MVD information. Motion vector resolution can be selected at the block level (e.g., in AMVR).

[0189] BCW can enable dual prediction of blocks with unequal weights. BCW signaling can be performed at the block (e.g., CU) level.

[0190] For example, an internal motion vector representation (e.g., in a video codec) can be implemented with 1 / 16 luminance sampling accuracy instead of 1 / 4 luminance sampling accuracy (e.g., in a video codec mechanism).

[0191] Figure 17A and 17B An example of a block-based (e.g., or non-sub-block-based) merge list construction process (e.g., for video codec schemes) is shown. Block-based merging modes (e.g., in video codec mechanisms) may also be referred to as regular merging modes. For example, the construction of the MVP candidate list for block-based merging modes may differ across various video codec mechanisms. Block-based merging modes can have multiple merging codec modes, which may include one or more of the following: Merging mode with MV difference (MMVD); Geometric Partitioning (GPM); and / or Combined Intra / Inter-Frame Prediction (CIIP).

[0192] The merged MVP candidate list consists of one or more MVP candidates of the following types: spatial candidate; temporal MVP candidate; HMVP candidate; pairwise average candidate; and / or zero MV candidate.

[0193] Spatial candidates in one video codec scheme may be similar to spatial candidates in another video codec mechanism, for example, except that the first two candidates can be swapped. Temporal MVP candidates in one video codec scheme may be similar to those in another video codec scheme. For example, HMVP candidates can be inserted into the merge list such that the merge list reaches the maximum allowed number of MVP candidates minus one (1). Pairwise averaged candidates (e.g., up to one pairwise averaged candidate) can be added to the merged candidate list. For example, pairwise candidates can be calculated as follows: The first two MVP candidates present in the list can be considered. The motion vectors of the first two MVP candidates present in the list can be averaged. The average can be calculated separately for each list of reference images. For example, if both MVPs are bidirectional, the motion vectors associated with both lists L0 and L1 can be averaged. If only one motion vector exists in the list of reference images, that motion vector can be used as is to form the pairwise candidate.

[0194] Figure 18 An example of permitted motion vector difference (MVD) is shown. The MMVD merging mode allows / enables encoding and decoding of a limited number of motion vector differences (MVD) over selected merged MVP candidates, for example, to represent motion information of the CU. MMVD encoding and decoding can be limited to four (4) vector directions and eight (8) magnitudes, for example, from 1 / 4 lumen sample to 32 lumen sample. MMVD can provide an intermediate level of accuracy, which can produce a middle trade-off between rate cost and MV accuracy to signal motion information.

[0195] Figure 19 An example representation of CU motion data in GPM mode is shown. GPM merging mode (e.g., in video codec schemes) can support inter-frame prediction. In the example, inter-frame CUs can be segmented, for example, along a straight line into (e.g., two) motion partitions, such as... Figure 19 As shown in the example, partitions can be non-rectangular. For example, if they are rectangular, asymmetric partitioning can be performed, which avoids the redundancy of binary partitioning at the CU level.

[0196] Figure 20 An example of GPM separation grouped at the same angle is shown. GPMs are signaled using a specific merging pattern, for example, a CU-level flag. The orientation and / or location of the separation line relative to the CU center can be signaled (e.g., at the CU level) via a (e.g., dedicated) GPM index. In the example, for in For example, excluding 8×64 and 64×8 (e.g., every possible) CU sizes, the geometric partitioning mode can support a total of 64 partitions. Figure 20 Various examples of achievable separation lines are shown, for example, at different offsets from the center of the CU, and for various separation line angles.

[0197] Each (e.g., each) geometric partition in a CU can use its own motion for inter-frame prediction. For each (e.g., each) partition, single prediction (e.g., single prediction only) may be allowed. Each (e.g., each) partition may have a motion vector and a reference image index. For example, a single-prediction motion constraint can be applied to limit the number of motion-compensated predictions per CU (e.g., only two), which is similar to regular dual prediction.

[0198] (For example, the motion vector of each) partition can be derived from (for example, up to two) merge indices used separately for (for example, each) partition, which can be similar to a regular whole-block-based merge pattern. Figure 21 An example of blending between two prediction partitions is shown in GPM. For instance, after predicting each part of a geometric partition, a blending process with adaptive weights can be used to adjust sample values ​​along the edges of the geometric partition (e.g., as shown in the image). Figure 21 (As shown in the example). This process can form the prediction signal for the entire CU. For example, as in other prediction modes, the transformation and quantization process can be applied to the entire CU (e.g., not to each partition).

[0199] CIIP merging modes may include combining inter-frame prediction signals with intra-frame prediction signals to predict the current CU. For example, the inter-frame prediction signal in CIIP mode can be derived using the same inter-frame prediction procedure applied to the regular merging mode. The intra-frame prediction signal can be derived after the regular intra-frame prediction procedure using the planar mode. For example, a weighted average can be used to combine the intra-frame and inter-frame prediction signals. For example, weight values ​​can be calculated based on the encoding / decoding modes of the top and left adjacent blocks. The weight values ​​can be calculated, for example, according to equation (1): Weight and The sum can be equal to a constant, for example, four (4). Weights and It can be constant throughout the CU.

[0200] For example, motion based on sub-blocks can be represented and encoded / decoded in a video encoding / decoding scheme. Figure 22 Examples of affine motion models based on control points are shown (e.g., supported by a video codec scheme). A 4-parameter affine model is shown on the left. A 6-parameter affine model is shown on the right.

[0201] Affine motion compensation can be performed. In video codec schemes, translational motion models (e.g., translation-only models) can be applied to motion-compensated temporal prediction (MCP). Translational motion may not (e.g., be able to) capture certain types of motion, such as zoom in, zoom out, rotation, perspective motion, and / or irregular motion. In video codec schemes, sub-block-based affine motion compensation prediction can be used at the CU level. Figure 22 As shown in the example, the affine motion field of a block can be described by motion information from (e.g., two) control point motion vectors (e.g., a 4-parameter affine motion model) or (e.g., three) control point motion vectors (e.g., a 6-parameter affine motion model). Figure 22 As shown, vector These can be control point motion vectors (CPMVs) associated with the block. These vectors can be used to represent the affine motion field of the block under consideration.

[0202] The 4-parameter affine motion model can, for example, derive the motion vector at the sample position (x, y) in the block from the 4-parameter affine motion field calculation shown in equation (2): The 6-parameter affine motion model can, for example, derive the motion vector at the sample position (x, y) in the block from the 6-parameter affine motion field shown in equation (3): Where (mv0x, mv0y) can be the motion vector of the top left control point, (mv1x, mv1y) can be the motion vector of the top right control point, and (mv2x, mv2y) can be the motion vector of the bottom left control point.

[0203] Figure 23 An example of an affine motion field representation based on 4×4 sub-blocks is shown. For example, affine motion compensation is performed on the basis of 4×4 sub-blocks. (e.g., the motion vector of each) 4×4 luminance sub-block can be derived, for example, by calculating the motion vector of the center sample of each sub-block according to equation (2) or equation (3) (e.g., as...). Figure 23 (As shown in the example). The calculated motion vector can be rounded to, for example, 1 / 16 fractional accuracy. A motion-compensated interpolation filter can (e.g., then) be applied to generate (e.g., each) a prediction for a sub-block using the derived motion vector. The sub-block size in the chroma component can (e.g., also) be, for example, 4×4. The MV of a 4×4 chroma sub-block can be calculated as the average of the MVs of the top-left and bottom-right luminance sub-blocks in the corresponding 8×8 luminance region.

[0204] Translational motion inter-frame prediction can achieve one or more (e.g., two main) affine inter-frame prediction modes, such as affine AMVP mode and / or affine merging mode.

[0205] Affine merging patterns can be sub-block-based motion coding and decoding patterns within sub-block merging patterns. For example, affine merging patterns can be applied to CUs based on width and / or height (e.g., if both width and height are greater than or equal to eight (8)). The CPMV of the current CU can be generated using affine merging patterns based on motion information of spatially adjacent CUs. There may be one or more (e.g., up to five) Control Point Motion Vector Predictions (CPMVP) candidates. Signaling indices can be used to indicate the candidates to be used for the current CU. One or more of the following three types of CPVM candidates can be used to form the list of affine merging candidates: inherited affine merging candidates extrapolated from the CPMVs of adjacent CUs; ​​CPMVPs that construct affine merging candidates, which can be derived using translational MVs of adjacent CUs; ​​and / or zero MVs.

[0206] There may be a maximum number (e.g., two) of inherited affine candidates that can be derived from the affine motion model of adjacent blocks, such as one from the left-adjacent CU and one from the top-adjacent CU.

[0207] Figure 24 The example in the image shows a candidate block. Figure 24 Examples of locations for inheriting affine motion predictions are shown. For example, for a left-side prediction, the scan order could be A0->A1. For example, for an upper-side prediction, the scan order could be B0->B1->B2. A first inheritance candidate from one side (e.g., each side) can be selected (e.g., only the first inheritance candidate from each side). For example, if / when a neighboring affine CU is identified, the control point motion vectors of the neighboring affine CUs can be used to derive CPMVP candidates from the affine merging list of the current CU.

[0208] Figure 25 An example of control point motion vector inheritance is shown. For example... Figure 25 As shown, for example, if the adjacent lower-left block A is encoded and decoded in affine mode, the motion vectors v2, v3, and v4 of the upper-left, upper-right, and lower-left corners of the CU including block A can be obtained. The two CPMVs of the current CU can be calculated based on v2 and v3, for example, if / when block A is encoded and decoded using a 4-parameter affine model. For example, if block A is encoded and decoded using a 6-parameter affine model, the three CPMVs of the current CU can be calculated based on v2, v3, and v4.

[0209] Figure 26 An example of localization for constructing candidate locations for an affine merging pattern is shown. Affine candidates can be constructed by combining the translational motion information of each control point's neighboring neighbors. The motion information of the control points can be derived from (e.g., specified) spatial and temporal neighbors, for example, as... Figure 26The example is shown in the diagram. CPMVk (k=1, 2, 3, 4) can represent the k-th control point. For CPMV1, the B2->B3->A2 block can be checked. The MV of the first available block can be used. For CPMV2, the B1->B0 block can be checked. For CPMV3, the A1->A0 block can be checked. For example, TMVP can be used as CPMV4 if possible.

[0210] For example, after obtaining the MVs of four control points, affine merging candidates can be constructed based on motion information. One or more of the following combinations of control point MVs can be used to generate affine merging candidates (e.g., in the following order): {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}.

[0211] For example, control point motion vectors {CPMV1, CPMV2, CPMV3} can be used. Control point motion vectors {CPMV1, CPMV2, CPMV3} can be used to generate the affine motion field of the CU (e.g., following equation (3)).

[0212] Combinations of three (3) CPMVs can construct 6-parameter affine merge candidates. Combinations of two (2) CPMVs can construct 4-parameter affine merge candidates. For example, if the reference indices of the control points are different, related combinations of control point MVs can be discarded, which can avoid motion scaling when the CPMVs point to different reference images.

[0213] For example, if the list is still not full, after considering appending inherited affine merge candidates and constructed affine merge candidates to the affine merge candidate list, zero MV can be inserted at the end of the list.

[0214] Affine AMVP mode can be applied to the CU based on its width and / or height (e.g., both width and height are greater than or equal to 16). Affine flags can be signaled in the bitstream (e.g., at the CU level) to indicate the use of the affine AMVP mode. Another flag can signal whether a 4-parameter or 6-parameter affine model is used. For example, the difference between the current CU's CPMV and its predicted CPMVP can be encoded and decoded in affine AMVP mode.

[0215] The CPMVP used to predict the CPMV of a CU can be taken from an affine AMVP candidate list, which can consist of two (2) elements. For example, the affine AMVP candidate list can be constructed using one or more CPVM candidates of the following types (e.g., in the following order): inherited affine AMVP candidates extrapolated from the CPMV of neighboring CUs; ​​affine AMVP candidate CPMVPs constructed using translational MVs of neighboring CUs; ​​translational MVs from neighboring CUs; ​​and / or zero MVs.

[0216] Item checks can be used. Checking potential candidates can include checking whether valid affine AMVP or affine merge candidates that predict the current CU for the affine CPMV are available and / or valid. Available and valid candidates can be added to the candidate list being constructed.

[0217] The checking order for inherited affine AMVP candidates can be the same as or similar to the checking order for inherited affine merged candidates. Differences (e.g., unique differences) can be (e.g., only) affine CUs with the same reference image as those in the current block that can be considered for AVMP candidates. For example, a pruning process may not be applied if / when inherited affine motion predictions are inserted into the candidate list.

[0218] Affine AMVP candidates can be constructed from (e.g., specified) spatial neighbors, such as... Figure 26 The example shown is illustrated. The inspection order used can be, for example, the same as the inspection order in the affine merge candidate construction. The reference picture indexes of adjacent blocks can also be checked, for example. The block that can be used can be the first block in the inspection order, which is inter-frame encoded and has the same reference picture as the current CU.

[0219] For example, if / when the current CU is encoded / decoded using a 4-parameter affine mode, and and If both are available, then MV and Three CPMVs can be added as candidates to the affine AMVP list. For example, if / when the current CU is encoded / decoded in a 6-parameter affine mode and three CPMVs are available, the three CPMVs can be added as candidates to the affine AMVP list. Otherwise, constructing AMVP candidates can be set to unavailable.

[0220] For example, if / when available, for instance, if after inserting valid inherited affine AMVP candidates and constructing AMVP candidates, the list of candidate affine AMVPs is still less than two (2), then MVs can be added sequentially, for example. , and As a translation MV, to predict the (e.g., all) control point MVs of the current CU. If the affine AMVP list is still not full, then Z zero MVs can (e.g., then) be used to fill the affine list.

[0221] Sub-block merging / skipping patterns can be merging patterns using a merging candidate list with (e.g., only) sub-block-based motion candidates (e.g., up to five (5)) elements. A merging index (e.g., for regular merging) can indicate the sub-block-based merging candidate used to derive motion data for the CU. The sub-block-based merging candidate list can consist of, for example, the following elements. A sub-block-based temporal motion vector prediction (SbTMVP) candidate can be placed first. Affine merging candidates can (e.g., then) be added to the list. For example, the sub-block merging list can be constructed using one or more of the following candidate lists: SbTMVP; inherited affine merging candidates; constructed affine merging candidate CPMVP, which can be derived using translation MV of adjacent CUs; ​​and / or zero MV.

[0222] Sub-block-based temporal motion vector prediction (SbTMVP) can utilize motion fields from co-located images, which can merge candidates similarly to TMVP rules. SbTMVP may differ from TMVP in one or more of the following ways: TMVP predicts motion at the CU level. SbTMVP predicts motion at the sub-CU level. TMVP can obtain temporal motion vectors from co-located blocks in a co-located image. SbTMVP can apply motion shifting before obtaining temporal motion information from the co-located image. Motion shifting can be obtained from motion vectors from spatially neighboring blocks of the current CU.

[0223] The modes used for motion representation of inter-frame prediction data (e.g., in a video codec scheme) may include one or more of the following: non-nearest neighbor spatial merge candidates (e.g., non-nearest neighbor spatial MV prediction (NASMVP) candidates), template matching (TM), adaptive reordering of merge candidates with template matching (ARMC-TM), multiple hypothesis prediction (MHP), GPM with MMVD, bidirectional matching AMVP merge mode (BM-AMVP merge), bidirectional matching MV refinement, support for AMVP merge mode in low-latency configurations, and / or support for MVP merge mode in affine motion representation modes.

[0224] Figure 27 Examples of spatially adjacent and non-nearest block locations that can be used to derive spatial merge candidates are shown. For example, non-nearest spatial merge candidates, such as Non-Nearest Spatial MV Predictions (NASMVP) candidates, can be added to the regular merge list, for instance, after TMVP merge candidates. The spatial locations of these merge candidates are determined by… Figure 27 The example shown is in the image.

[0225] Figure 28 An example of template matching (TM) over a search region surrounding the initial motion vector (MV) is shown. TM can be a decoder-side motion vector refinement method. TM can refine the motion vector of the CU, for example, by matching template regions above and to the left of the current CU with templates in the search region of the reference image. Figure 28 As shown in the example, a better MV can be searched around the initial MV of the current CU within the search range (e.g., within the [-8,+8] pixel search range).

[0226] For example, a refined MV can be obtained by minimizing the template matching cost between the templates around the current CU and the candidate templates in the reference image.

[0227] Template matching (TM) techniques can (e.g., also) be used to identify one or more (e.g., some) MVP candidates in a regular AMVP and / or regular merge pattern. For example (e.g., in AMVP), MVP candidates can be identified based on template matching error. MVP candidates with the lowest TM cost can be selected and further refined using TM.

[0228] Adaptive reordering of merge candidates with template matching (ARMC-TM) can be achieved. For example, in regular merge mode, affine merge mode, and / or template matching merge mode (e.g., merge mode with TMP refinement can be applied), merge candidates can be reordered based on their TM cost. The TM cost can be calculated as, for example, the sum of absolute differences (SAD) between samples of the template of the current block and their corresponding reference samples.

[0229] Multiple hypothesis prediction (MHP) can be implemented. In the MHP inter-frame prediction mode, one or more additional motion-compensated prediction signals can be signaled (e.g., in addition to the regular bi prediction signal). For example, the final total prediction signal can be obtained by weighted superposition of samples. For example, the resulting prediction signal can be obtained according to equation (4). : in It can represent the bi prediction signal, and This can represent the first additional inter-frame prediction signal / hypothesis.

[0230] The motion parameters for each additional prediction hypothesis can be signaled, for example, explicitly by specifying a reference index, a motion vector prediction index, and a motion vector difference, or implicitly by specifying a merge index. Individual multi-hypothesis merging flags can distinguish between signaling modes.

[0231] GPM with MMVD can be implemented. For example, GPM can be implemented by applying motion vector refinement to an existing GPM unidirectional MV. Signaling flags can be (e.g., firstly) assigned to the GPM CU to specify whether this mode is used. The (e.g., each) geometric partition of the GPM CU can determine whether to signal MVD, e.g., if / when this mode is used. For example, after a GPM merge candidate is selected, e.g., if signaling MVD is applied to the geometric partition, the motion of the partition can (e.g., further) be refined by information from the signaling MVD.

[0232] GPM with TM can be implemented. Template matching can be applied to GPM. For example, if / when GPM mode is enabled for CU, a CU-level flag can be signaled to indicate whether TM is applied to two geometric partitions. TM can be used to refine (e.g., each) the motion information of geometric partitions. For example, if / when TM is selected, a template can be constructed using neighboring samples on the left, top, or upper left, based on the segmentation angle.

[0233] It can implement bidirectional matching AMVP merging patterns (BM-AMVP merging). BM-AMVP merging patterns can represent CU motion data. BM-AMVP merging patterns can include AMVP motion predictions in an inter-frame direction associated with (e.g., one) a list of reference images, and merging patterns in the opposite direction associated with (e.g., another) a list of reference images.

[0234] For example, this MV refinement mode can be allowed if / when the selected merged forecast and AMVP forecast make at least one reference picture available in the past and at least one reference picture available in the future.

[0235] The AMVP portion of the pattern can be signaled, for example, as a regular one-way AMVP. For instance, the reference index and MVD can be signaled together, for example, with the derived MVP index, if template matching is used. For example, the MVP index can be signaled if / when template matching is disabled.

[0236] The AMVP direction (e.g., the reference image list) LX can include X, which can be zero (0) or one (1). The merging portion in the other direction (1-LX) can be (e.g., implicitly) derived, for example, by minimizing the bidirectional matching (BM) cost between the AMVP prediction and the merge prediction, for example, for a pair of AMVP and merge motion vectors. For example, the bidirectional matching cost can be computed using the merge candidate MV and AMVPMV of each merge candidate that may have motion vectors in the other direction (1-LX) from the merge candidate list. The merge candidate with the minimum cost can be selected. Bidirectional matching refinement can be applied to the coded block starting from the selected merge candidate MV and AMVP MV.

[0237] Figure 29 An example of bidirectional matching (BM) is shown. BM can involve bidirectional matching MV refinement. The bidirectional prediction operation can involve searching for a refined MV around (e.g., two) initial MVs (MV0 and MV1) in a list of reference images L0 and L1, such as... Figure 29 As illustrated in the example, a refined MV can be derived around the initial MV based on the minimum bidirectional matching cost between (e.g., two) reference blocks in L0 and L1.

[0238] BM can perform local searches to derive integer sample precision, such as intDeltaMV. For example, a local search can apply a 3×3 square search pattern to iterate through the horizontal search range [-sHor,sHor] and the vertical search range [-sVer,sVer].

[0239] The cost of bidirectional matching can be calculated, for example, according to equation (5): Here, `mvDistanceCost` can represent the cost of motion vector encoding / decoding, which can increase as a function of the MV value. For example, the local search of `intDeltaMV` can be terminated if / when (e.g., 3×3) the `bilCost` at the center point of the search pattern has the minimum cost. The current minimum cost search point can (e.g., otherwise) become the new center point of the search pattern (e.g., 3×3), and the search for the minimum cost continues until the end of the search range is reached, for example, if / when (e.g., 3×3) the `bilCost` at the center point of the search pattern does not have the minimum cost.

[0240] The final deltaMV can be derived by applying existing fractional sample refinement (e.g., further). The refined MV after the first pass can be derived (e.g., then) for example, according to equation (6): .

[0241] AMVP merge mode can be supported in low-latency configurations. Figure 30 An example of enabling AMVP merging for dual-prediction blocks with two reference images, one from the past and one from the future, is shown.

[0242] The BM-AMVP merge pattern (also known as the MVP merge pattern) can be used for B-images, for example, which have at least one past reference image and at least one future reference image, for example, in terms of image display order.

[0243] AMVP merging mode can (for example, also) be used in low-latency images. For instance, if both reference images of an inter-frame encoded image are in the past of the current image, such as having a picture order count (POC) or picture index in the display order that is lower than the current image's POC, then the inter-frame encoded image can be low-latency.

[0244] For example, AMVP merging can be used for low-latency images by removing bidirectional matching reordering and refinement from the AMVP merging prediction pattern.

[0245] For example, even in low-latency cases where the reference image pair is not a true bidirectional image, AMVP merging (e.g., amvpMerge) can be enabled for dual prediction blocks. For low-latency images, the restrictions on long-term and WP reference images for resampling can be relaxed.

[0246] For example, by setting the BM cost of amvpMerge candidates that are not truly bidirectional during merge candidate reordering, bidirectional matching reordering can be deactivated in low-latency blocks (e.g., in blocks that have past or future (2) reference pictures).

[0247] AMVP merging mode can be supported in affine motion representation mode. The AMVP merging mode for affine blocks can signal the reference image list index, reference image index, and / or MVP index to indicate the AMVP prediction value. For example, merge candidates can be derived by using ARMC reordering. For example, if / when the block size is equal to or greater than 8×8 and / or the current image is not a low-latency image, a signaling flag can be used to indicate whether the AMVP merged block is an affine codec block.

[0248] Enriching motion data representations (e.g., in ECM) can improve compression performance. For example, affine AMVP merging patterns for affine motion can be suitable for low-latency scenarios. Motion model representations can be extended to support other combinations of motion models, which may differ from AMVP merging in affine and / or non-affine modes. AMVP merging patterns can be extended to a richer set of ways to represent motion representations in inter-frame codec blocks.

[0249] Affine AMVP merging motion representation can be enabled for low-latency blocks and images. The affine AMVP merging mode can be enabled for low latency in a manner similar to how the non-affine AMVP merging mode is supported for low latency.

[0250] This paper describes CU motion parsing and CU motion derivation in the BM-AMVP merge mode. Specifically, it describes the parsing and reconstruction of CU motion data in the case of CTU encoding / decoding within the BM-AMVP merge.

[0251] Figure 31 and Figure 32 An example of parsing motion information associated with a codec unit (CU) is described, for example, if / when the CU is in AMVP merge mode. Figure 31 Examples of motion data parsing and information parsing in AMVP merge mode are shown. Figure 32 An example is shown illustrating the details of parsing the codec data related to the BM-AMVP merge mode.

[0252] Figure 31 The input to the process can be an encoding / decoding CU in AMVP mode. For example... Figure 31 As shown, for example, if the CU is not in AMVP mode, then Figure 31 The parsing process may not occur. It is possible to parse the BM-AMVP merge mode information present in the encoded / decoded bitstream. Figure 32 Detailed description Figure 31 The analysis is shown below.

[0253] like Figure 32 As shown, it can be determined whether the conditions for applying the AMVP merge mode to the current CU are met. For example, the condition could be that the current slice has at least two reference images with different images, such as in the case of a low-latency slice. For example, for a non-low-latency slice, the condition could be that the slice has at least one past reference image and one future reference image. The conditions can enable the use of the AMVP merge mode for both low-latency and non-low-latency slices. For example, if the condition is true, the AMVP merge mode can be enabled.

[0254] Flags indicating the use of AMVP merge mode for the current CU can be parsed. The values ​​of the decoding flags can be checked. For example, if the value of the decoding flag is false, the flags amvpMergeModeFlag[L0] and amvpMergeModeFlag[L1] can be set to false. Reference Figure 31 and Figure 32The operations shown indicate that the flags amvpMergeModeFlag[L0] and amvpMergeModeFlag[L1] can indicate that the merge mode is used for inter-frame prediction of the current CU from reference picture lists L0 and L1, respectively. Setting the flags to false indicates that AMVP mode is used for both reference picture lists.

[0255] For example, if the AMVP merge mode flag is true, the merge mode can be resolved for the inter-frame orientation of the current CU, such as the reference picture list. For example, the flag amvpMergeModeFlag[mergeDir] can be set to true, while the flag amvpMergeModeFlag[1-mergeDir] can be set to false to indicate that the merge mode is used for the reference picture list corresponding to mergeDir.

[0256] Back Figure 31 This process determines whether the current slice is a low-latency type, for example, whether the current slice has two reference images from the past. For example, if the current slice is not a low-latency type, the `affine_flag` syntax element can be parsed. Parsing can indicate whether the AMVP merging pattern is used for affine motion modeling.

[0257] For low-latency slices, affine AMVP merge mode may be allowed (e.g., only allowed).

[0258] Figure 31 The remaining operations shown may include resolving the AMVP inter-frame prediction data of the current CU in the inter-frame direction using an AMVP mode (e.g., instead of a merge mode). The flag amvpMergeModeFlag[L0] can be checked. For example, if the flag amvpMergeModeFlag[L0] is false, then AMVP can be used for the reference picture list L0. The reference picture index and motion vector difference of the current CU can be (e.g., then) resolved from the bitstream of L0. For example, if, for example, TM-based motion vector prediction values ​​are not allowed to be derived at the video sequence level, the motion vector prediction index MvpIdx[L0] can be resolved for L0.

[0259] For example, the flag amvpMergeModeFlag[L1] can be checked in the same way as the flag amvpMergeModeFlag[L0]. For example, if AMVP is enabled for the reference image list L1, AMVP prediction data can be parsed for L1 (e.g., similarly). Figure 31 The example process shown can be completed.

[0260] Figure 33 An example of motion data reconstruction and export in the BM-AMVP merging pattern is described. Figure 33 The diagram illustrates the motion data export process of a codec unit (CU) encoded and decoded in BM-AMVP merging mode on the decoder side. Motion export reconstructs (e.g., all) inter-frame prediction information associated with the CU (e.g., a list of reference images and motion vectors).

[0261] like Figure 33 As shown, this determines whether the CU is encoded / decoded in AMVP mode. It can also determine whether the CU is encoded / decoded in BM-AMVP merge mode. In the example, for instance, if the CU is not encoded / decoded in both AMVP and BM-AMVP merge modes, then... Figure 33 Motion export may not occur.

[0262] For example, if the CU is determined to be in BM-AMVP merge mode, the refListMerge and refListAMVP indices can be assigned to the reference picture list indices to which Merge and AMVP can be used, respectively. The reference picture indices and motion vector differences can be set equal to their resolved corresponding values ​​(e.g., for AMVP inter-frame orientation). For example, if TM-based MVP orientation is allowed, motion vector predictions for AMVP mode can be derived. For example, motion vector predictions for AMVP mode can be derived based on template matching. For example, the decoder can select the MVP with the lowest TM cost. For example, if the opposite is true, the MVP index can be set equal to its resolved value.

[0263] Figure 33 The remaining operations shown may include exporting merged prediction data associated with the inter-frame direction refListMerge. A list of merged MV prediction candidates can be constructed, for example, dedicated to the reference image list refListMerge. The construction of the merge list can be similar to other merge candidate list constructions (e.g., as described here). The difference may be performing tests to check whether the AMVP merge mode is allowed for the relevant candidates for each potential merge candidate to be added (e.g., each). The tests may depend on the affine_flag associated with the current CU (e.g., in...). Figure 31 (As parsed in the text). The test can check whether the Proof-of-Concept (POC) of the candidate merge candidate and the reference image of the current CU's AMVP prediction part are different, for example, whether the affine_flag is false. For example, if the affine_flag is true, the condition that a merge candidate is permissible in AMVP merge mode is that its reference image is in the opposite direction compared to the AMVP part of the current CU. In the affine case, for example, if in (e.g., two) related reference images, one is in the past and the other is in the future relative to the current image, then AMVP merge mode can be allowed.

[0264] A bidirectional matching mechanism can be used to select merge candidates from the list, for example, for the current CU. For instance, merge candidates that result in the minimum bidirectional matching cost can be considered. A bidirectional matching cost can be calculated for each candidate with two predicted values. For the AMVP and Merge-related reference image lists, bidirectional matching costs can be created separately for the AMVP motion vector predicted values ​​and for each candidate merged MVP.

[0265] The best merging candidate with respect to BM cost can be found. The motion information of the current CU can be obtained (e.g., then). The motion information of the current CU can be refined according to the BM motion vector refinement process, which refines the motion vector in both directions according to the BM cost.

[0266] The affine AMVP merge prediction pattern can be used for low-latency blocks (prediction units). The construction of the merge candidate list can be modified (e.g., as shown below). Figure 33 (As shown). For example, the test can be modified to check / determine whether affine merge candidates are acceptable for AMVP merge prediction patterns. In the example, the affine flag can be skipped when determining whether an affine merge candidate is acceptable for AMVP merge prediction. In the example, the POC value associated with the reference image related to the AMVP and merge portion of the candidate under consideration can be determined. It can be confirmed that the two POC values ​​are different. The merge candidate permission policy for the affine case can be aligned with the candidate permission policy for the non-affine case.

[0267] In the example, for affine AMVP merging motion representation patterns, the bidirectional matching-based reordering of affine merging candidates during the construction of the affine merging candidate list can be skipped. For example, during the construction of the affine AMP merging pattern list, bidirectional matching-based motion refinement in the affine AMVP merging pattern can also be skipped.

[0268] In the example, for instance, where bidirectional matching is possible, a bidirectional matching-based refinement of the affine merging candidates can be performed on the merging candidates. This bidirectional matching-based refinement of the affine merging candidates can be skipped (e.g., only) for low-latency affine merging candidates. For example, bidirectional matching refinement of the affine merging candidates can be performed if the candidate under consideration has a reference picture of its future (or past), and the AMVP portion of the considered AMVP merging CU (or PU) has a reference picture of its past (or future).

[0269] In the example, during the construction of the affine merge candidate list for the affine AMVP merge pattern, a partial sorting of the affine merge candidates can be performed, for example (only) among the non-low-latency merge candidates. The position of the low-latency candidates relative to the non-low-latency candidates in the merge list can remain unchanged.

[0270] In the example, in addition to the low-latency case, the affine AMVP merging extension associated with the low-latency case extends the AMVP merging motion representation pattern to (e.g., all) unidirectional and / or bi-prediction cases. If a given block has two reference images in the future of the current image, the block can be identified as unidirectional, rather than low-latency. In this case, both the AMVP merging pattern and the affine AMVP merging pattern can be allowed to represent the motion information of the block under consideration. The AMVP merging and affine AMVP merging patterns supporting unidirectional blocks can be implemented in a similar manner to those proposed in this paper for the low-latency case.

[0271] In the example, in addition to the affine motion representation case, an AMVP merging pattern can be supported for each (e.g., all) sub-block-based motion representation case. The merging portion of the affine AMVP merging motion representation pattern can include the construction of a sub-block merging candidate list. The merging candidate list can include affine merging candidates and / or at least one SbTMVP merging candidate.

[0272] The AMVP merge motion representation pattern can be extended to new merge affine merge motion representation patterns.

[0273] Alternatively, the AMVP merge motion representation pattern can be extended to a new merge subblock merge motion representation pattern.

[0274] For example, the AMVP merging concept can be extended using motion representation patterns, which can be called merging affine merging patterns. Merging affine merging patterns can include bipredictions for a given PU, which can mix affine and non-affine merging patterns.

[0275] For example, the AMVP merging concept can be extended using motion representation patterns, previously referred to as merge sub-block merging patterns. Merge sub-block merging patterns can include bipredictions for a given PU, which can mix sub-block merging and global block-based merging patterns.

[0276] In the following description, both variants are specified in the general terminology merging affine merging motion representation mode.

[0277] The PU corresponding to a dual prediction block may include, for example, a merge candidate in one inter-frame direction associated with a list of reference images (L0 or L1), and an affine merge candidate (or a sub-block-based merge candidate) in the opposite inter-frame direction associated with another list of reference images (L1 or L0).

[0278] For example, signaling can be used to merge affine merge flags (e.g., at the PU level) to indicate the use of a motion representation mode for a given block.

[0279] Figure 34An example of the motion analysis process under the affine merging mode is shown. Figure 34 As shown, the merged codec unit (CU) can be parsed. Figure 34 The parsing in the codec can be performed based on the motion codec representation mode. In the example, a flag can indicate that the merge affine merge mode is used to merge CUs. In the example, the flag can be signaled in the bitstream and parsed by the decoder. The inter-frame direction of the affine portion using the merge affine merge mode can be signaled and parsed by the decoder, which can indicate the list of reference pictures used by the affine merge mode to represent the motion data of the block under consideration.

[0280] In the example, the proposed merging affine merging and merging sub-block merging motion representation modes can be combined with the MMVD (merging with motion vector differences) merging motion encoding and decoding mechanism.

[0281] For example, for one or two merge affine merge candidates used to represent the motion of a given dual-prediction inter-frame block, the MMVD motion vector offset can be signaled in the form of an MMVD index.

[0282] In the example, in the case of bidirectional blocks in the merge affine merge mode, the bidirectional matching cost minimization process can be used to derive one or both of the MMVD indices of the merge affine merge (or merge based on merged sub-blocks) motion representations of the blocks under consideration.

[0283] Merging affine merging motion data for a given CU can be reconstructed without parsing syntax elements. Merging candidate lists and sub-block-based merging candidate lists can be constructed (e.g., alternatively). In affine and non-affine modes, each list can include (e.g., only) single-prediction merging candidates. For example, the final affine merging candidate selected to represent the motion data of the considered block can be determined based on a bidirectional matching cost minimization criterion. Bidirectional matching for each affine / non-affine pair can be computed. The pair with the minimum cost can be selected. The affine merging mode can be (e.g., only) used for non-low-latency blocks, such as those having a past reference image of the current image and another future reference image.

[0284] The signaling merge index can be used for the merge affine merge mode to indicate to the decoder which pair of merge and affine merge candidates can be used to represent the motion data of the current block s.

[0285] For example, a third list of merged affine merge pairs can be constructed by combining the single-prediction candidates from two lists (e.g., once the merged and affine merged single-prediction candidate lists have been constructed). For example, a pair of candidates can be allowed if (e.g., only) the candidates have different reference images. For example, if it is bidirectional (e.g., in the reference images of the candidates), for example, if one is in the past of the current image and the other is in the future of the current image, a pair of candidates can be allowed (e.g., alternatively).

[0286] Merge indexes (e.g., at the PU level encoding / decoding) can indicate which pair of candidates in the combination list is selected to represent the motion of the PU under consideration.

[0287] Merging affine merging motion representations can bring further diversity to the representation of block motion data with a very limited number of additional encoding and decoding syntaxes, which can improve compression efficiency.

[0288] For example, motion vector encoding and decoding can be implemented using advanced control of motion vector encoding and decoding methods.

[0289] Motion data encoding and decoding systems can be activated and / or deactivated (e.g., canonically) by (e.g., dedicated) Sequence Parameter Set (SPS) signaling flags.

[0290] Motion data encoding and decoding systems can be activated and / or deactivated (e.g., canonically) by (e.g., dedicated) Picture Parameter Set (PPS) signaling flags.

[0291] Motion data encoding and decoding systems can be activated and / or deactivated (e.g., canonically) by (e.g., dedicated) picture header syntax elements.

[0292] Motion data encoding and decoding systems can be activated and / or deactivated (e.g., canonically) by (e.g., dedicated) slice header syntax elements.

[0293] Motion data encoding and decoding systems can be activated and / or deactivated (e.g., canonically) by (e.g., dedicated) subpicture-level syntax elements.

[0294] Motion data encoding and decoding systems can be (e.g., canonically) activated and / or deactivated by (e.g., dedicated) codec tree unit (CTU) level syntax elements.

[0295] Although the features and elements have been described above in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware, which is contained in a computer-readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted via wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROMs and digital multifunction discs (DVDs). A processor associated with the software can be used to implement a radio frequency transceiver used in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A video encoding / decoding device, comprising: The processor is configured to at least: The first image sequence count (POC) value associated with the first reference image associated with the first adaptive motion vector prediction (AMVP) merge portion of the merge list candidate is determined to be different from the second POC value associated with the second reference image associated with the second AMVP merge portion of the merge list candidate; Use merge list candidates to generate the merge list; and Motion information associated with the current codec unit (CU) is obtained based on the merged list.

2. The video encoding and decoding device according to claim 1, wherein the current CU is in affine AMVP merging prediction mode.

3. The video encoding / decoding device according to claim 1, wherein the merge list candidate is a low-latency merge list candidate.

4. The video codec apparatus of claim 1, wherein the processor is configured to skip bidirectional matching-based reordering of low-latency merge list candidates in the merge list.

5. The video codec apparatus of claim 1, wherein the processor is configured to perform a bidirectional matching-based reordering of non-low-latency merge list candidates in the merge list.

6. A video encoding / decoding method, comprising: The first image sequence count (POC) value associated with the first reference image associated with the first adaptive motion vector prediction (AMVP) merge portion of the merge list candidate is determined to be different from the second POC value associated with the second reference image associated with the second AMVP merge portion of the merge list candidate; Use merge list candidates to generate the merge list; and Motion information associated with the current codec unit (CU) is obtained based on the merged list.

7. The video encoding and decoding method according to claim 6, wherein the current CU is in affine AMVP merging prediction mode.

8. The video encoding / decoding method according to claim 6, wherein the merge list candidate is a low-latency merge list candidate.

9. The video encoding / decoding method according to claim 6 further includes a bidirectional matching-based reordering of low-latency merge list candidates in the merge list.

10. The video encoding / decoding method of claim 6 further includes performing a bidirectional matching-based reordering of non-low-latency merge list candidates in the merge list.

11. A computer program product stored on a non-transitory computer-readable medium and comprising program code instructions, which, when executed by at least one processor, are used to implement the steps of the method according to at least one of claims 6 to 10.

12. A computer program comprising program code instructions, which, when executed by a processor, are used to implement the steps of the method according to at least one of claims 6 to 10.

13. A bitstream comprising information representing an encoded output generated by any one of the methods according to any one of claims 6 to 10.

Images