Interaction between neural network-based intra-prediction mode and normal intra-prediction mode

Abstract

Disclosed are systems, methods, and means for performing intra prediction of luminance coding blocks and / or chrominance coding blocks and associated signaling when neural network-based intra prediction is enabled. Information representing a neural network-based prediction mode and a block context may be obtained for a current coding block. A neural network-based predictor may be generated for the current coding block based on the block context and the neural network-based prediction mode. A non-neural network-based intra prediction mode corresponding to the generated neural network-based predictor may be determined. A prediction mode for a neighboring block may be determined based on the non-neural network-based intra prediction mode.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application claims the benefits of European Provisional Patent Application No. 21305546.0 filed on 28 April 2021, European Provisional Patent Application No. 21305806.8 filed on 11 June 2021, European Provisional Patent Application No. 21306297.9 filed on 20 September 2021, and European Provisional Patent Application No. 21306892.7 filed on 21 December 2021, the contents of which are incorporated herein by reference. [Background technology]

[0002] To achieve high compression efficiency, image and video coding schemes may utilize prediction and transformation to leverage spatial and temporal redundancy within the video content. Intra-prediction or inter-prediction may be used to leverage intra-frame correlation or inter-frame correlation. The difference between the original block and the predicted block, often expressed as prediction error or prediction residual, may be transformed, quantized, and entropicated. To reconstruct the video, the compressed data may be decoded by the reverse processes corresponding to entropicating, quantizing, transforming, and predicting. [Overview of the project]

[0003] When neural network-based intra prediction is enabled, systems, methods, and means for performing intra prediction of luminance coding blocks and / or chrominance coding blocks, and associated signaling, are disclosed. For a current coding block, information representing a neural network-based prediction mode and block context can be obtained. A neural network-based predictor can be generated for the current coding block based on the block context and the neural network-based prediction mode. A non-neural network-based intra prediction mode corresponding to the generated neural network-based predictor can be determined. The prediction mode for adjacent blocks can be determined based on the non-neural network-based intra prediction mode.

[0004] The systems, methods, and means described herein can include a decoder. In some embodiments, the systems, methods, and means described herein can include an encoder. In some embodiments, the systems, methods, and means described herein can include signals (e.g., from an encoder and / or received by a decoder). A computer-readable medium can include instructions for causing one or more processors to perform the methods described herein. A computer program product can include instructions that can cause one or more processors to perform the methods described herein when the program is executed by the one or more processors. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] [Figure 1A] FIG. 1 is a system diagram illustrating an exemplary communication system in which one or more of the disclosed embodiments can be implemented. [Figure 1B] FIG. 1A is a system diagram illustrating an exemplary wireless transmit / receive unit (WTRU) that can be used within the communication system illustrated in FIG. 1, according to one embodiment. [Figure 1C]This is a system diagram illustrating an exemplary radio access network (RAN) and an exemplary core network (CN) that may be used in a communication system illustrated in Figure 1A according to one embodiment. [Figure 1D] This is a system diagram illustrating a further exemplary RAN and a further exemplary CN that may be used in the communication system illustrated in Figure 1A according to one embodiment. [Figure 2] An example video encoder will be used as an illustration. [Figure 3] An exemplary video decoder is illustrated. [Figure 4] This document illustrates an example of a system in which various embodiments and configurations can be implemented. [Figure 5] An example of a context surrounding the rectangular block that should be predicted is shown. [Figure 6] This example shows a neural network-based intra-predictive mode signaling for W×H chrominance coding blocks (CBs). [Figure 7] This example shows how the encoder can infer the index of the normal intra-prediction mode that best represents the neural network prediction for the current block. [Figure 8] This example shows how the decoder infers the index of the normal intra-prediction mode that best represents the neural network prediction for the current block. [Figure 9] This example shows how the encoder can infer the index of the normal intra-prediction mode that best represents the neural network prediction for the current block. [Figure 10] This example shows how the decoder infers the index of the normal intra-prediction mode that best represents the neural network prediction for the current block. [Figure 11] This example shows how the encoder can infer the indices of k normal intra-prediction modes that best represent the neural network predictions for the current block. [Figure 12]This example shows how the decoder can infer the indices of k normal intra-prediction modes that best represent the neural network predictions for the current block. [Figure 13] This document provides illustrative definitions of intra predictive signaling and direct mode (DM) for chrominance coding blocks. [Figure 14] This section presents an exemplary intra-predictive signaling method for chrominance CB. [Figure 15] This document now presents an exemplary intra-predictive signaling for chrominance-based cross-validation (CB). [Figure 16] This example shows how the decoder infers the index of the normal intra-prediction mode that best represents the neural network prediction for the current block. [Figure 17] This shows an example of prediction using a neural network-based intra-prediction mode. [Figure 18] This shows an example of prediction using a neural network-based intra-prediction mode. [Figure 19] This shows an example of prediction using a neural network-based intra-prediction mode. [Figure 20] This shows an example of prediction using a neural network-based intra-prediction mode. [Figure 21] This example demonstrates block prediction using a neural network-based intra-prediction mode. [Figure 22] An example of a neural network-based pipeline on the encoder side is shown. [Figure 23] An example of a neural network-based pipeline on the decoder side is shown. [Figure 24] This shows an exemplary decision tree representing the predictive coding of trGrpIdx for predTrGrpIdx, resulting in a bit Bremainder. [Figure 25] An example of predictive decoding associated with predictive coding is shown. [Figure 26]An exemplary derivation of the list of most probable modes (MPMs) is shown. [Figure 27] This provides an exemplary derivation of a list of MPMs for video codecs, including neural network-based intra-predictive modes. [Figure 28] An illustrative derivation of the MPM list is shown below. [Figure 29] This provides an exemplary derivation of a list of MPMs for video codecs, including neural network-based intra-predictive modes. [Figure 30] This provides an exemplary derivation of a list of MPMs for video codecs, including neural network-based intra-predictive modes. [Figure 31] An illustrative derivation of the MPM list is shown below. [Figure 32] This example shows how the decoder can infer the index of the normal intra-prediction mode, which represents the neural network prediction for the current block. [Modes for carrying out the invention]

[0006] A more detailed understanding can be obtained from the following detailed explanation, which is provided in conjunction with the attached diagrams as examples.

[0007] Figure 1A illustrates an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, message transmission, and broadcast to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communication system 100 may use one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique-word OFDM (UW-OFDM), resource block filtering OFDM, and filter bank multicarrier (FBMC).

[0008] As shown in Figure 1A, the communication system 100 may include radio transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, but it will be understood that the disclosed embodiments intend any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a radio environment. For example, WTRU102a, 102b, 102c, and 102d, any of which may be referred to as “station” and / or “STA”, may be configured to transmit and / or receive radio signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscriber-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, radio sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearables, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other radio devices operating in an industrial and / or automated processing chain context), consumer electronics devices, and devices operating in commercial and / or industrial radio networks. Any of WTRU102a, 102b, 102c, and 102d may interchangeably be referred to as UE.

[0009] The communication system 100 may also include base stations 114a and / or base stations 114b. Each of the base stations 114a and 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks, such as CN 106 / 115, the Internet 110, and / or other networks 112. As an example, base stations 114a and 114b may be base transceiver stations (BTS), node B, enode B, home node B, home enode B, gNB, NR node B, site controller, access point (AP), wireless router, etc. Although base stations 114a and 114b are each depicted as single elements, it will be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0010] Base station 114a may be part of RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), and relay nodes. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be licensed spectra, unlicensed spectra, or a combination of licensed and unlicensed spectra. Cells may provide coverage of radio services to a particular geographic area that may be relatively fixed or change over time. Cells may be further divided into cell sectors. For example, a cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver per sector of the cell. In one embodiment, the base station 114a may use multiple-input multiple output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0011] Base stations 114a and 114b may communicate with one or more WTRUs 102a, 102b, 102c, and 102d via an air interface 116, which may be any suitable radio communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0012] More specifically, as described above, the communication system 100 may be a multiple access system and may use one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN 104 / 113 may implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish air interfaces 115 / 116 / 117 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0013] In one embodiment, base stations 114a and WTRUs 102a, 102b, and 102c may implement radio technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish an air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0014] In one embodiment, base stations 114a and WTRUs 102a, 102b, and 102c can implement radio technologies such as NR radio access, which can establish an air interface 116 using New Radio (NR).

[0015] In one embodiment, base station 114a and WTRU 102a, 102b, 102c may implement multiple radio access technologies. For example, base station 114a and WTRU 102a, 102b, 102c may implement LTE radio access and NR radio access together, for example, using the dual connectivity (DC) principle. Thus, the air interface utilized by WTRU 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions transmitted to and from multiple types of base stations (e.g., eNB and gNB).

[0016] In other embodiments, base stations 114a and WTRUs 102a, 102b, and 102c may implement wireless technologies such as IEEE 802.11 (i.e., Wireless Fidelity, WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access, WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), and GSM EDGE (GERAN).

[0017] The base station 114b in Figure 1A may be, for example, a wireless router, Home node B, Home e node B, or access point, and may utilize any suitable RAT to facilitate wireless connectivity in local areas such as offices, homes, vehicles, campuses, industrial facilities, aerial corridors (for use by drones), roads, etc. In one embodiment, the base station 114b and WTRU 102c, 102d may implement wireless technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, the base station 114b and WTRU 102c, 102d may implement wireless technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base stations 114b and WTRUs 102c, 102d may establish picocells or femtocells using cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). As shown in Figure 1A, base station 114b may have a direct connection to the internet 110. Therefore, base station 114b may not need to access the internet 110 via CN 106 / 115.

[0018] RAN104 / 113 can communicate with CN106 / 115, which may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRU102a, 102b, 102c, and 102d. The data may have various quality of service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, and mobility requirements. CN106 / 115 may provide call control, billing services, mobile location-based services, prepaid calls, internet connectivity, video distribution, etc., and / or perform high-level security functions such as user authentication. Although not shown in Figure 1A, it will be understood that RAN104 / 113 and / or CN106 / 115 may communicate directly or indirectly with other RANs employing the same RAT as RAN104 / 113 or different RATs. For example, in addition to being connected to RAN104 / 113 which can utilize NR radio technology, CN106 / 115 can also communicate with another RAN (not shown) using GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0019] CN106 / 115 may also function as a gateway for WTRU102a, 102b, 102c, 102d to access PSTN108, the Internet 110, and / or other networks 112. PSTN108 may include a circuit-switched telephone network providing Plain Old Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices, where these networks and devices use common communication protocols such as the transmission control protocol (TCP), the user datagram protocol (UDP), and / or the Internet protocol (IP) of the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may employ the same RAT as RAN104 / 113 or a different RAT.

[0020] Some or all of the WTRUs 102a, 102b, 102c, and 102d in the communication system 100 may include multimode capability (for example, WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different radio networks via different radio links). For example, WTRU 102c shown in Figure 1A may be configured to communicate with base station 114a, which may use cellular-based radio technology, and base station 114b, which may use IEEE 802 radio technology.

[0021] Figure 1B is a system diagram illustrating an exemplary WTRU 102. As shown in Figure 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any partial combination of the aforementioned elements while maintaining consistency with one embodiment.

[0022] The processor 118 may be a general-purpose processor, a dedicated processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functions that enable the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to a transceiver 120 which may be coupled to a transmit / receive element 122. Figure 1B depicts the processor 118 and transceiver 120 as separate components, but it will be understood that the processor 118 and transceiver 120 may be integrated together in an electronic package or chip.

[0023] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF signals and optical signals. It will be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of radio signals.

[0024] Although the transmit / receive element 122 is depicted as a single element in Figure 1B, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may utilize MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving radio signals via the air interface 116.

[0025] The transceiver 120 may be configured to modulate the signal transmitted by the transmit / receive element 122 and demodulate the signal received by the transmit / receive element 122. As described above, the WTRU 102 may have multimode capability. Therefore, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0026] The processor 118 of the WTRU102 may be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit) and may receive user input from these. The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from any type of suitable memory, such as non-removable memory 130 and / or removable memory 132, and store data in such memory. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from memory not physically located on the WTRU 102, such as on a server or home computer (not shown), and store data in such memory.

[0027] The processor 118 may receive power from the power supply 134, but may be configured to distribute and / or control power to other components in the WTRU 102. The power supply 134 may be any suitable device for supplying power to the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0028] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to or instead of the information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be understood that the WTRU 102 may acquire location information by any preferred location determination method while maintaining consistency with one embodiment.

[0029] The processor 118 may be further coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functions, and / or wired or wireless connectivity. For example, peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, and the like. The peripheral device 138 may include one or more sensors, which may be one or more of the following: gyroscope, accelerometer, Hall effect sensor, magnetometer, compass sensor, proximity sensor, temperature sensor, time sensor, geolocation sensor, altimeter, light sensor, touch sensor, magnetometer, barometer, gesture sensor, biometric sensor, and / or humidity sensor.

[0030] WTRU102 may include a full-duplex radio in which the transmission and reception of some or all of the signals (e.g., associated with specific subframes for both UL (e.g., transmission) and downlink (e.g., reception) may be in parallel and / or simultaneous. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference either through hardware (e.g., chokes) or signal processing via a processor (e.g., via a separate processor (not shown) or processor 118). In one embodiment, WRTU102 may include a half-duplex radio for the transmission and reception of any of the signals (e.g., associated with specific subframes for either UL (e.g., transmission) or downlink (e.g., reception)).

[0031] Figure 1C is a system diagram illustrating RAN104 and CN106 according to one embodiment. As described above, RAN104 can communicate with WTRU102a, 102b, and 102c via the air interface 116 using E-UTRA wireless technology. RAN104 can also communicate with CN106.

[0032] RAN104 may include e-nodes-B160a, 160b, and 160c, but it will be understood that RAN104 may include any number of e-nodes-B while maintaining consistency with one embodiment. Each of e-nodes-B160a, 160b, and 160c may include one or more transceivers for communicating with WTRU102a, 102b, and 102c via the air interface 116. In one embodiment, e-nodes-B160a, 160b, and 160c may implement MIMO technology. Thus, e-node-B160a may, for example, use multiple antennas to transmit radio signals to and / or receive radio signals from WTRU102a.

[0033] Each of the e-nodes-B160a, 160b, and 160c may be associated with a specific cell (not shown) and may be configured to handle wireless resource management decisions, handover decisions, user scheduling, etc., in UL and / or DL. As shown in Figure 1C, the e-nodes-B160a, 160b, and 160c may communicate with each other via the X2 interface.

[0034] The CN106 shown in Figure 1C may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or PGW) 166. Although each of the aforementioned elements is depicted as part of CN106, it will be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0035] The MME162 can be connected to each of the e-nodes B162a, 162b, and 162c in RAN104 via the S1 interface and can function as a control node. For example, the MME162 may perform roles such as authenticating users of WTRU102a, 102b, and 102c, activating / deactivating bearers, and selecting gateways for specific services during the initial attachment of WTRU102a, 102b, and 102c. The MME162 may provide control plane functionality for switching between RAN104 and other RANs (not shown) employing other radio technologies such as GSM and / or WCDMA.

[0036] The SGW164 can be connected to each of the e-nodes B160a, 160b, and 160c in RAN104 via the S1 interface. The SGW164 can generally route and forward user data packets to and from WTRU102a, 102b, and 102c. The SGW164 can also perform other functions, such as anchoring the user plane during e-node B handovers, triggering paging when DL data is available to WTRU102a, 102b, and 102c, and managing and remembering the context of WTRU102a, 102b, and 102c.

[0037] SGW164 may be connected to PGW166, which may provide WTRU102a, 102b, and 102c with access to a packet-switched network such as the Internet 110 to facilitate communication between WTRU102a, 102b, and 102c and IP-enabled devices.

[0038] CN106 can facilitate communication with other networks. For example, CN106 can provide WTRU102a, 102b, and 102c with access to a circuit-switched network such as PSTN108 to facilitate communication between WTRU102a, 102b, and 102c and conventional terrestrial line communication devices. For example, CN106 may include, or communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that functions as an interface between CN106 and PSTN108. In addition, CN106 may provide WTRU102a, 102b, and 102c with access to another network 112, which may include other wired and / or wireless networks owned and / or operated by other service providers.

[0039] Although the WTRU is described as a wireless terminal in Figures 1A to 1D, in certain representative embodiments, such a terminal is intended to be able to use a wired communication interface (e.g., temporary or permanent) with a communication network.

[0040] In a typical embodiment, the other network 112 may be a WLAN.

[0041] A WLAN in Infrastructure Basic Service Set (BSS) mode may have an Access Point (AP) of the BSS and one or more stations (STAs) associated with the AP. The AP may have access to or interfaces with another type of wired / wireless network that carries traffic entering and / or leaving the Distribution System (DS) or BSS. Traffic originating outside the BSS and destined for the STA may reach and be delivered to the STA via the AP. Traffic originating from the STA to destinations outside the BSS may be sent to the AP and then delivered to their respective destinations. Traffic between STAs within the BSS may be transmitted, for example, via the AP; a source STA may send traffic to the AP, and the AP may deliver the traffic to the destination STA. Traffic between STAs within the BSS may be considered and / or referred to as peer-to-peer traffic. Peer-to-peer traffic may be transmitted between a source STA and a destination STA (for example, directly between them) in a direct link setup (DLS). In certain representative embodiments, the DLS may use 802.11e DLS or 802.11z tunneled DLS (TDLS). A WLAN using Independent BSS (IBSS) mode may not have APs, and STAs within or using IBSS (e.g., all STAs) may communicate directly with each other. The IBSS mode of communication may be referred to herein as “ad hoc” communication mode.

[0042] When using the 802.11ac infrastructure operating mode or a similar operating mode, an AP may transmit beacons on a fixed channel, such as the primary channel. The primary channel may be of a fixed width (e.g., a 20 MHz bandwidth) or a width dynamically set via signaling. The primary channel may be the operating channel of the BSS and may be used by the STA to establish a connection with the AP. In certain typical embodiments, for example, in an 802.11 system, Carrier Sense Multiple Access / Collision Avoidance (CSMA / CA) may be implemented. In the case of CSMA / CA, the STA, including the AP (e.g., all STAs), may sense the primary channel. If the primary channel is sensed / detected and / or determined to be busy by a particular STA, that STA may be backed off. A single STA (e.g., only one station) may transmit at any given time on a given BSS.

[0043] High-throughput (HT) STAs may use a 40 MHz wide channel for communication, which may be formed, for example, through a combination of a primary 20 MHz channel and adjacent or non-adjacent 20 MHz channels.

[0044] Very High Throughput (VHT) STAs may support channels with widths of 20 MHz, 40 MHz, 80 MHz, and / or 160 MHz. The 40 MHz and / or 80 MHz channels mentioned above may be formed by combining multiple consecutive 20 MHz channels. A 160 MHz channel may be formed by combining eight consecutive 20 MHz channels, or by combining two non-contiguous 80 MHz channels, which may be referred to as an 80+80 configuration. In the 80+80 configuration, after channel coding, the data may pass through a segment parser that can split the data into two streams. Inverse Fast Fourier Transform (IFFT) and time-domain processing may be performed separately for each stream. The streams may be mapped to two 80 MHz channels, and the data may be transmitted by a transmitting STA. At the receiver of a receiving STA, the operation described above for the 80+80 configuration may be reversed, and the combined data may be transmitted to Medium Access Control (MAC).

[0045] Sub-1 GHz operating modes are supported by 802.11af and 802.11ah. Channel operating bandwidth and carrier are reduced in 802.11af and 802.11ah compared to those used in 802.11n and 802.11ac. 802.11af supports 5 MHz, 10 MHz, and 20 MHz bandwidths in the TV White Space (TVWS) spectrum, while 802.11ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz bandwidths using the non-TVWS spectrum. According to a typical embodiment, 802.11ah may support meter-type control / machine-type communications, such as MTC devices in a macro coverage area. MTC devices may have certain capabilities, including support for specific and / or limited bandwidths (e.g., support only for that). MTC devices may include batteries with battery life exceeding a threshold (e.g., to maintain very long battery life).

[0046] A WLAN system capable of supporting multiple channels and channel bandwidths such as 802.11n, 802.11ac, 802.11af, and 802.11ah includes a channel that can be designated as the primary channel. The primary channel may have a bandwidth equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel may be set and / or limited by an STA from among all STAs operating in a BSS that support the minimum bandwidth operating mode. In the 802.11ah example, the primary channel may be 1 MHz wide for an STA (e.g., an MTC type device) that supports (e.g., only) the 1 MHz mode, even if other STAs in the AP and BSS support 2 MHz, 4 MHz, 8 MHz, 16 MHz, and / or other channel bandwidth operating modes. Carrier sensing and / or Network Allocation Vector (NAV) settings may depend on the state of the primary channel. For example, if the primary channel is busy due to an STA (which only supports 1MHz operating mode) transmitting to the AP, a large portion of the frequency band may remain idle and could be considered busy, even if it were available.

[0047] In the United States, the available frequency band that can be used by 802.11ah is 902MHz to 928MHz. In South Korea, the available frequency band is 917.5MHz to 923.5MHz. In Japan, the available frequency band is 916.5MHz to 927.5MHz. The total bandwidth available for 802.11ah is 6MHz to 26MHz, depending on the country code.

[0048] Figure 1D is a system diagram illustrating RAN113 and CN115 according to one embodiment. As described above, RAN113 can communicate with WTRU102a, 102b, and 102c via air interface 116 using NR radio technology. RAN113 can also communicate with CN115.

[0049] RAN113 may include gNB180a, 180b, and 180c, but it will be understood that RAN113 may include any number of gNBs while maintaining consistency with one embodiment. Each of gNB180a, 180b, and 180c may include one or more transceivers for communicating with WTRU102a, 102b, and 102c via the air interface 116. In one embodiment, gNB180a, 180b, and 180c may implement MIMO technology. For example, gNB180a and 108b may use beamforming to transmit and / or receive signals to gNB180a, 180b, and 180c. Thus, gNB180a may, for example, use multiple antennas to transmit radio signals to and / or receive radio signals from WTRU102a. In one embodiment, gNB180a, 180b, and 180c may implement carrier aggregation technology. For example, gNB180a may transmit multiple component carriers to WTRU102a (not shown). A subset of these component carriers may be on the unauthorized spectrum, and the remaining component carriers may be on the authorized spectrum. In one embodiment, gNB180a, 180b, and 180c may implement coordinated multi-point (CoMP) technology. For example, WTRU102a may receive coordinated transmissions from gNB180a and gNB180b (and / or gNB180c).

[0050] WTRU102a, 102b, and 102c may communicate with gNB180a, 180b, and 180c using transmissions associated with scalable numerology. For example, OFDM symbol intervals and / or OFDM subcarrier intervals may vary for different transmissions, different cells, and / or different portions of the radio transmission spectrum. WTRU102a, 102b, and 102c may communicate with gNB180a, 180b, and 180c using subframes or transmission time intervals (TTIs) of varying or scalable lengths (e.g., containing varying numbers of OFDM symbols and / or having varying absolute time durations).

[0051] gNB180a, 180b, and 180c can be configured to communicate with WTRU102a, 102b, and 102c in standalone and / or non-standalone configurations. In a standalone configuration, WTRU102a, 102b, and 102c can communicate with gNB180a, 180b, and 180c without accessing other RANs (e.g., e-node-B160a, 160b, and 160c). In a standalone configuration, WTRU102a, 102b, and 102c can utilize one or more of gNB180a, 180b, and 180c as mobility anchor points. In a standalone configuration, WTRU102a, 102b, and 102c can communicate with gNB180a, 180b, and 180c using signals in unlicensed bands. In a non-standalone configuration, WTRU102a, 102b, and 102c can communicate with and connect to gNB180a, 180b, and 180c, while also communicating with and connecting to other RANs such as e-nodes-B160a, 160b, and 160c. For example, WTRU102a, 102b, and 102c can implement DC principles for substantially simultaneous communication with one or more gNB180a, 180b, and 180c and one or more e-nodes-B160a, 160b, and 160c. In a non-standalone configuration, e-nodes B160a, 160b, and 160c can function as mobility anchors for WTRU102a, 102b, and 102c, while gNB180a, 180b, and 180c can provide additional coverage and / or throughput to service WTRU102a, 102b, and 102c.

[0052] Each of the gNB180a, 180b, and 180c may be associated with a specific cell (not shown) and may be configured to handle wireless resource management decisions, handover decisions, user scheduling in UL and / or DL, support for network slicing, dual connectivity, interworking between NR and E-UTRA, routing of user plane data to User Plane Functions (UPFs) 184a and 184b, routing of control plane information to Access and Mobility Management Functions (AMFs) 182a and 182b, and so on. As shown in Figure 1D, the gNB180a, 180b, and 180c may communicate with each other via the Xn interface.

[0053] The CN115 shown in Figure 1D may include at least one AMF182a, 182b, at least one UPF184a, 184b, at least one Session Management Function (SMF)183a, 183b, and optionally a Data Network (DN)185a, 185b. Although each of the aforementioned elements is depicted as part of the CN115, it will be understood that any of these elements may be owned and / or operated by entities other than the CN operator.

[0054] AMF182a and 182b can be connected to one or more gNB180a, 180b, and 180c in RAN113 via the N2 interface and can function as control nodes. For example, AMF182a and 182b can perform roles such as user authentication for WTRU102a, 102b, and 102c, support network slicing (e.g., handling different PDU sessions with different requirements), selection of specific SMF183a and 183b, management of registration areas, termination of NAS signaling, and mobility management. Network slicing can be used by AMF182a and 182b to customize CN support for WTRU102a, 102b, and 102c based on the type of service utilizing WTRU102a, 102b, and 102c. For example, different network slices may be established for different use cases, such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced massive mobile broadband (eMBB) access, services for machine type communication (MTC) access, and / or similar. The AMF162 may provide control plane functionality for switching between RAN113 and other RANs (not shown) employing other radio technologies such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies such as WiFi.

[0055] SMF183a and 183b can be connected to AMF182a and 182b in CN115 via the N11 interface. SMF183a and 183b can also be connected to UPF184a and 184b in CN115 via the N4 interface. SMF183a and 183b can select and control UPF184a and 184b and configure the routing of traffic through UPF184a and 184b. SMF183a and 183b can perform other functions such as managing and assigning UE IP addresses, managing PDU sessions, controlling policy enforcement and QoS, and providing downlink data notifications. PDU session types can be IP-based, non-IP-based, Ethernet-based, etc.

[0056] UPF184a and 184b may be connected via the N3 interface to one or more gNB180a, 180b, and 180c in RAN113, thereby providing WTRU102a, 102b, and 102c with access to a packet-switched network such as the Internet 110 to facilitate communication between WTRU102a, 102b, and 102c and IP-enabled devices. UPF184 and 184b may perform other functions such as routing and forwarding packets, enforcing user plane policies, supporting multi-homed PDU sessions, handling user plane QoS, buffering downlink packets, and providing mobility anchoring.

[0057] CN115 can facilitate communication with other networks. For example, CN115 may include, or communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that functions as an interface between CN115 and PSTN108. In addition, CN115 may provide WTRU102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers. In one embodiment, WTRU102a, 102b, 102c may be connected to local data networks (DN) 185a, 185b via UPF184a, 184b through an N3 interface to UPF184a, 184b, and an N6 interface between UPF184a, 184b and DN185a, 185b.

[0058] As can be seen from Figures 1A to 1D and their corresponding descriptions, one or more of the functions described herein relating to one or more of the WTRU102a to d, base stations 114a to b, e-node-B160a to c, MME162, SGW164, PGW166, gNB180a to c, AMF182a to b, UPF184a to b, SMF183a to b, DN185a to b, and / or any other devices described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more of the functions described herein. For example, an emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0059] Emulation devices may be designed to implement testing of one or more other devices in a laboratory and / or operator network environment. For example, one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless network to test other devices in a communications network. One or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless network. Emulation devices may be directly coupled to another device for testing purposes and / or may perform testing using terrestrial radio communication.

[0060] One or more emulation devices may perform one or more functions, including all of the above, while not implemented / deployed as part of a wired and / or wireless communication network. For example, an emulation device may be used in a test laboratory test scenario, and / or in a wired and / or wireless communication network that is not deployed (e.g., for testing purposes), to implement testing of one or more components. One or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas) may be used by the emulation device to transmit and / or receive data.

[0061] This application describes various embodiments, including tools, features, examples, models, and methods. Many of these embodiments are described specifically, often in a way that may sound restrictive, at least to illustrate their individual characteristics. However, this is for the purpose of clarifying the description and not to limit the application or scope of those embodiments. In fact, all of the different embodiments can be combined and interchangeable to provide further embodiments. Moreover, embodiments can likewise be combined and interchangeable with embodiments described in prior applications.

[0062] The embodiments described and contemplated in this application can be implemented in many different forms. Figures 5 to 15 described herein may provide some embodiments, but other embodiments are also contemplated. The considerations in Figures 5 to 15 are not intended to limit the scope of implementation forms. At least one of the embodiments relates generally to video encoding and decoding, and at least one other embodiment relates generally to transmitting a generated or encoded bitstream. These and other embodiments can be implemented as a computer-readable storage medium storing instructions for encoding or decoding video data according to any of the methods, apparatus, or described methods, and / or a computer-readable storage medium storing a bitstream generated according to any of the described methods.

[0063] In this application, the terms “reconstructed” and “decoded” may be used interchangeably, the terms “pixel” and “sample” may be used interchangeably, and the terms “image,” “picture,” and “frame” may be used interchangeably.

[0064] Various methods are described herein, each of which includes one or more steps or actions to achieve the described method. Unless a particular order of steps or actions is required for the proper operation of the method, the order and / or use of any particular steps and / or actions may be modified or combined. In addition, terms such as “first” and “second” may be used in various embodiments to modify elements, components, steps, operations, etc., such as “first decoding” and “second decoding.” The use of such terms does not imply any ordering of the modified operations unless specifically required. Therefore, in this embodiment, the first decoding does not need to be performed before the second decoding, and may occur, for example, before the second decoding, during the second decoding, or during a time overlapping with the second decoding.

[0065] Various methods and other embodiments described herein may be used to modify modules of the video encoder 200 and decoder 300, for example, the decoding module, as shown in Figures 2 and 3. Furthermore, the subject matter disclosed herein may be applied to any type, format, or version of video coding, whether existing or future, whether described in standards or recommendations, and any extensions of such standards and recommendations. Unless otherwise indicated or technically excluded, the embodiments described herein may be used individually or in combination.

[0066] In the embodiments described in this application, various numerical values ​​are used, such as the number of bits and bit depth. These and other specific values ​​are for illustrative purposes only, and the embodiments described are not limited to these specific values.

[0067] Figure 2 shows an exemplary video encoder. Modifications of the exemplary encoder 200 are intended, but encoder 200 is described below for clarity without describing all expected modifications.

[0068] Before encoding, the video sequence may undergo pre-encoding processing 201, such as applying a color conversion to the input color picture (e.g., converting from RGB4:4:4 to YCbCr4:2:0), or performing a remapping of input picture components to obtain a signal distribution more resistant to compression (e.g., using one histogram equalization of color components). Metadata may be associated with its pre-processing and attached to the bitstream.

[0069] In encoder 200, the picture is encoded by encoder elements as described below. The picture to be encoded is divided 202 and processed in units of coding units (CUs). Each unit is encoded using either intra-mode or inter-mode, for example. If a unit is encoded in intra-mode, it performs intra-prediction 260. In inter-mode, it performs motion estimation 275 and motion compensation 270. The encoder determines whether to use intra-mode or inter-mode to encode a unit 205 and indicates the intra / inter decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting the prediction block from the original image block 210.

[0070] Next, this predicted residual is transformed 225 and quantized 230. The transformed coefficients after quantization, as well as the motion vector and other syntactic elements, are entropic coded 245 to output a bitstream. The encoder can skip the transformation and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transformation and quantization, i.e., the residual is coded directly without applying either the transformation or quantization process.

[0071] The encoder decodes the encoded blocks to provide a reference for further prediction. The quantized transformation coefficients are inversely quantized and inversely transformed to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined to reconstruct the image blocks. An in-loop filter is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce encoding artifacts. The filtered image is stored in a reference picture buffer (280).

[0072] Figure 3 shows an embodiment of a video decoder. In the exemplary decoder 300, the bitstream is decoded by the decoder elements as described below. The video decoder 300 generally performs a decoding pass which is the reverse of the encoding pass, as described in Figure 2. The encoder 200 also generally performs video decoding as part of encoding the video data.

[0073] In particular, the decoder input includes a video bitstream, which may be generated by a video encoder 200. First, the bitstream is entropy-decoded 330 to obtain transformation coefficients, motion vectors, and other coded information. Picture segmentation information indicates how the picture is segmented. Thus, the decoder can segment the picture 335 according to the decoded picture segmentation information. The transformation coefficients are inversely quantized 340 and inversely transformed 350 to decode the predicted residuals. The decoded predicted residuals and predicted blocks are combined 355 to reconstruct the image blocks. The predicted blocks may be obtained from intra-predictions 360 or from motion-compensated predictions (i.e., inter-predictions) 375 370. An in-loop filter 365 is applied to the reconstructed image. The filtered image is stored in a reference picture buffer 380.

[0074] The decoded picture may undergo further post-decoded processing 385, such as inverse color transformation (e.g., conversion from YCbCr4:2:0 to RGB4:4:4), or inverse remapping, which performs the reverse of the remapping process performed in pre-encoding processing 201. The post-decoded processing may use metadata derived in pre-encoding processing and signaled in the bitstream. In the embodiment, the decoded image (e.g., after applying the in-loop filter 365 and / or after post-decoded processing 385, if post-decoded processing is used) may be sent to a display device for rendering to the user.

[0075] Figure 4 shows an embodiment of a system in which various embodiments and examples described herein may be implemented. System 400 may be embodied as a device comprising various components described below and configured to perform one or more embodiments described herein. Examples of such a device include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of System 400 may be embodied individually or in combination as a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing and encoder / decoder elements of System 400 are distributed across multiple ICs and / or separate components. In various embodiments, System 400 is communicably coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, System 400 is configured to implement one or more embodiments described herein.

[0076] System 400 includes, for example, at least one processor 410 configured to execute instructions loaded into itself in order to implement various embodiments described herein. The processor 410 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 400 includes at least one memory 420 (e.g., a volatile memory device and / or a non-volatile memory device). System 400 includes a storage device 440 which may include non-volatile memory and / or volatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drives, and / or optical disk drives. The storage device 440 may, in non-limiting examples, include internal storage devices, accessory storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0077] System 400 includes, for example, an encoder / decoder module 430 configured to process data to provide encoded or decoded video, the encoder / decoder module 430 of which may include its own processor and memory. The encoder / decoder module 430 represents a module that may be included in the device to perform encoding and / or decoding functions. As is known, the device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 430 may be implemented as a separate element of System 400, or it may be incorporated into the processor 410 as a combination of hardware and software, as is known to those skilled in the art.

[0078] Program code loaded into the processor 410 or encoder / decoder 430 to perform various embodiments described herein may be stored in the storage device 440 and subsequently loaded onto the memory 420 for execution by the processor 410. According to various embodiments, one or more of the processor 410, memory 420, storage device 440, and encoder / decoder module 430 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video, or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, actions, and operational logic.

[0079] In some embodiments, the internal memory of the processor 410 and / or the encoder / decoder module 430 is used to store instructions and provide working memory for the processing required during encoding or decoding. However, in other embodiments, memory outside the processing device (for example, the processing device may be either the processor 410 or the encoder / decoder module 430) is used for one or more of these functions. The external memory may be memory 420 and / or storage device 440, for example, dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory is used, for example, to store the television's operating system. In at least one embodiment, high-speed external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations.

[0080] Inputs to the elements of System 400 may be provided through various input devices, as shown in Block 445. Such input devices include, but are not limited to, (i) a radio frequency (RF) section for receiving RF signals transmitted terrestrially by broadcasters, for example; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other embodiments not shown in Figure 4 include composite video.

[0081] In various embodiments, the input device of block 445 has associated input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency in a certain band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band in order to select a signal frequency band which may be referred to as a channel in a particular embodiment, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and / or (vi) demultiplexing to select a desired data packet stream. The RF portion of various embodiments includes one or more elements for performing these functions, e.g., frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and multiplexers. The RF portion may include tuners for performing various of these functions, e.g., down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive an RF signal transmitted by a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments perform similar or different functions by rearranging the order of the elements described above (and others), removing some of these elements, and / or adding other elements. Adding elements may include inserting elements between existing elements, such as inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF section includes an antenna.

[0082] The USB and / or HDMI terminals may include their respective interface processors for connecting the system 400 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or within the processor 410, as needed. Similarly, aspects of USB or HDMI interface processing may be implemented, for example, in a separate interface IC or within the processor 410, as needed. For example, the processor 410, along with various processing elements including an encoder / decoder 430 operating in conjunction with memory and storage elements, may process data streams as needed to provide demodulated, error-corrected, and demultiplexed streams for presentation on an output device.

[0083] Various elements of system 400 may be provided within an integrated housing. Within the integrated housing, the various elements are interconnected and can transmit data between them using a suitable connection array 425, for example, an internal bus known in the art, including an Inter-IC (I2C) bus, wiring, and a printed circuit board.

[0084] System 400 includes a communication interface 450 that enables communication with other devices via a communication channel 460. The communication interface 450 may, but is not limited to, include transceivers configured to transmit and receive data via the communication channel 460. The communication interface 450 may, but is not limited to, include a modem or a network card, and the communication channel 460 may be implemented, for example, in a wired and / or wireless medium.

[0085] In various embodiments, data is streamed to system 400 or provided in other ways using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 460 and a communication interface 450 adapted for Wi-Fi communication. In these embodiments, the communication channel 460 is typically connected to an access point or router that provides access to an external network, including the Internet, to enable streaming applications and other overhead communications. In other embodiments, the streamed data is provided to system 400 using a set-top box that provides data via an HDMI connection of input block 445. In yet another embodiment, the streamed data is provided to system 400 using an RF connection of input block 445. As described above, various embodiments provide data in non-streaming manners. In addition, various embodiments use wireless networks other than Wi-Fi, e.g., cellular networks or Bluetooth® networks.

[0086] System 400 can provide output signals to various output devices, including a display 475, a speaker 485, and other peripheral devices 495. In various embodiments, the display 475 includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 475 may be for a television, tablet, laptop, mobile phone, or other device. The display 475 can also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop). Other peripheral devices 495, in various embodiments, include one or more of a standalone digital video disc (or digital multipurpose disc) (DVD for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 495 that provide functions based on the output of System 400. For example, a disc player performs the function of playing the output of System 400.

[0087] In various embodiments, control signals are communicated between the system 400 and the display 475, speaker 485, or other peripheral devices 495 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable control between devices with or without user intervention. Output devices may be coupled to the system 400 via dedicated connections through their respective interfaces 470, 480, and 490. Alternatively, output devices may be connected to the system 400 via communication interface 450 and communication channel 460. The display 475 and speaker 485 may be integrated into a single unit with other components of the system 400 in an electronic device such as a television. In various embodiments, the display interface 470 includes a display driver, such as a timing controller (TCon) chip.

[0088] The display 475 and speaker 485 may, alternatively, be separate from one or more of the other components, for example, if the RF portion of input 445 is part of a separate set-top box. In various embodiments where the display 475 and speaker 485 are external components, the output signal may be provided via a dedicated output connection, such as an HDMI port, a USB port, or a COMP output.

[0089] These embodiments may be implemented by a processor 410, by hardware, or by computer software implemented by a combination of hardware and software. In non-limiting embodiments, these embodiments may be implemented by one or more integrated circuits. The memory 420 may be any type appropriate for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, in non-limiting examples. The processor 410 may be any type appropriate for the technical environment and may include one or more of microprocessors, general-purpose computers, dedicated computers, and processors based on multi-core architectures, in non-limiting examples.

[0090] Various implementations include decoding. As used in this application, “decoding” can encompass all or part of the processes performed on a received encoded sequence to produce a final output suitable for a display. In various embodiments, such processes typically include processes performed by a decoder, e.g., one or more of entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also include, or alternatively, processes performed by a decoder in the various implementations described in this application, e.g., inverse quantization, inverse transform, and obtaining information representing a neural network-based prediction mode and block context for the currently coded block, generating a neural network-based predictor for the currently coded block based on the block context and the neural network-based prediction mode, determining a non-neural network-based intra-prediction mode corresponding to the generated neural network-based predictor, and determining a prediction mode for adjacent blocks based on the non-neural network-based intra-prediction mode.

[0091] In further embodiments, in one embodiment, “decoding” refers only to entropy decoding; in another embodiment, “decoding” refers only to differential decoding; and in yet another embodiment, “decoding” refers to a combination of entropy decoding and differential decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or to a broader decoding process as a whole will become clear from the context of the specific descriptions and will be well understood by those skilled in the art.

[0092] Various implementations include encoding. Similar to the above consideration of "decoding," "encoding" as used in this application may encompass all or part of the processes performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes typically include one or more processes performed by an encoder, e.g., partitioning, differential coding, transformation, quantization, and entropy coding. In various embodiments, such processes also, or alternatively, include processes performed by an encoder in the various implementations described in this application, e.g., quantization, transformation, and obtaining information representing a neural network-based prediction mode and block context for the currently coded block, generating a neural network-based predictor for the currently coded block based on the block context and the neural network-based prediction mode, determining a non-neural network-based intra-prediction mode corresponding to the generated neural network-based predictor, and determining a prediction mode for adjacent blocks based on the non-neural network-based intra-prediction mode.

[0093] In further embodiments, in one embodiment, "encoding" refers only to entropy coding; in another embodiment, "encoding" refers only to differential coding; and in yet another embodiment, "encoding" refers to a combination of differential coding and entropy coding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to a broader encoding process as a whole will become clear from the context of the specific descriptions and will be well understood by those skilled in the art.

[0094] It should be noted that the coding syntax used herein for syntax elements such as intra-prediction mode index, probability, and block context are descriptive terms. Therefore, these do not preclude the use of other syntax element names.

[0095] If a diagram is presented as a flowchart, it should be understood that the diagram also provides a block diagram of the corresponding device. Similarly, if a diagram is presented as a block diagram, it should be understood that the diagram also provides a flowchart of the corresponding method / process.

[0096] The implementation forms and embodiments described herein may be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even if considered only in the context of a single implementation form (for example, considered only as a method), the implementation forms of the considered features may also be implemented in other forms (for example, apparatus or programs). Apparatus may be implemented, for example, with appropriate hardware, software, and firmware. The method may be implemented, for example, in a processor, where processor generally refers to a processing device, and includes, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, mobile phones, and portable / mobile information terminals ("Personal Digital Assistants, PDAs"), which facilitate the communication of information between end users.

[0097] The phrases "one example" or "an example" or "one implementation" or "an implementation," as well as references to other variations thereof, mean that the specific features, structures, characteristics, etc. described in relation to the examples are included in at least one example. Therefore, the phrases "in one example" or "in an example" or "in one implementation" or "in an implementation," which appear in various places throughout this application, as well as any other variations, do not necessarily all refer to the same example.

[0098] In addition, this application may refer to “determining” various types of information. Determining information may include, for example, one or more of the following: estimating information, calculating information, predicting information, or retrieving information from memory. Acquiring may include receiving, retrieving, constructing, generating, and / or determining.

[0099] Furthermore, this application may refer to “accessing” various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0100] In addition, this application may refer to "receiving" various types of information. Receiving is intended to be a broad term, similar to "accessing." Receiving information may include, for example, accessing information or retrieving information (for example, from memory). Furthermore, "receiving" is typically involved in operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0101] For example, in the cases of "A / B", "A and / or B", and "at least one of A and B", it should be understood that the use of any of the following " / ", "and / or", and "at least one of" is intended to encompass the selection of only the first listed option (A), only the second listed option (B), or both options (A and B). In further embodiments, in the cases of "A, B, and / or C" and "at least one of A, B, and C," such expressions are intended to encompass the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or the selection of all three options (A, B, and C). This can be extended to the number of listed items, as will be apparent to those skilled in the art in this and related fields.

[0102] Furthermore, as used herein, the term “signaling” specifically refers to indicating something to a corresponding decoder. An encoder signal may include, for example, an encoding function for an input for a block using a predictive mode. Thus, in one embodiment, the same parameters are used on both the encoder and decoder sides. Therefore, for example, the encoder can transmit certain parameters to the decoder so that the decoder can use the same specific parameters (explicit signaling). Conversely, if the decoder already has certain parameters as well as other parameters, signaling can be used to allow the decoder to simply know and select those parameters without transmission (implicit signaling). Bit saving is achieved in various embodiments by avoiding the transmission of any actual functions. It should be understood that signaling can be achieved in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. The above relates to the verb form of the word “signal,” which may also be used as a noun herein.

[0103] As will be apparent to those skilled in the art, implementations can generate various signals formatted to carry information that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the implementations described. For example, a signal may be formatted to carry the bitstream of the embodiment described. Such a signal may be formatted, for example, as an electromagnetic wave (using, for example, the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The signal carried by the signal may be, for example, analog or digital information. The signal may be transmitted by various different wired or wireless links, as is known. The signal may be stored in a processor-readable medium or accessed or received from a processor-readable medium.

[0104] Many embodiments are described herein. Features of embodiments may be provided individually or in any combination across various categories and types of claims. Furthermore, embodiments may include one or more of the features, devices, or embodiments described herein individually or in any combination across various categories and types of claims. For example, features described herein may be implemented in a bitstream or signal containing information generated as described herein. The information may enable a decoder to decode the bitstream, encoder, bitstream, and / or decoder according to any of the embodiments described herein. For example, features described herein may be implemented by creating and / or transmitting and / or receiving and / or decoding a bitstream or signal. For example, features described herein may be implemented in a method, process, apparatus, medium for storing data, medium for storing instructions, or signal. For example, features described herein may be implemented in a TV, set-top box, mobile phone, tablet, or other electronic device that performs decoding. A TV, set-top box, mobile phone, tablet, or other electronic device may display the resulting image (e.g., an image from residual reconstruction of a video bitstream) (e.g., using a monitor, screen, or other type of display). A TV, set-top box, mobile phone, tablet, or other electronic device may receive and decode a signal containing an encoded image.

[0105] Neural network-based intra-prediction modes can be signaled in block-based video codecs. Typically, intra-prediction modes may refer to directional intra-prediction modes such as 65, non-directional modes such as PLANAR mode and DC mode, and non-neural network-based intra-prediction modes such as matrix-based intra-prediction (MIP) mode. Interaction may refer to the interaction between the intra-prediction mode selected to predict the current block and the intra-prediction mode selected to predict the blocks surrounding the current block (e.g., adjacent blocks of the current block). Interaction may relate to how the signaling cost of the intra-prediction mode selected to predict the current block may be affected by the intra-prediction mode selected to predict the blocks around the current block.

[0106] A neural network associated with a neural network-based intra-prediction mode can be trained to determine (e.g., infer its index) the normal intra-prediction mode (e.g., a non-neural network intra-prediction mode) that best represents (e.g., best represents) the neural network prediction for the current block. For example, a neural network-based predictor can be generated for the current block based on the block context and the neural network-based intra-prediction mode. Blocks predicted by the neural network-based intra-prediction mode can be represented via the index of the normal intra-prediction mode that best represents the neural network prediction. For example, if the derivation of a list of MPMs is used to signal the intra-prediction mode selected to predict the current luminance CB, the derivation may use the normal intra-prediction mode index inferred by the neural network-based intra-prediction mode when predicting luminance CBs adjacent to the current luminance CB (e.g., adjacent luminance CBs) (e.g., the index of a non-neural network-based intra-prediction mode may be added to the derived MPM list).

[0107] A neural network belonging to a neural network-based intra-prediction mode can be trained to infer the indices of several (e.g., k) normal intra-prediction modes that best represent the neural network prediction of the current block. A neural network associated with a neural network-based intra-prediction mode can be trained in ranking based on how (e.g., how well) a normal intra-prediction mode (e.g., each normal intra-prediction mode) represents the neural network prediction of the current block (e.g., the generated neural network-based predictor). A block predicted by a neural network-based intra-prediction mode can be represented, for example, through k inferred indices. For example, if a list of MPMs is used to signal an intra-prediction mode selected to predict a current luminance CB, then several non-neural network-based intra-prediction mode indices (e.g., k normal intra-prediction mode indices) can be inferred using a neural network-based intra-prediction mode. For example, the indices can be added to the derived MPM list when predicting luminance CBs adjacent to the current luminance CB (e.g., adjacent luminance CBs).

[0108] Machine learning-based intra-prediction modes and normal intra-prediction modes may be used when encoding and decoding video content. Mapping between machine learning-based intra-prediction mode indices (e.g., each machine learning-based intra-prediction mode index) and normal intra-prediction mode indices may be used, for example, to facilitate interaction between machine learning-based intra-prediction modes and normal intra-prediction modes. Machine learning-based intra-prediction modes may be ignored and / or disabled, for example, to facilitate interaction between machine learning-based intra-prediction modes and normal intra-prediction modes.

[0109] A mapping can be obtained between machine learning-based intra-prediction mode indices (e.g., each machine learning-based intra-prediction mode index) and indices of normal intra-prediction modes. MIP modes may belong to a group of machine learning-based intra-prediction modes, for example, if a matrix of MIP modes is learned. MIP modes may be fixed on the encoder side and / or decoder side. In an embodiment, the mapping can be changed, for example, if the intra-prediction mode selected to predict a luminance CB that is above or to the left of the current luminance CB (e.g., the current luminance CB in the derivation of the list of MPMs) is an MIP mode, then the index of the MIP mode can be changed to the index of a normal intra-prediction mode (e.g., used by the derivation of the list of MPMs).

[0110] In the following embodiments, the argument to PU:getIntraDirLuma may be a prediction unit (PU) (e.g., shown as pu in the following embodiments). A PU may be equivalent to a coding unit (CU) of a PU. In embodiments, the PU supplied to PU:getIntraDirLuma (e.g., shown as pu in the following embodiments) may be the PU of a CU of a luminance CB that is currently located above or to the left of a luminance CB. The mapping from MIP mode indices (e.g., each MIP mode index) to the indices of normal intra-prediction modes may be replaced by a mapping from MIP mode indices (e.g., each MIP mode index) to the PLANAR index.

[0111] [Table 1]

[0112] In the above embodiment, PU:getIntraDirLuma may be provided in the file UnitTools.cpp. isMip may return true, for example, if the argument PU of isMip (e.g., shown as pu in the above embodiment) is predicted via MIP mode. In the above embodiment, pu.intraDir[CHANNEL_TYPE_LUMA] may be the index of the intra prediction mode selected to predict the luminance component of PU (e.g., shown as pu in the above embodiment). In the above embodiment, g_mapMipToAngular65 may be a mapping from MIP mode indexes (e.g., each MIP mode index) to the index of the normal intra prediction mode.

[0113] The above embodiment may be applied to linear and / or affine machine learning-based intra-prediction modes. A mapping (e.g., a fixed mapping) can be constructed from a machine learning-based intra-prediction mode index (e.g., each machine learning-based intra-prediction mode index) to an index of a typical intra-prediction mode.

[0114] For example, when deriving the list of MPMs for a given luminance CB, if a luminance CB located above the current luminance CB is predicted by a neural network-based intra-prediction mode, the information associated with the prediction via the neural network-based intra-prediction mode may be discarded. The derivation of the list of MPMs may be adapted to depend on the remaining information. For example, if a luminance CB located to the left of the current luminance CB is predicted by a neural network-based intra-prediction mode, the information associated with the prediction by the neural network-based intra-prediction mode may be ignored. The derivation of the list of MPMs may depend on the remaining information. In the following embodiment, for example, when L==A==NN_IDX is the current luminance CB, the default list of MPMs may be derived. The indices of the intra-prediction modes selected to predict two adjacent luminance CBs (e.g., each of the two adjacent luminance CBs) may not provide information for deriving the list of MPMs.

[0115] The semantics of the following examples may include the following: L is

[0116]

number

[0117]

number

[0118] The initialization of the MPM list can be provided, for example, as follows:

[0119] [Table 2-1]

[0120] [Table 2-2]

[0121] [Table 2-3]

[0122] [Table 2-4]

[0123] In the above embodiment, the neural-based intra-prediction mode may include a neural network capable of predicting the current luminance CB. For example, an intra-prediction mode signaling denoted as S0 may be used, which includes the nnFlag flag property of the neural network intra-prediction mode, and isSignaling0 may be shown as true. Otherwise, an intra-prediction mode signaling denoted as S1, which does not include the nnFlag flag, may be used, and isSignaling0 may be shown as false.

[0124] In the above embodiment, the derivation of the list of MPMs for current luminance CBs may, for example, not consider (e.g., no longer consider) the correlation in the selected prediction mode index from one luminance CB to a subsequent luminance CB (e.g., the next luminance CB) when adjacent luminance CBs (e.g., two adjacent CBs) are predicted via a neural network-based intra-prediction mode.

[0125] A neural network-based intra-prediction mode in chrominance can be signaled. In an embodiment, the neural network-based intra-prediction mode may be constructed to apply exclusively to luminance blocks when adding the neural network-based intra-prediction mode to a block-based video codec.

[0126] Neural network-based modes may be provided for luminance blocks and chrominance blocks (e.g., both luminance and chrominance blocks). Intra-predictive signaling associated with the neural network modes for luminance blocks and chrominance blocks (e.g., both luminance and chrominance blocks) may be provided. The intra-predictive signaling of a current chrominance CB may be used, for example, to describe a chrominance CB of a current CU that shares the same intra-predictive signaling (e.g., two chrominance CBs).

[0127] Unified signaling of neural network-based modes in luminance and chrominance may be provided. In an embodiment, intra-predictive signaling may include setting a neural network flag at the CU level when a neural network-based intra-predictive mode (e.g., an additional neural network-based intra-predictive mode) is applied to both luminance and chrominance blocks. For a current CU, a transform unit (TU) in the current CU (e.g., luminance TB and chrominance TB) may select a neural network-based intra-predictive mode, for example, if the flag is equal to 1. In CUs in separate trees between luminance and chrominance, local partitioning in luminance may differ from local partitioning in chrominance.

[0128] Neural network-based mode signaling in chrominance can be provided via direct mode (DM). Currently, in chrominance CB, prediction via neural network-based intra-prediction modes can be enabled via DM (e.g., exclusively via DM). DM can be, for example, a neural network-based intra-prediction mode that is a candidate for rate-distortion search in chrominance on the encoder side, if the luminance CB collated with the chrominance CB is predicted via a neural network-based intra-prediction mode.

[0129] Figure 5 shows an example of the context surrounding the predicted rectangular block. In the embodiment, in the neural network-based intra-prediction mode, θ H,W A neural network f parameterized by H,W (.;θ H,W The function can take the context X of the decoded pixels around the W×H block Y (e.g., reconstructed pixels on the decoder side, already encoded pixels on the encoding side, both currently adjacent to CBY) and return a prediction for Y.

[0130] Figure 6 shows an example of signaling for a neural network-based intra-prediction mode for a current W×H chrominance CB. The top-left pixel of the CB may be located at (x,y) in the current frame. NN_IDX may be the index of the neural network-based intra-prediction mode (e.g., 72). The current W×H chrominance CB and collated luminance CB can be predicted by the neural network-based intra-prediction mode. T may represent a set of height and width pairs for a block, for example, where for each height H and width W, the neural network-based intra-prediction mode may include a neural network capable of predicting a block of size W×H. For example, for a given W×H chrominance CB whose top-left pixel is located at (x,y) in the current frame, the chrominance CB and collated luminance CB can be predicted by the neural network-based intra-prediction mode such that (H,W)∈T and x≧n l and y≧n a If so, DM can be the index of the neural network intra-prediction mode. In the embodiment, DM may be set to PLANAR.

[0131] An index can be determined for the normal intra-prediction (e.g., non-neural network intra-prediction mode) that best represents the neural network prediction of the current block.

[0132] Figure 7 shows an example where the encoder infers the index of the normal intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) that best represents the neural network prediction (e.g., the generated neural network-based predictor) for the current block.700 The neural network-based intra-prediction mode takes in (e.g., retrieves) the context X adjacent to the current block Y (e.g., context X may be pixels adjacent to block Y that have already been encoded) and makes a prediction for Y (e.g., generates a neural network-based predictor by testing potential neural-based prediction modes on adjacent pixels of context X).

[0133]

number

[0134]

number

[0135] In this embodiment, the block-based video codec is m∈N * This may include typical intra-prediction modes (e.g., non-neural network-based intra-prediction modes). For a given block, a neural network-based intra-prediction mode is a vector P ∈ [0,1] mIt can be returned (for example, calculated) from one or more other outputs, and the coefficient of this index i may correspond to the probability that the normal intra-prediction mode of index i (e.g., a non-neural network-based intra-prediction mode) represents (e.g., best represents) the block's neural network prediction. In the embodiment, several probabilities (e.g., representation probabilities) associated with each non-neural network-based intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) are given in the block context (e.g., by applying the non-neural network-based intra-prediction mode to adjacent pixels to determine which mode best represents the generated neural network-based predictor), and the neural network-based prediction mode Based on P, an index repIdx ∈ [|0, m-1|] of the typical intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) corresponding to (e.g., best representing) the neural network prediction of a block (e.g., a generated neural network-based predictor) can be selected (e.g., selected from each non-neural network-based intra-prediction mode). In embodiments, probabilities for a subset of typical intra-prediction modes (e.g., non-neural network-based intra-prediction modes) can be calculated, and non-neural network-based intra-prediction modes can be selected from the subset. In embodiments, repIdx can be used as a substitute when a block is involved in intra-predicted dependencies between neighboring blocks.

[0136] The methods, systems, and means described herein may be implemented on an encoding device, such as an encoder 200 described with respect to Figure 2. In an embodiment, a neural network-based intra-prediction mode 705 and a prediction of Y based on the context X of the current block Y are used.

[0137]

number

[0138]

number

[0139] Figure 8 shows an example where the decoder infers the index of the normal intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) that best represents the neural network prediction (e.g., the generated neural network-based predictor) for the current block.800 The neural network-based intra-prediction mode takes in (e.g., retrieves) the context X adjacent to Y (e.g., context X may be pixels adjacent to block Y that have already been reconstamped) and makes a prediction for Y (e.g., generates a neural network-based predictor by testing potential neural-based prediction modes on adjacent pixels of context X).

[0140]

number

[0141]

number

[0142] The methods, systems, and means described herein may be implemented on a decoding device such as the decoder 300 described with respect to Figure 3. As shown in Figure 8, the prediction of Y

[0143]

number

[0144]

number

[0145]

number

[0146]

number

[0147]

Number

[0148]

Number

[0149]

Number

[0150] As shown in FIGS. 7 and 8, for blocks in an exemplary block-based video codec, primary transformation and secondary transformation can be applied. In an embodiment, a combination of transformations (e.g., primary transformation, secondary transformation, and tertiary transformation) can be applied.

[0151] The normal intra prediction mode index in a subset of the set of normal mode indexes can be inferred. In an embodiment, the inferred normal intra prediction mode index can belong to a subset of the set of normal intra prediction mode indexes in an exemplary block-based video codec. For example, S⊂[|0,m - 1|] can be provided such that card(S)=1. For example, in FIGS. 7 and 8, P∈[0,1] l and repIdx∈S.

[0152] In an embodiment, S = [|0,66|] can represent a set of indexes of 65 directional intra prediction modes, PLANAR, and DC. In an embodiment, l can be equal to 67. For example, the set of intra prediction mode indexes excluding S (e.g., the intra prediction mode indexes in an exemplary codec) can be equal to the set of MIP mode indexes.

[0153] The MPM for an adjacent block can be derived based on the non-neural network-based intra-prediction mode index inferred for the current block (which is predicted to be encoded / decoded using neural network-based predictions, for example). For example, an index can be obtained associated with the non-neural network-based intra-prediction mode that corresponds to (e.g., best represents) the generated neural network-based predictor. The index can be added to the MPM list for the adjacent block. For a current luminance CB, the calculation of index A of the intra-prediction mode selected to predict luminance CBs located above the current luminance CB, and the calculation of index L of the intra-prediction mode selected to predict luminance CBs located to the left of the current luminance CB, can be illustrated with the following example.

[0154] [Table 3]

[0155] As shown in the example above, the calculation of index A of the intra-prediction mode selected to predict luminance CBs currently located above the luminance CB, and the calculation of index L of the intra-prediction mode selected to predict luminance CBs currently located to the left of the luminance CB may be performed. In embodiments, A and L may be used, for example, to derive a list of MPMs for the current luminance CB after calculations (e.g., two calculations). 'puLeft' may be a pointer to the left PU. 'puLeft' being NULL may indicate that there are no PUs (e.g., no CUs) to the left of the current luminance CB. 'puAbove' may be a pointer to the above PU. 'puAbove' being NULL may indicate that there are no PUs (e.g., no CUs) above the current luminance CB. NN_IDX may indicate the index of the neural network-based intra-prediction mode. 'PU::getIntraDirLuma' may return the index of the intra-prediction mode selected to predict the luminance CB of the argument PU's CU. 'PU::getIntraDirLuma' can, for example, replace the MIP mode index with PLANAR_IDX when the MIP mode is used to predict the luminance CB of the CU of the argument PU. In the embodiment, repIdx left ∈[|0,66|] may represent the index of the normal intra-prediction mode that the neural network-based intra-prediction mode would infer, for example, if the neural network-based intra-prediction mode is currently used to predict the luminance CB to the left of the luminance CB. In the embodiment, repIdx above ∈[|0,66|] may represent, for example, the index of the normal intra-prediction mode that the neural network-based intra-prediction mode predicts when the neural network-based intra-prediction mode predicts a luminance CB that is currently higher than the luminance CB.

[0156] In video codecs with neural network-based intra-predictive modes, the derivation of a list of MPMs (e.g., the list of MPMs described herein) can now be applied to luminance CB. For example, the comparison between A and NN_IDX, and the comparison between L and NN_IDX (e.g., both are shown in Figure 3) can be bypassed.

[0157] The semantics of the following examples may include the following: L may represent the index of the intra-predictive mode of the luminance CB on the left, which is,

[0158]

number

[0159]

number

[0160] The initialization of the MPM list can be provided, for example, as follows:

[0161] [Table 4-1]

[0162] [Table 4-2]

[0163] [Table 4-3]

[0164] As shown in the above embodiment, a list of MPMs can be derived if, for example, the index L of the intra-prediction mode selected to predict the left luminance CB and the index A of the intra-prediction mode selected to predict the upper luminance CB arise from one or more embodiments described herein. In the embodiment, a neural network-based intra-prediction mode may include a neural network that can be used to predict the current luminance CB. For example, the indication may indicate the presence of a neural network in a neural network-based intra-prediction mode that can be used to predict the current luminance CB, if the neural network-based intra-prediction mode includes a neural network that can predict the current luminance CB. For example, an intra-prediction mode signaling denoted as S0 may be used, which may include a flag nnFlag characteristic of the neural network-based intra-prediction mode. For example, if the neural-based intra-prediction mode includes a neural network that can currently predict luminance CB, then an intra-prediction mode signaling denoted as S0, which includes the nnFlag flag property of the neural network intra-prediction mode, may be used, and isSignaling0 may be shown as true. Otherwise, an intra-prediction mode signaling denoted as S1, which does not include the nnFlag flag, may be used, and isSignaling0 may be shown as false.

[0165] Different output vectors can be computed, for which an index can be computed for a typical intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) that best represents a neural network prediction. In embodiments, the output of a neural network-based intra-prediction mode may vary. In embodiments, for a given block, a neural network-based intra-prediction mode may be used to compute at least one vector based on the context surrounding the block, where the coefficient of index i is the probability that repIdx=i. In embodiments, a neural network-based intra-prediction mode may return a scalar, vector, and / or tensor for which repIdx can be computed. For example, a neural network-based intra-prediction mode may compute a vector where the coefficient of index i is the log-probability opposite to the log-probability that repIdx=i.

[0166]

number

[0167] Figure 9 shows an example where the encoder infers the index of the normal intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) that best represents the neural network prediction (e.g., the generated neural network-based predictor) for the current block. The neural network-based intra-prediction mode takes in (e.g., retrieves) the context X adjacent to the current block Y (e.g., context X may be pixels adjacent to block Y that have already been encoded) and makes a prediction for Y (e.g., generates a neural network-based predictor by testing the potential neural network-based prediction mode for the adjacent pixels of context X).

[0168]

number

[0169]

number

[0170]

number

[0171] Figure 10 shows an example where the decoder infers the index of the normal intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) corresponding to the neural network prediction (e.g., a generated neural network-based predictor) for the current block. The neural network-based intra-prediction mode takes in the context X adjacent to the current block Y (e.g., context X may be pixels adjacent to block Y that have already been reconstructed) and predicts Y (e.g., generates a neural network-based predictor by testing the potential neural network-based prediction mode for adjacent pixels in context X).

[0172]

number

[0173]

number

[0174]

number

[0175] A neural network-based intra-prediction mode can be used to infer an index of several normal intra-prediction modes that can represent (e.g., best represent) the predictions returned by the neural network-based intra-prediction mode.

[0176] Figure 11 shows an example where the encoder infers the index of the normal intra-prediction mode (e.g., k non-neural network-based intra-prediction modes) that best represents the neural network prediction (e.g., the generated neural network-based predictor) of the current block.900 The inferred normal intra-prediction mode can be added to the MPM list of the adjacent block.

[0177] The neural network-based intra-prediction mode takes in (e.g., retrieves) the context X adjacent to block Y (e.g., context X may be pixels adjacent to block Y that have already been encoded) and predicts Y (e.g., generates a neural network-based predictor by testing the potential neural-based prediction mode on adjacent pixels of context X).

[0178]

number

[0179]

number

[0180] The methods, systems, and means described herein may be implemented on an encoding device, for example, the encoder described with respect to Figure 2. In an embodiment, a neural network-based intra-prediction mode 905 and the current block Y are used to predict Y based on the context X.

[0181]

number

[0182]

number

[0183] Figure 12 shows an exemplary index of the normal intra-prediction mode (e.g., non-neural network intra-prediction mode) that best represents the neural network prediction (e.g., generated neural network-based predictor) of the current block on the decoder side.

[0184] The neural network-based intra-prediction mode takes in (e.g., retrieves) the context X adjacent to block Y (e.g., context X may be pixels adjacent to block Y that have already been reconstructed) and predicts Y (e.g., generates a neural network-based predictor by testing the potential neural-based prediction mode on adjacent pixels of context X).

[0185]

number

[0186]

Number

[0187] The methods, systems, and means described in this specification may be implemented on a decoding device, e.g., the decoder described with respect to FIG. 3. As shown in FIG. 10, the prediction of Y

[0188]

Number

[0189]

Number

[0190]

Number

[0191]

Number

[0192]

Number

[0193] from the reconstructed residual ​​

number

[0194]

number

[0195] A list of MPMs can be derived for the current luminance CB. For example, if k=2, a neural network-based intra-prediction mode is selected to predict luminance CBs located to the left of the current luminance CB and luminance CBs located above the current luminance CB. When predicting luminance CBs located to the left of the current luminance CB, the indices of two normal intra-prediction modes can be inferred, and when predicting luminance CBs located above the current luminance CB, the indices of two normal intra-prediction modes can be inferred. The semantics of this embodiment may include the following: L is, for example,

[0196]

number

[0197]

number

[0198] [Table 5] ‘puLeft’ can be a pointer to the left PU. In an embodiment, ‘puLeft’ being NULL may indicate that there is no PU (e.g., no CU) to the left of the current luminance CB. ‘puAbove’ can be a pointer to the upper PU. In an embodiment, ‘puAbove’ being NULL may indicate that there is no PU (e.g., no CU) above the current luminance CB. ‘PU::getIntraDirLuma’ can return the index of the intra prediction mode selected to predict the luminance CB of the CU of the argument PU. ‘PU::getIntraDirLuma’ can replace the MIP mode index with PLANAR_IDX, for example, when the MIP mode is used to predict the luminance CB of the CU of the argument PU. For the current luminance CB, a list of MPMs can be derived as follows.

[0199] [Table 6-1]

[0200] [Table 6-2]

[0201] [Table 6-3]

[0202] [Table 6-4] 'fillListMPMsNNLeftAbove' can be a function (e.g., any function) that puts 'ptrIndicesRepNNLeft[0]', 'ptrIndicesRepNNLeft[1]', 'ptrIndicesRepNNAbove[0]', and 'ptrIndicesRepNNAbove[1]' into a list of MPMs pointed to by 'MPM'. In an embodiment, 'fillListMPMsNNLeftAbove' may be designed so that the MPMs (e.g., all MPMs) are different.

[0203] In the embodiment, 'fillListMPMsNNLeftAbove' may behave as described herein. The priority in the list of MPMs may be 'ptrIndicesRepNNLeft[0]', 'ptrIndicesRepNNAbove[0]', 'ptrIndicesRepNNLeft[1]', and 'ptrIndicesRepNNAbove[1]', as illustrated below.

[0204] [Table 7-1]

[0205] [Table 7-2]

[0206] [Table 7-3] 'areRedundantInSecondArray' can be used to set 'isIdx0Redundant' to true if, for example, 'ptrIndicesRepNNAbove[0]'='ptrIndicesRepNNLeft[0]' or 'ptrIndicesRepNNAbove[0]'='ptrIndicesRepNNLeft[1]'. 'areRedundantInSecondArray' can be used to set 'isIdx1Redundant' to true if, for example, 'ptrIndicesRepNNAbove[1]'='ptrIndicesRepNNLeft[0]' or 'ptrIndicesRepNNAbove[1]'='ptrIndicesRepNNLeft[1]'.

[0207] As illustrated above, the condition 'isIdx0Redundant' && 'isIdx1Redundant' can indicate that the set {'ptrIndicesRepNNAbove[0]','ptrIndicesRepNNAbove[1]'} is redundant with the set {'ptrIndicesRepNNLeft[0]','ptrIndicesRepNNLeft[1]'}. "Redundant" can mean that the two mentioned sets are equal, regardless of their order (see the concept of set equivalence in the Python programming language). Based on the determination that {'ptrIndicesRepNNAbove[0]','ptrIndicesRepNNAbove[1]'} is redundant with the set {'ptrIndicesRepNNLeft[0]','ptrIndicesRepNNLeft[1]'}, 'ptrIndicesRepNNAbove[0]' and 'ptrIndicesRepNNAbove[1]' may be excluded from the MPM derivation list.

[0208] 'fillUpToSix' can be used to fill a list of MPMs pointed to by 'MPM' from list index 'idxInc' up to 5. 'fillUpToSix' can use a procedure to perform the filling. In an example, 'fillUpToSix' may generate a list of MPMs containing different intra-prediction mode indices.

[0209] In the embodiment, the list of MPMs may be derived such that the indices of two normal intra-prediction modes inferred during the prediction of luminance CBs currently located to the left of the luminance CB may appear in the derived list of MPMs. For example, if a neural network-based intra-prediction mode is selected to predict luminance CBs currently located to the left of the luminance CB, and a normal intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) is selected to predict luminance CBs currently located above the luminance CB, then the indices of the two normal intra-prediction modes inferred during the prediction of luminance CBs currently located to the left of the luminance CB may be included in the derived list of MPMs. In the example diagram above, 'fillListMPMsWithoutInitialization' may look like this:

[0210] [Table 8-1]

[0211] [Table 8-2]

[0212] [Table 8-3]

[0213] In the embodiment, the list of MPMs may be derived such that the indices of two normal intra-prediction modes (e.g., non-neural network-based intra-prediction modes) inferred during the prediction of luminance CBs currently located above luminance CBs may be included in the derived list of MPMs. For example, if a neural network-based intra-prediction mode is selected to predict luminance CBs currently located above luminance CBs, and a normal intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) is selected to predict luminance CBs currently located to the left of luminance CBs, then the indices of the two normal intra-prediction modes inferred during the prediction of luminance CBs currently located above luminance CBs may be included in the derived list of MPMs. In the illustrative diagram, 'fillListMPMsWithoutInitialization' may look like this:

[0214] [Table 9-1]

[0215] [Table 9-2]

[0216] [Table 9-3]

[0217] [Table 9-4]

[0218] Instructions related to the neural network-based mode in intra-prediction at chrominance may be signaled. In embodiments, instructions such as the flag nnFlagChrominance, which may characterize the neural network-based intra-prediction mode, may be placed before directFlag in the intra-prediction signaling of the current chrominance CB. The neural network-based intra-prediction mode may be used to predict the current chrominance CB, for example, if nnFlagChrominance is equal to 1. To reduce redundancy, the encoder may decide, based on one or more conditions, whether to include instructions (e.g., the flag nnFlagChrominance) in the signaling of the neural network-based intra-prediction mode in the intra-prediction signaling of the current chrominance CB. The decoder may decide, based on one or more conditions, whether to analyze instructions (e.g., the flag nnFlagChrominance) in the signaling of the neural network-based intra-mode in the intra-prediction signaling of the current chrominance CB. One or more conditions may ensure that nnFlagChrominance does not appear in the intra-predictive signaling of the current chrominance CB if DM is a neural network-based intra-predictive mode (e.g., as shown in Figures 14 and 15). To save bits, if prediction of the current chrominance CB cannot be performed via a neural network-based intra-predictive mode, for example, the condition may ensure that nnFlagChrominance does not appear in the intra-predictive signaling of the current chrominance CB. A neural network-based intra-predictive mode may not be performed based on one or more of the following: the context of the current chrominance CB cannot be extracted, and / or there is no neural network in the neural network-based intra-predictive mode that could be used to predict the current chrominance CB (e.g., as shown in Figures 14 and 15).

[0219] Figure 13 shows exemplary intra-predictive signaling and DM for the current W×H chrominance CB, where its top-left pixel is located at (x,y) in the current frame. The parameter n of the context of the current chrominance CB. l and n a However, it can be specified (for example, as shown in Figure 5). NN IDX This may represent an index of the neural network-based intra-prediction mode. C 10 ,C 11 , and signaling related to C0 can be obtained (for example, as shown in Figure 15).

[0220] Figure 14 shows an exemplary intra-predictive signaling for current chrominance CB. In state S101, the truncated binary coding of the non-DM mode among the four non-DM modes can be signaled. As shown in the figure, the flag characteristic of the neural network-based mode can be placed before directFlag.

[0221] Figure 15 shows exemplary intra-predictive signaling for current chrominance CB. In the embodiment, the nnFlagChrominance property of a neural network-based intra-predictive mode may be placed after directFlag in the intra-predictive signaling for current chrominance CB. The intra-predictive signaling may follow the examples described herein. In state S101, the non-DM truncated binary coding of the four non-DM modes may be signaled. As shown in Figure 15, in the intra-predictive signaling for current chrominance CB, an indicator such as the flag nnFlagChrominance may be placed after directFlag, which may indicate the properties of a neural network-based intra-predictive mode. The neural network-based intra-predictive mode may be used to predict current chrominance CB, for example, if nnFlagChrominance is equal to 1. Whether to signal the flag nnFlagChrominance may be determined based on the value of directFlag. If directFlag indicates that prediction of the current chrominance CB cannot be performed via the neural network-based intra-prediction mode (e.g., directFlag=1, direct mode is used), then signaling of the flag nnFlagChrominance may be skipped. If directFlag indicates that prediction of the current chrominance CB can be performed via the neural network-based intra-prediction mode (e.g., directFlag=0, direct mode is not used), then the flag nnFlagChrominance may be included in the bitstream. On the decoder side, whether or not to parse the flag nnFlagChrominance may be determined based on the value of directFlag. If directFlag indicates that prediction of the current chrominance CB cannot be performed via the neural network-based intra-prediction mode (e.g., directFlag=1, direct mode is used), then the parsing of the flag nnFlagChrominance may be skipped.If directFlag indicates that prediction of the current chrominance CB can be performed via a neural network-based intra-prediction mode (e.g., directFlag=0, direct mode is not used), then the flag nnFlagChrominance can be parsed from the bitstream.

[0222] Figure 16 shows an example of an index on the decoder side for the typical intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) that corresponds to (e.g., best represents) the neural network prediction of the current block.

[0223] At 1600, information representing a neural network-based prediction and / or block context may be obtained. In the embodiment, the information may be obtained for the current block in the picture and / or video. The information may be obtained as described with reference to Figures 5 and 6. For example, the block context may include the set of decoded pixels surrounding the current block.

[0224] In 1605, neural network-based intra-prediction may be performed. For example, neural network-based intra-prediction may be performed on the current block based on the block context and / or a neural network-based intra-prediction mode. Neural network-based intra-prediction may be performed as described with reference to Figures 8 and 10. In embodiments, neural network-based intra-prediction may be performed on the luminance and / or chrominance components of the current block. Signaling information from the bitstream may be decoded. The signaling information may include flags indicating whether the neural network-based prediction mode predicts the current block, as described with reference to Figures 14 and 15, for example.

[0225] In 1610, a non-neural network-based intra-prediction mode can be determined. For example, a non-neural network-based intra-prediction mode may correspond to a performed neural network-based intra-prediction. The non-neural network-based intra-prediction mode can be determined as described with reference to Figures 8 and 10. In embodiments, multiple non-neural network-based intra-prediction modes can be determined as described with reference to Figure 12. The non-neural network-based intra-prediction mode can be determined based on a metric associated with the non-neural network-based intra-prediction mode representing the neural network-based intra-prediction. The metric may include probabilities as described with reference to Figure 8. For example, probabilities can be determined based on a block context and / or a neural network-based prediction mode. In embodiments, the determined non-neural network-based intra-prediction mode may be associated, for example, with the highest probability among multiple representation probabilities associated with multiple non-neural network-based intra-prediction modes.

[0226] In 1615, the prediction mode may be determined. The prediction mode may be determined based on the determined non-neural network-based intra-prediction mode. The prediction mode may be determined for blocks adjacent to the current block. In the embodiment, the MPM may be determined for adjacent blocks based on the determined non-neural network-based intra-prediction mode.

[0227] In an exemplary neural network-based intra-prediction mode, a neural network may be used to predict blocks of different sizes by transposing (e.g., selectively transposing) the context of the current block to be predicted and the neural network prediction of the block, and / or by downsampling the context of the current block and interpolating the neural network prediction of the block. For example, if the context of the current block to be predicted and the neural network prediction of the block are transposed, the inferred index repIdx of the typical intra-prediction mode that represents (e.g., best represents) the neural network prediction of the current block may also be transposed (e.g., likewise).

[0228] Figure 17 shows an example of prediction using a neural network-based intra-prediction mode. In this embodiment, for a given pair of block heights H and widths W, the neural network-based intra-prediction mode may or may not include a neural network that predicts a block of size W × H. Vertical downsampling coefficient γ∈N * and horizontal downsampling coefficient δ∈N * Regarding this, the neural network-based intra-prediction mode is,

[0229]

number

[0230]

number

[0231]

number

[0232] In the embodiment, if the inferred normal intra-prediction mode index belongs to a set of indices for 65 directions of normal intra-prediction modes, PLANAR, and / or DC (e.g., m=67), the transpose of the inferred index may correspond to the transpose of the inferred normal intra-prediction mode direction with respect to the diagonal from the top left to the bottom right. For example, the transpose of the inferred index may be obtained as follows:

[0233] [Table 10]

[0234] PLANAR_IDX may represent the index of PLANAR (e.g., 0). DC_IDX may represent the index of DC (e.g., 1). 68 may correspond to the number of typical intra-prediction modes (e.g., non-neural network-based intra-prediction modes). Different numbers may be used for different codecs based on the number of intra-prediction modes in the codec. For example, this number may exclude the MIP mode and / or CCLM mode plus 1. The transpose of the inferred typical intra-prediction mode index described above may be similar to 2780 in Figure 17.

[0235] In the embodiment, Figure 17 may be adapted when multiple indices of typical intra-prediction modes (e.g., non-neural network-based intra-prediction modes) that represent (e.g., best represent) the neural network predictions of the current block are inferred, as described with respect to Figures 11 and 12. For example, 2780 is the inferred typical intra-prediction mode index repIdx using the same transpose. i This may include the transpose of . For example, in 2780, repIdx i , repIdxTr i It can be changed to repIdxTr i This can be used, for example, to derive a list of MPMs for blocks adjacent to the current block.

[0236] In the example, Figure 17 shows a neural network.

[0237]

number

[0238] Figure 18 shows an example of prediction using a neural network-based intra-prediction mode. In this embodiment, a neural network

[0239]

number

[0240] For blocks to be predicted via a neural network-based intra-prediction mode, the neural network-based mode inference of the index of the normal intra-prediction mode representing (e.g., best representing) the neural network prediction of the block may be combined with the neural network-based mode inference and / or prediction of the implicit transform signaling for the transform to be applied to the residuals of the neural network prediction. For example, the neural network-based intra-prediction mode may be used to predict the implicit transform signaling for the low-frequency non-separable transform (LFNST) to be applied to the block of linear transform coefficients resulting from the horizontal and vertical application of the discrete cosine transform 2 (DCT2) to the residuals of the neural network prediction. If the context of the current luminance CB to be predicted and the neural network prediction of the CB are transposed, the inferred index repIdx of the normal intra-prediction mode representing (e.g., best representing) the neural network prediction of the CB may be transposed (e.g., likewise transposed), and lfnstIdx Y = Predicted pIdx of LFNST group index for i,i∈{1,2} i Y It can be transposed (for example, it can also be transposed). lfnstIdx Y ∈{0,1,2} can currently represent the LFNST index of the luminance CB. For example, lfnstIdx Y =0 may indicate that LFNST is not currently applicable to luminance CB. lfnstIdx Y ∈{1,2} is, for example, the index lfnstIdx in a pair of LFNST kernels given by the LFNST group index. YUsing the -1 LFNST kernel, it can be shown that LFNST is applied to blocks of linear transformation coefficients resulting from applying DCT2 horizontally and vertically to the residuals of neural network predictions. The LFNST group index can indicate the index of the pair of LFNST kernels to be considered and whether the blocks of linear transformation coefficients resulting from the horizontal and vertical application of DCT2 to the residuals of neural network predictions can be transposed.

[0241] Figure 19 shows an example of prediction using a neural network-based intra-prediction mode. In this embodiment, the neural network prediction of a block may be combined with the inference and / or prediction of implicit transformation signaling for transformations to be applied to the residuals of the neural network prediction, and / or the inference of a typical intra-prediction mode index representing (e.g., best representing) the neural network prediction of a block.

[0242]

number

[0243]

number

[0244] In the example, lfnstIdx Y = Predicted pIdx of LFNST group index for i,i∈{1,2} i Y The transpose of can be obtained as follows:

[0245] [Table 11]

[0246] lfnstIdx explained above Y The transpose of the LFNST group index prediction for =i,i∈{1,2} may be similar to 3001 and 3002 described herein with respect to Figure 19.

[0247] In the example, Figure 18 shows a neural network.

[0248]

number

[0249]

number

[0250] Figure 20 shows an example of prediction using a neural network-based intra-prediction mode. In this example, the vector P1 returned by the neural network is shown. Y It can be transposed. For example, in 3003, the index P1 belonging to [|1,6|] Y The coefficients in (for example, each coefficient) are P1 Y The coefficients can be swapped with the coefficients that are symmetric with respect to the midpoint of [1:7]. In the example, the vector P2 returned by the neural network Y It can be transposed. In 3004, the P2 of the index belonging to [|1,6|] Y The coefficients in (for example, each coefficient) are P2 Y [1:7] can be swapped with the coefficient that is symmetric with respect to the midpoint. The final predicted value of the current block, the value of repIdxTr, and predTrGrpIdx are explained with respect to Figure 19. Y The values ​​of these can be equal to those in Figure 20.

[0251] In a combination of inference by a neural network-based intra-prediction mode of an index in a typical intra-prediction mode that represents (e.g., best represents) a neural network prediction of a block, and inference and / or prediction by a neural network-based intra-prediction mode of implicit transformation signaling for a transformation to be applied to the residuals of the neural network prediction, the transformation targeted by the inference and / or prediction may be a first-order, second-order, third-order, and / or other level transformation. The transformation targeted by the inference and / or prediction (e.g., each transformation) may be applied to a block of transformation coefficients that are the residuals of the neural network prediction, or the output of a different transformation (e.g., a previous transformation) associated with a luminance block or a chrominance block.

[0252] Figure 21 shows an example of prediction using a neural network-based intra-prediction mode. In this embodiment, the neural network-based intra-prediction mode may or may not include a neural network that predicts a block of size W × H for a pair of block height H and width W. Vertical downsampling coefficient γ∈N * and horizontal downsampling coefficient δ∈N * Regarding this, the neural network-based intra-prediction mode is,

[0253]

number

[0254]

number

[0255]

number

[0256]

number

[0257]

number

[0258]

number

[0259] Figure 22 shows an example of a neural network-based pipeline on the encoder side 4100. In 4101, the neural network-based pipeline predicts Y from the context X of block Y.

[0260]

number

[0261]

number

[0262] Figure 23 shows an example of a neural network-based pipeline on the decoder side.4200.4201 In this pipeline, the neural network-based pipeline predicts the current block from context X.

[0263]

number

[0264]

number

[0265]

number

[0266]

number

[0267]

number

[0268]

number

[0269]

number

[0270] If the block is currently coded in intra mode, and the neural network-based intra prediction mode returns the predicted predTrGrpIdx for the selected LFNST group index, then the neural network-based mode can be used to predict the implicit signaling of the LFNST. Figure 24 shows bit B remainder This shows an exemplary decision tree representing the predictive coding of trGrpIdx for predTrGrpIdx, which results in [the desired outcome]. Other predictive coding schemes may be compatible with the example described with respect to Figure 22.

[0271] Predictive decoding can be performed if the block is currently coded in intra mode and a neural network-based intra prediction mode returns a predicted predTrGrpIdx for the selected LFNST group index. Figure 25 shows an exemplary predictive decoding associated with predictive coding as described herein with respect to Figure 24.

[0272] For a given block predicted by a neural network-based intra-prediction mode, the index of the normal intra-prediction mode that represents (e.g., best represents) the neural network prediction of the block can be used to represent the intra-prediction of the block at any time, for example, during video encoding or video decoding. The index of the normal intra-prediction mode that represents the neural network prediction of a block can be used even if the normal intra-prediction mode index is used during either encoding or decoding of another block that may be located in a different coding tree unit (CTU) than the one hosting the previous block. The index of the normal intra-prediction mode that represents the neural network-based prediction of a block can be used even if the normal intra-prediction mode index is used during either encoding or decoding of another block that may be located in a different frame than the one hosting the previous block. This can be illustrated in the context of deriving a list of MPMs for a given luminance CB. The derivation of the list of MPMs can be implemented, for example, by a computing system running software. Figures 26, 27, 28, 29, and 30 illustrate the derivation of the list of MPMs for current luminance CB, illustrating how the list of MPMs can be derived for current luminance CB in a system including a neural network-based intra-predictive mode.

[0273] Figure 26 illustrates an exemplary derivation of the list of MPMs for current luminance CBs. The derivation can be implemented, for example, using software running on a computing system. The derivation may involve filling in a general list of 22 MPMs. In Figure 26, it can be assumed that luminance CBs directly adjacent to current luminance CBs are predicted within the intra. This may be true if the current luminance CBs are within the intra frame. When filling in the general list of MPMs, if an intra-predictive mode index is proposed to fill the cell at index k, and the intra-predictive mode index does not already exist in the general list of MPMs, the index may be placed in the cell at index k, and filling may resume at the cell at index k+1. Otherwise (for example, if the intra-predictive mode index already exists in the general list of MPMs), the intra-predictive mode index may be ignored, and filling may resume at the cell at index k. Referring to Figure 26, in 4300, PLANAR IDXThis can be proposed to fill the cell at index 0 of the MPM list. In 4301, if the height of the current luminance CB is greater than its width, an index of the intra-prediction mode that predicts the luminance CB overlapping square "1", which can be called the above CB, can be proposed to fill the general list of MPMs. Otherwise, an index of the intra-prediction mode that predicts the luminance CB overlapping square "2", which can be called the left CB, can be proposed to fill the general list of MPMs. In 4302, if the height of the current luminance CB is greater than its width, an index of the intra-prediction mode that predicts the luminance CB overlapping square "2" can be proposed to fill the general list of MPMs. Otherwise, an index of the intra-prediction mode that predicts the luminance CB overlapping square "1" can be proposed to fill the general list of MPMs. In 4303, an index of the intra-prediction mode that predicts the luminance CB overlapping square "4", which can be called the bottom-left CB, can be proposed to fill the general list of MPMs. In 4304, an index of an intra-predictive mode that predicts a luminance CB overlapping square "3", which may be called the upper right CB, may be proposed to fill in the general list of MPMs. In 4305, an index of an intra-predictive mode that predicts a luminance CB overlapping square "0", which may be called the upper left CB, may be proposed to fill in the general list of MPMs.

[0274] In 4306, an index of a first intra-prediction mode derived by decoder-side intra-mode derivation (DIMD) may be proposed to fill the general list of MPMs. In 4307, an index of a second intra-prediction mode derived by DIMD may be proposed to fill the general list of MPMs. In 4308, if the index of the second intra-prediction mode in the current general list of MPMs is neither PLANAR_IDX nor DC_IDX, an index of the eight directional intra-prediction modes (e.g., each of the eight directional intra-prediction modes) that are neighbors of this second intra-prediction mode in the current general list of MPMs with respect to directivity may be proposed to fill the general list of MPMs. In 4309, if the third intra-predictive mode index in the current general list of the MPM is neither PLANAR_IDX nor DC_IDX, then indices of the eight directional intra-predictive modes (e.g., each of the eight directional intra-predictive modes) that are neighbors of this third intra-predictive mode in the current general list of the MPM with respect to directivity may be proposed to fill the general list of the MPM. In 4310, if the second or third intra-predictive mode index in the current general list of the MPM is DC IDX Equally, if the fourth intra-prediction mode index in the current general list of MPMs is neither PLANAR_IDX nor DC_IDX, then the indices of the six directional intra-prediction modes adjacent to this fourth intra-prediction mode in the current general list of MPMs with respect to directivity (e.g., each of the six directional intra-prediction modes) may be proposed to fill the general list of MPMs. In 4311, the remaining cells in the general list of MPMs may be filled with default intra-prediction mode indices. Once the general list of 22 MPMs is filled, the first six MPMs may be grouped into a list of primary MPMs, and the last 16 MPMs may be aggregated into a list of secondary MPMs.

[0275] Figure 27 illustrates an exemplary application of the derivation depicted in Figure 26 of an implementation described herein, which typically involves inferring an index of intra-prediction modes to represent the neural network prediction of the current block. Figure 27 involves the derivation of a list of MPMs for the current luminance CB in a video codec, for example, which includes a neural network-based intra-prediction mode. The derivation may be implemented, for example, on a computing system running software. In Figure 27, as in Figure 26, luminance CBs directly adjacent to the current luminance CB can be predicted intra-. The exemplary derivation in Figure 27 proceeds similarly to the derivation in Figure 26, except that, for example, for each of the five luminance CBs (e.g., each of the five luminance CBs) that currently enclose a luminance CB and overlap with squares "0", "1", "2", "3", and "4", respectively, if the index of the intra-prediction mode selected to predict the CB is equal to the index NN_IDX of the neural network-based intra-prediction mode, then the index may be replaced by the index repIdx of the best-representing normal intra-prediction mode that represents the neural network prediction of the adjacent luminance CB. For each of these five luminance CBs (e.g., each of the five luminance CBs), this potential replacement may be performed before proposing the index of the intra-prediction mode selected to predict the adjacent luminance CB in order to fill the general list of MPMs.

[0276] Figure 28 illustrates an exemplary derivation of a list of MPMs for a current luminance CB when multiple luminance CBs directly adjacent to the current luminance CB are predicted within the inter. The implementation can be, for example, run on a computing system running software. Figure 28 illustrates an exemplary case where (e.g., adjacent) luminance CBs around the current luminance CB overlapping squares "2" and "4" respectively can be predicted within the inter, while (e.g., adjacent) luminance CBs around the current luminance CB overlapping squares "0", "1", and "3" respectively can be predicted intra. The width of a current luminance CB can be greater than its height. Referring to Figure 28, in 4400, PLANAR_IDX may be proposed to fill the cell at index 0 of the list of MPMs. In 4401, an index for an intra-prediction mode that predicts a luminance CB overlapping square "1", which may be called an upper CB, may be proposed to fill the general list of MPMs. In 4402, for example, a portion of the code (e.g., a subroutine) that could be called "getIpmInfo" may operate on a luminance CB overlapping square "2", which could be called the left CB. The intra-prediction mode index returned by "getIpmInfo" may be proposed to fill in a general list of MPMs. Using the decoded motion vector, "getIpmInfo" may find the last decoded CU predicted intra-intra, linked to the luminance CB argument of "getIpmInfo" via motion, in past decoded frames. "getIpmInfo" may search the history of motion vectors that give rise to the luminance CB argument of "getIpmInfo" to find the last CU predicted intra-intra. "getIpmInfo" may return the index of the intra-prediction mode selected to predict the luminance CB of the found CU. In 4403, an index of the intra-prediction mode that predicts a luminance CB overlapping square "3", which could be called the upper right CB, may be proposed to fill in a general list of MPMs.In 4404, an index of intra-predictive modes that predict the luminance CB overlapping square "0", which may be called the upper-left CB, may be proposed to fill a general list of MPMs. In 4405, "getIpmInfo" may be called on the luminance CB overlapping square "4", which may be called the lower-left CB, and the returned intra-predictive mode index may be proposed to fill a general list of MPMs. The processes in 4406, 4407, 4408, 4409, 4410, and 4411 in Figure 28 may proceed in accordance with those described in relation to 4306, 4307, 4308, 4309, 4310, and 4311 in Figure 26, respectively.

[0277] Figure 29 illustrates the application of the implementation described herein to the derivation depicted in Figure 28, which typically involves inferring an index of an intra-prediction mode to represent the neural network prediction of the current block. Figure 29 is accompanied by the derivation of a list of MPMs for the current luminance CB, including a neural network-based intra-prediction mode. In Figure 29, for example, as in Figure 28, luminance CBs around (e.g., adjacent) current luminance CBs overlapping squares "2" and "4," respectively, can be predicted intra-internally, and luminance CBs around (e.g., adjacent) current luminance CBs overlapping squares "0," "1," and "3," respectively, can be predicted intra-intra. The width of a current luminance CB can be greater than its height. In comparison with Figure 28, the first difference in Figure 29 is that, for three luminance CBs (e.g., each of the three luminance CBs) that are currently adjacent to a luminance CB and overlap in squares "0", "1", and "3", respectively, if the index of the intra-prediction mode selected to predict this CB is equal to the index NN_IDX of the neural network-based intra-prediction mode, then the index may be replaced by the index repIdx of the normal intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) that represents (e.g., best represents) the neural network prediction of the adjacent luminance CB. The second difference is that, for two luminance CBs (e.g., each of the two luminance CBs) that are currently adjacent to a luminance CB and overlap squares "2" and "4" respectively, during a call to "getIpmInfo", if the index of the intra-prediction mode selected to predict the luminance CB of the last CU predicted intra-, is equal to the index NN_IDX of the neural network-based intra-prediction mode, then this index may be replaced by the index repIdx of the normal intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) that represents (e.g., best represents) the neural network prediction of the luminance CB of the found CU.

[0278] Figures 28 and 29 illustrate the derivation of a list of MPMs for a current luminance CB when the width of the current luminance CB is greater than its height, with luminance CBs (e.g., adjacent) around the current luminance CB overlapping with squares "2" and "4" predicted inter-inter, and luminance CBs (e.g., adjacent) around the current luminance CB overlapping with squares "0", "1", and "3" predicted intra-intra, respectively. However, exemplary derivations (e.g., depicted in these figures) may be applicable to other cases involving different luminance CBs (e.g., adjacent) around the current luminance CB predicted inter- and / or intra-intra. For example, Figure 30 illustrates another exemplary application of the derivation depicted in Figure 28 of the implementation described herein, which typically involves inferring an index of an intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) to represent the neural network prediction of the current block. Figure 30 illustrates an exemplary derivation of a list of MPMs for current luminance CBs in a video codec, for example, including a neural network-based intra-prediction mode. The derivation can be performed, for example, by software running on a computing system. Luminance CBs around (e.g., adjacent) current luminance CBs overlapping with squares "1" and "3" can be predicted intra-internally, and luminance CBs around current luminance CBs overlapping with squares "0", "2", and "4" can be predicted intra-intra. The height of a current luminance CB can be greater than its height.

[0279] The principle shown in Figures 26, 27, 28, 29, and 30 can be applied when a neural network-based intra-prediction mode can infer the indices of k normal intra-prediction modes that best represent the neural network predictions of the current block, instead of inferring the index of a normal intra-prediction mode that best represents the neural network predictions of the current block.

[0280] In one embodiment, “getIpmInfo” may be programmed to perform a different operation. For example, during the encoding of a given video, for a first non-intra frame following a given reconfigured intra-frame, the index of the intra-prediction mode selected to predict the luminance CB (e.g., each luminance CB) in a given reconfigured intra-frame may be propagated to the intra-predicted luminance CB in the non-intra frame following the associated motion vector. This process may result in a map of intra-prediction mode indices, where the intra-predicted luminance CB (e.g., each luminance CB) in the non-intra frame possesses its propagated intra-prediction mode index. This process may be repeated for non-intra frames. “getIpmInfo” may use its deflection luminance CB motion vector to determine the CU involved in motion compensation, resulting in a prediction of its deflection luminance CB and the propagated intra-prediction mode index of the found CU. “getIpmInfo” may return the propagated intra-prediction mode index. A similar process can be applied to decoding a given video. In an embodiment, for a given luminance CB, the substitution of the index NN_IDX of the neural network-based intra-prediction mode with the index repIdx of the normal intra-prediction mode (e.g., non-neural network-based prediction mode) that best represents the neural network prediction of the luminance CB may not occur during the call to "getIpmInfo". In an embodiment, this may occur during the creation of a map of propagating intra-prediction mode indices when moving from one video frame to the next. Figure 31 illustrates the corresponding exemplary derivation of the list of MPMs.

[0281] During the coding phase, for a given block predicted by a neural network-based intra-prediction mode, an index of the normal intra-prediction mode that best represents the neural network prediction of the block may be used at any time during the coding of the block following the intra-prediction process. If the implicit signaling of the indices of the transformations to be applied during the coding of the block uses the index of the selected intra-prediction mode, the inferred index of the normal intra-prediction mode may replace the index of the neural network-based intra-prediction mode. Similarly, during the decoding phase, for a given block predicted by a neural network-based intra-prediction mode, an index of the normal intra-prediction mode that best represents the neural network prediction of the block may be used at any time during the decoding of the block following the intra-prediction process. If the implicit signaling of the indices of the transformations to be applied during the decoding of the block uses the index of the selected intra-prediction mode, the inferred index of the normal intra-prediction mode may replace the index of the neural network-based intra-prediction mode. Figure 32 presents an adaptation of Figure 8 to such an implementation. Figure 32 shows an example in which the decoder side infers the index of a normal intra-prediction mode (e.g., a non-neural network-based intra-prediction mode) that can (e.g., best) represent the neural network prediction of the current block.4500 The neural network-based intra-prediction mode may take (e.g., obtain) the context X surrounding Y to provide a prediction Y^ and a vector P for Y, the coefficient of this index i may correspond on the decoder side to the probability that the normal intra-prediction (e.g., a non-neural network-based intra-prediction mode) of index i represents (e.g., best represents) the neural network prediction Y^.4501 repIdx may be equal to argmax(P).4502 In the embodiment, the inverse quantizer 4505 reconstructs the quadratic transformation coefficients

[0282]

number

[0283]

number

[0284]

number

[0285]

number

[0286]

number

[0287]

number

[0288]

number

[0289] The principle presented in Figure 32 can be applied to indices of several typical intra-prediction modes corresponding to (e.g., best represented) neural network predictions of a block. While features and elements are described above in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with other features and elements. In addition, the methods described herein can be implemented in computer programs, software, or firmware embedded in computer-readable media for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted via wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor associated with software can be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A video decoding device, The processor comprises, Currently, for coding blocks, we are obtaining information representing the neural network-based prediction mode and block context, Based on the block context and the neural network-based prediction mode, a neural network-based predictor is generated for the currently coding block. Based on the multiple pixels adjacent to the currently coding block, a non-neural network-based intra-prediction mode corresponding to the generated neural network-based predictor is determined. Based on the aforementioned non-neural network-based intra-prediction mode, a prediction mode for the adjacent block is determined, A video decoding device configured to decode the adjacent blocks based on the prediction mode.

2. The device according to claim 1, wherein the current coding block includes a current luminance block and a current chrominance block, and the generated neural network-based predictor includes a luminance predictor for the current luminance block and a chrominance predictor for the current chrominance block.

3. Determining the aforementioned non-neural network-based intra-prediction mode is Based on the block context and the neural network-based prediction mode, multiple representation probabilities are obtained associated with multiple non-neural network-based intra-prediction modes. The device according to claim 1, further comprising selecting the determined non-neural network-based intra-prediction mode from the plurality of non-neural network-based intra-prediction modes based on the plurality of representation probabilities.

4. The device according to claim 3, wherein a first representation probability associated with a first non-neural network-based intra-prediction mode is configured to indicate the probability that using the first non-neural network-based intra-prediction mode for the currently coding block results in a representation of the generated neural network-based predictor, where a higher probability indicates that the representation is more similar to the generated neural network-based predictor compared to a lower probability, and the non-neural network-based intra-prediction mode associated with the highest representation probability is selected.

5. The aforementioned processor, Obtain an intra-prediction index associated with the determined non-neural network-based intra-prediction mode corresponding to the generated neural network-based predictor, The device according to claim 1, further configured to add the intra-prediction index associated with the determined non-neural network-based intra-prediction mode to the most probable mode (MPM) list of the adjacent block.

6. The aforementioned processor, The device according to claim 1, further configured to apply at least one of a linear inverse transform or a quadratic inverse transform based on the non-neural network-based intra-prediction mode.

7. The aforementioned processor, The device according to claim 1, further configured to derive the most likely mode (MPM) for the adjacent block based on the determined non-neural network-based intra-prediction mode.

8. A video encoding device, The processor comprises, Currently, for coding blocks, we are obtaining information representing the neural network-based prediction mode and block context, Based on the block context and the neural network-based prediction mode, a neural network-based predictor is generated for the currently coding block. Based on the multiple pixels adjacent to the currently coding block, a non-neural network-based intra-prediction mode corresponding to the generated neural network-based predictor is determined. Based on the aforementioned non-neural network-based intra-prediction mode, a prediction mode for the adjacent block is determined, A video encoding device configured to encode the adjacent blocks based on the prediction mode.

9. The device according to claim 8, wherein the current coding block includes a current luminance block and a current chrominance block, and the generated neural network-based predictor includes a luminance predictor for the current luminance block and a chrominance predictor for the current chrominance block.

10. Determining the non-neural network-based intra-prediction mode is performed by the processor, Based on the block context and the neural network-based prediction mode, multiple representation probabilities are obtained associated with multiple non-neural network-based intra-prediction modes. The device according to claim 8, further comprising being configured to select the determined non-neural network-based intra-prediction mode from the plurality of non-neural network-based intra-prediction modes based on the plurality of representation probabilities.

11. The device according to claim 10, wherein a first representation probability associated with a first non-neural network-based intra-prediction mode is configured to indicate the probability that using the first non-neural network-based intra-prediction mode for the currently coding block results in a representation of the generated neural network-based predictor, where a higher probability indicates that the representation is more similar to the generated neural network-based predictor compared to a lower probability, and the non-neural network-based intra-prediction mode associated with the highest representation probability is selected.

12. A method for video decoding, Currently, for coding blocks, we are obtaining information representing the neural network-based prediction mode and block context, Based on the block context and the neural network-based prediction mode, a neural network-based predictor is generated for the currently coding block. Based on the multiple pixels adjacent to the currently coding block, a non-neural network-based intra-prediction mode corresponding to the generated neural network-based predictor is determined. Based on the aforementioned non-neural network-based intra-prediction mode, a prediction mode for the adjacent block is determined, A method comprising decoding the adjacent block based on the prediction mode.

13. The method according to claim 12, wherein the current coding block includes a current luminance block and a current chrominance block, and the generated neural network-based predictor includes a luminance predictor for the current luminance block and a chrominance predictor for the current chrominance block.

14. Determining the aforementioned non-neural network-based intra-prediction mode is Based on the block context and the neural network-based prediction mode, multiple representation probabilities are obtained associated with multiple non-neural network-based intra-prediction modes. The method according to claim 12, further comprising selecting the determined non-neural network-based intra-prediction mode from the plurality of non-neural network-based intra-prediction modes based on the plurality of representation probabilities.

15. The method according to claim 14, wherein a first representation probability associated with a first non-neural network-based intra-prediction mode is configured to indicate the probability that using the first non-neural network-based intra-prediction mode for the currently coding block results in a representation of the generated neural network-based predictor, where a higher probability indicates that the representation is more similar to the generated neural network-based predictor compared to a lower probability, and the non-neural network-based intra-prediction mode associated with the highest representation probability is selected.

16. Obtain an intra-prediction index associated with the determined non-neural network-based intra-prediction mode corresponding to the generated neural network-based predictor, The method according to claim 12, further comprising adding the intra-prediction index associated with the determined non-neural network-based intra-prediction mode to the most probable mode (MPM) list of adjacent blocks.

17. The method according to claim 12, further comprising applying at least one of a linear inverse transform or a quadratic inverse transform based on the non-neural network-based intra-prediction mode.

18. The method according to claim 12, further comprising deriving the most likely mode (MPM) for the adjacent block based on the determined non-neural network-based intra-prediction mode.

19. A video encoding method, Currently, for coding blocks, we are obtaining information representing the neural network-based prediction mode and block context, Based on the block context and the neural network-based prediction mode, a neural network-based predictor is generated for the currently coding block. Based on the multiple pixels adjacent to the currently coding block, a non-neural network-based intra-prediction mode corresponding to the generated neural network-based predictor is determined. Based on the aforementioned non-neural network-based intra-prediction mode, a prediction mode for the adjacent block is determined, A method comprising encoding the adjacent blocks based on the prediction mode.

20. The method according to claim 19, wherein the current coding block includes a current luminance block and a current chrominance block, and the generated neural network-based predictor includes a luminance predictor for the current luminance block and a chrominance predictor for the current chrominance block.

21. Determining the aforementioned non-neural network-based intra-prediction mode is Based on the block context and the neural network-based prediction mode, multiple representation probabilities are obtained associated with multiple non-neural network-based intra-prediction modes. The method according to claim 19, further comprising selecting the determined non-neural network-based intra-prediction mode from the plurality of non-neural network-based intra-prediction modes based on the plurality of representation probabilities.

22. The method according to claim 21, wherein a first representation probability associated with a first non-neural network-based intra-prediction mode is configured to indicate the probability that using the first non-neural network-based intra-prediction mode for the currently coding block results in a representation of the generated neural network-based predictor, where a higher probability indicates that the representation is more similar to the generated neural network-based predictor compared to a lower probability, and the non-neural network-based intra-prediction mode associated with the highest representation probability is selected.

23. Obtain an intra-prediction index associated with the determined non-neural network-based intra-prediction mode corresponding to the generated neural network-based predictor, The method according to claim 19, further comprising adding the intra-prediction index associated with the determined non-neural network-based intra-prediction mode to the most probable mode (MPM) list of adjacent blocks.

24. The method described above is The method according to claim 19, further comprising applying at least one of a linear inverse transform or a quadratic inverse transform based on the non-neural network-based intra-prediction mode.

Images