Explicit predictive coding for point cloud compression

Abstract

Some embodiments of a method may include: obtaining a previously decoded point cloud frame as a reference point cloud frame; obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame; determining a predictor feature map based on the reference point cloud frame and the motion features; obtaining a current feature map representing the current point cloud frame, given the predictor feature map as a condition; and reconstructing the current point cloud frame based on the current feature map and using the predictor feature map as a condition.

Description

[0001] Cross-references to related applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 543,479, filed on October 10, 2023, entitled “EXPLICIT PREDICTIVE CODING FORPOINT CLOUD COMPRESSION”, which is hereby incorporated herein by reference in its entirety.

[0002] Merging by reference This application incorporates, by reference in its entirety, U.S. Provisional Patent Application Serial No. 63 / 543,484, filed October 10, 2023, entitled “IMPLICITPREDICTIVE CODING FOR POINT CLOUD COMPRESSION”; U.S. Provisional Patent Application Serial No. 63 / 460,798, filed April 20, 2023, entitled “Generative-based Predictive Coding for Point Cloud Compression” (“'798 Application”); U.S. Provisional Patent Application Serial No. 63 / 526,130, filed July 11, 2023, entitled “Generative-based Predictive Coding for LiDAR Point Cloud Compression” (“'130 Application”); and U.S. Provisional Patent Application Serial No. 63 / 526,130, filed July 12, 2022, entitled “Deep Distribution-Aware Point Feature Extractor for AI-Based Point Cloud”. The U.S. Provisional Patent Application Serial No. 63 / 388,600 (“'600 Application”) filed on January 10, 2023, entitled “Context-Aware Voxel-Based Upsampling for Point Cloud Processing”; the U.S. Provisional Patent Application Serial No. 63 / 438,212 (“'212 Application”) filed on January 10, 2023, entitled “Context-Aware Voxel-Based Upsampling for Point Cloud Processing”; and the International Patent Application Serial No. 63 / 424,421 (“'421 Application”) filed on November 10, 2022, entitled “Heterogeneous Mesh Autoencoders”. Background Technology

[0003] Point cloud is a data format used across several commercial sectors, from autonomous driving, robotics, AR / VR, civil engineering, computer graphics, to animation / film. 3D LiDAR sensors have been deployed in autonomous vehicles, and affordable LiDAR sensors have been released from sources such as Velodyne Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR Camera L515. With advancements in sensing technology, 3D point cloud data is becoming more readily available than ever before and is expected to be the ultimate enabler in the applications mentioned.

[0004] It is also argued that point cloud data consumes a significant portion of network traffic, for example, in immersive communications (VR / AR) and automotive applications connected via 5G networks. Efficient representation formats can be used with point clouds and communications. Specifically, raw point cloud data is organized and processed for world modeling and sensing purposes. Compression of raw point clouds can be used for data storage and transmission within relevant scenes. Summary of the Invention

[0005] A first example method according to some embodiments may include: obtaining a previously decoded point cloud frame as a reference point cloud frame; obtaining motion features describing the motion between the reference point cloud frame and a current point cloud frame; determining a predictor feature map based on the reference point cloud frame and the motion features; obtaining a current feature map representing the current point cloud frame, given the predictor feature map as a condition; and reconstructing the current point cloud frame based on the current feature map and using the predictor feature map as a condition.

[0006] For some embodiments of the first example method, obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame includes: performing motion analysis using the reference point cloud frame and the current point cloud frame as input to generate a motion feature map; and entropy encoding the motion feature map to generate the motion features.

[0007] For some embodiments of the first example method, obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame includes: performing feature extraction on the reference point cloud frame to generate a first feature map; performing feature extraction on the current point cloud frame to generate a second feature map; fusing the first feature map and the second feature map to generate a fused feature map; and performing feature aggregation on the fused feature map to generate the motion features.

[0008] In some embodiments of the first example method, the reference point cloud frame and the current point cloud frame are voxel-based representations, and performing the feature extraction on the reference point cloud frame includes: passing the reference point cloud frame through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the first feature map, and performing the feature extraction on the current point cloud frame includes: passing the current point cloud frame through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the second feature map.

[0009] In some embodiments of the first example method, the reference point cloud frame and the current point cloud frame are point-based representations, and performing the feature extraction on the reference point cloud frame includes: performing block partitioning on the reference point cloud frame; and performing block-by-block feature extraction on the block-partitioned reference point cloud frame to generate the first feature map, and performing the feature extraction on the current point cloud frame includes: performing block partitioning on the current point cloud frame; and performing block-by-block feature extraction on the block-partitioned current point cloud frame to generate the second feature map.

[0010] In some embodiments of the first example method, obtaining the motion features includes performing motion analysis training.

[0011] In some embodiments of the first example method, determining the predictor feature map based on the reference point cloud frame and the motion features includes: performing feature extraction on the reference point cloud frame to generate a first feature map; concatenating the first feature map with the motion features; passing the concatenated first feature map through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the predictor feature map.

[0012] In some embodiments of the first example method, performing the feature extraction on the reference point cloud frame includes performing point synthesis.

[0013] In some embodiments of the first example method, performing the feature extraction on the current point cloud frame includes performing point synthesis.

[0014] A first example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0015] A second example method according to some embodiments may include: obtaining a previously encoded point cloud frame as a reference point cloud frame; obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame; determining a predictor feature map based on the reference point cloud frame and the motion features; determining a current feature map representing the current point cloud frame given the predictor feature map as a condition; and encoding the current feature map into a bitstream using the predictor feature map as a condition.

[0016] For some embodiments of the second example method, obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame includes: performing motion analysis using the reference point cloud frame and the current point cloud frame as input to generate a motion feature map; and entropy encoding the motion feature map to generate the motion features.

[0017] In some embodiments of the second example method, obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame includes: performing feature extraction on the reference point cloud frame to generate a first feature map; performing feature extraction on the current point cloud frame to generate a second feature map; fusing the first feature map and the second feature map to generate a fused feature map; and performing feature aggregation on the fused feature map to generate the motion features.

[0018] In some embodiments of the second example method, the reference point cloud frame and the current point cloud frame are voxel-based representations, and performing the feature extraction on the reference point cloud frame includes: passing the reference point cloud frame through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the first feature map, and performing the feature extraction on the current point cloud frame includes: passing the current point cloud frame through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the second feature map.

[0019] In some embodiments of the second example method, the reference point cloud frame and the current point cloud frame are point-based representations, and performing the feature extraction on the reference point cloud frame includes: performing block partitioning on the reference point cloud frame; and performing block-by-block feature extraction on the block-partitioned reference point cloud frame to generate the first feature map, and performing the feature extraction on the current point cloud frame includes: performing block partitioning on the current point cloud frame; and performing block-by-block feature extraction on the block-partitioned current point cloud frame to generate the second feature map.

[0020] In some embodiments of the second example method, obtaining the motion features includes performing motion analysis training.

[0021] In some embodiments of the second example method, determining the predictor feature map based on the reference point cloud frame and the motion features includes: performing feature extraction on the reference point cloud frame to generate a first feature map; concatenating the first feature map with the motion features; passing the concatenated first feature map through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the predictor feature map.

[0022] In some embodiments of the second example method, determining the current feature map includes: performing feature extraction on the current point cloud frame to generate a second feature map; concatenating the second feature map with the predictor feature map; passing the concatenated second feature map through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the current feature map.

[0023] For some embodiments of the second example method, performing the feature extraction on at least one of the reference point cloud frame and the current point cloud frame includes using an Inception-ResNet block.

[0024] A second example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0025] A third example apparatus according to some embodiments may include at least one processor configured to perform any of the methods listed above.

[0026] A fourth example apparatus according to some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any of the methods listed above.

[0027] A fifth example apparatus according to some embodiments may include: at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any of the methods listed above.

[0028] Example signals according to some embodiments may include bitstreams generated according to any of the methods listed above. Attached Figure Description

[0029] Figure 1A This is a system diagram illustrating an example communication system according to some embodiments.

[0030] Figure 1B The illustration is based on some embodiments and can be used Figure 1AThe diagram shows a system diagram of an example wireless transmit / receive unit (WTRU) used in a communication system.

[0031] Figure 1C This is a system diagram illustrating a set of example interfaces of a system according to some embodiments.

[0032] Figure 2A This is a functional block diagram of a block-based video encoder (such as an encoder for multi-function video coding (VVC)) according to some embodiments.

[0033] Figure 2B This is a functional block diagram of a block-based video decoder (such as a decoder for VVC) according to some embodiments.

[0034] Figure 3A This is a schematic side view of an example waveguide display that can be used with extended reality (XR) applications according to some embodiments.

[0035] Figure 3B This is a schematic side view illustrating example alternative display types that can be used with extended reality applications according to some embodiments.

[0036] Figure 3C This is a schematic side view illustrating example alternative display types that can be used with extended reality applications according to some embodiments.

[0037] Figure 4 This is a process diagram illustrating the encoding and decoding process of an example codec with explicit prediction according to some embodiments.

[0038] Figure 5 This is a process diagram illustrating an example training process for motion analysis according to some embodiments.

[0039] Figure 6 This is a flowchart illustrating an example training process for generating blocks of predictor features according to some embodiments.

[0040] Figure 7 This is a flowchart illustrating an example motion analysis process according to some embodiments.

[0041] Figure 8 This is a diagram illustrating an example feature extraction process for a voxel-based representation according to some embodiments.

[0042] Figure 9 This is a diagram illustrating an example feature extraction process for a point-based representation according to some embodiments.

[0043] Figure 10 This is a flowchart illustrating the process of example feature aggregation (FC block) according to some embodiments.

[0044] Figure 11 This is a diagram illustrating the process of generating an example feature predictor (FA block) according to some embodiments.

[0045] Figure 12 This is a diagram illustrating the process of example feature aggregation / extraction in an encoder (FE block) according to some embodiments.

[0046] Figure 13 This is a diagram illustrating the process of example feature aggregation / extraction in a decoder (FD block) according to some embodiments.

[0047] Figure 14 This is a diagram illustrating the process of example feature aggregation / extraction in a decoder (FD block) according to some embodiments.

[0048] Figure 15 This is a flowchart illustrating an example conditional feature decoder according to some embodiments.

[0049] Figure 16 This is a process diagram illustrating the synthesis of example point blocks (PS blocks) according to some embodiments.

[0050] Figure 17 This is a process diagram illustrating an example Inception-ResNet block for feature aggregation according to some embodiments.

[0051] Figure 18 This is a process diagram illustrating an example transformer block for feature aggregation according to some embodiments.

[0052] Figure 19 This is a process diagram illustrating an example self-attention block according to some embodiments.

[0053] Figure 20 This is a flowchart illustrating an example decoding process according to some embodiments.

[0054] Figure 21 This is a flowchart illustrating an example encoding process according to some embodiments.

[0055] The entities, connections, arrangements, etc., depicted and combined with the descriptions in the various figures are presented by way of example and not by way of limitation. Accordingly, any and all statements or other indications about what a particular figure “depicts,” what a particular element or entity in a particular figure “is” or “has,” and any and all similar statements (which may be interpreted as absolute and therefore limiting in isolation and out of context) may only be correctly interpreted as having a clause such as “In at least one embodiment, …” presupposedly appeared above. For the sake of brevity and clarity, this implicit introductory clause will not be repeated verbatim in the detailed description. Detailed Implementation

[0056] Figure 1A This is a schematic diagram illustrating an example communication system 100 in which one or more of the disclosed embodiments may be implemented. The communication system 100 may be a multi-access system that provides content such as voice, data, video, messages, and broadcasts to multiple wireless users. The communication system 100 enables multiple wireless users to access such content by sharing system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tail Unique Word DFT Spread Spectrum OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multicarrier (FBMC), etc.

[0057] like Figure 1A As shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106, Public Switched Telephone Network (PSTN) 108, Internet 110, and other networks 112. However, it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, WTRUs 102a, 102b, 102c, and 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain environments), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. Any of WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0058] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a and 114b may be any type of device configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks, such as CN 106, the Internet 110, and / or other networks 112. For example, base stations 114a and 114b may be base transceiver stations (BTS), NodeBs, eNodeBs, home NodeBs, home eNodeBs, gNBs, NR NodeBs, site controllers, access points (APs), wireless routers, etc. Although base stations 114a and 114b are each depicted as a single element, it should be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0059] Base station 114a may be part of RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage of a specific geographic area, which may be relatively fixed or may change over time. The cell may be further divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Therefore, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In one embodiment, base station 114a may employ multiple-input multiple-output (MIMO) technology and may use multiple transceivers for each sector of the cell. For example, beamforming can be used to transmit and / or receive signals in a desired spatial direction.

[0060] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116. Air interface 116 can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Any suitable radio access technology (RAT) can be used to establish air interface 116.

[0061] More specifically, as described above, the communication system 100 can be a multi-access system and can employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN 104 / 113 can implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can establish an air interface 116 using Wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0062] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement radio technologies such as evolved UMTS terrestrial radio access (E-UTRA), which may use Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro) to establish air interface 116.

[0063] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as NR radio access, which can establish an air interface 116 using a new radio (NR).

[0064] In one embodiment, base station 114a and WTRUs 102a, 102b, and 102c can implement multiple radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can jointly implement LTE radio access and NR radio access, for example, using the dual connectivity (DC) principle. Therefore, the air interface used by WTRUs 102a, 102b, and 102c can be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0065] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate GSM Evolution (EDGE), GSM EDGE (GERAN), etc.

[0066] For example, Figure 1A Base station 114b can be a wireless router, home Node B, home eNode B, or access point, and can utilize any suitable RAT to facilitate wireless connectivity in a local area, such as commercial locations, homes, vehicles, campuses, industrial facilities, air corridors (e.g., for drone use), roads, etc. In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d can utilize cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish picocells or femtocells. Figure 1A As shown, base station 114b can be directly connected to Internet 110. Therefore, base station 114b may not need to access Internet 110 via CN 106.

[0067] RAN 104 / 113 can communicate with CN 106, which can be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRUs 102a, 102b, 102c, and 102d. Data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. CN 106 can provide call control, billing services, location-based services, prepaid calling, internet connectivity, video distribution, and / or perform advanced security functions such as user authentication. Although in Figure 1AAlthough not shown, it should be understood that RAN 104 / 113 and / or CN106 can communicate directly or indirectly with other RANs that use the same RAT as or a different RAT than RAN 104 / 113. For example, in addition to connecting to RAN 104 / 113, which may utilize NR radio technology, CN 106 can also communicate with another RAN (not shown) that uses GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0068] CN 106 can also serve as a gateway for WTRUs 102a, 102b, 102c, and 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN 104 / 113 or a different RAT.

[0069] Some or all of the WTRUs 102a, 102b, 102c, and 102d in the communication system 100 may include multi-mode capabilities (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example... Figure 1A The WTRU 102c shown can be configured to communicate with base station 114a, which may employ cellular-based radio technology, and to communicate with base station 114b, which may employ IEEE 802 radio technology.

[0070] Figure 1B This is a system diagram illustrating the example WTRU 102. (Example: ...) Figure 1B As shown, among other things, WTRU 102 may include, in particular, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a global positioning system (GPS) chipset 136, and / or other peripheral devices 138, etc. It should be understood that WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with the embodiments.

[0071] Processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. Processor 118 may perform signal encoding, data processing, power control, input / output processing, and / or any other functions that enable WTRU 102 to operate in a wireless environment. Processor 118 may be coupled to transceiver 120, which may be coupled to transmitting / receiving element 122. Although Figure 1B The processor 118 and transceiver 120 are depicted as separate components, but it should be understood that the processor 118 and transceiver 120 may be integrated together in an electronic package or chip.

[0072] Transmitting / receiving element 122 can be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over air interface 116. For example, in one embodiment, transmitting / receiving element 122 can be an antenna configured to transmit and / or receive RF signals. In one embodiment, transmitting / receiving element 122 can be, for example, a transmitter / detector configured to transmit and / or receive IR, UV, or visible light signals. In yet another embodiment, transmitting / receiving element 122 can be configured to transmit and / or receive both RF and optical signals. It should be understood that transmitting / receiving element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0073] Although the transmitting / receiving element 122 is in Figure 1B While depicted as a single element, WTRU 102 may include any number of transmit / receive elements 122. More specifically, WTRU 102 may employ MIMO technology. Thus, in one embodiment, WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals on air interface 116.

[0074] Transceiver 120 can be configured to modulate signals transmitted by transmitting / receiving element 122 and demodulate signals received by transmitting / receiving element 122. As described above, WTRU 102 can have multi-mode capability. Therefore, for example, transceiver 120 may include multiple transceivers to enable WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0075] The processor 118 of WTRU 102 can be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit) and can receive user input data therefrom. The processor 118 can also output user data to the speaker / microphone 124, keypad 126, and / or display / touchpad 128. Furthermore, the processor 118 can access and store information from any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. Non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 may include a user identification module (SIM) card, memory stick, secure digital storage (SD) card, etc. In other embodiments, the processor 118 can access and store information from memory that is not physically located on WTRU 102 (e.g., a server or home computer (not shown)).

[0076] The processor 118 can receive power from the power supply 134 and can be configured to distribute and / or control power to other components in the WTRU 102. The power supply 134 can be any suitable device that powers the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0077] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to, or instead of, information from the GPS chipset 136, the WTRU 102 may receive location information on the air interface 116 from base stations (e.g., base stations 114a, 114b) and / or determine its location based on the timing of signals received from two or more nearby base stations. It should be understood that the WTRU 102 may acquire location information using any suitable location determination method while remaining consistent with the embodiments.

[0078] The processor 118 may be further coupled to other peripheral devices 138, which may include one or more software and / or hardware modules providing additional features, functions, and / or wired or wireless connectivity. For example, peripheral devices 138 may include accelerometers, electronic compasses, satellite transceivers, digital cameras (for photos and / or videos), Universal Serial Bus (USB) ports, vibration devices, television transceivers, hands-free headsets, Bluetooth® modules, FM radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, etc. Peripheral devices 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors; geolocation sensors, altimeters, light sensors, touch sensors, magnetometers, barometers, attitude sensors, biosensors, and / or humidity sensors.

[0079] WTRU 102 may include a full-duplex radio for which the transmission and reception of some or all signals (e.g., signals associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference via hardware (e.g., a choke) or via signal processing by a processor (e.g., a separate processor (not shown) or via processor 118). In one embodiment, WTRU 102 may include a half-duplex radio for which the transmission and reception of some or all signals (e.g., signals associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)) may be concurrent and / or simultaneous.

[0080] Despite WTRU in Figure 1A-1B While described as a wireless terminal, it is conceivable that, in some representative embodiments, such a terminal may use (e.g., temporarily or permanently) a wired communication interface with a communication network.

[0081] In a representative embodiment, another network 112 may be a WLAN.

[0082] Given Figure 1A-1B As described herein, one or more, or all, of the functions described herein can be performed by one or more emulation devices (not shown). An emulation device can be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, an emulation device can be used to test other devices and / or simulate network and / or WTRU functions.

[0083] Simulation devices can be designed to perform one or more tests on other devices in laboratory and / or carrier network environments. For example, one or more simulation devices can perform one or more functions, or all functions, while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. One or more simulation devices can perform one or more functions, or all functions, while being temporarily implemented / deployed as part of a wired and / or wireless communication network. Simulation devices can be directly coupled to another device for testing purposes and / or can perform tests using over-the-air wireless communication.

[0084] One or more simulation devices may perform one or more functions, including all functions, rather than being implemented / deployed as part of a wired and / or wireless communication network. For example, simulation devices may be used to test test scenarios in laboratory and / or non-deployment (e.g., testing) wired and / or wireless communication networks to implement the testing of one or more components. One or more simulation devices may be test devices. Simulation devices may transmit and / or receive data using direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas).

[0085] Figure 1C This is a system diagram illustrating a set of example interfaces for a system according to some embodiments. An extended reality display device, along with its control electronics, can be implemented for some embodiments. System 150 can be embodied as a device including the various components described below and configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 150 can be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various embodiments, system 150 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 150 is configured to implement one or more of the aspects described in this document.

[0086] System 150 includes at least one processor 152 configured to execute instructions loaded thereon for implementing various aspects, such as those described in this document. Processor 152 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 150 includes at least one memory 154 (e.g., a volatile memory device and / or a non-volatile memory device). System 150 may include a storage device 158, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 158 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0087] System 150 includes an encoder / decoder module 156 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 156 may include its own processor and memory. The encoder / decoder module 156 represents one or more modules that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, as is known to those skilled in the art, the encoder / decoder module 156 may be implemented as a separate element of system 150, or may be incorporated within processor 152 as a combination of hardware and software.

[0088] Program code to be loaded onto processor 152 or encoder / decoder 156 to execute the various aspects described herein may be stored in storage device 158 and subsequently loaded onto memory 154 for execution by processor 152. According to various embodiments, one or more of processor 152, memory 154, storage device 158, and encoder / decoder module 156 may store one or more items of various kinds during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0089] In some embodiments, the memory within processor 152 and / or encoder / decoder module 156 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be processor 152 or encoder / decoder module 152) is used for one or more of these functions. External memory may be memory 154 and / or storage device 158, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding; also known as H.265 and MPEG H Part 2), or VVC (Multi-Functional Video Coding, a new standard being developed by the Joint Video Experts Group JVET).

[0090] Inputs to the components of system 150 can be provided by various input devices, as indicated in block 172. Such input devices include, but are not limited to: (i) an RF section that receives radio frequency (RF) signals, for example, transmitted over the air by a broadcasting device; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1C Other examples not shown include composite videos.

[0091] In various embodiments, as known in the art, the input device of block 172 has associated respective input processing elements. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) further limiting the signal to a narrower frequency band to select, for example, a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements to perform these functions, such as frequency selectors, signal selectors, frequency limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0092] Additionally, the USB and / or HDMI terminals may include their respective interface processors for connecting system 150 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) can be implemented as needed, for example, within a separate input processing IC or within processor 152. Similarly, aspects of USB or HDMI interface processing can be implemented as needed, either within a separate interface IC or within processor 152. Demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 152 and encoder / decoder 156, to operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0093] Various components of system 150 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data between them using a suitable connection arrangement 174 (e.g., internal buses as known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0094] System 150 includes a communication interface 160 that enables communication with other devices via a communication channel 162. The communication interface 160 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 162. The communication interface 160 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 162 may be implemented, for example, in a wired and / or wireless medium.

[0095] In various embodiments, data is streamed to or otherwise provided to system 150 using a wireless network such as a Wi-Fi network, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 162 and a communication interface 160 suitable for Wi-Fi communication. The communication channel 162 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that delivers data via an HDMI connection of input block 172 to provide streaming data to system 150. Still other embodiments use an RF connection of input block 172 to provide streaming data to system 150. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks instead of Wi-Fi, such as cellular networks or Bluetooth networks.

[0096] System 150 can provide output signals to various output devices, including display 176, speaker 178, and other peripheral devices 180. Display 176 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 176 can be used in televisions, tablets, laptops, cellular phones (mobile phones), or other devices. Display 176 can also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop). In various examples of embodiments, other peripheral devices 180 include one or more of a standalone digital video disc (or digital multifunction disc) (DVR for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 180 that provide functionality based on the output of system 150. For example, a disc player performs the function of playing the output of system 150.

[0097] In various embodiments, signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols enabling inter-device control with or without user intervention is used to communicate control signals between system 150 and display 176, speaker 178, or other peripheral devices 180. Output devices can be communicatively coupled to system 150 via dedicated connections through their respective interfaces 164, 166, and 168. Alternatively, output devices can be connected to system 150 via communication interface 160 using communication channel 162. Display 176 and speaker 178 can be integrated into a single unit with other components of system 150 in electronic devices such as, for example, a television set. In various embodiments, display interface 164 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0098] For example, if the RF portion of input 172 is part of a standalone set-top box, then display 176 and speaker 178 may alternatively be separate from one or more other components. In various embodiments where display 176 and speaker 178 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0099] System 150 may include one or more sensor devices 168. Examples of sensor devices that may be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as the user's position and orientation. Where system 150 is used as a control module (such as control modules 124, 132) for an extended reality display, the user's position and orientation can be used to determine how image data is rendered so that the user perceives the correct portion of a virtual object or scene from the correct viewpoint. In the case of a head-mounted display device, the device's own position and orientation can be used to determine the user's position and orientation for the purpose of rendering virtual content. In the case of other display devices such as telephones, tablets, computer monitors, or televisions, other inputs can be used to determine the user's position and orientation for the purpose of rendering content. For example, the user can use a touchscreen, keypad or keyboard, trackball, joystick, or other inputs to select and / or adjust the desired viewpoint and / or viewing direction. When the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation used for rendering content can be selected and / or adjusted based on the movement of the display device.

[0100] The embodiments can be implemented by computer software implemented by processor 152, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, memory 154 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 152 can be of any type suitable for the technical environment and can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0101] Block-based video coding Like HEVC, VVC is built on a block-based hybrid video coding framework. Figure 2A A block diagram of a block-based hybrid video coding system 200 is given. Variations of this encoder 200 are envisioned, but for clarity, the encoder 200 is described below without describing all anticipated variations.

[0102] Before being encoded, the video sequence may undergo pre-coding (204), for example, applying color transformations to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of the input image components to obtain a signal distribution more resilient to compression (e.g., using histogram equalization with one of the color components). Metadata may be associated with pre-processing and appended to the bitstream.

[0103] The input video signal 202, including the image to be encoded, is partitioned (206) and processed block by block, for example, in units of CUs. Different CUs can have different sizes. In VTM-1.0, a CU can reach 128x128 pixels. However, unlike HEVC, which partitions blocks solely based on quadtrees, in VTM-1.0, coding tree units (CTUs) are divided into CUs to accommodate the varying local characteristics based on quadtrees / binaries / tritrees. Additionally, the concept of multiple partitioning unit types in HEVC is removed, so the separation of CUs, prediction units (PUs), and transform units (TUs) no longer exists in VVC-1.0; instead, each CU is always used as the basic unit for both prediction and transformation without further partitioning. In the multi-type tree structure, CTUs are first partitioned using a quadtree structure. Then, each quadtree leaf node can be further partitioned using binary and ternary tree structures. Different partitioning types can be used, such as quad partitioning, vertical binary partitioning, horizontal binary partitioning, vertical ternary partitioning, and horizontal ternary partitioning.

[0104] exist Figure 2AIn the encoder, spatial prediction (208) and / or temporal prediction (210) can be performed. Spatial prediction (or “intra-frame prediction”) uses pixels from samples (referred to as reference samples) of adjacent blocks already encoded in the same video picture / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also referred to as “inter-frame prediction” or “motion-compensated prediction”) uses reconstructed pixels from already encoded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU can be signaled by one or more motion vectors (MVs) indicating the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index can be sent separately, which is used to identify which reference picture in the reference picture storage device (212) the temporal prediction signal comes from.

[0105] The mode decision block (214) in the encoder selects the optimal prediction mode, for example, based on a rate-distortion optimization method. This selection can be made after spatial and / or temporal predictions have been performed. Intra-frame / inter-frame decisions can be indicated, for example, by a prediction mode flag. The prediction block (216) is subtracted from the current video block to generate the prediction residual. The prediction residual is decorrelated using a transform (218) and quantization (220). (For some blocks, the encoder can bypass both the transform and quantization, in which case the residual can be encoded directly without applying the transform or quantization process.) The quantized residual coefficients are inversely quantized (222) and inversely transformed (224) to form the reconstructed residual, which is then added back to the prediction block (226) to form the reconstructed signal of the CU. Before the reconstructed CU is placed in the reference image storage device (212) and used to encode future video blocks, the reconstructed CU can be further filtered (228) with in-loop filtering (such as deblocking / SAO (sample adaptive offset) filtering) to reduce coding artifacts. In order to form the output video bitstream 230, the coding mode (inter-frame or intra-frame), prediction mode information, motion information and quantization residual coefficients are all sent to the entropy coding unit (108) for further compression and packing to form the bitstream.

[0106] Figure 2B A block diagram of a block-based video decoder 250 is given. In decoder 250, the bitstream is decoded by decoder elements as described below. Figure 2A As described above, the video decoder 250 typically performs decoding rounds in the reverse order of the encoding rounds. The encoder 200 also typically performs video decoding as part of the encoded video data.

[0107] Specifically, the decoder's input includes a video bitstream 252, which can be generated by the video encoder 200. The video bitstream 252 is first unpacked and entropy decoded at the entropy decoding unit 254 to obtain transform coefficients, motion vectors, and other encoded information. Picture partitioning information indicates how the picture is partitioned. Therefore, the decoder can partition (256) the picture based on the decoded picture partitioning information. The encoding mode and prediction information are sent to the spatial prediction unit 258 (if intra-frame coded) or the temporal prediction unit 260 (if inter-frame coded) to form a prediction block. The residual transform coefficients are sent to the inverse quantization unit 262 and the inverse transform unit 264 to reconstruct the residual block. Then, at 266, the prediction block and the residual block are added together to generate the reconstructed block. The reconstructed block can be further subjected to in-loop filtering 268 before being stored in the reference picture storage device 270 for predicting future video blocks.

[0108] The decoded image 272 can undergo further post-decoding processing (274), such as inverse color transformation (e.g., a conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping (which performs the inverse of the remapping process performed in the pre-encoding process (204)). The post-decoding processing can utilize metadata derived in the pre-encoding process and signaled in the bitstream. The decoded video can be sent to a display device 276. The display device 276 can be a device separate from the decoder 250, or the decoder 250 and the display device 276 can be components of the same device.

[0109] The various methods and other aspects described in this application can be used to modify modules of the video encoder 200 or decoder 250. Furthermore, the systems and methods disclosed herein are not limited to VVC or HEVC and can be applied to, for example, other standards and recommendations, whether pre-existing or future-developed, and any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this application can be used individually or in combination.

[0110] Figure 3A This is a schematic side view of an example waveguide display that can be used with extended reality (XR) applications according to some embodiments. The image is projected by an image generator 302. The image generator 302 can project the image using one or more of a variety of technologies. For example, the image generator 302 can be a laser beam scanning (LBS) projector, a liquid crystal display (LCD), a light-emitting diode (LED) display (including organic LED (OLED) or micro LED (µLED) displays), a digital light processor (DLP), a liquid crystal on silicon (LCoS) display, or other types of image generators or light engines.

[0111] Light representing image 312 generated by image generator 302 is coupled into waveguide 304 via in-coupler 306. In-coupler 306 diffracts the light representing image 312 into one or more diffraction orders. For example, ray 308 representing one of the rays from the bottom portion of the image is diffracted by in-coupler 306, and one of the diffraction orders 310 (e.g., the second order) is at an angle capable of propagating through waveguide 304 via total internal reflection. Image generator 302 displays the image as instructed by control module 324, which operates to render image data, video data, point cloud data, or other displayable data.

[0112] At least a portion of the light 310, already coupled into waveguide 304 by diffraction-in coupler 306, is coupled out of the waveguide by diffraction-out coupler 314. At least some of the light coupled out of waveguide 304 replicates the angle of incidence of the light coupled into the waveguide. For example, in the illustration, the coupled rays 316a, 316b, and 316c replicate the angle of the coupled ray 308. Because the light leaving the out coupler replicates the direction of the light entering the in coupler, the waveguide essentially replicates the original image 312. The user's eye 318 can focus on the replicated image.

[0113] exist Figure 3A In the example, the outgoing coupler 314 couples out only a portion of the light on each reflection, thus allowing a single input beam (such as beam 308) to generate multiple parallel output beams (such as beams 316a, 316b, and 316c). In this way, even if the user's eye is not perfectly aligned with the center of the outgoing coupler, at least some of the light originating from each part of the image may reach the user's eye. For example, if the eye 318 moves downwards, beam 316c may enter the eye even if beams 316a and 316b do not, so the user can still perceive the bottom of image 312 despite the positional shift. Therefore, the outgoing coupler 314 partially functions as an exit pupil dilator in the vertical direction. The waveguide may also include one or more additional exit pupil dilators (…). Figure 3A (not shown in the image), so that the exit pupil expands in the horizontal direction.

[0114] In some embodiments, waveguide 304 is at least partially transparent with respect to light originating from outside the waveguide display. For example, at least some of the light 320 from a real-world object (e.g., object 322) passes through waveguide 304, allowing the user to see the real-world object when using the waveguide display. Since the light 320 from the real-world object also passes through diffraction grating 314, there will be multiple diffraction orders, and therefore multiple images. To minimize the visibility of multiple images, it is desirable that the zeroth order (not deflected by 314) has high diffraction efficiency for both light 320 and the zeroth order, while higher diffraction orders have lower energy. Therefore, in addition to expanding and coupling out virtual images, the out-coupler 314 is preferably configured to allow the zeroth order of the real image to pass through. In such embodiments, the image displayed by the waveguide display can appear to be superimposed on the real world.

[0115] Figure 3B This is a schematic side view illustrating example alternative display types that can be used with extended reality applications according to some embodiments. In the XR head-mounted display device 330, a control module 332 controls a display 334 (which may be an LCD) to display images. The head-mounted display includes a partially reflective surface 336 that reflects (and in some embodiments, both reflects and focuses) the image displayed on the LCD to make the image visible to the user. The partially reflective surface 336 also allows at least some external light to pass through, thereby allowing the user to see their surroundings.

[0116] Figure 3C This is a schematic side view illustrating example alternative display types that can be used with extended reality applications according to some embodiments. In the XR head-mounted display device 340, a control module 342 controls a display 344 (which may be an LCD) to display an image. The image is focused by one or more lenses of a display optics 346 to make the image visible to the user. Figure 3C In this example, the external light does not reach the user's eyes directly. However, in some such embodiments, an external camera 348 can be used to capture images of the external environment and display such images on a display 344 along with any virtual content that may also be displayed.

[0117] The embodiments described herein are not limited to any particular type or structure of XR display device.

[0118] This application belongs to the field of point cloud compression and processing. This field aims to develop tools for the compression, analysis, interpolation, representation, and understanding of point cloud signals.

[0119] Point clouds are a common data format across several commercial sectors, from autonomous driving, robotics, AR / VR, civil engineering, computer graphics, to animation / film. 3D LiDAR sensors have already been deployed in autonomous vehicles, and affordable LiDAR sensors have been released from Velodyne Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR Camera L515. With advancements in sensing technology, 3D point cloud data is becoming more feasible than ever before and is expected to be the ultimate enabler in the applications mentioned.

[0120] It is also believed that point cloud data consumes a significant portion of network traffic, for example, in cars connected to 5G networks and in immersive communications (VR / AR). An efficient representation format is essential for point cloud understanding and communication. In particular, raw point cloud data needs to be properly organized and processed for world modeling and sensing purposes. Compression of the raw point cloud is indispensable when data needs to be stored and transmitted in relevant scenarios.

[0121] Furthermore, point clouds can represent continuous scans of the same scene containing multiple moving objects. These are called dynamic point clouds in contrast to static point clouds captured from static scenes or static objects. Dynamic point clouds are typically organized into frames, where different frames are captured at different times. Dynamic point clouds may require processing and compression to become real-time or with low latency.

[0122] The automotive industry and autonomous vehicles are areas where point clouds can be used. Autonomous vehicles should be able to "detect" their environment to make sound driving decisions based on the realities of their immediate surroundings. Typical sensors like LiDAR generate (dynamic) point clouds used by perception engines. These point clouds are not intended to be seen by the human eye, and they are typically sparse, not necessarily colored, and dynamic at a high capture frequency. They can possess other properties, such as reflectivity provided by LiDAR, as this property indicates the material of the sensed object and can aid in decision-making.

[0123] Virtual reality (VR) and immersive worlds have become a hot topic and are envisioned by many as the future of 2D flat video. The basic idea is to immerse the viewer in all the environments around them, as opposed to a standard TV where they can only see a virtual world in front of them. Depending on the viewer's degree of freedom within the environment, there are several levels of immersion. Point clouds are a good candidate format for distributing VR worlds. They can be static or dynamic and typically have an average size, say, no more than a few million points at a time.

[0124] Point clouds can also be used for various purposes, such as cultural heritage / buildings, where objects like statues or buildings are 3D scanned to share the spatial configuration of the object without sending or accessing it. Furthermore, this is a way to ensure the preservation of knowledge about an object in case it might be destroyed, for example, a temple damaged by an earthquake. Such point clouds are typically static, colored, and enormous.

[0125] Another use case is in topography and cartography, where 3D representation is used, maps are not limited to a flat plane and can include bumps and depressions. Google Maps is now a good example of a 3D map, but it uses a grid instead of point clouds. However, point clouds may be a suitable data format for 3D maps, and such point clouds are typically static, colored, and huge.

[0126] World modeling and sensing via point clouds may be a fundamental technology that allows machines to gain knowledge about the 3D world around them, which is crucial for the applications discussed above.

[0127] This application was conceived in consideration of the above circumstances.

[0128] 3D point cloud data is essentially discrete samples of the surface of an object or scene. To fully represent the real world using these point samples, a massive number of points are actually needed. For example, a typical VR immersive scene contains millions of points, while a point cloud typically contains hundreds of millions. Therefore, processing such large-scale point clouds is computationally expensive, especially for consumer devices with limited computing power, such as smartphones, tablets, and car navigation systems.

[0129] The first step in any processing or inference of point clouds is having an efficient storage method. To store and process the input point cloud at an affordable computational cost, one solution is to first downsample it, where the downsampled point cloud summarizes the geometry of the input point cloud while having far fewer points. This downsampled point cloud is then fed into subsequent machine tasks for further consumption. However, further reductions in storage space can be achieved by converting the original point cloud data (raw or downsampled) into a bitstream using entropy coding techniques for lossless compression.

[0130] Beyond lossless coding, many scenarios seek lossy coding to significantly improve compression ratios while maintaining the introduced distortion at a specific quality level. To achieve less lossy coding, efficient point feature extractors are needed to improve reconstruction accuracy within a given resource budget.

[0131] Learning-based dynamic point cloud compression In practice, the requirement is not to compress point cloud frames independently, but rather to compress dynamic point cloud sequences consisting of a series of frames. Therefore, inter-frame prediction has been introduced for point cloud compression that includes both point cloud geometry and point cloud attributes to take advantage of temporal redundancy.

[0132] The specific challenges of learning-based inter-frame prediction for dynamic point clouds are how to predict the current frame based on a reference frame and how to encode the current frame using the generated prediction.

[0133] Dynamic point clouds captured by LiDAR in autonomous driving systems or for VR / AR applications can present significant challenges in storage and transmission due to their massive data volume. In this application, we attempt to address these challenges in deep learning-based dynamic point cloud compression. We specifically investigate how to use deep learning techniques to perform inter-frame prediction. Two challenges need to be addressed: how to generate predictions based on reference point cloud frames; and how to utilize these predictions to assist in the encoding of the current frame. Assuming the existence of explicit motion analysis, the technique proposed in this application is termed explicit predictive coding.

[0134] This application provides inter-frame predictive coding for dynamic point cloud compression. The proposal is termed an explicit prediction technique because it features explicit motion analysis on the encoder and explicit decoding of motion information on the decoder. In existing explicit prediction techniques, predictor point cloud frames are constructed in 3D space when their predictor features are generated. Such predictor frame construction involves a point cloud synthesis step, which may involve a heavy network. In this application, it is proposed to generate predictor features without constructing predictor frames. Instead, predictor features are directly computed based on a reference frame and motion features. Finally, a two-level conditional coding paradigm is deployed to compress the current point cloud frame conditionally with motion features and then conditionally with predictor features.

[0135] Encoder and decoder Figure 4 This is a process diagram illustrating the encoding and decoding process of an example codec with explicit prediction according to some embodiments. Figure 4 The figure shows the overall diagram 400, which features an inter-frame predictive coding scheme for dynamic point clouds with explicit prediction. Figure 4 The left side is encoder section 402, and the right side is decoder section 404.

[0136] encoder Figure 4 The encoding section 402 comprises three segments. The first segment is used for motion analysis, specifically to analyze the dynamics between two point cloud frames. This segment will... and reference point cloud The input is taken as the input. The MA block 406 performs motion analysis on this pair of point cloud frames, and the output of the MA block 406 is a motion feature map. As a descriptor for motion between the reference point cloud frame and the current point cloud frame. Note that... It is a feature map, where each feature is associated with a point or a partition in a point cloud frame (the spatial resolution may vary based on the MA architecture). Then, it is processed by ENC. m Block 408 pairs of motion features Entropy encoding is performed on this ENC. m Block 408 outputs motion bitstream and reconstructed motion characteristics The reconstructed motion features yes This is a quantized version. Note that motion information is output and encoded as a feature map in the feature domain, rather than as motion vectors in the 3D spatial domain. By representing motion in the feature domain, the feature vectors typically have a much higher dimension than the 3D world; for example, a feature vector can be a 16-dimensional vector. Therefore, high-dimensional motion features are more favorable for compression with greater representability compared to the original 3D motion vectors. Therefore, in the motion analysis section, it is not always necessary to generate a 3D motion field containing the original 3D motion vectors.

[0137] The second section (predictor feature generator block 410) will reference the point cloud frame. and reconstructed motion characteristics Take it as input. Then generate the predictor feature map. .

[0138] The third section is called the coding section, or the deep feature-based coding section. This section mainly consists of two steps: a feature encoder / feature extraction step (FE) followed by entropy coding. FE block 412 encodes the current point cloud... It takes the feature map as its main input and outputs the feature map. Entropy coding begins with rounding or quantization to allow entropy (arithmetic) encoding of the symbol sequence. This is then done by the encoder. Block 414 generates a bit stream Furthermore, in this application, the feature extraction block is subjected to the feature map used as a predictor. The conditional inputs (see '798 and '130 applications) are restricted. Later, on the decoding side, the same conditions will be applied to the corresponding decoding block. In summary, a conditional autoencoder architecture is employed in the proposed method.

[0139] Note that the predictor generates FA blocks 410 and 418 subject to the conditional feature map. The limitations. Simultaneously, the output of FA is used as a condition for the (main) coded segment. Therefore, Figure 4 The architecture proposed in this paper can be called a second-level conditional autoencoder.

[0140] decoder Figure 4 The decoding section 404 consists of three segments corresponding to the decoding segment. These three segments are the motion reconstruction segment, the predictor feature generation segment, and the depth feature-based decoding segment. The basic idea is to first use the motion reconstruction segment to decode the features from... The motion is decoded, and then segments are generated based on the reference frame using feature generation. The predictor features are calculated, and then the reconstructed point cloud is decoded using a depth-based decoding segment.

[0141] The first section is the motion reconstruction section. There is no motion analysis step on the decoding side. Instead, the entropy decoder... 416 from bit stream Reconstructing motion feature maps .

[0142] The second segment of the decoder is exactly the same as the second segment of the encoder (which is the predictor feature generation FA block 418). The decoder's FA block 418 uses the same input as the encoder's FA block 410, which are reference point cloud frames. and reconstructed motion characteristics Therefore, the same predictor feature map is reproduced at the decoder. .

[0143] The third segment of decoding (depth feature-based decoding) consists of two steps: entropy decoding. 420 pairs of bit streams Decode to reconstruct feature maps The feature decoder FD 422 performs feature aggregation to perform feature map-based aggregation. Reconstructing point clouds As previously noted, the feature decoder FD 422 is conditionally fed by the predictor feature map. Restrictions.

[0144] The graph used for inter-frame predictive coding is characterized by having a bitstream dedicated to motion information. This leads to asymmetric architecture designs that allow for lighter decoders and heavier encoders. The reason is that motion analysis is typically computationally intensive and is only needed during encoding. For the decoder, it directly decodes the motion information to construct the motion information, and then the predictor information. Decoding motion information from the bitstream is generally much easier than motion analysis. On the other hand, by performing motion analysis and sending the motion information as overhead in the bitstream, it typically provides a higher quality predictor feature map compared to implicit prediction methods that create the predictor purely based on a reference frame.

[0145] The graph used for inter-frame predictive coding is further characterized by a conditional autoencoder architecture. In the main coding section, both the encoder and decoder are subject to the same conditional input constraints.

[0146] Motion analysis As previously presented, considering that feature maps are easier to compress and more representative, Motion Analysis (MA) outputs motion feature maps instead of motion fields. However, training such a Motion Analysis block that outputs motion feature maps instead of directly outputting motion fields can be challenging, and this is what we will address next.

[0147] Figure 5 This is a process diagram illustrating an example training procedure for motion analysis according to some embodiments. In some embodiments of the training procedure 500, training is performed end-to-end. Figure 4 Before the codec in the codec, the motion analysis block MA502 is pre-trained. For example... Figure 5 As shown, a pre-training method for MA block 502 is proposed. In some embodiments, motion synthesis blocks (MS) are used to train the motion analysis. Specifically, the MS block is appended to MA block 502 and After block 504, MS block 506 will... Take as input and generate 3D motion field . for Each point in contains and The 3D motion vectors between them. The MS block has been introduced; the pre-training of the MA block can be supervised by the motion field using a benchmark ground truth motion field. To execute.

[0148] In some embodiments, MS block 506 is introduced for pre-training of MA block 502. Once pre-training is complete, MS block 506 is no longer used elsewhere. The ground truth motion field is the true and correct motion between the current point cloud and the reference point cloud. The ground truth is used for pre-training the MA block.

[0149] To perform such training, readily available datasets for motion or scene flow estimation can be used, making the description and A ground truth motion field for the actual motion between the points is available. When a ground truth motion field is not available, a self-supervised training strategy can be used to train the MA block. For some embodiments, the strategy is as follows: During each training step, a motion field is computed. .Then With reference point cloud Swap the inputs to the MA block to obtain another motion field. The self-supervised training strategy is that if MA block 502 and MS block 506 are working correctly, then... The direction of motion should be in In the opposite direction. Therefore, consistency measures can be... and- The distance between points is calculated and used as a loss function for training purposes. In some embodiments, the chamfer distance of the point set is used as a consistency measure, such that... , where CD represents the chamfer distance.

[0150] exist Figure 5 In the example, the MS block will The input is taken as the input, which undergoes some distortion from rounding or quantization used for entropy encoding. In this way, the effects of rounding or quantization can be considered during pre-training. However, it should be noted that rounding / quantization is non-differentiable and prevents backpropagation. Finally, such distortion is simulated by adding noise signals for training purposes. In some embodiments, the MS block will incorporate motion features. Take it directly as input and generate the sports field. Once the pre-training of the MS block is complete, its neural network parameters will be loaded into the proposed codec. Figure 4 The MS block was introduced for pre-training of the MA block. Therefore, once pre-training is complete, the MS block is no longer used. This is why the MS block does not appear in... Figure 4 In some embodiments, the output of the MA block is directly fed into the MS block.

[0151] Figure 6 This is a flowchart illustrating an example training process for generating blocks of predictor features according to some embodiments. Figure 6 Example training method 600 is shown. In some embodiments, a dedicated step can be used to train the FA block 602. Feature synthesis (FS) block 604 synthesizes the feature maps... Take as input and synthesize point clouds in 3D space. .

[0152] Figure 7 This is a flowchart illustrating an example motion analysis process according to some embodiments. Figure 7 An example design of the motion analysis (MA) block 700 is shown. First, the reference point cloud frame and the current point cloud frame are sent to the same spatial feature extraction (FB) blocks 702 and 704 to output a spatial feature map. and Spatial features are typically associated with downsampled point clouds. They are abstract, high-level descriptors that subsequent modules can leverage to extract motion information. In some embodiments, the fusion (FU) block 706 can accumulate and / or concatenate feature maps.

[0153] Then, the fusion (FU) block 706 is applied to fuse the two spatial feature maps. For the current feature map... At each location in the reference feature map, the motion analysis process is performed. Find its k nearest neighbors. Assume Position A in the vector has a feature vector Its reference feature map The k nearest neighbor positions in the array eigenvectors on Associated. In one embodiment, the FU block is associated with all vectors at position A. Link. This is executed not only for position A, but also for... Executed from all other locations within it. The feature map representing the connections will be output by FU block 706. In some embodiments, FU block 706 outputs all vectors at position A. and Summation. Similarly, this is performed not only for position A, but also for... Perform the summation at all other positions within the graph. (Feature map of summation) The output will be handled by the FU block. Note that after the fusion step of the FU block, the output feature map contains features from... and Information about both.

[0154] Finally, feature aggregation (FC) is performed to further aggregate the temporal feature maps from FU block 706. In one example, the output motion features... Previously, four (4) convolutional layers were used to implement the FC block 708. The dimensions of the convolutional layers were (128, 128, 64, 64, 32), so the motion feature map... It has 32 channels. See the process diagram below. Figure 10 .

[0155] Spatial Feature Extractor We discuss the design of the Spatial Feature Extractor (FB) block here. The Spatial Feature Extractor (FB) will extract point clouds... Take it as input and output its feature map.

[0156] Figure 8 This is a flowchart illustrating an example feature extraction process for a voxel-based representation according to some embodiments. In some embodiments, when and When dealing with voxelized point clouds, FB block 800 consists of two convolutional layers 806 and 808 (with a span of 2 for downsampling) and two multilayer perceptron (MLP) layers 802 and 804, as follows. Figure 8As shown in the diagram. For example, the input-output dimensions of these four layers are (3, 32, 64, 128, 64), making the output feature map... and Both will have 64 channels.

[0157] Figure 9 This is a diagram illustrating an example feature extraction process based on point-based representations according to some embodiments. In another embodiment, when and When represented using point-based representation, Figure 9 The design of FB block 900 is provided. In this embodiment, FB block 900 first performs block partitioning 902 to uniformly divide the point cloud into blocks, and then applies a shared block-by-block feature extractor PN 904 to compute the feature vector for each block. All such block-by-block features constitute the output feature map of FB. In some embodiments, PN block 904 adopts a PointNet design. See Qi, Charles R. et al.'s "..." Pointnet: Deep Learning on Point sets for 3D Classification and Segmentation (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)). It consists of several pointwise MLP layers with dimensions (3, 32, 64, 128, 128) and a global max pooling to obtain block features with 128 channels. Then, it is followed by three MLP layers with dimensions (128, 64, 64) to make the output feature map and It has 64 channels.

[0158] Predictor generation In the second segment, both the encoder and decoder share the same design for the predictor generation block. Predictor generation plays a crucial role in inter-frame prediction, which is beneficial for compression.

[0159] In one embodiment, the predicted value of the point cloud frame in 3D space is explicitly generated intermediately during the generation of the FA block by the predictor. Examples are described in '798 and '130 applications.

[0160] However, in this application, we propose to avoid generating point cloud frames in 3D space within the FA block. One motivation is that the FA block needs to operate on both the encoder and decoder, and we hope to reduce complexity as much as possible. On the other hand, synthesizing point clouds in 3D space from feature space can be a fairly computationally intensive task. Therefore, in one example, we propose not to synthesize point clouds during the predictor generation process. However, this introduces additional challenges in how to train the FA block.

[0161] In this application, we propose using a dedicated step to pre-train the FA module. The training method is as follows: Figure 6 As shown. For training purposes, we introduce a Feature Synthesis Block (FS). The FS block synthesizes the feature map... Take as input and synthesize point clouds in 3D space. Using synthesized point clouds and current point cloud The error between them is used for supervision.

[0162] In one embodiment, when the predictor features The dimension and this feature During alignment, the feature decoder FD block can be used as the FS block.

[0163] Notice, Figure 5 Motion analysis blocks and Figure 6 Both the predictor-generated blocks and the network itself can be considered "encoding" networks. To train those "encoding" networks, we use a dedicated "decoding" network to enable the training procedure. It should be noted that, due to the backpropagation mechanism, a better selection of the "decoding" network can help us obtain a better "encoding" network.

[0164] Figure 10 This is a flowchart illustrating the process of example feature aggregation (FC block) according to some embodiments. For some embodiments, Figure 7 FC blocks can be like Figure 10 The example FC block 1000 is shown for implementation. Figure 7 The output of the FU block is specified as Figure 10 Input feature map This feature map It can be processed through a series of convolutional encoders 1002, 1004, 1006, and 1008 and used as a motion feature map. Output. Figure 10 Four convolutional encoders are shown, but the number may vary in some embodiments.

[0165] Figure 11 This is a diagram illustrating the process of generating an example feature predictor (FA block) according to some embodiments. Figure 11 An example design of the predictor generation block 1100 is shown. First, the spatial feature extractor FB 1102 is used as a reference point cloud frame. Extract feature maps For motion analysis, FB block 1102 can be related to... Figure 7 The FB blocks 702 and 704 in the design are identical. Then, this feature... With motion characteristics link( Figure 11 "in (symbol 1104), followed by a series of MLPs 1106, 1108 and convolutional layers 1110, 1112 to output the predictor feature map. . Figure 11 The dimensions of the layers are (128, 256, 256, 64).

[0166] In some embodiments, an IRN or a transformer block can be used to enhance feature aggregation. See Szegedy, Christian et al. Inception-v4, inception-resnet and the impact of residual connections on learning "(31:1, In Proceedings of the AAAI Conference on Artificial Intelligence (2017) (" Szegedy ")); Zhao, Hengshuang et al.'s " Point Transformer (ICCV 16239-16248 (2021) (“ Zhao ")); Mao, jiageng et al." Voxel transformer for 3d object detection ”(In Proceedings of the IEEE / CVFInternational Conference on Computer Vision 3164-3173 (2021) (“ Mao ")); and Zhang, Cheng et al.'s " PVT: Point-Voxel Transformer for Point Cloud Learning "(arXiv preprint arXiv: 2108.06076 (2021) (" Zhang ")).

[0167] Feature encoder Figure 12 This is a diagram illustrating the example feature aggregation / extraction process in an encoder (FE block) according to some embodiments. In the third section, the encoder and decoder have a symmetrical design. They are located in the predictor feature map. The encoding and decoding conditions are executed above. An example of the conditional feature encoder FE 1200 is provided in this subsection; see [link to relevant documentation]. Figure 12 .

[0168] First, the current point cloud It is first fed into FB block 1202 for initial feature extraction, resulting in the output Then, it is compared with the predictor features. Link. Assumption and Both have 64 channels, so the connected features have a total of 128 channels. This is then passed to several neural network layers to generate the output feature map. The purpose of these layers is to remove and from of Redundant information in both, so as to obtain the feature map of the result. It contains less information and results in a smaller bitstream. .

[0169] In one embodiment, we make these layers a series of MLPs 1204, 1206 and convolutional layers 1208, 1210 to output feature maps. . Figure 12 The dimensions of the four layers are (128, 256, 256, 64), making It has 64 channels. In other embodiments, IRN or advanced transformers can be used instead of convolutional or MLP layers to enhance the feature encoder. See also Szegedy ; Zhao ; Mao ;and Zhang .

[0170] Direct residual computation can be performed instead of join operations, followed by computation on the predictor feature map. and spatial feature map Redundancy removal between time intervals. Given its simplicity, this might be the preferred method. However, direct residual computation can be inefficient in compression, while dedicated neural network layers may be more efficient for removing temporal redundancy.

[0171] Feature Decoder Figure 13 This is a diagram illustrating the process of example feature aggregation / extraction in a decoder (FD block) according to some embodiments. In this subsection, we illustrate the design of the conditional feature decoder FD 1300, as... Figure 13 As shown in the image. First, the decoded feature map... With conditional predictor feature map Link. Assumption and Both have 64 dimensions, so the connected features have 128 dimensions. The connected features are fed into several neural network layers to generate feature maps. Through this process, the predictor information based on the reference frame can be fused back into the feature map. In some embodiments, these layers are a series of MLPs 1302, 1304 and convolutional layers 1306, 1308 to output feature maps. . Figure 13The dimensions of the four layers are (128, 256, 256, 64), making It has 64 channels. In some embodiments, IRN or advanced transformers can be used to replace convolutional or MLP layers to enhance the feature decoder. See also Szegedy ; Zhao ; Mao ;and Zhang Subsequently, this feature Feeds are sent to point composite PS block 1310 to reconstruct the current point cloud in 3D space. .

[0172] Figure 14 This is a diagram illustrating the process of example feature aggregation / extraction in a decoder (FD block) according to some embodiments. In some embodiments, such as Figure 14 As shown, the connection occurs slightly later in the pipeline. (Feature map of entropy decoding) It is used as input by a series of convolutional layers 1402, 1404 and MLPs 1406, 1408. Then, it is processed through the predictor feature map. To connect the aggregated features. Finally, the connected features The data is sent to the point synthesis PS block to synthesize a point cloud in 3D space. In this example implementation, the fusion of predictor information is deferred to point synthesis PS block 1410.

[0173] Figure 15 This is a process diagram illustrating an example conditional feature decoder according to some embodiments. Figure 15 In some embodiments of the example feature decoder 1500, the fusion of predictor information can be performed before processing by the PS block 1518 by concatenating another series of convolutional layers / CNNs 1502, 1504, 1514, 1516 and MLPs 1506, 1508, 1510, 1512 after the concatenation. In this case, the series of convolutional layers / CNNs 1502, 1504, 1514, 1516 and MLPs 1506, 1508, 1510, 1512 can be located either before or after the feature maps are concatenated.

[0174] Figure 16 This is a process diagram illustrating an example point composition block (PS block) according to some embodiments. We provide a design for PS block 1600 here. In one embodiment, if it is desired to reconstruct using a point-based representation... The point-to-point block consists of two convolutional layers 1602 and 1604 and two MLP layers 1606 and 1608, as follows: Figure 16 As shown in the image. Figure 16 The dimensions of the four layers in the image are (128, 256, 256, 5×3), which makes... Each feature vector in the reconstructed current point cloud Five (five) 3D points are generated. In another embodiment, if it is desired to reconstruct using a voxel-based representation... Then the point-to-point block only contains convolutional layers, which is different from the " Multiscale Point Cloud Geometry Compression The decoder design is the same as that in the '798 application (2021 Data Compression Conference (DCC). IEEE (2021)).

[0175] End-to-end training The proposed explicit predictive coding network needs to be trained on a dynamic point cloud sequence dataset. Training methods can include those described above. Figure 5 The pre-training of the motion analysis block MA described above, and as described in the text, are performed as follows: Figure 6 The predictor generation block FA is pre-trained as described in the paper. Based on the pre-trained motion analysis MA and the predictor generation FA, the complete architecture is trained in an end-to-end manner. Figure 4 In some embodiments, the MA block and FA block can be trained end-to-end from scratch along with other blocks. In some embodiments, the MA block and FA block can be pre-trained. During end-to-end training, it is common practice to replace the entropy encoding / decoding steps with entropy bottleneck layers. See also people's " Variational image compression with a scale hyperprior "(International Conference on Learning Representations (2018)); and David Minnen et al." Joint Autoregressive and Hierarchical Priors for Learned Image Compression (Advances in Neural Information Processing Systems (2018)). In actual training, bitrate and distortion are jointly minimized. Therefore, the bitrate budget can be modeled in a differentiable way to optimize the codec. Entropy bottleneck layers serve this purpose. Entropy bottlenecks estimate the bitrate of the input feature map in a differentiable way to enable end-to-end training of the feature compression pipeline. In some embodiments, two entropy bottleneck layers can be used. Specifically, one entropy bottleneck layer is used to estimate the bitrate of motion information, while the second entropy bottleneck layer is used to estimate the feature bitrate in the main coding segment.

[0176] Advanced feature aggregation In our application, we use MLPs and convolutional layers as the underlying microarchitecture, forming the basic design of the proposed functional architecture. It should be noted that these can be replaced with or supplemented with higher-level feature aggregation microarchitectures, such as InceptionResNet (IRN) or transformer blocks. See also Szegedy ; Zhao ; Mao ;and Zhang .

[0177] Inception ResNet (IRN) Figure 17 This is a process diagram illustrating an example Inception-ResNet block for feature aggregation according to some embodiments. Figure 17 The IRN architecture 1700 is provided. This example illustrates the architecture of the IRN block 1700 to aggregate features with D channels. Here, "CONV" N "meaning to accept having" N A convolutional layer with input channels of 1000 channels. In this example, each CONV D / 4 block 1702, 1706, 1712 is followed by Recti-Linear Unit (ReLU) blocks 1704, 1708, 1714. The outputs of each CONV D / 2 block in CONV D / 2 blocks 1710, 1716 are concatenated together by connection block 1718. (Plus sign (" 1720 indicates summation.

[0178] converter Figure 18 This is a process diagram illustrating an example transformer block for feature aggregation according to some embodiments. Figure 18 The diagram shows a converter block 1800, in which, " (1804, 1808) again indicates a request for a summation. Figure 18 The basic diagram of converter block 1800 is shown, which consists of self-attention block 1802 with residual connections and MLP block 1806 (composed of several MLP layers) with residual connections.

[0179] Figure 19 This is a process diagram illustrating an example self-attention block according to some embodiments. A block diagram of self-attention block 1900 is shown below. Figure 19 As shown. Details are as follows.

[0180] Given voxel position Associated current feature vector (1902) and its position relative to voxels Associated neighbors Features ,in yes In the input sparse tensor The self-attention block 1900 strives based on all neighboring features. (1910) to update features (1902). First, by making features (1902) After kNN block 1926 based on The coordinates are used to perform a k-nearest neighbor (kNN) search to obtain the points. (1904). Then, use the following formula to calculate. Query embedding : , Where the current feature vector is made (1902) is generated via MLP block 1906. (1908).

[0181] After that, calculation All nearest neighbor key embeddings (1920) Sum embedding (1922): , in, (1906) (1912) and (1914) are MLP layers used to obtain queries, keys, and values ​​respectively, while (1918) is a voxel and The positional code between them is calculated using the following formula: , in (1916) is the MLP layer used to obtain positional encoding. and These are 3D coordinates, and they are voxels. and The center of position A. The output feature of position A through the self-attention block is: , in It is the Softmax normalization function. It is an eigenvector The length, and It is a predefined constant.

[0182] The transformer block updates the feature maps at all locations in the same manner and then outputs the updated feature maps. Note that in a simplified embodiment, , , ,and It can contain only a single fully connected layer, which corresponds to a linear projection.

[0183] Figure 20 This is a flowchart illustrating an example decoding process according to some embodiments. In some embodiments, example process 2000 may include obtaining 2002 a previously decoded point cloud frame as a reference point cloud frame. In some embodiments, example process 2000 may further include obtaining 2004 motion features describing the motion between the reference point cloud frame and the current point cloud frame. In some embodiments, example process 2000 may further include determining 2006 a predictor feature map based on the reference point cloud frame and the motion features. In some embodiments, example process 2000 may further include obtaining 2008 a current feature map representing the current point cloud frame, given the predictor feature map as a condition. In some embodiments, example process 2000 may further include reconstructing 2010 the current point cloud frame based on the current feature map and using predicted features as conditions.

[0184] Figure 21 This is a flowchart illustrating an example encoding process according to some embodiments. In some embodiments, example process 2100 may include obtaining 2102 a previously encoded point cloud frame as a reference point cloud frame. In some embodiments, example process 2100 may further include obtaining 2104 motion features describing the motion between the reference point cloud frame and the current point cloud frame. In some embodiments, example process 2100 may further include determining 2106 a predictor feature map based on the reference point cloud frame and the motion features. In some embodiments, example process 2100 may further include determining 2108 a current feature map representing the current point cloud frame, given the predictor feature map as a condition. In some embodiments, example process 2100 may further include encoding 2110 the current feature map into a bitstream using the predictor features as a condition.

[0185] While the methods and systems according to some embodiments are generally discussed in the context of extended reality (XR), some embodiments can be applied to any XR context, such as, for example, virtual reality (VR) / mixed reality (MR) / augmented reality (AR) contexts. Furthermore, although the term "head-mounted display (HMD)" is used herein according to some embodiments, for some embodiments, some embodiments can be applied to wearable devices capable of, for example, XR, VR, AR, and / or MR (which may or may not be attached to the head).

[0186] A first example method / apparatus according to some embodiments may include: obtaining a previously decoded point cloud frame as a reference point cloud frame; obtaining motion features describing the motion between the reference point cloud frame and a current point cloud frame; determining a predictor feature map based on the reference point cloud frame and the motion features; obtaining a current feature map representing the current point cloud frame, given the predictor feature map as a condition; and reconstructing the current point cloud frame based on the current feature map and using the predictor feature map as a condition.

[0187] For some embodiments of the first example method, obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame includes: performing motion analysis using the reference point cloud frame and the current point cloud frame as input to generate a motion feature map; and entropy encoding the motion feature map to generate the motion features.

[0188] For some embodiments of the first example method, obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame includes: performing feature extraction on the reference point cloud frame to generate a first feature map; performing feature extraction on the current point cloud frame to generate a second feature map; fusing the first feature map and the second feature map to generate a fused feature map; and performing feature aggregation on the fused feature map to generate the motion features.

[0189] In some embodiments of the first example method, wherein the reference point cloud frame and the current point cloud frame are voxel-based representations, and wherein performing the feature extraction on the reference point cloud frame includes: passing the reference point cloud frame through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the first feature map, and wherein performing the feature extraction on the current point cloud frame includes: passing the current point cloud frame through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the second feature map.

[0190] In some embodiments of the first example method, wherein the reference point cloud frame and the current point cloud frame are point-based representations, and wherein performing the feature extraction on the reference point cloud frame includes: performing block partitioning on the reference point cloud frame; and performing block-by-block feature extraction on the block-partitioned reference point cloud frame to generate the first feature map, and wherein performing the feature extraction on the current point cloud frame includes: performing block partitioning on the current point cloud frame; and performing block-by-block feature extraction on the block-partitioned current point cloud frame to generate the second feature map.

[0191] In some embodiments of the first example method, obtaining the motion features includes performing motion analysis training.

[0192] In some embodiments of the first example method, determining the predictor feature map based on the reference point cloud frame and the motion features includes: performing feature extraction on the reference point cloud frame to generate a first feature map; concatenating the first feature map with the motion features; passing the concatenated first feature map through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the predictor feature map.

[0193] In some embodiments of the first example method, performing the feature extraction on the reference point cloud frame includes performing point synthesis.

[0194] In some embodiments of the first example method, performing the feature extraction on the current point cloud frame includes performing point synthesis.

[0195] A first example method / apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0196] A second example method / apparatus according to some embodiments may include: obtaining a previously encoded point cloud frame as a reference point cloud frame; obtaining motion features describing the motion between the reference point cloud frame and a current point cloud frame; determining a predictor feature map based on the reference point cloud frame and the motion features; determining a current feature map representing the current point cloud frame, given the predictor feature map as a condition; and encoding the current feature map into a bitstream using the predictor feature map as a condition.

[0197] For some embodiments of the second example method, obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame includes: performing motion analysis using the reference point cloud frame and the current point cloud frame as input to generate a motion feature map; and entropy encoding the motion feature map to generate the motion features.

[0198] In some embodiments of the second example method, obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame includes: performing feature extraction on the reference point cloud frame to generate a first feature map; performing feature extraction on the current point cloud frame to generate a second feature map; fusing the first feature map and the second feature map to generate a fused feature map; and performing feature aggregation on the fused feature map to generate the motion features.

[0199] In some embodiments of the second example method, wherein the reference point cloud frame and the current point cloud frame are voxel-based representations, and wherein performing the feature extraction on the reference point cloud frame includes: passing the reference point cloud frame through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the first feature map, and wherein performing the feature extraction on the current point cloud frame includes: passing the current point cloud frame through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the second feature map.

[0200] In some embodiments of the second example method, wherein the reference point cloud frame and the current point cloud frame are point-based representations, and wherein performing the feature extraction on the reference point cloud frame includes: performing block partitioning on the reference point cloud frame; and performing block-by-block feature extraction on the block-partitioned reference point cloud frame to generate the first feature map, and wherein performing the feature extraction on the current point cloud frame includes: performing block partitioning on the current point cloud frame; and performing block-by-block feature extraction on the block-partitioned current point cloud frame to generate the second feature map.

[0201] In some embodiments of the second example method, obtaining the motion features includes performing motion analysis training.

[0202] In some embodiments of the second example method, determining the predictor feature map based on the reference point cloud frame and the motion features includes: performing feature extraction on the reference point cloud frame to generate a first feature map; concatenating the first feature map with the motion features; passing the concatenated first feature map through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the predictor feature map.

[0203] In some embodiments of the second example method, determining the current feature map includes: performing feature extraction on the current point cloud frame to generate a second feature map; concatenating the second feature map with the predictor feature map; passing the concatenated second feature map through one or more multilayer perceptron (MLP) blocks; and passing the outputs of the one or more MLP blocks through one or more convolutional layers to generate the current feature map.

[0204] For some embodiments of the second example method, performing the feature extraction on at least one of the reference point cloud frame and the current point cloud frame includes using an Inception-ResNet block.

[0205] A second example method / apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0206] A third example apparatus according to some embodiments may include at least one processor configured to perform any of the methods listed above.

[0207] A fourth example apparatus according to some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any of the methods listed above.

[0208] A fifth example apparatus according to some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any of the methods listed above.

[0209] Example signals according to some embodiments may include bitstreams generated according to any of the methods listed above.

[0210] This application describes a wide variety of aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in detail and are generally described in a manner that may sound limiting, at least to illustrate individual characteristics. However, this is for the purpose of clarity and does not limit the application or scope of those aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, the aspects described can also be combined and interchanged with those described in earlier applications.

[0211] The aspects described and contemplated in this application can be implemented in many different forms. While some embodiments are specifically illustrated, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to the transmission of a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, a computer-readable storage medium having instructions stored thereon for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having a bitstream generated according to any of the described methods stored thereon.

[0212] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "pixel" and "sample," and the terms "image," "picture," and "frame." Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0213] The terms HDR (High Dynamic Range) and SDR (Standard Dynamic Range) generally convey specific values ​​of dynamic range to those skilled in the art. However, additional embodiments are also contemplated, in which references to HDR are understood to mean "higher dynamic range," and references to SDR are understood to mean "lower dynamic range." Such additional embodiments are not bound by any specific value of dynamic range that may frequently be associated with the terms "high dynamic range" and "standard dynamic range."

[0214] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first," "second," etc., can be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply an ordering of the modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or in a time period overlapping with the second decoding.

[0215] For example, various numerical values ​​may be used in this application. Specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0216] The embodiments described herein can be implemented by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be of any type suitable for the technical environment and can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0217] Various implementations involve decoding. As used herein, “decoding” can encompass all or part of a process performed, for example, on a received encoded sequence, to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes, or alternatively includes, processes performed by a decoder of the various implementations described herein, such as extracting an image from a tile (packed) image, determining an upsampling filter to use and then upsampling the image, and flipping the image back to its intended orientation.

[0218] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. It will be clear, based on the specific context of the description, whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to a broader decoding process.

[0219] Various implementations involve encoding. In a manner similar to the discussion above regarding “decoding,” the term “encoding,” as used herein, can encompass all or part of a process performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such a process includes one or more processes typically performed by an encoder, such as segmentation, differential coding, transform, quantization, and entropy coding. In various embodiments, such a process also includes, or alternatively includes, processes performed by the encoders of the various implementations described herein.

[0220] As a further example, in one embodiment, "encoding" refers only to entropy encoding; in another embodiment, "encoding" refers only to differential encoding; and in yet another embodiment, "encoding" refers to a combination of differential and entropy encoding. It will be clear, based on the specific context of the description, whether the phrase "encoding process" is intended to specifically refer to a subset of operations or generally to a broader encoding process.

[0221] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0222] Various embodiments refer to rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between bit rate and distortion is typically considered, often taking into account constraints on computational complexity. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of bit rate and distortion. Different approaches exist to address the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, and a complete evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save encoding complexity, particularly by calculating approximate distortion based on predicting or predicting the residual signal rather than the reconstructed signal. A hybrid of these two approaches can also be used, such as by using approximate distortion only for some possible encoding options and full distortion for others. Other methods evaluate only a subset of possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of encoding costs and associated distortion.

[0223] The implementations and aspects described herein can be implemented, for example, in a method or process, apparatus, software program, data stream, or signal. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the features in question can be implemented in other forms (e.g., apparatus or program). An apparatus can be implemented, for example, in suitable hardware, software, and firmware. A method can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. A processor also includes communication devices, such as, for example, a computer, a cellular phone, a portable / personal digital assistant (“PDA”), and other devices that facilitate communication of information between end-users.

[0224] The references to "an embodiment" or "an embodiment" or "an implementation" or "implementation," and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the appearance of the phrases "in an embodiment" or "in an embodiment" or "in an implementation" or "in an implementation," and any other variations appearing throughout this application, do not necessarily refer to the same embodiment.

[0225] Additionally, this application may refer to the term "determined" information. Determined information may include, for example, one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory.

[0226] Furthermore, this application may refer to the various pieces of information referred to as "access". Accessed information may include, for example, one or more of the following: receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0227] Furthermore, this application may refer to the term "receive" for each piece of information. Like "access," "receive" is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) one or more. Moreover, "receive" is generally referred to in one manner or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0228] To understand, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to as many listed items as possible.

[0229] Furthermore, as used herein, the term "signaling" specifically refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals a particular one of several parameters used for region-based filter parameter selection for artifact removal filtering. In this way, in one embodiment, the same parameter is used on both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be done in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the verb form of the term "signaling" has been referred to above, the term "signal" can also be used as a noun herein.

[0230] The implementation can generate various signals that are formatted to carry information, such as information that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the described embodiments. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via various wired or wireless links. The signal may be stored on a processor-readable medium.

[0231] We have described numerous embodiments. Features of these embodiments may be provided individually or in any combination across various claim classes and types. Furthermore, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination, across various claim classes and types: • Adapt the residual at the encoder according to any of the embodiments discussed. • Includes one or more bitstreams or signals of the described syntax element or its variants. • Includes a bitstream or signal that conveys the syntax of information generated according to any of the described embodiments. • Inserting information into the signaling syntax allows the decoder to adapt elements of the residual in a manner corresponding to that used by the encoder. • Create and / or transmit and / or receive and / or decode bit streams or signals including one or more of the described syntax elements or their variants. • Create and / or transmit and / or receive and / or decode according to any of the described embodiments. • A method, process, apparatus, medium for storing instructions, medium for storing data, or signal according to any of the described embodiments. • TVs, set-top boxes, cellular phones, tablets, or other electronic devices that perform adaptive filter parameters according to any of the described embodiments. • TVs, set-top boxes, cellular phones, tablets, or other electronic devices that perform adaptive filter parameters according to any of the described embodiments and display (e.g., using a monitor, screen, or other type of display) the resulting image. • TVs, set-top boxes, cellular phones, tablets, or other electronic devices that select (e.g., using a tuner) a channel to receive signals including encoded images and perform adaptive filter parameters according to any of the described embodiments. • TVs, set-top boxes, cellular phones, tablets, or other electronic devices that receive signals including encoded images over the air (e.g., using an antenna) and perform adaptive filter parameters according to any of the described embodiments.

[0232] Note that the various hardware elements in one or more embodiments described herein are referred to as “modules”, which perform (i.e., execute, implement, etc.) the various functions described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that a person skilled in the art would consider suitable for a given implementation. Each described module may further include executable instructions for performing one or more functions described as being performed by the respective module, and note that these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, and may be stored in one or more suitable non-transitory computer-readable media, such as commonly referred to as RAM, ROM, etc.

[0233] Although the features and elements have been described above in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor storage devices, magnetic media (such as internal hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROMs and digital versatile discs (DVDs)). The processor associated with the software can be used to implement a radio frequency transceiver used in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: Obtain the previously decoded point cloud frame as the reference point cloud frame; Obtain motion features that describe the motion between the reference point cloud frame and the current point cloud frame; The predictor feature map is determined based on the reference point cloud frame and the motion features; Given the predictor feature map as a condition, a current feature map representing the current point cloud frame is obtained; as well as The current point cloud frame is reconstructed based on the current feature map and using the predictor feature map as a condition.

2. The method according to claim 1, wherein, Obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame includes: Motion analysis is performed using the reference point cloud frame and the current point cloud frame as input to generate a motion feature map; and The motion feature map is entropy encoded to generate the motion features.

3. The method according to claim 1, wherein, Obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame includes: Feature extraction is performed on the reference point cloud frame to generate a first feature map; Perform feature extraction on the current point cloud frame to generate a second feature map; The first feature map and the second feature map are fused to generate a fused feature map; and Feature aggregation is performed on the fused feature map to generate the motion features.

4. The method according to claim 3, in, The reference point cloud frame and the current point cloud frame are voxel-based representations, and The feature extraction performed on the reference point cloud frame includes: The reference point cloud frame is passed through one or more multilayer sensing (MLP) blocks; and The output of the one or more MLP blocks is passed through one or more convolutional layers to generate the first feature map. The feature extraction performed on the current point cloud frame includes: The current point cloud frame is passed through one or more multilayer sensing (MLP) blocks; and The output of the one or more MLP blocks is passed through one or more convolutional layers to generate the second feature map.

5. The method according to claim 3, in, The reference point cloud frame and the current point cloud frame are point-based representations, and The feature extraction performed on the reference point cloud frame includes: Perform block partitioning on the reference point cloud frame; and Block-by-block feature extraction is performed on the block-partitioned reference point cloud frame to generate the first feature map, and The feature extraction performed on the current point cloud frame includes: Perform block partitioning on the current point cloud frame; and Block-by-block feature extraction is performed on the current point cloud frame that has been partitioned into blocks to generate the second feature map.

6. The method according to any one of claims 1-5, wherein, Obtaining the motion characteristics includes performing motion analysis training.

7. The method according to any one of claims 1-6, wherein, Determining the predictor feature map based on the reference point cloud frame and the motion features includes: Feature extraction is performed on the reference point cloud frame to generate a first feature map; Connect the first feature map with the motion feature; The first feature map of the connection is passed through one or more multilayer perceptron (MLP) blocks; and The output of the one or more MLP blocks is passed through one or more convolutional layers to generate the predictor feature map.

8. The method according to any one of claims 1-7, in, Performing the feature extraction on the reference point cloud frame includes performing point synthesis.

9. The method according to any one of claims 1-8, in, Performing the feature extraction on the current point cloud frame includes performing point synthesis.

10. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by a processor, are operable to cause the apparatus to perform the method of any one of claims 1 to 9.

11. A method comprising: Obtain the previously encoded point cloud frame as the reference point cloud frame; Obtain motion features that describe the motion between the reference point cloud frame and the current point cloud frame; The predictor feature map is determined based on the reference point cloud frame and the motion features; Given the predictor feature map as a condition, determine the current feature map representing the current point cloud frame; as well as The current feature map is encoded into a bitstream using the predictor feature map as a condition.

12. The method of claim 11, wherein, Obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame includes: Motion analysis is performed using the reference point cloud frame and the current point cloud frame as input to generate a motion feature map; and The motion feature map is entropy encoded to generate the motion features.

13. The method according to claim 12, wherein, Obtaining motion features describing the motion between the reference point cloud frame and the current point cloud frame includes: Feature extraction is performed on the reference point cloud frame to generate a first feature map; Perform feature extraction on the current point cloud frame to generate a second feature map; The first feature map and the second feature map are fused to generate a fused feature map; and Feature aggregation is performed on the fused feature map to generate the motion features.

14. The method according to claim 13, in, The reference point cloud frame and the current point cloud frame are voxel-based representations, and The feature extraction performed on the reference point cloud frame includes: The reference point cloud frame is passed through one or more multilayer sensing (MLP) blocks; and The output of the one or more MLP blocks is passed through one or more convolutional layers to generate the first feature map. The feature extraction performed on the current point cloud frame includes: The current point cloud frame is passed through one or more multilayer sensing (MLP) blocks; and The output of the one or more MLP blocks is passed through one or more convolutional layers to generate the second feature map.

15. The method according to claim 13, in, The reference point cloud frame and the current point cloud frame are point-based representations, and The feature extraction performed on the reference point cloud frame includes: Perform block partitioning on the reference point cloud frame; and Block-by-block feature extraction is performed on the block-partitioned reference point cloud frame to generate the first feature map, and The feature extraction performed on the current point cloud frame includes: Perform block partitioning on the current point cloud frame; and Block-by-block feature extraction is performed on the current point cloud frame that has been partitioned into blocks to generate the second feature map.

16. The method according to any one of claims 11-15, wherein, Obtaining the motion characteristics includes performing motion analysis training.

17. The method according to any one of claims 11-16, wherein, Determining the predictor feature map based on the reference point cloud frame and the motion features includes: Feature extraction is performed on the reference point cloud frame to generate a first feature map; Connect the first feature map with the motion feature; The first feature map of the connection is passed through one or more multilayer perceptron (MLP) blocks; and The output of the one or more MLP blocks is passed through one or more convolutional layers to generate the predictor feature map.

18. The method according to any one of claims 11-17, wherein, Determining the current feature map includes: Perform feature extraction on the current point cloud frame to generate a second feature map; Connect the second feature map to the predictor feature map; The second feature map of the connection is passed through one or more multilayer perceptron (MLP) blocks; and The output of the one or more MLP blocks is passed through one or more convolutional layers to generate the current feature map.

19. The method according to any one of claims 11-18, wherein, Performing the feature extraction on at least one of the reference point cloud frame and the current point cloud frame includes using an Inception-ResNet block.

20. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by a processor, are operable to cause the apparatus to perform the method of any one of claims 11 to 19.

21. An apparatus comprising at least one processor configured to perform the method of any one of claims 1-9 and 11-19.

22. An apparatus comprising a computer-readable medium storing instructions for causing one or more processors to perform the method of any one of claims 1-9 and 11-19.

23. An apparatus comprising at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform the method of any one of claims 1-9 and 11-19.

24. A signal comprising a bit stream generated according to any one of claims 1-9 and 11-19.

Images