Deep distribution-aware point feature extractor for AI-based point cloud compression

Abstract

Some embodiments of the method may include a learning-based point cloud geometry processing block method, the method including: accessing a first feature map, the first feature map having a number of C channels and being an input to a processing block, the first feature map being generated by a first set of neural network layers; accessing a set of distribution parameters; transforming the first feature map into a second feature map based on the set of distribution parameters; and encoding the second feature map into a bitstream. These example processes may be applicable to both an encoder and a decoder in an AI-based point cloud compression (PCC) framework.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is a nonprovisional adaptation of U.S. Provisional Patent Application No. 63 / 388,600, entitled "Deep Distribution-Aware Point Feature for AI-Based Point Cloud Compression," filed July 12, 2022 (the "'600 Application"), and claims the benefit under 35 U.S.C. § 119(e), which is incorporated herein by reference in its entirety. The following examples are incorporated herein by reference in their entirety: U.S. Provisional Patent Application No. 63 / 252,482, filed October 5, 2021 (the "'482 Application"), entitled "Method and Apparatus for Point Cloud Compression Using Hybrid Deep Entropy Coding," U.S. Provisional Patent Application No. 63 / 297,894, filed January 10, 2022 (the "'894 Application"), entitled "Coordinate Refinement and Upsampling from Quantized Point Cloud Reconstruction," and U.S. Provisional Patent Application No. 63 / 297,869, filed January 10, 2022 (the "'869 Application"), entitled "Scalable Framework for Point Cloud Compression." [Background technology]

[0002] Point clouds are data that can be used in many business fields, from autonomous driving, robotics, AR / VR, civil engineering, and computer graphics to the animation / film industry. 3D LiDAR sensors are being deployed in self-driving cars, and affordable LiDAR sensors include the Velodyne Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR Camera L515. With advances in sensing technology, 3D point cloud data is becoming more widespread in the applications and industries mentioned above and others. Summary of the Invention

[0003] An exemplary learning-based point cloud geometry processing block method according to some embodiments may include accessing a first feature map, the first feature map having multiple C channels and being an input to a processing block, the first feature map being generated by a first set of neural network layers; accessing a set of distribution parameters; converting from the first feature map to a second feature map based on the set of distribution parameters; and encoding the second feature map into a bitstream.

[0004] Some embodiments of the example learning-based point cloud geometry coding block method may further include updating the first feature map by normalizing elements of the first feature map.

[0005] In some embodiments of the exemplary learning-based point cloud geometry encoding block method, normalizing the elements of the first feature map may include determining a respective length of each feature vector associated with one of the elements of the first feature map and dividing each element of the first feature map by the respective length.

[0006] In some embodiments of the exemplary learning-based point cloud geometry encoding block method, normalizing the elements of the first feature map may include arranging each of the one or more vectors by reshaping an associated feature channel in the first feature map; determining a respective length of each of the one or more vectors for each feature channel; and dividing each vector element of each vector by the respective length for each feature channel, where each vector element is one of the elements of the first feature map.

[0007] In some embodiments of the exemplary learning-based point cloud geometry encoding block method, normalizing the elements of the first feature map may include: arranging each of the one or more vectors by reshaping an associated feature channel in the first feature map; determining a respective standard deviation of each of the one or more vectors for each feature channel; determining a respective mean of each of the one or more vectors for each feature channel; updating each vector element of each vector by subtracting the respective mean; and dividing each updated vector element of each vector by the respective standard deviation for each feature channel, wherein each vector element is one of the elements of the first feature map.

[0008] In some embodiments of the exemplary learning-based point cloud geometry coding block method, the set of distribution parameters is determined using back-propagation techniques during a training period.

[0009] In some embodiments of the exemplary learning-based point cloud geometry encoding block method, a set of distribution parameters is determined for each feature channel.

[0010] Some embodiments of the exemplary learning-based point cloud geometry coding block method may further include updating the second feature map by performing downsampling using an average pooling or max pooling function.

[0011] Some embodiments of the exemplary learning-based point cloud geometry coding block method may further include determining a third feature map by filtering the second feature map with a smoothing filter, and updating the second feature map by concatenating the third feature map to the second feature map.

[0012] In some embodiments of the example learning-based point cloud geometry encoding block method, performing processing prior to encoding the second feature map may include accessing a second set of distribution parameters and updating the second feature map by transforming the second feature map based on the second set of distribution parameters.

[0013] In some embodiments of the exemplary learning-based point cloud geometry encoding block method, performing processing prior to encoding the second feature map may include aggregating the second feature map using a second neural network.

[0014] In some embodiments of the exemplary learning-based point cloud geometry coding block method, the second neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

[0015] In some embodiments of the exemplary learning-based point cloud geometry coding block method, aggregating the second feature map may include using a Residual Network (ResNet) architecture.

[0016] Some embodiments of the exemplary learning-based point cloud geometry encoding block method may further include, in parallel with converting the first feature map to the second feature map, determining a fourth feature map by aggregating the first feature maps using a neural network, and updating the second feature map by concatenating the fourth feature map to the second feature map.

[0017] In some embodiments of the exemplary learning-based point cloud geometry coding block method, aggregating the first feature map using a neural network may include using a residual network (ResNet) architecture.

[0018] In some embodiments of the exemplary learning-based point cloud geometry encoding block method, performing processing prior to converting the first feature map into the second feature map includes aggregating the first feature map using a third neural network.

[0019] In some embodiments of the exemplary learning-based point cloud geometry coding block method, the third neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

[0020] An exemplary learning-based point cloud geometry encoding block apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to cause the apparatus to perform any of the above-listed claims.

[0021] An exemplary learning-based point cloud geometry decoder method according to some embodiments may include decoding a first feature map from a bitstream, accessing a set of distribution parameters, transforming the first feature map into a second feature map based on the set of distribution parameters, and reconstructing a point cloud from the second feature map.

[0022] Some embodiments of the example learning-based point cloud geometry decoder method may further include updating the first feature map by normalizing elements of the first feature map.

[0023] In some embodiments of the example learning-based point cloud geometry decoder method, normalizing the elements of the first feature map may include determining a respective length of each feature vector associated with one of the elements of the first feature map, and dividing each element of the first feature map by the respective length.

[0024] In some embodiments of the example learning-based point cloud geometry decoder method, normalizing the elements of the first feature map may include: arranging each of the one or more vectors by reshaping an associated feature channel in the first feature map; determining a respective length of each of the one or more vectors for each feature channel; and dividing each vector element of each vector by the respective length for each feature channel, wherein each vector element is one of the elements of the first feature map.

[0025] In some embodiments of the example learning-based point cloud geometry decoder method, normalizing the elements of the first feature map may include: arranging each of the one or more vectors by reshaping an associated feature channel in the first feature map; determining a respective standard deviation of each of the one or more vectors for each feature channel; determining a respective mean of each of the one or more vectors for each feature channel; updating each vector element of each vector by subtracting the respective mean; and dividing each updated vector element of each vector by the respective standard deviation for each feature channel, wherein each vector element is one of the elements of the first feature map.

[0026] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the set of distribution parameters is determined during a training period using back-propagation techniques.

[0027] In some embodiments of the exemplary learning-based point cloud geometry decoder method, a set of distribution parameters is determined for each feature channel.

[0028] In some embodiments of the exemplary learning-based point cloud geometry decoder method, performing processing prior to encoding the second feature map includes accessing a second set of distribution parameters and updating the second feature map by transforming the second feature map based on the second set of distribution parameters.

[0029] In some embodiments of the exemplary learning-based point cloud geometry decoder method, performing processing prior to encoding the second feature map includes aggregating the second feature map using a second neural network.

[0030] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the second neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

[0031] In some embodiments of the exemplary learning-based point cloud geometry decoder method, aggregating the second feature map may include using a residual network (ResNet) architecture.

[0032] In some embodiments of the exemplary learning-based point cloud geometry decoder method, performing processing prior to converting the first feature map into the second feature map includes aggregating the first feature map using a third neural network.

[0033] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the third neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

[0034] An exemplary learning-based point cloud geometry decoder device according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to cause the device to perform any of the methods recited above.

[0035] The embodiments described herein include methods used in video encoding and decoding (collectively "coding").

[0036] In further embodiments, encoder and decoder devices are provided for performing the methods described herein. The encoder or decoder devices may include a processor configured to perform the methods described herein. The devices may include a computer-readable medium (e.g., a non-transitory medium) that stores instructions for performing the methods described herein. In some embodiments, the computer-readable medium (e.g., a non-transitory medium) stores video encoded using any of the methods described herein.

[0037] One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for performing bidirectional optical flow and encoding or decoding video data according to any of the methods described above. The present embodiments also provide a computer-readable storage medium having stored thereon a bitstream generated according to the methods described above. The present embodiments also provide a method and apparatus for transmitting a bitstream generated according to the methods described above. The present embodiments also provide a computer program product including instructions for performing any of the methods described. [Brief explanation of the drawings]

[0038] [Figure 1A] FIG. 1 is a system diagram illustrating an exemplary communication system according to some embodiments. [Figure 1B]1B is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that may be used within the communication system shown in FIG. 1A, in accordance with some embodiments. [Figure 1C] FIG. 1 is a functional block diagram of a system according to some embodiments. [Figure 2A] 1 is a functional block diagram of a block-based video encoder, such as an encoder used for Versatile Video Coding (VVC), according to some embodiments. [Figure 2B] FIG. 2 is a functional block diagram of a block-based video decoder, such as a decoder used in VVC, according to some embodiments. [Figure 3] FIG. 1 is a functional block diagram illustrating an example encoder architecture for point cloud (PC) feature extraction according to some embodiments. [Figure 4] FIG. 1 is a functional block diagram illustrating an example of how a PC group size K can be determined for a given voxel quantization for PC reconstruction in a point cloud compression (PCC) framework according to some embodiments. [Figure 5] FIG. 1 is a functional block diagram illustrating exemplary point-wise feature extraction according to some embodiments. [Figure 6] FIG. 1 is a functional block diagram illustrating an example group feature distribution tensor according to some embodiments. [Figure 7] FIG. 1 is a functional block diagram illustrating an example group feature distribution with learnable transformation parameters according to some embodiments. [Figure 8] FIG. 1 is a functional block diagram illustrating exemplary transformed feature aggregation, augmentation, and dimension matching, according to some embodiments. [Figure 9] FIG. 1 is a functional block diagram illustrating exemplary point-wise residual feature extraction and group-wise feature aggregation and augmentation according to some embodiments. [Figure 10]FIG. 1 is a functional block diagram illustrating an example encoder architecture for PC feature extraction according to some embodiments. [Figure 11] FIG. 2 is a functional block diagram illustrating an exemplary decoder architecture according to some embodiments. [Figure 12A] FIG. 1 is a functional block diagram showing an encoder and a decoder in autoencoder-based lossy geometry compression. [Figure 12B] FIG. 1 is a functional block diagram illustrating an exemplary application of the PCC framework to an encoder and decoder in autoencoder-based lossy geometry compression, according to some embodiments. [Figure 13A] FIG. 1 is a functional block diagram illustrating point analysis and point synthesis within a scalable PCC framework. [Figure 13B] FIG. 1 is a functional block diagram illustrating an exemplary application of the PCC framework to point analysis and point synthesis within a scalable PCC framework, according to some embodiments. [Figure 14A] FIG. 1 is a functional block diagram illustrating set abstraction (SA) processing in the PointContextNet environment. [Figure 14B] FIG. 1 is a functional block diagram illustrating an exemplary application of the PCC framework to Set Abstraction (SA) processing in a PointContextNet environment, according to some embodiments. [Figure 15A] FIG. 1 is a functional block diagram illustrating set abstraction (SA) processing in the coordinate refinement module (CRM). [Figure 15B] FIG. 1 is a functional block diagram illustrating an exemplary application of the PCC framework to set abstraction (SA) processing in a coordinate refinement module (CRM), according to some embodiments. [Figure 16] 1 is a flowchart illustrating an exemplary process for point cloud feature extraction, according to some embodiments. [Figure 17]FIG. 1 is a functional block diagram illustrating a deep feature-based PCC pipeline, according to some embodiments. [Figure 18] FIG. 1 is a functional block diagram illustrating a DDA-Net encoder architecture, according to some embodiments. [Figure 19] FIG. 1 is a functional block diagram illustrating a probability estimator distribution network, according to some embodiments. [Figure 20] 1 is a flowchart illustrating an exemplary process for point cloud feature extraction, according to some embodiments. [Figure 21] 1 is a flowchart illustrating an exemplary process for point cloud feature extraction, according to some embodiments. [Figure 22] 1 is a flowchart illustrating an exemplary learning-based point cloud geometry processing, according to some embodiments. [Figure 23] 1 is a flowchart illustrating an exemplary learning-based point cloud geometry processing, according to some embodiments.

[0039] The entities, connections, arrangements, etc. shown in and described in connection with the various figures are presented by way of example and not limitation. Thus, any and all statements or other indications regarding what a particular figure "shows," what a particular element or entity of a particular figure "is," or what "has," and any and all similar statements that, alone and out of context, may be construed as absolute and therefore limiting, may properly be read only as structurally preceded by a clause such as "in at least one embodiment...." For brevity and clarity of presentation, this implied precedent clause will not be repeated multiple times in the Detailed Description. DETAILED DESCRIPTION OF THE INVENTION

[0040] Exemplary Network for Implementing the Embodiments A wireless transmit / receive unit (WTRU) may be used, for example, to perform point cloud (PC) extraction in some embodiments described herein.

[0041] 1A is a diagram illustrating an example communications system 100 in which one or more disclosed embodiments may be implemented. Communications system 100 may be a multiple-access system that provides content, such as voice, data, video, messaging, broadcasts, etc., to multiple wireless users. Communications system 100 may enable multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multicarrier (FBMC), etc.

[0042] 1A, communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106, public switched telephone network (PSTN) 108, Internet 110, and other networks 112, although it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a "station" and / or "STA," may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, mobile phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (loT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., for remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain contexts), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be referred to interchangeably as a UE.

[0043] The communications system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communications networks, such as the CN 106, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNodeB, a Home Node B, a Home eNodeB, a gNB, an NR Node B, a site controller, an access point (AP), a wireless router, etc. Although the base stations 114a, 114b are each depicted as a single element, it will be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0044] The base station 114a may be part of the RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage for a particular geographic area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In some embodiments, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers per sector of the cell, for example, using beamforming to transmit and / or receive signals in desired spatial directions.

[0045] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communications link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0046] More specifically, as noted above, the communications system 100 may be a multiple-access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base stations 114a of the RANs 104 / 113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 116 using wideband CDMA (WCDMA). WCDMA may include communications protocols such as High-Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink Packet Access (HSDPA) and / or High-Speed ​​Uplink Packet Access (HSUPA).

[0047] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0048] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as New Radio (NR) radio access, which may establish the air interface 116 using NR.

[0049] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may jointly implement LTE radio access and NR radio access, e.g., using dual connectivity (DC) principles. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0050] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a wireless technology such as IEEE 802.11 (i.e., Wireless Fidelity, WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access, WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), or the like.

[0051] 1A may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point and may utilize any suitable RAT to facilitate wireless connectivity in a local area such as a business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, etc. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may establish a picocell or a femtocell using a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). As shown in FIG. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not need to access the Internet 110 through the CN 106.

[0052] The RAN 104 / 113 may communicate with the CN 106, which may be any type of network configured to provide voice, data, application, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have various quality of service (QoS) requirements, such as different throughput, latency, error tolerance, reliability, data throughput, and mobility requirements. The CN 106 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, and / or perform high-level security functions such as user authentication. Although not shown in FIG. 1A , it will be understood that the RAN 104 / 113 and / or the CN 106 may communicate directly or indirectly with other RANs employing the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which may utilize NR radio technology, the CN 106 may also communicate with another RAN (not shown) employing GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0053] The CN 106 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices, which use common communication protocols such as the transmission control protocol (TCP), the user datagram protocol (UDP), and / or the internet protocol (IP) of the TCP / IP Internet protocol suite. The networks 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the network 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RANs 104 / 113 or a different RAT.

[0054] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links.) For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with a base station 114a, which may employ a cellular-based wireless technology, and a base station 114b, which may employ an IEEE 802.2 wireless technology.

[0055] 1B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0056] The processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. While FIG. 1B depicts the processor 118 and the transceiver 120 as separate components, it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0057] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR signals, UV signals, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF signals and light signals. It will be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0058] 1B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0059] The transceiver 120 may be configured to modulate signals transmitted by the transmit / receive element 122 and demodulate signals received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as, for example, NR and IEEE 802.11.

[0060] The processor 118 of the WTRU 102 may be coupled to and may receive user-entered data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. Additionally, the processor 118 may access information from and store data in any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In other embodiments, the processor 118 may access information from and store data in memory that is not physically located on the WTRU 102, such as on a server or home computer (not shown).

[0061] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control the power to other components in the WTRU 102. The power source 134 may be any suitable device for providing power to the WTRU 102. For example, the power source 134 may include one or more dry batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0062] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or instead of, information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0063] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, etc. The peripheral device 138 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, a direction sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0064] The WTRU 102 may include a full-duplex radio where transmission and reception of some or all of the signals associated with a particular subframe (e.g., for both the UL (e.g., for transmission) and downlink (e.g., for reception)) may be parallel and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and or substantially eliminate self-interference through either hardware (e.g., a choke) or signal processing via a processor (e.g., via a separate processor (not shown) or processor 118). In an embodiment, the WTRU 102 may include a half-duplex radio for transmission and reception of some or all of the signals (e.g., associated with a particular subframe for either the UL (e.g., for transmission) or downlink (e.g., for reception)).

[0065] Although the WTRU is described in FIGS. 1A-1B as a wireless terminal device, it is contemplated that in certain representative embodiments, such a terminal device may use a wired communication interface (e.g., temporarily or permanently) with the communication network.

[0066] In a representative embodiment, the other network 112 may be a WLAN.

[0067] 1A-1B and the corresponding description, one or more or all of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more or all of the functions described herein. For example, an emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0068] The emulation devices may be designed to implement one or more tests of other devices in a lab environment and / or an operator network environment. For example, one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless communications network to test other devices in the communications network. One or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless communications network. The emulation devices may be directly coupled to another device for testing purposes and / or may use terrestrial wireless communications to perform the tests.

[0069] One or more emulation devices may perform one or more functions, inclusive, while not being implemented / deployed as part of a wired and / or wireless communications network. For example, the emulation devices may be utilized in test scenarios in a test lab and / or in an undeployed (e.g., test) wired and / or wireless communications network to implement testing of one or more components. One or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (which may include, e.g., one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0070] Exemplary System The embodiments described herein are not limited to being implemented on a WTRU. Such embodiments may be implemented using other systems, such as the system of FIG. 1C. FIG. 1C is a block diagram of an example system in which various aspects and embodiments may be implemented. System 150 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 150, singly or in combination, may be embodied in a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 150 are distributed across multiple ICs and / or separate components. In various embodiments, system 150 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, the system 150 is configured to implement one or more of the aspects described herein.

[0071] System 150 includes at least one processor 152 configured to execute instructions loaded therein, for example, to implement various aspects described herein. Processor 152 can include embedded memory, input / output interfaces, and various other circuitry known in the art. System 150 includes at least one memory 154 (e.g., a volatile memory device and / or a non-volatile memory device). System 150 includes storage device 158, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Flash, magnetic disk drives, and / or optical disk drives. Storage device 158 may include, by way of non-limiting example, an internal storage device, an attached storage device (including removable and non-removable storage devices), and / or a network-accessible storage device.

[0072] System 150 includes an encoder / decoder module 156 configured to process data to provide, for example, encoded or decoded video, which may include its own processor and memory. Encoder / decoder module 156 represents a module that may be included in a device for performing encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, encoder / decoder module 156 may be implemented as a separate element of system 150 or may be incorporated within processor 152 as a combination of hardware and software, as is known to those skilled in the art.

[0073] Program code to be loaded into the processor 152 or the encoder / decoder 156 to perform various aspects described herein may be stored in the storage device 158 and subsequently read into the memory 154 for execution by the processor 152. According to various embodiments, one or more of the processor 152, the memory 154, the storage device 158, and the encoder / decoder module 156 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and computational logic.

[0074] In some embodiments, memory internal to the processor 152 and / or the encoder / decoder module 156 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 152 or the encoder / decoder module 156) is used for one or more of these functions. The external memory may be the memory 154 and / or the storage device 158, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store, for example, the television's operating system. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET).

[0075] Inputs to the elements of system 150 can be provided through various input devices indicated in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcast station, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high definition multimedia interface (HDMI) input terminal. Another example, not shown in FIG. 1C, is composite video.

[0076] In various embodiments, the input devices of block 1130 have associated respective input processing elements known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulating the downconverted, band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, downconverting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, omit some of these elements, and / or add other elements that perform similar or different functions. Adding elements can include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0077] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 150 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or within processor 152, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, as desired, in a separate interface IC or within processor 152. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 152 and encoder / decoder 156, which operates in combination with memory and storage elements, to process the data streams necessary for presentation to an output device.

[0078] The various elements of system 150 may be provided within an integrated housing in which the various elements may be interconnected and data may be transmitted between them using suitable connection devices 1140, such as internal buses known in the art, including Inter-IC (I2C) buses, wiring, and printed circuit boards.

[0079] System 150 includes a communication interface 160 that enables communication with other devices over a communication channel 162. Communication interface 160 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 162. Communication interface 160 may include, but is not limited to, a modem or a network card, and communication channel 162 may be implemented in a wired medium and / or a wireless medium, for example.

[0080] In various embodiments, data is streamed or otherwise provided to system 150 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in such embodiments is received via communication channel 162 and communication interface 160 adapted for Wi-Fi communication. Typically, communication channel 162 in such embodiments is connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, streamed data is provided to system 150 using a set-top box that delivers data via an HDMI connection in input block 1130. In yet other embodiments, streamed data is provided to system 150 using an RF connection in input block 1130. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0081] System 150 can provide output signals to various output devices, including a display 170, speakers 172, and other peripheral devices 174. Display 170 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 170 can be for a television, a tablet, a laptop, a mobile phone, or other device. Display 170 can also be integrated with other components (e.g., as in a smartphone) or can be separate (e.g., an external monitor for a laptop). In various example embodiments, other peripheral devices 174 include one or more of a standalone digital video disc (or digital versatile disc) (both terms referred to as a DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 174 that provide functionality based on the output of system 150. For example, a disc player performs the function of playing the output of system 150.

[0082] In various embodiments, control signals are communicated between system 150 and display 170, speakers 172, or other peripheral devices 174 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that allow control between devices with or without user intervention. Output devices can be communicatively coupled to system 150 via dedicated connections through respective interfaces 164, 166, and 168. Alternatively, output devices can be connected to system 150 via communication interface 160 using communication channel 162. Display 170 and speakers 172 may be integrated into a single unit with other components of system 150 in an electronic device such as a television. In various embodiments, display interface 164 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0083] Alternatively, display 170 and speakers 172 may be separate from one or more of the other components, for example, if the RF portion of input 1130 is part of a separate set-top box. In various embodiments in which display 170 and speakers 172 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0084] The embodiments may be implemented by computer software implemented by the processor 152, by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 154 may be of any type appropriate to the technology environment and may be implemented using any suitable data storage technology, including, by way of non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 152 may be of any type appropriate to the technology environment and may include, by way of non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a multi-core architecture-based processor.

[0085] Block-Based Video Coding Like HEVC, VVC is built on a block-based hybrid video coding framework. Figure 2A shows a block diagram of a block-based hybrid video encoding system 200. While variations of this encoder 200 are contemplated, the following describes the encoder 200 for clarity without listing all possible variations.

[0086] Before encoding, the video sequence may undergo encoding preprocessing (204), such as applying a color transformation to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata can be associated with the preprocessing and added to the bitstream.

[0087] An input video signal 202 containing a picture to be coded is divided (206) into blocks, for example, by CU, and processed. Different CUs may have different sizes. In VTM-1.0, a CU can be up to 128 x 128 pixels. However, unlike HEVC, which divides blocks only based on a quad tree, in VTM-1.0, coding tree units (CTUs) are divided into CUs to accommodate various local characteristics based on a quad / binary / ternary tree. Furthermore, the concept of multiple partition unit types in HEVC is eliminated, such that the separation of CUs, prediction units (PUs), and transform units (TUs) no longer exists in VVC-1.0. Instead, each CU is always used as the basic unit for both prediction and transformation without further partitioning. In a multi-type tree structure, CTUs are first divided by a quad tree structure. Then, each quad tree leaf node can be further divided by a binary and ternary tree structure. Different split types may be used, such as 4-way split, 2-way vertical split, 2-way horizontal split, 3-way vertical split, and 3-way horizontal split.

[0088] In the encoder of FIG. 2A, spatial prediction (208) and / or temporal prediction (210) may be performed. Spatial prediction (or "intra-prediction") uses pixels from samples of previously coded neighboring blocks (called reference samples) within the same video picture / slice to predict the current video block. Spatial prediction reduces spatial redundancy inherent in video signals. Temporal prediction (also called "inter-prediction" or "motion-compensated prediction") uses reconstructed pixels from previously coded video pictures to predict the current video block. Temporal prediction reduces temporal redundancy inherent in video signals. The temporal prediction signal for a given CU may be signaled by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. Also, if multiple reference pictures are supported, a reference picture index may additionally be transmitted, which is used to identify which reference picture in the reference picture store (212) the temporal prediction signal comes from.

[0089] A mode decision block (214) in the encoder selects the best prediction mode, for example, based on a rate-distortion optimization method. This selection may be made after spatial prediction and / or temporal prediction is performed. The intra / inter decision may be indicated, for example, by a prediction mode flag. The prediction block is subtracted (216) from the current video block to generate a prediction residual. The prediction residual is decorrelated (218) using a transform and quantized (220). (For some blocks, the encoder may bypass both the transform and quantization, in which case the residual may be coded directly without applying a transform or quantization process.) The quantized residual coefficients are inverse quantized (222) and inverse transformed (224) to form a reconstructed residual, which is then added back to the prediction block (226) to form the reconstructed signal for the CU. Further in-loop filtering, such as deblocking filtering / sample adaptive offset (SAO) filtering, may be applied to the reconstructed CUs (228) before they are placed in the reference picture store (212) and used to code future video blocks to reduce encoding artifacts. To form the output video bitstream 230, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit (108) to be further compressed and packed to form the bitstream.

[0090] Figure 2B shows a block diagram of a block-based video decoder 250. In the decoder 250, a bitstream is decoded by decoder elements, as described below. The video decoder 250 generally performs a decoding path that is the reverse of the encoding path described in Figure 2A. The encoder 200 also generally performs video decoding as part of encoding the video data.

[0091] In particular, the decoder's input includes a video bitstream 252, which may be generated by the video encoder 200. The video bitstream 252 is first unpacked and entropy decoded in an entropy decoding unit 254 to obtain transform coefficients, motion vectors, and other coding information. Picture partition information indicates how the picture is partitioned. Thus, the decoder may partition the picture according to the decoded picture partition information (256). Coding mode and prediction information are sent to either a spatial prediction unit 258 (if intra-coded) or a temporal prediction unit 260 (if inter-coded) to form a prediction block. The residual transform coefficients are sent to an inverse quantization unit 262 and an inverse transform unit 264 to reconstruct a residual block. The prediction block and the residual block are then added together at 266 to generate a reconstructed block. The reconstructed block may further undergo in-loop filtering 268 before being stored in a reference picture store 270 for use in predicting future video blocks.

[0092] The decoded picture 272 may further undergo post-decoding processing (274), such as an inverse color transform (e.g., converting from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (204). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream. The decoded and processed video may be sent to a display device 276. The display device 276 may be a separate device from the decoder 250, or the decoder 250 and display device 276 may be components of the same device.

[0093] Various methods and other aspects described in this disclosure may be used to modify modules of video encoder 200 or decoder 250. Furthermore, the systems and methods of this disclosure are not limited to VVC or HEVC, but may also be applied to, for example, other standards and recommendations, whether existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, aspects described in this disclosure may be used individually or in combination.

[0094] This disclosure discusses point cloud compression and processing, which may include tools for compression, analysis, interpolation, representation, and understanding of point cloud signals.

[0095] Point cloud data may consume a large portion of network traffic, for example, between cars connected via 5G networks and in immersive communications (VR / AR / MR). Efficient representation formats can be used for point cloud communication. In particular, raw point cloud data can be organized and processed for modeling and sensing of the world, environment, or scene, etc. Compression of the raw point cloud can be used for data storage and transmission.

[0096] Furthermore, a point cloud can represent a continuous scan of a scene that may contain multiple moving objects. Such a point cloud is called a dynamic point cloud, as compared to a static point cloud that may be captured from a static scene and / or static objects. A dynamic point cloud may be organized into frames, where different frames are captured at different times. Processing and compression of the dynamic point cloud may be performed in real time or with low latency.

[0097] For example, the automotive industry, including autonomous vehicles, may use point clouds. Autonomous vehicles should be able to "probe" their environment and make good driving decisions based on their immediate surroundings. Typically, LiDAR sensors generate (dynamic) point clouds that are used by perception engines. Furthermore, these point clouds are typically dynamic with a high capture frequency, sparse, not necessarily color-coded, and invisible to the human eye. Such point clouds can indicate the material of the sensed object and can include other attributes such as reflectivity provided by the LiDAR that can be used in making decisions.

[0098] Virtual reality (VR) and immersive worlds are a hot topic and are predicted by many to be the future of 2D flat video. Viewers can be immersed in a full-surrounding environment, as opposed to standard television, which only sees a virtual world in front of the viewer. There are several degrees of immersion depending on the viewer's degrees of freedom within the environment. Point cloud formats can be used to deliver VR world and environment data. Such point clouds can be static or dynamic, and typically have an average size of several million points at a time.

[0099] Point clouds can also be used for a variety of other purposes, such as scanning cultural heritage objects and / or buildings, where objects such as statues or buildings are scanned in 3D. The spatial configuration data of an object can be shared without dispatching or visiting the actual object or building. This data can also be used to preserve knowledge about an object when an earthquake destroys the object or building, such as a temple. Such point clouds are typically static, color-coded, and huge in size.

[0100] Another use case is in topography and cartography, which uses 3D representations, where maps are not limited to flat surfaces but may include relief. Google Maps, for example, may use meshes instead of point clouds for their 3D maps. Nevertheless, point clouds may be the preferred data format for three-dimensional maps, where such point clouds are usually static, color-coded, and huge in size.

[0101] World modeling and sensing via point clouds can enable machines to record and use spatial configuration data in the 3D world around them, which can be used in the applications mentioned above.

[0102] 3D point cloud data includes discrete samples on the surface of an object or scene. A huge number of points can be used to fully represent the real world with point samples. For example, a typical VR immersive scene can include millions of points, while a point cloud can typically include hundreds of millions of points. Therefore, processing such large point clouds is computationally expensive, especially for consumer devices that may have limited computing power, such as smartphones, tablets, and automobile navigation systems.

[0103] Any processing or inference of point clouds may use efficient storage methodologies. To store and process an input point cloud at an affordable computational cost, the input point cloud may be downsampled, with the downsampled point cloud encapsulating the geometry of the input point cloud while having far fewer points. The downsampled point cloud is input to a subsequent machine task for further processing. However, further reduction in storage space may be achieved by converting the raw point cloud data (original or downsampled) into a bitstream through entropy coding techniques for lossless compression.

[0104] In addition to lossless coding, many scenarios use lossy coding to significantly improve the compression ratio while maintaining the induced distortion under a certain quality level. To achieve less lossy coding, efficient point feature extractors can be used to improve the accuracy of reconstruction within a given resource budget.

[0105] Several papers have shown interest in the application of sparse convolutions, including: Graham, Benjamin, Sparse 3D Convolutional Neural Networks, ARXIV PREPRINT, arXiv:1505.02890 (2015); Liu, Baoyuan et al., Sparse Convolutional Neural Networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 806-814 (2015); Graham, Benjamin et al., 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9224-9232 (2018); Choy, Christopher et al., 4D Spatio-Temporal Convnets: Minkowski Convolutional Neural Networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3075-3084 (2019). In line with these papers, point feature extraction from point cloud (PC) data is used in artificial intelligence (AI)-based PC analysis, such as classification, segmentation, registration, and compression. Point cloud compression (PCC) in such applications may employ a trade-off between complexity (e.g., computational cost or storage consumption) and performance (e.g., accuracy) for PC reconstruction, and a balanced architecture between the encoder and decoder.

[0106] Although local group analysis may allow for the extraction of more representative features, the method should not become too complex due to the encoding and decoding times in the PCC framework. Furthermore, point feature extractors may be memory-limited, allowing for deeper architectures without significantly increasing feature dimensionality.

[0107] Recent AI-based end-to-end frameworks and deep entropy models for point cloud compression (PCC) have focused heavily on applying sparse voxel convolutions and less on point-based feature extraction from point cloud data. Point-by-point feature analysis may play a more significant role as the bit depth of the input data increases. Furthermore, the geometric representation of point clouds impacts the ability to efficiently decompress highly abstracted features to lower bit rates without significant computational cost. As such point cloud datasets continue to grow, efficient AI-based feature extractors for PCC can be used.

[0108] Qi, Charles R. et al., "Pointnet: Deep Learning on Point Set for 3D Classification and Segmentation," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652-660 (2017), describes the PointNet architecture for a point-based feature extractor. The PointNet architecture consists of a series of pointwise fully connected multi-layer perceptron (MLP) layers with specific feature dimensions, followed by pooling across all points. The PointNet architecture lacks feature details such as local geometric information.

[0109] The PointNet++ architecture, described by Qi, Charles R. et al., "Pointnet++: Deep Hierarchical Feature Learning on Point Set in a Metric Space," Advances in Neural Information Processing Systems 30 (2017), introduces a set abstraction layer within the architecture. This architecture utilizes local geometric information hierarchically, but each sampling process requires a sampling step, such as farthest point sampling, followed by a grouping function, such as a ball query. Additionally, each set abstraction layer requires the execution of a mini-PointNet, which can require some computational overhead.

[0110] The PointNet and PointNet++ methods extract features from given discrete point locations. To generalize this problem, points from 3D space are randomly sampled, and then a function is fitted to give the probability of point occupancy at any given coordinate in 3D space. In Mescheder, Lars, et al., "Occupancy Networks: Learning 3D Reconstruction in Function Space," Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 4460-4470 (2019), the occupancy network attempts to learn this non-discrete function with the help of a conditional batch normalization parameter fed into each corresponding step of the occupancy probability generator. To better extract surface details of points, Peng, Songyou, et al., "Convolutional Occupancy Networks," European Conference on Computer Vision, pp. 523-540 (2020), note that occupancy networks can be further improved by adding a convolutional layer, such as a U-Net, before the fully connected layer. Again, these methods can be overly complex and difficult to deploy in a PCC framework. Ma, Xu et al., "Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework," ARXIV PREPRINT arXiv:2202.07123 (2022) discuss the PointMLP architecture. Ran, Haoxi et al., "Surface Representation for Point Cloud," ARXIV PREPRINT arXiv:2205.05740 (2022) discuss the RepSurf architecture. These architectures are introduced to improve the ensemble abstraction layer. The PointMLP architecture introduces a hierarchical multi-stage architecture with an affine local geometry extractor. The RepSurf architecture adds local information, such as triangle and umbrella orientations, to point cloud data. All of these methods require costly multi-stage sampling and grouping processes.

[0111] The above approaches target point cloud classification or segmentation problems. They are not fully suited to the PCC framework due to the trade-off between accuracy and complexity. Furthermore, the optimization of the loss function formulated for PCC (rate and distortion) is different from that used for classification problems (cross-entropy).

[0112] Yan, Wei et al., Deep Autoencoder-Based Lossy Geometry Compression for Point Cloud, ARXIV PREPRINT arXiv:1905.03691 (2019) ("Yan") paper introduces an autoencoder-based PCC network. An MLP with an aggregation layer is abstracted for global features that are finally sent to an entropy encoder. On the decoder side, the decoded feature codes are input to an MLP decoder to reconstruct the point cloud. Based on the PointNet network, Yan applies an exemplary point feature extractor within the end-to-end PCC framework.

[0113] The '482 application, for example, describes the PointContextNet algorithm, while the '894 application, for example, describes the Coordinate Refinement Module (CRM) algorithm. Both the '482 application and the '894 application use their respective exemplary algorithms for deep entropy coding within end-to-end (AI-based) PCC networks, for example, applying a series of set abstraction (SA) layers through (costly) sampling and grouping layers with hierarchical point feature analysis.

[0114] The '869 application introduces a scalable PCC framework by, for example, implementing the feature extractor (in both the encoder and decoder) with a sparse convolutional network. For example, the feature extractor in this end-to-end framework processes local point group information by integrating K-nearest neighbor oriented analysis for each point. The microarchitecture used in some examples can be enhanced in the area of ​​feature extraction.

[0115] This application, according to some embodiments, introduces a set of point cloud (PC) feature processes that take 3D points or features as input and extract group-wise feature points at the encoder output of some embodiments. While such an architecture can be adapted to any point cloud feature extraction for various purposes, such as point cloud classification, point cloud model segmentation, point cloud flow analysis, autonomous driving applications, etc., this architecture is adapted to an AI-based point cloud compression (PCC) framework. Details of the overall process and architecture are first described according to some examples, and then how to apply the process in the PCC framework is described.

[0116] The point feature extractor described below according to some embodiments is applicable to all of the above exemplary PCC frameworks. Regarding the architecture of Yan and the '869 application, the architecture and processing of the present application may improve the representation of point features on both the encoder and decoder sides. Regarding the exemplary architectures of the '482 and '894 applications, the exemplary architecture and processing of the present application according to some embodiments may streamline the computational complexity of local geometric information.

[0117] FIG. 3 is a functional block diagram illustrating an example encoder architecture for point cloud (PC) feature extraction according to some embodiments. Features are multidimensional vectors that represent the output of neural network layers. In point cloud applications, these vectors are often extracted point-by-point via MLP networks. An overall example architecture 300 of the present application according to some embodiments is shown in FIG. 3. One version of the feature extractor is a single point-by-point block 306. A more advanced version queries for group points 304, extracts features for all points 306, aggregates by group point, and performs group-by-group feature extraction 318 to generate group features 320. These two versions of the example architecture may perform cost-effective feature extraction but may lack precision and detailed feature representation.

[0118] Features, such as per-point features or per-group features, may be represented as high-dimensional vectors. Features are tensors of 3D points that are embedded in a higher dimensional space. In some embodiments, a machine learning process that performs the embedding may be a necessary step. In some embodiments, features are sets of 3D points in a 3D point cloud model of an environment or scene. Features may be generated by a machine learning process in some embodiments. The feature tensors may be used as point cloud input 302 to point cloud query 304 in FIG. 3 in some embodiments. In some embodiments, feedback may be represented by a multi-stage architecture. The multi-stage architecture may be one of many examples of a learning-based point cloud compression (PCC) framework. In some embodiments, features are data that passes through a learning-based PCC framework.

[0119] This application focuses on more advanced feature aggregation for better (e.g., more accurate) representation of PC features. In some embodiments, a deep distribution-aware point feature extractor 306 may be used. In some embodiments, point features 308 may be input to a group feature distribution transformation. In some embodiments, the transformation is performed on the group feature distribution (310), and the transformed features are aggregated and matched (312). In addition, deep residual-based local features and global features may be combined (314, 316) to further improve feature extraction. The sequence of these processes 310, 312, 314, 316 may be more deeply coupled to further improve the quality of the representation. The dashed line within the aggregation / augmentation block 316 indicates a branch for additional output by the augmentation process. Augmentation is used when a new series continues through the feedback loop 316, 310. Further details about this branching are shown in FIG. 9.

[0120] In some embodiments, transformed feature aggregation matching (dashed process 312) is another layer that can be added to blend group features. In some embodiments, group feature distribution transformation process 310 can directly receive point features instead of embedded 3D points via pointwise feature extraction 306. In some embodiments, the point cloud query (block (a)) can use an optimal number of group points for better (e.g., more accurate) reconstruction quality. In some embodiments, the feedback path in FIG. 3 can go from the output of the group feature aggregation and augmentation block (316 in FIG. 3) to the input of the pointwise residual network (314 in FIG. 3). This is possible because both blocks 310, 314 in FIG. 3 take the same feature dimension input.

[0121] Group queries and analyses can classify the shape of a local neighborhood around a query location. Processing features by such local neighborhoods / groups allows exploiting the hierarchical structure of the point cloud and often performs more accurate feature extraction of PCs.

[0122] For PC classification or segmentation, downsampling may often be accompanied by a group query. This practice may be applied to a PCC framework, such as an end-to-end compression architecture, where PC quantization results in point downsampling. In some embodiments, any grouping algorithm, such as a ball query or a K-Nearest-Neighbors (KNN) algorithm, may also be applied. While a ball query extracts more precise geometric details, KNN may be efficient for a PCC framework and also a more straightforward process.

[0123] FIG. 4 is a functional block diagram illustrating an example of how a PC group size K may be determined for a given voxel quantization for PC reconstruction in a point cloud compression (PCC) framework according to some embodiments. As an example process 400 shown in FIG. 4, the group size K may be efficiently selected based on the quantization step and the density of occupied voxels 404 within the grid. The left grid 402 of FIG. 4 is the original voxel representation of a 4-bit PC model. The left side 402 of FIG. 4 shows the original (occupied) point cloud voxels before quantization. The right side 408 of FIG. 4 shows a quantized voxel with a query point 410. Both the occupied point 404 and the query point 410 are considered to reside at the center of each voxel. For example, by using a quantization 406 with a step size of 4, the original 4×4×4 voxel grid is quantized to 1×1×1 voxels, as shown in the right grid 408 of FIG. 4. Every occupied point (circle) in the original grid 402 is represented by a single voxel point (star) in the quantized grid 408. In some embodiments, quantization or voxel quantization 406 may be used to perform grouping, and features may be extracted from the resulting groups.

[0124] The average number of original points per quantized grid point is the size of the neighborhood processed for a given quantization step. In other words, the quantity K of the entire point cloud model avg is the quantized point

[0125]

number

[0126]

number

[0127]

number

[0128]

number

[0129]

number

[0130]

number

[0131] In some embodiments, multiple training models with different K values ​​may be pre-trained. pccan be formulated as Equation 2.

[0132]

number

[0133] K pc The trained network with the closest K compared to k may be used for inference. In this example context, inference refers to the reconstruction (which may include decoding) of a detailed point cloud from a quantized point cloud model. In some embodiments, a quantized point cloud is a simplified point cloud having fewer points than the original point cloud before quantization. In some embodiments, the quantized point cloud model may be generated using training data in a point cloud compression (PCC) framework. Inference may be performed on the training data.

[0134] FIG. 5 is a functional block diagram illustrating exemplary pointwise feature extraction according to some embodiments. The present method and process seeks to extract meaningful PC features through a neural network. The three-dimensional (N×K) locations 502, along with group points from each query point, are embedded into a desired feature dimension D, where N is the number of query points and K is the size of the group of output features F 508. Such a process 500 can be used as a connector for the following methods and processes. As shown in FIG. 5, the feature dimension gradually increases to the desired size and then connects pointwise to the next module. In some embodiments, the unit microarchitecture may combine a batch normalization (BN) layer between a fully connected (FC) layer and an activation function (ACT). The FC layer contains neurons that apply a linear transformation to the input vector via a weight matrix. A nonlinear transformation is then applied via ACT. The BN layer, typically located between the FC and ACT, normalizes the weights to improve training performance. The fully connected layers, batch normalization layers, and activation layers may be used to construct a convolutional neural network (CNN) or multi-layer perceptron (MLP) 504 architecture. Referring to Figures 3 and 5, the group feature output may have (N x K) features 506 for some embodiments.

[0135] Utilizing local groups of point clouds can be used for point-based feature extraction. However, hierarchical approaches using set abstraction (SA) layers use sampling functions such as farthest point sampling (FPS), which may be indistinguishable depending on the intended use. Furthermore, for each SA layer, both grouping and a separate pass of the PointNet network may be required, which is costly for the PCC framework. To avoid potential degradations in accuracy and cost-effectiveness, a fully distinguishable and unique method can be created that emphasizes the shape of local geometric information. In this sense, grouping methods can analyze the group distribution of each feature dimension to better distinguish between local groups in an efficient manner.

[0136] In some embodiments, distribution-aware features within the PCC framework represent how groups of points surrounding a quantized point are processed or interpreted. In some embodiments, quantization may be performed from the original voxel representation and / or the quantized voxel representation. Such distribution-aware features may be used to reconstruct a point cloud. In some embodiments, the point cloud compression (PCC) framework may be expressed as a learning-based point cloud geometry or an artificial intelligence (AI)-based point cloud compression framework. In some embodiments, the geometry processing block may be a processing block within the learning-based PCC framework.

[0137] 6 is a functional block diagram illustrating an example group feature distribution tensor according to some embodiments. The detailed calculation process 600 of the group distribution is calculated as follows: As shown in FIG. 6, the feature tensor F 608 per input point has N groups 604 of K points 602 per group multiplied by the dimension D of the feature size. In some embodiments, the items F in the items 602 of FIG. 6 11 is an example of a feature vector. In some embodiments, the feature vector may be row-oriented or column-oriented. The length of the feature vector is the norm value calculated over the elements of the feature vector. This point feature F(F P) is the pointwise row feature [F 11 ,F 12 ,...,F jk ,...,F NK ] T For each group j, the mean feature value is calculated by Equation 3:

[0138]

number

[0139]

number

[0140] In some embodiments, a feature map can be a set of channels (e.g., multiple C channels). The columns of items 616 and 626 in FIG. 6 are examples of feature maps with multiple D channels (e.g., D=C in this example). In some embodiments, an example of a reshaped vector is the f i and f' i In some embodiments, the length of the reshaped feature vector is a norm value calculated over the elements of the reshaped feature vector. In some embodiments, the result of updating each of the series of vector elements by subtracting their respective means is the ΔFjk In FIG. 6, the term "calculate feature distribution" divides the elements by their respective standard deviations.

[0141] FIG. 7 is a functional block diagram illustrating an exemplary group feature distribution with learnable transformation parameters according to some embodiments. In some embodiments, γ (gamma) and β (beta) are learned from training. In some embodiments, the distribution parameters may include transformation parameters or transform parameters. In some embodiments, the distribution parameters may be predetermined, for example, by neural network training. In some embodiments, the distribution parameters, of which γ (gamma) and β (beta) are examples, may be determined using an exemplary backpropagation technique during a training period. Such a training period may be for training a neural network. In some embodiments, the group distribution may be constructed by a process 700 (see FIG. 7) on a feature tensor F′ 702, 704, where each column of the feature is represented by a one-dimensional vector f′ i 706, 708, 710. Each column forms a bell curve or query point centered around the group mean. These bell curves represent the group distributions for each feature dimension (D). Feature map F' 704 in FIG. 7 is an example of a feature map with a D channel. In some embodiments, these group distributions 712, 714, 716 can be individually transformed via transformation parameters γ and β so that the feature representation can further distinguish the shape of the corresponding feature dimension during point cloud (PC) reconstruction. In some embodiments, γ and β are examples of distribution parameters. In some embodiments, the term distribution parameters can be considered as transformation parameters or transform parameters. As shown in FIG. 7 and in Equation 5 below, f' i is a one-dimensional feature f i is the (final) distribution of

[0142]

number

[0143]

number

[0144] In some embodiments, the transformation parameters γ and β can be separated not only by feature dimension but also by group, which can further emphasize local distributions at little additional cost. In this case, the number of parameters increases from D to (K×D), where D is the size of the feature dimension and K is the size of the group of output features F. In some embodiments, a density factor can be introduced to further weight the importance of each transformation parameter in the tensor. In some embodiments, slices can be divided into groups. One-dimensional feature elements are f i These elements f i may be further divided by local group elements. If the point cloud has N local groups, the f i terms may be further divided into N groups of elements. In some embodiments, the feature map may be a concatenation of feature elements per slice, which may be computed by normalizing corresponding slices from another feature map.

[0145] In point-by-point features, each point is a member of a group. For each group member point, a mini-distribution can be calculated over all group points centered around the group mean point. These processes are repeated for all points in the point cloud scene. In some embodiments, this process can generate updated point-by-point features that contain floating-point numbers in matrix form. This process can be separated by feature dimension (which can be matrix columns or channels). For each set of point cloud data, the distribution of channel elements can be varied, for example, into range, amplitude, or mean. In some embodiments, these distributions can be computed by learnable transformation parameters during a training process. These distribution curves can be transformed to allow the feature extractor to better distinguish or emphasize each point feature.

[0146] FIG. 8 is a functional block diagram illustrating exemplary transformed feature aggregation, augmentation, and dimension matching, according to some embodiments. As shown in the exemplary process 800 of FIG. 8, global and local representations may be mixed, and further augmentation may be performed on features F′ 802. Local features may be aggregated, and then the augmented tensor 804 may be augmented with global features. In some embodiments, dimension matching 810 may be performed, and updated features F″ 808 are output. The dimensions F′ and F″ are identical. Thus, the augmented features may be used with group feature distribution transformations and pointwise residual networks (310 and 314, respectively, in FIG. 3). In some embodiments, the term pooling refers to a function that aggregates multiple point features into one point feature by averaging or taking the maximum value within the features. In FIG. 8, the aggregated feature F′ 804 is the result of a pooling operation. In some embodiments, the pooling operation may be a function of, for example, average pooling or max pooling. In some embodiments, a smoothing filter may be applied to the feature map. For example, the expanded feature F' in Figure 8 AE is an example of the output of such a smoothing filter.

[0147] An efficient analysis of fully distinct local groups can be obtained. As shown in FIG. 8, the output features F' can be used independently in some embodiments, or in some embodiments, aggregated and expanded features (F') per group can be used. AE ) may be concatenated 806. A matching layer may follow that matches the feature dimensions to the input size. In some embodiments, the output 812 of the matching block 810 may reflect a transformation of the dimensions of the concatenated output, which is (N×K)×(2D), to the dimensions of the input, which is (N×K)×D.

[0148] The PointNet architecture is a cost-effective PC feature extractor. Due to its simplicity, the PointNet architecture can be used in a PCC framework. However, in some applications, the PointNet architecture lacks sufficient detail for the PCC framework, particularly for lower bitrate compression. To overcome this potential problem, according to some embodiments, a deeper network with a combination of global and local features across the network can be used. A residual network is a specific network that learns, for example, residuals. A residual network can be used to design a deeper neural network. In some contexts, a deeper network can be used as a more general term compared to a residual network.

[0149] FIG. 9 is a functional block diagram illustrating exemplary pointwise residual feature extraction and groupwise feature aggregation and augmentation according to some embodiments. As shown in FIG. 9, there is an architecture 900 that can be used with deeper networks. To avoid degradation in deeper networks, a ResNet-like design is used. In some embodiments, the residual network block 922, such as the example shown in FIG. 9, can be based on a ResNet architecture. He, Kaiming, Deep Residual Learning for Image Recognition, ARXIV PREPRINT arXiv:1512.03385 (2015) (“He”), describes an exemplary ResNet architecture. (See, e.g., FIG. 3, p. 4, right.) In the exemplary modified implementation shown in FIG. 9, according to some embodiments, the image input of the ResNet architecture is replaced with a point cloud input 902. The convolutional layers of the ResNet architecture are replaced with fully connected (FC) layers. To match the feature dimensions of the input and output, the input to the fully connected layer, which may be pointwise features 904, is downscaled. Therefore, the input to the lower FC layer 916 is downscaled. In some embodiments, the input to the upper second FC layer 912 (closer to the center of FIG. 9) is downscaled. In some embodiments, the downscaling is a 2x downscaling for both the inputs to the upper and lower FC layers. Such a 2x downscaling may be performed to counteract the augmentation performed in the AUG block 926. In some embodiments, the input to the upper first FC layer 906 (closer to the left side of FIG. 9) is downscaled. Both global and local features are preserved by the "aggregate-then-augment" process. Unlike image inputs, 3D point cloud structures are irregular. Therefore, convolutional layers are replaced with fully connected (FC) layers. FC layers link the output of a preceding processing block to the input of a subsequent processing block. In some embodiments, FC layers output a linear transformation of the input.Each FC layer 906, 912, 916 is followed by a batch normalization (BN) 908, 914, 918 layer and an activation function (ACT) 910, 920. In some embodiments, the BN layer may be used to train the network and help it converge faster. The activation (ACT) function may be, for example, a rectifier linear unit (ReLU) function, in which negative values ​​are replaced with zero. In some embodiments, the BN layer may be omitted, such that the output of the FC layer is fed to the ACT layer. To connect some of these processes sequentially and make them compatible with the distribution transformation tensor introduced previously, the shape of the input may match the output tensor. The feature dimension is downscaled (divided by 2) for the FC layer in both the residual path (top path in FIG. 9 ) and the shortcut path (bottom path in FIG. 9 ). The outputs of both the top and bottom paths are displayed with a plus sign.

[0150]

number

[0151]

number

[0152]

number

[0153] So far, pointwise features have been processed. Each group is aggregated (AGG) 924 to create a per-group feature 928. In some embodiments, the per-group feature is output (932). In some embodiments, the successive blocks may be repeated with the output of the combined expansion (EXP) 930 and augmentation (AUG) 926 processes. During augmentation, both the output features of the residual network (before the AGG block) and the expanded per-group features (output of the EXP block) are concatenated. This concatenated feature is first sent to repeat the successive blocks. In some embodiments, the aggregation (AGG) block in FIG. 9 is similar to the aggregation process in FIG. 8. In some embodiments, the expansion (EXP) block in FIG. 9 is similar to the expansion process in FIG. 8. According to some embodiments, the augmentation (AUG) block in FIG. 9 is similar to the concatenation process in FIG. 8.

[0154] In some embodiments, the architecture shown in FIG. 9 may be inserted into the architecture of FIG. 3 where the residual network block of FIG. 9 is used for the pointwise network (block (e) of FIG. 3). The AGG, AUG, and EXP blocks of FIG. 9 are used for the group feature aggregation and augmentation block (block (f) of FIG. 3). In some embodiments, the feedback path of FIG. 3 may go from the output of the group feature aggregation and augmentation block (block (f) of FIG. 3) to the input of the pointwise residual network (block (e) of FIG. 3). Such a configuration of the feedback path of FIG. 3 may be used when the architecture shown in FIG. 9 is inserted into the architecture of FIG. 3, and the serially concatenated feedback path of FIG. 9 is the feedback path of FIG. 3.

[0155] In some embodiments, this deep residual processing can advantageously work in conjunction with distribution-aware processing. In some embodiments, the distribution-aware processing can be blocks 310 and 312 of FIG. 3. In some embodiments, the distribution-aware processing can be block 310 of FIG. 3. In some embodiments, the deep residual processing can be blocks 314 and 316 of FIG. 3. At each stage, features of both processes can be connected either serially (see FIG. 3) or in parallel (see FIG. 10). Rich representations are propagated through the deep network. In other words, deep extraction processes can be connected sequentially without degradation while preserving local geometric information. Furthermore, maintaining in / out dimension compatibility facilitates creating a variety of designs for different purposes.

[0156] Figure 10 is a functional block diagram illustrating an example encoder architecture for PC feature extraction according to some embodiments. While the overall architecture assembles several processes in series, some embodiments may use a parallel architecture 1000, as shown in Figure 10. In some embodiments, a point cloud 1002 is input to a point cloud query 1004 that outputs to point-wise feature extraction 1006.

[0157] Compared to FIG. 3 , one difference is that the distribution transformations 1008, 1010 in FIG. 10 use independent parallel paths (path 1 in 1008, 1010, and path 2 in 1012, 1014), and then augment these outputs with features from the residual module (1016). The dimensions of the output 1016 are later matched with a microarchitecture (matching) similar to that introduced in FIG. 8 to generate group features 1020. To finish the deep stage, a final aggregation (below the dashed line in block 1016) is performed, followed by group-wise feature extraction 1018. In some embodiments, the group size K may be 1, in which case the group features may be referred to as point features.

[0158] FIG. 11 is a functional block diagram illustrating an example decoder architecture according to some embodiments. In an end-to-end PCC framework, a decoder is used in conjunction with an encoder. In some embodiments, another architecture 1100 for the decoder is applicable, as shown in FIG. 11. In some embodiments, a first pointwise extractor may increase the feature dimension. Some of the processing described above may occur between the first pointwise extractor and the second pointwise extractor. The second pointwise extractor may reduce the feature dimension to 3D to reconstruct the final decompressed point cloud. In some embodiments, there is no aggregation / grouping in the decoder, and the dimensionality of the decoder input data 1102 is N groups with feature dimension D (N×D).

[0159] In some embodiments, a decoder architecture may obtain group feature data. The decoder process may extract group-wise features from the group feature data (1104). The size of the groups may then be matched / expanded to the size of the points (1106). The expanded point-wise features may be sent to a group feature distribution transformation process 1108, which may perform a transformation on the group feature distribution. The transformed features may be input to a point-wise residual network 1110, which may generate residual-based feature data. The decoder process may perform point-wise feature extraction 1112 to extract a reconstructed point cloud 1114.

[0160] PC feature extraction in AI-based PCC architectures is a relatively new area compared to PC classification or segmentation. In Yan and the '869 application, the PC feature extractor defines, for example, an MLP layer followed by an aggregation step. While the PCC framework calls for a cost-effective architecture, there is still room for better representation of PC features. In some embodiments, the feature extractors described above can be used in the PCC framework, and compression performance can be improved. The enhanced feature extractor can be used for other tasks, such as segmentation and point cloud classification.

[0161] 12A is a functional block diagram illustrating an example application of the PCC framework to an encoder and a decoder in autoencoder-based lossy geometry compression. As shown in FIG. 12A, Yan uses an end-to-end PCC architecture 1200. The encoder 1204 and decoder 1220 are designed using MLPs 1206, 1222 and a pooling layer 1208.

[0162] Yan discusses four modules: a PointNet-based encoder, a uniform quantizer, an entropy estimation block, and a nonlinear synthesis transform module. Yan uses an autoencoder as the compression platform. As stated in the first column of page 4323 of Yan: First, the input point cloud is downsampled by a sampling layer S to create point clouds with different point densities. The downsampled point set then passes through an autoencoder-based codec. The codec consists of an encoder E that takes the unordered point set as input and produces a compressed representation, a quantizer Q, and a decoder D that takes the quantized representation produced by Q and produces a reconstructed point cloud.

[0163] As shown in Figure 12A, the downsampled point cloud is used as input points 1202. These points serve as a multi-layer set of points that undergo (max) pooling as part of the encoding process. The encoder output, a latent code 1210, is sent through an entropy encoder 1212 to generate a global representation, a bitstream 1214 shown on the right side of Figure 12A between the entropy encoder 1212 and the entropy decoder 1216. The global representation is passed through the entropy decoder 1216 to generate quantized code 1218. The quantized code 1218 is passed through a decoder to reconstruct a multi-layer set of points, the output point cloud 1224.

[0164] Figure 12B is a functional block diagram illustrating an example application of the PCC framework to an encoder and decoder in autoencoder-based lossy geometry compression, according to some embodiments. In some embodiments of an encoder / decoder architecture 1250, the left encoder and decoder of Figure 12A may be replaced by the architecture of Figure 3 or Figure 10, as shown in Figure 12B. For example, in some embodiments, the encoder of Figure 12A may be replaced by a distribution-aware process 1254, a deep residual process 1256, and a feedback path, as shown in Figure 12B. Similarly, in some embodiments, the decoder of Figure 12A may be replaced by a distribution-aware process 1268, a deep residual process 1270, and a feedback path, as shown in Figure 12B. As a result, higher-quality latent codes 1258 and higher-quality quantized codes 1266 may be generated. Such codes may contain more representative PC features with less increased computational cost. In some embodiments, the encoder of Figure 12A is replaced by the encoder architecture of Figure 10. For input point 1252 of Figure 12B, the input point of Figure 12A is replaced by the point cloud input to the point cloud query (block 1004 of Figure 10). The group feature output of group-wise feature extraction (block 1018 of Figure 10) is the latent code of Figure 12A. In some embodiments, latent code 1258 is input to entropy encoder 1260, which outputs bitstream 1262. In some embodiments, bitstream 1262 is input to entropy decoder 1264, which outputs quantized code 1266. In some embodiments, the decoder of Figure 12A is replaced by the decoder architecture of Figure 11, the quantized code of Figure 12A is the group feature input to group-wise feature extraction (block 1104 of Figure 11), and for output point 1272 of Figure 12B, the reconstructed point cloud output of point-wise feature extraction (block 1112 of Figure 11) replaces the output point of Figure 12A.

[0165] Another example from the '869 application is shown in FIG. 13A. FIG. 13A is a functional block diagram illustrating an exemplary application of the PCC framework to point analysis and point composition within a scalable PCC framework. The '869 application introduces a scalable end-to-end PCC framework. This application focuses on feature analysis and composition, rather than point analysis and composition. Again, in this framework, the MLP and aggregation layers are combined in both the "res-to-feature converter" (point analysis in the encoder) block and the "feature-to-res converter" (in the decoder) block. In some embodiments, the geometry processing block within the trend-based PCC framework may be one or more of the blocks in FIG. 13A, such as, for example, blocks 1304, 1308, 1310, 1312, 1316, 1318, 1320, and / or 1322.

[0166] The '869 application discusses a lossy point cloud compression scheme that uses deep neural networks to encode point cloud geometry. In such a scheme 1300, a coarse version of an input point cloud 1302 is encoded 1304 as a first bitstream 1306 (BS0 in Figure 5 of the '869 application), and residual data (fine geometric details) is encoded 1312 as per-point features in a second bitstream 1314 (BS1 in Figure 5 of the '869 application). The residual data may be generated by point analysis 1308 and feature analysis 1310.

[0167] On the decoding side, the coarse point cloud (PC1 in Figure 14 of the '869 Application) is decoded 1316 from the first bitstream (BS0 in Figure 14 of the '869 Application). Residual data (R' in Figure 14 of the '869 Application) is decoded 1318 from the point-wise features (F' in Figure 14 of the '869 Application) and added 1320, 1322 to the coarse point cloud to retrieve a decoded version 1324 of the original input point cloud (PC0 in Figure 14 of the '869 Application).

[0168] FIG. 13B is a functional block diagram illustrating an exemplary application of the PCC framework to point analysis and point synthesis within a scalable PCC framework, according to some embodiments. In some embodiments, input points 1352 may be input to an octree encoder 1354 and a distribution-aware process 1358. The octree encoder 1354 outputs a base bitstream 1356 that is input to an octree decoder 1368. In end-to-end PCC compression 1350, the selection of the group size K may affect the level of compression enhancement and performance. Furthermore, a high-quality representation of PC features may improve reconstruction performance, especially at lower bit rates. For example, the point analysis process of FIG. 13A may be replaced by the architecture proposed in FIG. 3 or FIG. 10, with additional flexibility to add or remove processes, such as block 312 of FIG. 3 and block 1010 of FIG. 10. In some embodiments, the point synthesis process of FIG. 13A may be replaced with the architecture proposed in FIG. 11. For example, in some embodiments, the point analysis process of Figure 13A may be replaced by a distribution-aware process 1358, a deep residual process 1360, and a feedback path, as shown in Figure 13B. Similarly, in some embodiments, the point synthesis process of Figure 13A may be replaced by a distribution-aware process 1374, a deep residual process 1376, and a feedback path, as shown in Figure 13B. In some embodiments, the point analysis of Figure 13A is replaced with the encoder architecture of Figure 10, where the input points of Figure 13A are the point cloud input to the point cloud query (block 1004 of Figure 10), and the group feature output of the group-wise feature extraction (block 1018 of Figure 10) are the input to the feature analysis of Figure 13A. In some embodiments, the point synthesis of FIG. 13A is replaced by the decoder architecture of FIG. 11, with the feature synthesis output of FIG. 13A being the group feature input to group-wise feature extraction (block 1104 of FIG. 11), and the reconstructed point cloud output of point-wise feature extraction (block 1112 of FIG. 11) being the output points of FIG. 13A.

[0169] In some embodiments, the deep residual process 1360 outputs to a feature analysis process 1362 which in turn outputs to an entropy encoder 1364. The output of the entropy encoder is an enhanced bitstream 1366, which in turn is the input to an entropy decoder 1370. The entropy decoder 1370 outputs to a feature synthesis 1372. In some embodiments, the feature synthesis 1372 receives inputs from the octree decoder 1368 and the entropy decoder 137 and outputs to a distribution recognition process 1374. The output of the deep residual process 1376 is a set of output points 1378.

[0170] Figure 14A is a functional block diagram illustrating an example application of the PCC framework to set abstraction (SA) processing in the PointContextNet environment. Figure 14A shows an example method 1400 from the '482 application that uses an AI-based octree-structured entropy model.

[0171] The '482 application discusses searching a compressed point cloud based on a tree structure and searching for points near nodes in the tree structure. Two features 1406 are calculated 1404 from the search data and their locations 1402. The '482 application blends these two features with one or more known features of the node, and finally determines the occupancy 1408 of the current node from the coded bitstream and a predicted occupancy symbol distribution.

[0172] FIG. 14B is a functional block diagram illustrating an example application of the PCC framework to set abstraction (SA) processing in a PointContextNet environment, according to some embodiments. The architectures of FIG. 3 or FIG. 10 may be replaced with the SA processing / modules of FIG. 14A. For example, in some embodiments, the SA processing 1404 of FIG. 14A may be replaced by a distribution recognition processing 1454, a deep residual processing 1456, and a feedback path, as shown in FIG. 14B. In some embodiments, the set abstraction (SA) processing 1404 of FIG. 14A is replaced with the encoder architecture of FIG. 10, where the output of the input point context block 1402 of FIG. 14A is the point cloud input to the point cloud query (1004 of FIG. 10), and the group feature output of the group-wise feature extraction (1018 of FIG. 10) is the SA features 1406 of FIG. 14A.

[0173] In some embodiments, the exemplary process 1450 may input point context 1452 into a distribution recognition process 1454. The output of the deep residual process 1456 may be SA features 1458, which may in turn be input to occupancy probability prediction 1460.

[0174] The architecture extracts cost-effective features with high-level representations via feature distribution transformation and deep residual architecture within the octree entropy model of the PCC framework. In addition to the SA layer, some of the processes mentioned above, such as processes 310, 312, 314, 316, and 318 in Figure 3, can be used instead of the FC layer in both the PointContextNet and CRM architectures.

[0175] Figure 15A is a functional block diagram illustrating an example application of the PCC framework to set abstraction (SA) processing in a coordinate refinement module (CRM). The '894 application discusses coordinate refinement and upsampling of quantized and reconstructed point cloud data. Neighboring points of the decoded point cloud may be determined by an AI-based coordinate refinement process 1500. Based on characteristics of one of those neighboring points, refinement features may be determined using neural network techniques. The refinement features may be used to predict 1508 the refinement of the decoded point cloud.

[0176] FIG. 15B is a functional block diagram illustrating an example application of the PCC framework to set abstraction (SA) processing in a coordinate refinement module (CRM), according to some embodiments. The architecture of FIG. 3 or FIG. 10 may be replaced with the SA processing / module of FIG. 15A. For example, in some embodiments, the SA processing 1506 of FIG. 15A may be replaced by a distribution recognition processing 1554, a deep residual processing 1556, and a feedback path, as shown in FIG. 15B. In some embodiments, the set abstraction (SA) processing 1504 of FIG. 15A is replaced with the encoder architecture of FIG. 10, where the output of the input point context block 1502 of FIG. 15A is the point cloud input to the point cloud query (1004 of FIG. 10), and the group feature output of the group-wise feature extraction (1018 of FIG. 10) is the SA features of FIG. 15A.

[0177] In some embodiments of process 1550, input point context 1552 is input to distribution recognition process 1554. In some embodiments, the output of deep residual process 1556 is SA features 1558, which may in turn be input to offset prediction 1560.

[0178] Compared to the method of Yan and the '869 application, the methods of the '482 and '894 applications apply a more advanced feature extractor with a set abstraction (SA) module and further enhancements with multi-resolution (MRG) or multi-scale (MSG) grouping. The SA layer is followed by a series of FC layers. While these architectures can extract good representative PC features in a hierarchical and multi-level approach, they may have high computational costs that may be less favorable in some implementations.

[0179] This application describes a deep distribution-aware point feature extractor for point cloud data. This architecture can be used in an AI-based point cloud compression (PCC) framework as well as other architectures. The PCC framework uses a balanced tradeoff between reconstruction accuracy and computational cost. To facilitate such a goal, a per-channel feature distribution transformation process can be used. A deep residual-based network with an iterative mixture of global and local information can be used for further enhancement. Two of the previously described processes can be used in both the encoder and decoder of a given PCC framework.

[0180] In some embodiments, the inputs and outputs of the feature extractors described herein may be pointwise features. Thus, the feature extractors described herein may be plugged into the frameworks shown in Figures 12A, 13A, 14A, and 15A, for example, as shown in Figures 12B, 13B, 14B, and 15B, respectively.

[0181] FIG. 16 is a flowchart illustrating an exemplary process for point cloud feature extraction, according to some embodiments. In some embodiments, the exemplary process 1600 may include querying local point groups for each point in the point cloud having a selected group size (1602). In some embodiments, the exemplary process may further include extracting first per-point features (1604). In some embodiments, the exemplary process may further include performing a first pass and a second pass through a feedback process (1606). In some embodiments, the exemplary feedback process of the exemplary process may include converting the first per-point features into second per-point per-channel distribution features based on a set of transformation parameters (1608). In some embodiments, the exemplary feedback process of the exemplary process may further include extracting third per-point features from the second per-point features through a deep network (1610). In some embodiments, the exemplary feedback process of the exemplary process may further include aggregating the third per-point features based on the local point groups to form first per-group features (1612). In some embodiments, the exemplary feedback process of the exemplary process may further include enhancing the third pointwise features with an expanded version of the first groupwise features (1614). In some embodiments, the exemplary feedback process of the exemplary process may further include obtaining the next stage first pointwise features (1616) via the feedback process for use as the first pointwise features for the second pass.

[0182] Deep feature coding for PCC Learning-based PCC approaches can be divided into two main groups: deep octree-based PCC and deep feature-based PCC. In octree-based PCC, voxel occupancies are entropy coded directly into the bitstream. Learning-based methods are used to predict the probabilities of these voxel occupancies. In deep feature-based PCC, geometric features are quantized and compressed in an end-to-end manner.

[0183] Figure 17 is a functional block diagram illustrating a deep feature-based PCC pipeline, according to some embodiments. The dashed gray arrows in Figure 17 indicate the data flow of a general deep feature-based PCC framework 1700. Encoder E1 1704 extracts intermediate features from the input point cloud X 1702. Encoder E2 1708 further compresses the intermediate features to optimize compression with entropy encoder E3 1710. At the decoder, entropy decoder D3 1714 decodes the coded features from the bitstream 1712. Decoder D2 1716 processes the intermediate features, and decoder D1 1720 outputs the final decoded point cloud X.

[0184]

number

[0185] To extract PCC-specific features, a deep distribution-aware network (DDA-Net) can be used in combination with a general deep feature-based PCC pipeline, as shown in Figure 17. DDA-Net blocks 1706 and 1718 are inserted between the E1 and E2 encoders on the encoder side of the bitstream, and between the D1 and D2 decoders on the decoder side of the bitstream. The DDA-Net blocks take the intermediate features generated by the E1 encoder (or D2 decoder) as input and manipulate their distribution to further differentiate them for point cloud compression (PCC). The modified features are input to the E2 encoder (or D1 decoder).

[0186] The DDA-Net block can be applied to GRASP-Net, a PCC framework that combines both point-based and voxel-based architectures. GRASP-Net is described in Pang, J. et al., GRASP-Net: Geometric Residual Analysis and Synthesis for Point Cloud Compression, Proceedings of the 1st International Workshop on Advances in Point Cloud Compression, Processing, and Analysis, pp. 11-19 (2022) ("Pang"). In particular, the encoders E1 and E2 correspond to PointNet (point analysis in Pang) and CNN-based downsampling (feature analysis in Pang), respectively. On the decoder, D1 and D2 correspond to MLP layers (point synthesis in Pang) and CNN-based upsampling (feature synthesis in Pang). Input / Output Pairs

[0187]

number

[0188] Distribution-aware feature manipulation Pointwise features, where l is the point index and D is the feature dimension.

[0189]

number

[0190]

number

[0191]

number

[0192] Standardization Process

[0193]

number

[0194] DDA-Net Architecture 18 is a functional block diagram illustrating a DDA-Net encoder architecture, according to some embodiments. The deep distributional neural network (DDA-Net) uses the feature map F in Equation 7. p Enter 1802, as shown in Figure 18,

[0195]

number

[0196] Initial feature map F p is the feature with the modified distribution

[0197]

number

[0198]

number

[0199] Probability Distribution Estimation A local group of points surrounding each query point can be collected to gather local geometric information. The input "point features" are shown in Equation 8:

[0200]

number

[0201] Figure 19 is a functional block diagram illustrating a probability estimator distribution network, according to some embodiments. As shown on the left side of Figure 19, local grouping and NK-point feature embedding follow the process of the GRASP-Net architecture described in Pang. In the example PCC framework 1900 of Figure 19, these local grouping and NK-point features are the input 1902 of the E1 encoder.

[0202] In probability distribution estimation, F p is embedded into D / 2 dimensions by the shared MLP 1904 to obtain F. This feature F is then converted into point features

[0203]

number

[0204]

number

[0205] Each feature row F jk For , the corresponding group mean

[0206]

number

[0207]

number

[0208]

number

[0209]

number

[0210] For each standardized i-th column, f' i The distributions of the groups form a bell curve centered on the group mean or query point. These group distributions are then transformed into learnable parameters

[0211]

number

[0212]

number

[0213] The pooling operation is processed for each group j, resulting in the group feature of Equation 13, shown in FIG.

[0214]

number

[0215]

number

[0216]

number

[0217] Residual Network In parallel with the probability distribution estimation, a pointwise residual network can be generated in stages. See Figure 9 for an example. Based on He's ResNet architecture, the residual network is designed to fit the PCC framework. The convolutional layers are replaced with fully connected (FC) layers to adapt the network to the point cloud input. As shown in Figure 9, the feature dimension D can be reduced by half by the FC layers. Similar to the probability distribution estimation process, the residual network also generates branched outputs.

[0218]

number

[0219]

number

[0220]

number

[0221] FIG. 20 is a flowchart illustrating an exemplary process for point cloud feature extraction, according to some embodiments. In some embodiments, the exemplary learning-based point cloud geometry encoder process may include accessing a first feature map, where the first feature map is an input to the encoder, the first feature map being generated by a first set of neural network layers. In some embodiments, the exemplary process may further include normalizing elements of the first feature map to generate a second feature map. In some embodiments, the exemplary process may further include accessing a set of distribution parameters. In some embodiments, the exemplary process may further include transforming the second feature map into a third feature map based on the set of distribution parameters. In some embodiments, the exemplary process may further include aggregating the third feature map into a fourth feature map.

[0222] FIG. 21 is a flowchart illustrating an exemplary process for point cloud feature extraction, according to some embodiments. In some embodiments, a further exemplary learning-based point cloud geometry encoder process may include accessing a first feature map, where the first feature map is an input to the encoder, the first feature map being generated by a first set of neural network layers. In some embodiments, the further exemplary process may further include normalizing elements of the first feature map to generate a second feature map. In some embodiments, the further exemplary process may further include accessing a set of distribution parameters. In some embodiments, the further exemplary process may further include transforming the second feature map into a third feature map based on the set of distribution parameters. In some embodiments, the further exemplary process may further include aggregating the third feature map into a fourth feature map. In some embodiments, the further exemplary process may further include expanding the fourth feature map to the size of the third feature map. In some embodiments, the further exemplary process may further include augmenting the expanded feature map with the third feature map. In some embodiments, further exemplary processing may further include repeating the learning-based point cloud geometry encoder one or more times, where the enhanced feature map is used as the first feature map and the next set of distribution parameters is used as the distribution parameters.

[0223] 22 is a flowchart illustrating an exemplary learning-based point cloud geometry process according to some embodiments. Some embodiments of the exemplary process 2200 may include accessing a first feature map (2202), where the first feature map has a number of C channels and is an input to a processing block, the first feature map being generated by a first set of neural network layers. In some embodiments, the exemplary process may further include accessing a set of distribution parameters (2204). In some embodiments, the exemplary process may further include converting the first feature map to a second feature map (2206) based on the set of distribution parameters. In some embodiments, the exemplary process may further include encoding the second feature map into a bitstream (2208).

[0224] 23 is a flowchart illustrating an exemplary learning-based point cloud geometry process according to some embodiments. Some embodiments of the exemplary process 2300 may include decoding a first feature map from a bitstream (2302). In some embodiments, the exemplary process may further include accessing a set of distribution parameters (2304). In some embodiments, the exemplary process may further include converting the first feature map to a second feature map based on the set of distribution parameters (2306). In some embodiments, the exemplary process may further include reconstructing a point cloud from the second feature map (2308).

[0225] Although methods and systems according to some embodiments are generally described in the context of extended reality (XR), some embodiments may be applied to any XR context, such as, for example, a virtual reality (VR) / mixed reality (MR) / augmented reality (AR) context, etc. Also, while the term "head-mounted display (HMD)" is used herein according to some embodiments, some embodiments may be applied to, for example, a wearable device (which may or may not be head-mounted) capable of XR, VR, AR, and / or MR in some embodiments.

[0226] An exemplary learning-based point cloud geometry processing block method according to some embodiments may include accessing a first feature map, the first feature map having multiple C channels and being an input to a processing block, the first feature map being generated by a first set of neural network layers; accessing a set of distribution parameters; converting from the first feature map to a second feature map based on the set of distribution parameters; and encoding the second feature map into a bitstream.

[0227] Some embodiments of the example learning-based point cloud geometry coding block method may further include updating the first feature map by normalizing elements of the first feature map.

[0228] In some embodiments of the exemplary learning-based point cloud geometry encoding block method, normalizing the elements of the first feature map may include determining a respective length of each feature vector associated with one of the elements of the first feature map and dividing each element of the first feature map by the respective length.

[0229] In some embodiments of the exemplary learning-based point cloud geometry encoding block method, normalizing the elements of the first feature map may include arranging each of the one or more vectors by reshaping an associated feature channel in the first feature map; determining a respective length of each of the one or more vectors for each feature channel; and dividing each vector element of each vector by the respective length for each feature channel, where each vector element is one of the elements of the first feature map.

[0230] In some embodiments of the exemplary learning-based point cloud geometry encoding block method, normalizing the elements of the first feature map may include: arranging each of the one or more vectors by reshaping an associated feature channel in the first feature map; determining a respective standard deviation of each of the one or more vectors for each feature channel; determining a respective mean of each of the one or more vectors for each feature channel; updating each vector element of each vector by subtracting the respective mean; and dividing each updated vector element of each vector by the respective standard deviation for each feature channel, wherein each vector element is one of the elements of the first feature map.

[0231] In some embodiments of the exemplary learning-based point cloud geometry coding block method, the set of distribution parameters is determined using back-propagation techniques during a training period.

[0232] In some embodiments of the exemplary learning-based point cloud geometry encoding block method, a set of distribution parameters is determined for each feature channel.

[0233] Some embodiments of the exemplary learning-based point cloud geometry coding block method may further include updating the second feature map by performing downsampling using an average pooling or max pooling function.

[0234] Some embodiments of the exemplary learning-based point cloud geometry coding block method may further include determining a third feature map by filtering the second feature map with a smoothing filter, and updating the second feature map by concatenating the third feature map to the second feature map.

[0235] In some embodiments of the example learning-based point cloud geometry encoding block method, performing processing prior to encoding the second feature map may include accessing a second set of distribution parameters and updating the second feature map by transforming the second feature map based on the second set of distribution parameters.

[0236] In some embodiments of the exemplary learning-based point cloud geometry encoding block method, performing processing prior to encoding the second feature map may include aggregating the second feature map using a second neural network.

[0237] In some embodiments of the exemplary learning-based point cloud geometry coding block method, the second neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

[0238] In some embodiments of the exemplary learning-based point cloud geometry coding block method, aggregating the second feature map may include using a residual network (ResNet) architecture.

[0239] Some embodiments of the exemplary learning-based point cloud geometry encoding block method may further include, in parallel with converting the first feature map to the second feature map, determining a fourth feature map by aggregating the first feature maps using a neural network, and updating the second feature map by concatenating the fourth feature map to the second feature map.

[0240] In some embodiments of the exemplary learning-based point cloud geometry coding block method, aggregating the first feature map using a neural network may include using a residual network (ResNet) architecture.

[0241] In some embodiments of the exemplary learning-based point cloud geometry encoding block method, performing processing prior to converting the first feature map into the second feature map includes aggregating the first feature map using a third neural network.

[0242] In some embodiments of the exemplary learning-based point cloud geometry coding block method, the third neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

[0243] An exemplary learning-based point cloud geometry encoding block apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to cause the apparatus to perform any of the above-enumerated claims.

[0244] An exemplary learning-based point cloud geometry decoder method according to some embodiments may include decoding a first feature map from a bitstream, accessing a set of distribution parameters, transforming the first feature map into a second feature map based on the set of distribution parameters, and reconstructing a point cloud from the second feature map.

[0245] Some embodiments of the example learning-based point cloud geometry decoder method may further include updating the first feature map by normalizing elements of the first feature map.

[0246] In some embodiments of the example learning-based point cloud geometry decoder method, normalizing the elements of the first feature map may include determining a respective length of each feature vector associated with one of the elements of the first feature map, and dividing each element of the first feature map by the respective length.

[0247] In some embodiments of the example learning-based point cloud geometry decoder method, normalizing the elements of the first feature map may include: arranging each of the one or more vectors by reshaping an associated feature channel in the first feature map; determining a respective length of each of the one or more vectors for each feature channel; and dividing each vector element of each vector by the respective length for each feature channel, wherein each vector element is one of the elements of the first feature map.

[0248] In some embodiments of the example learning-based point cloud geometry decoder method, normalizing the elements of the first feature map may include: arranging each of the one or more vectors by reshaping an associated feature channel in the first feature map; determining a respective standard deviation of each of the one or more vectors for each feature channel; determining a respective mean of each of the one or more vectors for each feature channel; updating each vector element of each vector by subtracting the respective mean; and dividing each updated vector element of each vector by the respective standard deviation for each feature channel, wherein each vector element is one of the elements of the first feature map.

[0249] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the set of distribution parameters is determined during a training period using back-propagation techniques.

[0250] In some embodiments of the exemplary learning-based point cloud geometry decoder method, a set of distribution parameters is determined for each feature channel.

[0251] In some embodiments of the exemplary learning-based point cloud geometry decoder method, performing processing prior to encoding the second feature map includes accessing a second set of distribution parameters and updating the second feature map by transforming the second feature map based on the second set of distribution parameters.

[0252] In some embodiments of the exemplary learning-based point cloud geometry decoder method, performing processing prior to encoding the second feature map includes aggregating the second feature map using a second neural network.

[0253] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the second neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

[0254] In some embodiments of the exemplary learning-based point cloud geometry decoder method, aggregating the second feature map may include using a residual network (ResNet) architecture.

[0255] In some embodiments of the exemplary learning-based point cloud geometry decoder method, performing processing prior to converting the first feature map into the second feature map includes aggregating the first feature map using a third neural network.

[0256] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the third neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

[0257] An exemplary learning-based point cloud geometry decoder device according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to cause the device to perform any of the methods recited above.

[0258] An exemplary method according to some embodiments may include querying a local point group for each point in a point cloud having a selected group size, extracting first per-point features, and performing a first pass and a second pass via a feedback process, wherein the feedback process may include converting the first per-point features into second per-point channel-based distribution features based on a set of transformation parameters, extracting third per-point features from the second per-point features via a deep network, aggregating the third per-point features based on the local point group to form first per-group features, enriching the third per-point features with an enhanced version of the first per-group features, and obtaining the next-stage first per-point features for use as the first per-point features for the second pass via the feedback process.

[0259] In some embodiments of the exemplary method, the set of transformation parameters is a learnable set of transformation parameters.

[0260] In some embodiments, the example method may further include expanding the first group of features to generate expanded versions of the first group of features.

[0261] In some embodiments of the exemplary method, the first pointwise feature of the next stage may be an enhanced third pointwise feature.

[0262] In some embodiments, the example method may further include aggregating the second pointwise features based on the local point groups to form second groupwise features, augmenting the second pointwise features with an expanded version of the second groupwise features to form updated second pointwise features, obtaining fourth pointwise features, and matching the updated second pointwise features to the fourth pointwise features.

[0263] In some embodiments, the example method may further include expanding the third pointwise features to generate expanded versions of the second groupwise features.

[0264] In some embodiments of the exemplary method, the latent code may include a third point-by-point feature.

[0265] In some embodiments of the exemplary method, the collection abstraction feature may include a feature for each second group.

[0266] An exemplary apparatus according to some embodiments may include a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to query local point groups for each point in a point cloud having a selected group size, extract first point-wise features, and perform first and second passes via a feedback process; The instructions, when executed by the processor, are further operable to cause the apparatus to perform a feedback process, transform the first pointwise features into second pointwise channelwise distribution features based on a set of transformation parameters, extract third pointwise features from the second pointwise features via a deep network, aggregate the third pointwise features based on local point groups to form first groupwise features, enhance the third pointwise features with enhanced versions of the first groupwise features, and obtain, via the feedback process, the next-stage first pointwise features for use as first pointwise features for a second pass.

[0267] An additional exemplary method according to some embodiments may include querying a local point group for each point in a point cloud having a selected group size, extracting first per-point features, and performing a first pass and a second pass via a feedback process, wherein the feedback process extracts second per-point features from the first per-point features via a residual-based network, aggregating the second per-point features based on the local point group to form first per-group features, transforming the first per-point features into third per-point channel-based distribution features based on a set of transformation parameters, and aggregating the first per-point features into third per-point channel-based distribution features via a deep network. aggregating the fourth pointwise features based on the local point groups to form second groupwise features; augmenting the second pointwise features with an expanded version of the first groupwise features to form next-stage first pointwise features for use as first pointwise features for the second pass via a feedback process; and augmenting the fourth pointwise features with an expanded version of the second groupwise features to form next-stage third pointwise features for use as third pointwise features for the second pass via a feedback process.

[0268] In some embodiments of additional exemplary methods, the set of transformation parameters may be a set of learnable transformation parameters.

[0269] In some embodiments, additional example methods may further include expanding the first group-wise features to generate expanded versions of the first group-wise features.

[0270] In some embodiments, additional exemplary methods may further include expanding the second group-wise features to generate expanded versions of the second group-wise features.

[0271] In some embodiments, additional example methods may further include extracting fifth pointwise features, transforming the fifth pointwise features into a sixth pointwise channel-wise feature distribution based on a second set of transformation parameters, and extracting seventh pointwise features from the sixth pointwise features via a second residual-based network.

[0272] In some embodiments of the additional exemplary method, the second set of transformation parameters may be a learnable second set of transformation parameters.

[0273] In some embodiments, additional exemplary methods may further include outputting a seventh pointwise feature.

[0274] In some embodiments of the additional exemplary method, the latent code may include at least one of a first pointwise feature and a third pointwise feature.

[0275] In some embodiments of the additional exemplary method, the collection abstraction feature may include at least one of a first pointwise feature and a third pointwise feature.

[0276] An additional example apparatus according to some embodiments may include a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to query local point groups for each point in a point cloud having a selected group size, extract first pointwise features, and perform a first pass and a second pass through a feedback process, the instructions, when executed by the processor, performing the feedback process, extracting second pointwise features from the first pointwise features through a residual-based network, aggregating the second pointwise features based on the local point groups to form first pointwise features, and aggregating the first pointwise features based on a set of transformation parameters. aggregating the fourth pointwise features based on the local point groups to form second-stage first pointwise features for use as first pointwise features for a second pass via a feedback process; and augmenting the fourth pointwise features with an augmented version of the second groupwise features to form next-stage third pointwise features for use as third pointwise features for a second pass via a feedback process.

[0277] A further exemplary method according to some embodiments may include querying local point groups for each point in a point cloud having a selected group size; extracting first per-point features; transforming the first per-point features into second per-point channel-wise distribution features based on a set of transformation parameters; extracting third per-point features from the second per-point features via a deep network; aggregating the third per-point features based on the local point groups to form first per-group features; and extracting second per-group features from the expanded version of the first per-group features.

[0278] In some embodiments of further exemplary methods, the set of transformation parameters may be a set of learnable transformation parameters.

[0279] In some embodiments, further exemplary methods may further include expanding the first group of features to generate expanded versions of the first group of features.

[0280] In some embodiments, further exemplary methods may further include outputting the second group-wise features.

[0281] A further exemplary method may further include extracting fourth pointwise features from the first pointwise features via a residual-based network, aggregating the fourth pointwise features based on the local point groups to form third groupwise features, and extracting the fourth groupwise features from an expanded version of the third groupwise features.

[0282] In some embodiments, further exemplary methods may further include expanding the features for each of the third groups to generate expanded versions of the features for each of the third groups.

[0283] In some embodiments of further exemplary methods, the latent code may include a second grouping of features.

[0284] In some embodiments of further exemplary methods, the collection abstraction feature may include a second group-by-group feature.

[0285] A further exemplary apparatus according to some embodiments may include a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to query local point groups for each point in a point cloud having a selected group size, extract first per-point features, transform the first per-point features into second per-point channel-wise distribution features based on a set of transformation parameters, extract third per-point features from the second per-point features via a deep network, aggregate the third per-point features based on the local point groups to form first per-group features, and extract second per-group features from the expanded version of the first per-group features.

[0286] In some embodiments of further exemplary apparatus, the instructions, when executed by the processor, may be further operable to cause the apparatus to extract fourth pointwise features from the first pointwise features via a residual-based network, aggregate the fourth pointwise features based on the local point groups to form third groupwise features, and extract the fourth groupwise features from an expanded version of the third groupwise features.

[0287] Still additional exemplary methods according to some embodiments may include extracting first group-wise features, expanding the first group-wise features into first point-wise features, transforming the first point-wise features into second point-wise channel-wise distribution features based on a set of transformation parameters, extracting third point-wise features from the second point-wise features via a deep network, and extracting fourth point-wise features from the third point-wise features.

[0288] Yet additional exemplary apparatus according to some embodiments may include one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform any one of the methods recited above.

[0289] An exemplary signal according to some embodiments may include a bitstream formed by performing any one of the methods listed above.

[0290] An exemplary computer-readable storage medium according to some embodiments may store instructions for encoding or decoding a point cloud according to any one of the methods recited above.

[0291] An exemplary learning-based point cloud geometry processing block method according to some embodiments may include accessing a first feature map, where the first feature map is an input to the processing block, and where the first feature map is generated by a first set of neural network layers; normalizing elements of the first feature map to generate a second feature map; accessing a set of distribution parameters; transforming the second feature map into a third feature map based on the set of distribution parameters; and aggregating the third feature map into a fourth feature map.

[0292] In some embodiments of the exemplary learning-based point cloud geometry processing block method, the first feature map is a feature map for each queried point of the local group.

[0293] In some embodiments of the exemplary learning-based point cloud geometry processing block method, the second feature map is a concatenation of feature elements per slice computed by normalizing the corresponding slice from the first feature map.

[0294] In some embodiments of the exemplary learning-based point cloud geometry processing block method, the set of distribution parameters is obtained through training.

[0295] In some embodiments of the exemplary learning-based point cloud geometry processing block method, a set of distribution parameters is defined and applied for each slice.

[0296] In some embodiments of the exemplary learning-based point cloud geometry processing block method, the fourth feature map is a concatenation of the local group-wise feature aggregations to generate a group-wise feature map.

[0297] In some embodiments of the exemplary learning-based point cloud geometry processing block method, the fourth feature map is encoded into the bitstream.

[0298] Some embodiments of the exemplary learning-based point cloud geometry processing block method may further include expanding the fourth feature map to the size of the third feature map, augmenting the expanded feature map with the third feature map, and repeating the learning-based point cloud geometry encoder method of claim 34 one or more times, where the augmented feature map is used as the first feature map and the next set of distribution parameters is used as the distribution parameters.

[0299] In some embodiments of the exemplary learning-based point cloud geometry processing block method, the next set of distribution parameters is obtained via training.

[0300] In some embodiments of the exemplary learning-based point cloud geometry processing block method, a next set of distribution parameters is defined for each iteration of the learning-based point cloud geometry encoder method of the preceding claims.

[0301] In some embodiments of the exemplary learning-based point cloud geometry processing block method, the third feature map is a point-wise feature map.

[0302] In some embodiments of the exemplary learning-based point cloud geometry processing block method, the fourth feature map is a group-wise feature map.

[0303] In some embodiments of the exemplary learning-based point cloud geometry processing block method, the third feature map is a fine feature map.

[0304] In some embodiments of the exemplary learning-based point cloud geometry processing block method, the fourth feature map is a coarse feature map.

[0305] Some embodiments of the exemplary learning-based point cloud geometry processing block method may further include connecting a residual network to the learning-based point cloud geometry encoder in parallel.

[0306] Some embodiments of the exemplary learning-based point cloud geometry processing block method may further include serially connecting a residual network to the learning-based point cloud geometry encoder.

[0307] Some embodiments of the example learning-based point cloud geometry processing block method may further include inputting the fourth feature map to a second set of neural network layers and encoding the output of the second set of neural network layers into a bitstream.

[0308] An exemplary learning-based point cloud geometry processing block apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to cause the apparatus to perform the methods recited above.

[0309] An exemplary learning-based point cloud geometry decoder method according to some embodiments may include accessing a first feature map, where the first feature map is an input, where the first feature map is decoded from a bitstream; expanding the first feature map into a second feature map; normalizing elements of the second feature map to generate a third feature map; accessing a set of distribution parameters; and transforming the third feature map into a fourth feature map based on the set of distribution parameters.

[0310] In some embodiments of the example learning-based point cloud geometry decoder method, extending the first feature map into the second feature map may include not pooling the first feature map into the second feature map.

[0311] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the first feature map is a decoded group-wise feature map.

[0312] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the second feature map is generated by a first set of neural network layers.

[0313] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the second feature map is expanded to the size of the point-wise feature map.

[0314] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the third feature map is a concatenation of feature elements per slice, and the third feature map is computed by normalizing the corresponding slice from the second feature map.

[0315] In some embodiments of the exemplary learning-based point cloud geometry decoder method, at least one of the corresponding slices is further divided into two or more groups.

[0316] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the set of distribution parameters is obtained via training.

[0317] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the fourth feature map is input to a second set of neural network layers to reconstruct the point cloud.

[0318] Some embodiments of the exemplary learning-based point cloud geometry decoder method may further include embedding a fourth feature map through a set of neural network layers and repeating the learning-based point cloud geometry decoder method one or more times, where the embedded feature map is used as the second feature map and the next set of distribution parameters is used as the distribution parameters.

[0319] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the next set of distribution parameters is obtained via training.

[0320] In some embodiments of the exemplary learning-based point cloud geometry decoder method, a next set of distribution parameters is defined for each iteration of the learning-based point cloud geometry decoder method.

[0321] Some embodiments of the exemplary learning-based point cloud geometry decoder method may further include aggregating the fourth feature map into a fifth feature map, expanding the fifth feature map to the size of the fourth feature map, augmenting the expanded feature map with the fourth feature map, and repeating the learning-based point cloud geometry decoder method of claim 84 one or more times, wherein the augmented feature map is used as the second feature map and the next set of distribution parameters is used as the distribution parameters.

[0322] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the next set of distribution parameters is obtained via training.

[0323] In some embodiments of the exemplary learning-based point cloud geometry decoder method, a next set of distribution parameters is defined for each iteration of the learning-based point cloud geometry decoder method of claim 84.

[0324] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the fourth feature map is a point-wise feature map.

[0325] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the fifth feature map is a group-wise feature map.

[0326] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the fourth feature map is a fine feature map.

[0327] In some embodiments of the exemplary learning-based point cloud geometry decoder method, the fifth feature map is a coarse feature map.

[0328] Some embodiments of the example learning-based point cloud geometry decoder method may further include connecting a residual network to the learning-based point cloud geometry decoder in parallel.

[0329] Some embodiments of the exemplary learning-based point cloud geometry decoder method may further include serially connecting a residual network to the learning-based point cloud geometry decoder.

[0330] An exemplary learning-based point cloud geometry decoder device according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to cause the device to perform one of the methods recited above.

[0331] In some embodiments of the exemplary learning-based point cloud geometry processing block method, at least one of the distribution parameters is predetermined.

[0332] An exemplary method according to some embodiments may include performing processing including querying local point groups for each point in a point cloud having a selected group size, extracting first per-point features, transforming the first per-point features into second per-point features according to a per-channel distribution defined by a set of transformation parameters, extracting third per-point features from the second per-point features via a neural network layer, aggregating the third per-point features based on the local point groups to form first per-group features, enriching the third per-point features with an enhanced version of the first per-group features, and obtaining next-stage first per-point features for use as first per-point features for the next stage of processing.

[0333] This disclosure describes various aspects, including tools, features, embodiments, models, approaches, etc. Many of these aspects are described with specificity, often in a manner that may sound limiting, at least to indicate their individual characteristics. However, this is for purposes of clarity of description and does not limit the disclosure or scope of the aspects. In fact, all of the different aspects can be combined and substituted to provide further aspects. Furthermore, these aspects can also be similarly combined and substituted with aspects described in previous applications.

[0334] Aspects described and contemplated in this disclosure may be implemented in many different forms. While some embodiments are specifically illustrated, other embodiments are contemplated, and description of a particular embodiment is not intended to limit the breadth of implementations. At least one of the above aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects may be implemented as a method, an apparatus, a computer-readable storage medium having stored therein instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored therein a bitstream generated according to any of the described methods.

[0335] In this disclosure, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image," "picture," and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side, and the term "decoded" is used on the decoder side.

[0336] The terms high dynamic range (HDR) and standard dynamic range (SDR) often convey specific values ​​of dynamic range to those skilled in the art. However, additional embodiments are contemplated in which reference to HDR is understood to mean "higher dynamic range" and reference to SDR is understood to mean "lower dynamic range." Such additional embodiments are not constrained by any specific values ​​of dynamic range that might often be associated with the terms "high dynamic range" and "standard dynamic range."

[0337] Various methods are described herein, each of which includes one or more steps or acts for achieving the described method. Unless a specific order of steps or acts is required for the proper operation of the method, the order and / or use of specific steps and / or acts may be modified or combined. It should be noted that terms such as "first," "second," and the like may be used in various embodiments to modify elements, components, steps, operations, etc., e.g., "first decode" and "second decode." The use of such terms does not imply any ordering of the modified operations unless specifically required. Thus, in this example, the first decode need not be performed before the second decode, but may occur, for example, before, during, or during an overlapping period with the second decode.

[0338] In this disclosure, for example, various numerical values ​​may be used, and the specific values ​​are for illustrative purposes only, and the described aspects are not limited to these specific values.

[0339] The embodiments described herein may be implemented by computer software implemented by a processor, or by other hardware, or by a combination of hardware and software. As a non-limiting example, embodiments may be implemented by one or more integrated circuits. The processor may be of any type appropriate to the technology environment, and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a multi-core architecture-based processor.

[0340] Various implementations involve decoding. As used in this disclosure, "decoding" may encompass all or some of the processing performed on a received encoded sequence, e.g., to generate a final output suitable for display. In various embodiments, such processing includes one or more of the processing typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processing also or alternatively includes processing performed by the decoder of various implementations described in this disclosure, such as, for example, extracting a picture from a tiled (packed) picture, determining the upsampling filter to use, then upsampling the picture, and flipping the picture back to its intended orientation.

[0341] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process as a whole will become clear based on the context of the specific description.

[0342] Various implementations involve encoding. In a manner similar to the above discussion regarding "decoding," "encoding," as used in this disclosure, may encompass all or some of the processing performed on an input video sequence, e.g., to generate an encoded bitstream. In various embodiments, such processing includes one or more of the processing typically performed by an encoder, such as, e.g., partitioning, differential encoding, transforming, quantization, and entropy coding. In various embodiments, such processing also or alternatively includes processing performed by the encoders of various implementations described in this disclosure.

[0343] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to the broader encoding process as a whole will be clear based on the context of a specific description.

[0344] Where a figure is presented as a flow diagram, it should be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it should be understood that the figure also provides a flow diagram of the corresponding method / process.

[0345] Various embodiments refer to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is usually considered, often due to computational complexity constraints. Rate-distortion optimization is usually formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. There are various approaches to solving the rate-distortion optimization problem. For example, these techniques may be based on extensive testing of all encoding options, including all considered modes or coding parameter values, but with a thorough evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. To reduce encoding complexity, more rapid techniques may also be used, particularly with calculation of approximate distortion based on a prediction or prediction residual signal rather than a reconstructed signal. A mixture of these two approaches may also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for others. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization does not necessarily involve a thorough evaluation of both the encoding cost and the associated distortion.

[0346] Implementations and aspects described herein may be implemented as, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the discussed feature may also be implemented in other forms (e.g., an apparatus or a program). For example, an apparatus may be implemented in appropriate hardware, software, and firmware. The method may be implemented in a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Furthermore, processors include, for example, communication devices such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0347] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation," as well as any other variations, appearing in various places throughout this disclosure, do not necessarily all refer to the same embodiment.

[0348] Additionally, this disclosure may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0349] Additionally, this disclosure may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0350] Additionally, this disclosure may refer to "receiving" various pieces of information. Receiving, like "accessing," is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from memory). Furthermore, "receiving" typically involves in some way an operation such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0351] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded as many times as there are listed items.

[0352] Also, as used herein, the term "signaling" specifically refers to indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a specific one of multiple parameters for selecting a region-based filter parameter for de-artifact filtering. Thus, in some embodiments, the same parameter is used at both the encoder and decoder sides. Thus, for example, an encoder can transmit a specific parameter to a decoder (explicit signaling) so that the decoder can use the same specific parameter. In contrast, if the decoder already has the specific parameter as well as other parameters, signaling without transmission (implicit signaling) can be used to simply allow the decoder to know and select the specific parameter. By avoiding transmitting any actual function, bit savings are realized in various embodiments. It will be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder in various embodiments. While the above relates to the verb form of the term "signal," the term "signal" may also be used herein as a noun.

[0353] Implementations can generate various signals formatted to carry information that can be stored or transmitted, for example. Information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. For example, such a signal can be formatted as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored on a processor-readable medium.

[0354] Several embodiments are described. Features of these embodiments may be provided alone or in any combination across various claim categories and types. Furthermore, embodiments may include one or more of the following features, devices, or aspects across various claim categories and types, alone or in any combination: Adapting residues in an encoder according to any of the embodiments discussed. A bitstream or signal containing one or more of the syntax elements described, or variations thereof. A bitstream or signal containing syntax carrying information generated according to any of the described embodiments. Inserting syntax elements into the signaling that allow the decoder to adapt the residues in a way that corresponds to the way used by the encoder. Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements or variations thereof. Creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments. A method, process, apparatus, instruction storage medium, data storage medium, or signal according to any of the described embodiments. A TV, set-top box, mobile phone, tablet, or other electronic device that performs filter parameter adaptation according to any of the described embodiments. A TV, set-top box, mobile phone, tablet, or other electronic device that performs filter parameter adaptation according to any of the described embodiments and displays the resulting image (e.g., using a monitor, screen, or other type of display). A TV, set-top box, mobile phone, tablet, or other electronic device that selects a channel (e.g., using a tuner) to receive a signal containing encoded images and performs filter parameter adaptation according to any of the described embodiments. A TV, set-top box, mobile phone, tablet, or other electronic device that receives an over-the-air signal (e.g., using an antenna) that includes an encoded image and performs filter parameter adaptation according to any of the described embodiments.

[0355] It should be noted that one or more of the various hardware elements of the described embodiments are referred to as “modules,” which, in combination with the respective modules, perform (i.e., implement, perform, etc.) the various functions described herein. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by one of ordinary skill in the art for a given implementation. It should be noted that each described module may also include executable instructions for performing one or more functions described as being performed by the respective module, which may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may be stored on any suitable non-transitory computer-readable medium, or medium such as commonly referred to as RAM, ROM, etc.

[0356] While features and elements are described above in particular combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with the other features and elements. Additionally, the methods described herein can be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. 1. A learning-based point cloud geometry processing block method, comprising: accessing a first feature map; the first feature map has multiple C channels and is an input to the processing block; the first feature map is generated by a first set of neural network layers; and accessing a set of distribution parameters; transforming the first feature map into a second feature map based on the set of distribution parameters; encoding the second feature map into a bitstream; A method comprising:

2. The method of claim 1 , further comprising updating the first feature map by normalizing vector elements of the first feature map.

3. Normalizing the vector elements of the first feature map includes: determining a respective length of each feature vector associated with one of the vector elements of the first feature map; Dividing each of the vector elements of the first feature map by the respective length; The method of claim 2 , comprising:

4. Normalizing the vector elements of the first feature map includes: arranging vectors by reshaping each associated feature channel in the first feature map; determining a respective length of each reshaped vector for each feature channel; and dividing each of the vector elements of each reshaped vector for each feature channel by the respective length; each vector element is one of the elements of the first feature map; The method of claim 2.

5. Normalizing the vector elements of the first feature map includes: arranging vectors by reshaping each associated feature channel in the first feature map; determining a respective standard deviation of each reshaped vector for each feature channel; determining a respective mean of each reshaped vector for each feature channel; updating each vector element of each vector by subtracting the respective mean; Dividing each updated vector element of each vector by the respective standard deviation for each feature channel; Including, each vector element is one of the elements of the first feature map; The method of claim 2.

6. The method of claim 1 , wherein the set of distribution parameters is determined during a training period using back-propagation techniques.

7. The method of claim 1 , wherein the set of distribution parameters is determined for each feature channel.

8. The method of claim 1 , further comprising updating the second feature map by performing downsampling using an average pooling or max pooling function.

9. determining a third feature map by filtering the second feature map using a smoothing filter; updating the second feature map by concatenating the third feature map to the second feature map; The method of claim 1 , further comprising:

10. Performing processing prior to encoding the second feature map may include: accessing a second set of distribution parameters; updating the second feature map by transforming the second feature map based on the second set of distribution parameters; 10. The method of claim 1, comprising:

11. 11. The method of claim 1, wherein performing processing prior to encoding the second feature map comprises aggregating the second feature map using a second neural network.

12. 12. The method of claim 11, wherein the second neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

13. 13. The method of claim 11, wherein aggregating the second feature map comprises using a residual network (ResNet) architecture.

14. determining a fourth feature map by aggregating the first feature maps using a neural network in parallel with transforming the first feature map into the second feature map; updating the second feature map by concatenating the fourth feature map to the second feature map; 14. The method of any one of claims 1 to 13, further comprising:

15. 15. The method of claim 14, wherein aggregating the first feature map using a neural network includes using a residual network (ResNet) architecture.

16. 16. The method of claim 1, wherein performing processing prior to converting the first feature map to the second feature map comprises aggregating the first feature map using a third neural network.

17. 17. The method of claim 16, wherein the third neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

18. 1. An apparatus comprising: a processor; a non-transitory computer-readable medium storing instructions operable when executed by the processor to cause the apparatus to perform any one of claims 1 to 17; An apparatus comprising:

19. 1. A learning-based point cloud geometry decoder method, comprising: decoding a first feature map from the bitstream; accessing a set of distribution parameters; transforming the first feature map into a second feature map based on the set of distribution parameters; reconstructing the point cloud from the second feature map; A method comprising:

20. 20. The method of claim 19, further comprising updating the first feature map by normalizing elements of the first feature map.

21. Normalizing the elements of the first feature map determining a respective length of each feature vector associated with one of the elements of the first feature map; Dividing each element of the first feature map by the respective length; 21. The method of claim 20, comprising:

22. Normalizing the elements of the first feature map arranging each of the one or more vectors by reshaping an associated feature channel in the first feature map; determining a respective length of each of the one or more vectors for each feature channel; Dividing each vector element of each vector by the respective length for each feature channel; each vector element is one of the elements of the first feature map; 21. The method of claim 20, comprising:

23. Normalizing the elements of the first feature map arranging each of the one or more vectors by reshaping an associated feature channel in the first feature map; determining a respective standard deviation of each of the one or more vectors for each feature channel; determining the respective mean of each of the one or more vectors for each feature channel; updating each vector element of each vector by subtracting the respective mean; Dividing each updated vector element of each vector by the respective standard deviation for each feature channel; each vector element is one of the elements of the first feature map; 21. The method of claim 20, comprising:

24. 24. The method of any one of claims 19 to 23, wherein the set of distribution parameters is determined during a training period using back-propagation techniques.

25. 24. The method of any one of claims 19 to 23, wherein the set of distribution parameters is determined for each feature channel.

26. Performing processing prior to encoding the second feature map may include: accessing a second set of distribution parameters; updating the second feature map by transforming the second feature map based on the second set of distribution parameters; 28. The method of any one of claims 19 to 27, comprising:

27. 27. The method of claim 19, wherein performing processing prior to encoding the second feature map comprises aggregating the second feature map using a second neural network.

28. 28. The method of claim 27, wherein the second neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

29. 29. The method of any one of claims 27-28, wherein aggregating the second feature map comprises using a residual network (ResNet) architecture.

30. 30. The method of claim 19, wherein performing processing prior to converting the first feature map to the second feature map comprises aggregating the first feature map using a third neural network.

31. 31. The method of claim 30, wherein the third neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

32. 1. An apparatus comprising: a processor; a non-transitory computer-readable medium storing instructions operable when executed by the processor to cause the apparatus to perform any one of claims 19 to 31; and An apparatus comprising:

33. 1. A method comprising: Querying a local point group for each point in the point cloud having a selected group size; Extracting first pointwise features; and performing a first pass and a second pass via a feedback process; transforming the first pointwise features into second pointwise channelwise distribution features based on a set of transformation parameters; extracting third pointwise features from the second pointwise features via a deep network; and aggregating the third point-wise features based on the local point groups to form a first group-wise feature; Enhancing the third pointwise features with an expanded version of the first groupwise features; obtaining, via the feedback process, next-stage first pointwise features for use as the first pointwise features for the second pass; and a method comprising:

34. 34. The method of claim 33, wherein the set of transformation parameters is a learnable set of transformation parameters.

35. 35. The method of any one of claims 33 to 34, further comprising expanding the first group of features to generate the expanded version of the first group of features.

36. 36. The method of any one of claims 33 to 35, wherein the next-stage first pointwise feature is the enhanced third pointwise feature.

37. aggregating the second point-wise features based on the local point groups to form second group-wise features; augmenting the second pointwise features with an expanded version of the second groupwise features to form updated second pointwise features; obtaining a fourth pointwise feature; and matching the updated second pointwise features to the fourth pointwise features; 37. The method of any one of claims 33 to 36, further comprising:

38. 38. The method of claim 37, further comprising expanding the third pointwise features to generate the expanded version of the second groupwise features.

39. 39. The method of any one of claims 33 to 38, wherein a latent code comprises the third pointwise feature.

40. 40. The method of any one of claims 33 to 39, wherein a set abstraction feature comprises a feature for each of the second groups.

41. 1. An apparatus comprising: a processor; A non-transitory computer-readable medium that, when executed by the processor, Querying the local point group for each point in the point cloud having the selected group size; Extracting first pointwise features; a non-transitory computer-readable medium storing instructions operable to cause the apparatus to perform a first pass and a second pass via a feedback process; The instructions, when executed by the processor, perform the feedback process; transforming the first pointwise features into second pointwise channelwise distribution features based on a set of transformation parameters; extracting third pointwise features from the second pointwise features via a deep network; aggregating the third point-wise features based on the local point groups to form a first group-wise feature; Enhancing the third pointwise features with an expanded version of the first groupwise features; Obtaining a next-stage first pointwise feature via the feedback process to use as the first pointwise feature for the second pass. and further operable to cause the device to: Device.

42. 1. A method comprising: Querying a local point group for each point in the point cloud having a selected group size; Extracting first pointwise features; and performing a first pass and a second pass through a feedback process, the feedback process comprising: extracting second pointwise features from the first pointwise features via a residual-based network; aggregating the second point-wise features based on the local point groups to form first group-wise features; transforming the first pointwise features into third pointwise channelwise distribution features based on a set of transformation parameters; extracting fourth pointwise features from the third pointwise features via a deep network; and aggregating the fourth point-wise features based on the local point groups to form second group-wise features; augmenting the second pointwise features with an expanded version of the first groupwise features to form next-stage first pointwise features for use as the first pointwise features for the second pass via the feedback process; augmenting the fourth pointwise features with an expanded version of the second groupwise features to form next-stage third pointwise features for use as the third pointwise features for the second pass via the feedback process; A method comprising:

43. 43. The method of claim 42, wherein the set of transformation parameters is a learnable set of transformation parameters.

44. 44. The method of any one of claims 42 to 43, further comprising expanding the first group of features to generate the expanded version of the first group of features.

45. 45. The method of any one of claims 42 to 44, further comprising expanding the second group of features to generate the expanded version of the second group of features.

46. extracting fifth pointwise features; and transforming the fifth pointwise features into a sixth pointwise channelwise feature distribution based on a second set of transformation parameters; extracting seventh pointwise features from the sixth pointwise features via a second residual-based network; and 46. ​​The method of any one of claims 42 to 45, further comprising:

47. 47. The method of claim 46, wherein the second set of transformation parameters is a learnable second set of transformation parameters.

48. 47. The method of claim 46, further comprising outputting a seventh pointwise feature.

49. 49. The method of any one of claims 42 to 48, wherein a latent code includes at least one of the first pointwise feature and the third pointwise feature.

50. 50. The method of any one of claims 42 to 49, wherein collection abstraction features include at least one of the first pointwise feature and the third pointwise feature.

51. 1. An apparatus comprising: a processor; A non-transitory computer-readable medium that, when executed by the processor, Querying the local point group for each point in the point cloud having the selected group size; Extracting first pointwise features; a non-transitory computer-readable medium storing instructions operable to cause the apparatus to perform a first pass and a second pass via a feedback process; The instructions, when executed by the processor, perform the feedback process; extracting second pointwise features from the first pointwise features via a residual-based network; aggregating the second point-wise features based on the local point groups to form first group-wise features; transforming the first pointwise features into third pointwise channelwise distribution features based on a set of transformation parameters; extracting fourth pointwise features from the third pointwise features via a deep network; aggregating the fourth point-wise features based on the local point groups to form second group-wise features; augmenting the second pointwise features with an expanded version of the first groupwise features to form next-stage first pointwise features for use as the first pointwise features for the second pass via the feedback process; augmenting the fourth pointwise features with an expanded version of the second groupwise features to form next-stage third pointwise features for use as the third pointwise features for the second pass via the feedback process. The apparatus further functions to cause the apparatus to:

52. 1. A method comprising: Querying a local point group for each point in the point cloud having a selected group size; Extracting first pointwise features; transforming the first pointwise features into second pointwise channelwise distribution features based on a set of transformation parameters; extracting third pointwise features from the second pointwise features via a deep network; and aggregating the third point-wise features based on the local point groups to form a first group-wise feature; extracting second group-wise features from the expanded version of the first group-wise features; A method comprising:

53. 53. The method of claim 52, wherein the set of transformation parameters is a learnable set of transformation parameters.

54. 54. The method of any one of claims 52 to 53, further comprising expanding the first group of features to generate the expanded version of the first group of features.

55. 55. The method of any one of claims 52 to 54, further comprising outputting the second group of features.

56. extracting fourth pointwise features from the first pointwise features via a residual-based network; aggregating the fourth point-wise features based on the local point groups to form a third group-wise feature; and extracting fourth groupwise features from an expanded version of the third groupwise features; and 56. The method of any one of claims 52 to 55, further comprising:

57. 57. The method of any one of claims 52 to 56, further comprising expanding the features for each of the third groups to generate expanded versions of the features for each of the third groups.

58. 58. The method of any one of claims 52 to 57, wherein a latent code includes features for each of the second groups.

59. 59. The method of any one of claims 52 to 58, wherein a set abstraction feature comprises a feature for each of the second groups.

60. 1. An apparatus comprising: a processor; A non-transitory computer-readable medium that, when executed by the processor, Querying the local point group for each point in the point cloud having the selected group size; Extracting first pointwise features; transforming the first pointwise features into second pointwise channelwise distribution features based on a set of transformation parameters; extracting third pointwise features from the second pointwise features via a deep network; aggregating the third point-wise features based on the local point groups to form a first group-wise feature; extracting second group-wise features from the expanded version of the first group-wise features; a non-transitory computer-readable medium storing instructions operable to cause the device to: An apparatus comprising:

61. The instructions, when executed by the processor, extracting fourth pointwise features from the first pointwise features via a residual-based network; aggregating fourth point-wise features based on the local point groups to form third group-wise features; extracting fourth group-wise features from the expanded version of the third group-wise features; 61. The apparatus of claim 60, further operable to cause the apparatus to:

62. 1. A method comprising: Extracting features for each first group; Expanding the first group-wise features into first point-wise features; transforming the first pointwise features into second pointwise channelwise distribution features based on a set of transformation parameters; extracting third pointwise features from the second pointwise features via a deep network; and extracting fourth pointwise features from the third pointwise features; and A method comprising:

63. 1. An apparatus comprising: one or more processors; at least one memory coupled to the one or more processors; Equipped with 63. An apparatus, wherein the one or more processors are configured to perform any one of claims 33-40, 42-50, 52-59, and 62.

64. 63. A signal comprising a bitstream formed by performing any one of claims 33 to 40, 42 to 50, 52 to 59 and 62.

65. 63. A computer readable storage medium having stored thereon instructions for encoding or decoding a point cloud according to any one of claims 33 to 40, 42 to 50, 52 to 59 and 62.

66. 1. A learning-based point cloud geometry processing block method, comprising: accessing a first feature map; the first feature map is an input to the processing block; the first feature map is generated by a first set of neural network layers; and normalizing the elements of the first feature map to generate a second feature map; accessing a set of distribution parameters; transforming the second feature map into a third feature map based on the set of distribution parameters; aggregating the third feature map into a fourth feature map; A method comprising:

67. 67. The method of claim 66, wherein the first feature map is a feature map for each queried point of a local group.

68. 68. The method of any one of claims 66 to 67, wherein the second feature map is a concatenation of feature elements for each slice computed by normalizing the corresponding slice from the first feature map.

69. 69. The method of any one of claims 66 to 68, wherein the set of distribution parameters is obtained through training.

70. 70. The method of any one of claims 66 to 69, wherein the set of distribution parameters is defined and applied on a slice-by-slice basis.

71. 71. The method of any one of claims 66 to 70, wherein the fourth feature map is a concatenation of local group-wise feature aggregations to generate a group-wise feature map.

72. 72. The method of any one of claims 66 to 71, wherein the fourth feature map is encoded into a bitstream.

73. Expanding the fourth feature map to the size of the third feature map; augmenting the expanded feature map with the third feature map; and repeating the learning-based point cloud geometry encoder method of claim 34 one or more times. the enhanced feature map is used as the first feature map; The following set of distribution parameters is used as the distribution parameters:

73. The method of any one of claims 66 to 72.

74. 74. The method of claim 73, wherein the next set of distribution parameters is obtained through training.

75. 75. The method of any one of claims 73 to 74, wherein a next set of distribution parameters is defined for each iteration of the learning-based point cloud geometry encoder method of claim 34.

76. 76. The method of any one of claims 73 to 75, wherein the third feature map is a point-wise feature map.

77. 76. The method of any one of claims 73 to 75, wherein the fourth feature map is a group-wise feature map.

78. 76. The method of any one of claims 73 to 75, wherein the third feature map is a fine feature map.

79. 76. The method of any one of claims 73 to 75, wherein the fourth feature map is a coarse feature map.

80. 80. The method of any one of claims 66 to 79, further comprising connecting a residual network in parallel to the learning-based point cloud geometry encoder.

81. 80. The method of any one of claims 66 to 79, further comprising serially connecting a residual network to the learning-based point cloud geometry encoder.

82. inputting the fourth feature map into a second set of neural network layers; encoding the outputs of the second set of neural network layers into a bitstream; 82. The method of any one of claims 66 to 81, further comprising:

83. 1. An apparatus comprising: a processor; a non-transitory computer-readable medium storing instructions operable when executed by the processor to cause the apparatus to perform any one of claims 66 to 82; and An apparatus comprising:

84. 1. A learning-based point cloud geometry decoder method, comprising: accessing a first feature map; the first feature map is an input; the first feature map is decoded from a bitstream; and Expanding the first feature map into a second feature map; normalizing the elements of the second feature map to generate a third feature map; accessing a set of distribution parameters; transforming the third feature map into a fourth feature map based on the set of distribution parameters; A method comprising:

85. 85. The method of claim 84, wherein expanding the first feature map into the second feature map comprises unpooling the first feature map into the second feature map.

86. 86. The method of any one of claims 84 to 85, wherein the first feature map is a decoded group-wise feature map.

87. 87. The method of any one of claims 84 to 86, wherein the second feature map is generated by a first set of neural network layers.

88. 88. The method of any one of claims 84 to 87, wherein the second feature map is expanded to the size of a point-wise feature map.

89. the third feature map is a concatenation of feature elements for each slice; the third feature map is calculated by normalizing a corresponding slice from the second feature map.

89. The method of any one of claims 84 to 88.

90. 90. The method of claim 89, wherein at least one of the corresponding slices is further divided into two or more groups.

91. 91. The method of any one of claims 84 to 90, wherein the set of distribution parameters is obtained through training.

92. 92. The method of any one of claims 84 to 91, wherein the fourth feature map is input to a second set of neural network layers to reconstruct a point cloud.

93. Embedding a fourth feature map through a set of neural network layers; and and repeating the learning-based point cloud geometry decoder method of claim 84 one or more times. The embedded feature map is used as the second feature map, The following set of distribution parameters is used as the distribution parameters:

93. The method of any one of claims 84 to 92.

94. 94. The method of claim 93, wherein the next set of distribution parameters is obtained through training.

95. 95. The method of any one of claims 92 to 94, wherein a next set of distribution parameters is defined for each iteration of the learning-based point cloud geometry decoder method of claim 84.

96. aggregating the fourth feature map into a fifth feature map; Expanding the fifth feature map to the size of the fourth feature map; augmenting the expanded feature map with the fourth feature map; and repeating the learning-based point cloud geometry decoder method of claim 84 one or more times. The enhanced feature map is used as the second feature map; The following set of distribution parameters is used as the distribution parameters:

93. The method of any one of claims 84 to 92.

97. 97. The method of claim 96, wherein the next set of distribution parameters is obtained through training.

98. 98. The method of any one of claims 96 to 97, wherein a next set of distribution parameters is defined for each iteration of the learning-based point cloud geometry decoder method of claim 84.

99. 99. The method of any one of claims 96 to 98, wherein the fourth feature map is a point-wise feature map.

100. 99. The method of any one of claims 96 to 98, wherein the fifth feature map is a group-wise feature map.

101. 99. The method of any one of claims 96 to 98, wherein the fourth feature map is a fine feature map.

102. 99. The method of any one of claims 96 to 98, wherein the fifth feature map is a coarse feature map.

103. 103. The method of any one of claims 84 to 102, further comprising connecting a residual network in parallel to the learning-based point cloud geometry decoder.

104. 104. The method of any one of claims 84 to 103, further comprising serially connecting a residual network to the learning-based point cloud geometry decoder.

105. 1. An apparatus comprising: a processor; a non-transitory computer-readable medium storing instructions operable when executed by the processor to cause the apparatus to perform any one of claims 84 to 104; and An apparatus comprising:

106. 77. The method of claim 76, wherein at least one of the distribution parameters is predetermined.

107. 1. A method comprising: Querying a local point group for each point in the point cloud having a selected group size; Extracting first pointwise features; performing a process, Transforming the first pointwise features into second pointwise features according to a channel-wise distribution defined by a set of transformation parameters; extracting third pointwise features from the second pointwise features via a neural network layer; and aggregating the third point-wise features based on the local point groups to form a first group-wise feature; Enhancing the third pointwise feature with an expanded version of the first groupwise feature; obtaining a next stage first pointwise feature for use as a first pointwise feature for the next stage processing; and performing a process including: A method comprising:

Images