Depth distribution perceptual point feature extractor for AI-based point cloud compression

Abstract

Some embodiments of a method may include a learning-based point cloud geometry processing block method comprising: accessing a first feature map, where the first feature map has C channels and is an input to the processing block, and where the first feature map is generated by a first set of neural network layers; accessing a set of distribution parameters; transforming the first feature map into a second feature map based on the set of distribution parameters; and encoding the second feature map into a bitstream. These example processes may be applicable to both encoders and decoders of an AI-based Point Cloud Compression (PCC) framework.

Description

[0001] Cross-references to related applications

[0002] This application is a non-provisional filing of U.S. Provisional Patent Application Serial No. 63 / 388,600 (“600 Application”), filed on July 12, 2022, entitled “Deep Distribution-Aware Point Feature for AI-Based Point Cloud Compression”, and claims the benefit of that U.S. Provisional Patent Application pursuant to 35 U.S. SC § 119(e), the entire contents of which are incorporated herein by reference. The entire contents of the following cases are incorporated herein by reference: U.S. Provisional Patent Application Serial No. 63 / 252,482 (“'482 application”), filed October 5, 2021, entitled “Method and Apparatus for Point Cloud Compression Using Hybrid DeepEntropy Coding”; U.S. Provisional Patent Application Serial No. 63 / 297,894 (“'894 application”), filed January 10, 2022, entitled “Coordinate Refinement and Upsampling from Quantized Point Cloud Reconstruction”; and U.S. Provisional Patent Application Serial No. 63 / 297,869 (“'869 application”), filed January 10, 2022, entitled “Scalable Framework for Point Cloud Compression”. Background Technology

[0003] Point cloud data is applicable to numerous business sectors, such as autonomous driving, robotics, AR / VR, civil engineering, computer graphics, and the animation / film industry. 3D LiDAR sensors have been deployed in autonomous vehicles, and affordable LiDAR sensors include the Velodyne Velabit, Apple iPad Pro 2020, and the Intel RealSense LiDAR camera L515. With advancements in sensing technology, 3D point cloud data is becoming increasingly prevalent in the applications and industries mentioned above. Summary of the Invention

[0004] Examples of learning-based point cloud geometry processing blocks, according to some implementation schemes, may include: accessing a first feature map, wherein the first feature map has C channels and is the input to the processing block, and wherein the first feature map is generated by a first set of neural network layers; accessing a set of distribution parameters; transforming the first feature map into a second feature map based on the set of distribution parameters; and encoding the second feature map into a bitstream.

[0005] Some implementations of the example-based point cloud geometry encoding block method may also include updating the first feature map by normalizing the elements of the first feature map.

[0006] For some implementations of the example learning-based point cloud geometry encoding block method, normalizing the element of the first feature map may include: determining the corresponding length of each feature vector associated with an element of the first feature map; and dividing each element of the first feature map by the corresponding length.

[0007] For some implementations of the example learning-based point cloud geometry encoding block method, normalizing the element of the first feature map may include: arranging each of one or more vectors by reshaping the associated feature channels in the first feature map; determining a corresponding length for each of the one or more vectors for each feature channel; and dividing each vector element of each vector by the corresponding length for each feature channel, wherein each vector element is an element of the element of the first feature map.

[0008] For some implementations of the example learning-based point cloud geometry encoding block method, normalizing the element of the first feature map may include: arranging each of one or more vectors by reshaping the associated feature channels in the first feature map; determining the corresponding standard deviation of each of the one or more vectors for each feature channel; determining the corresponding average of each of the one or more vectors for each feature channel; updating each vector element of each vector by subtracting the corresponding average; and dividing each updated vector element of each vector by the corresponding standard deviation for each feature channel, wherein each vector element is an element of the element of the first feature map.

[0009] For some implementations of this example of a learning-based point cloud geometry encoding block method, the set of distribution parameters is determined using backpropagation during the training period.

[0010] For some implementations of this example of a learning-based point cloud geometry encoding block method, the set of distribution parameters is determined on a per-feature-channel basis.

[0011] Some implementations of the example of the learning-based point cloud geometry encoding block method may also include updating the second feature map by performing downsampling using a function of average pooling or max pooling.

[0012] Some implementations of the example of the learning-based point cloud geometry encoding block method may also include: determining a third feature map by filtering the second feature map using a smoothing filter; and updating the second feature map by concatenating the third feature map to the second feature map.

[0013] For some implementations of this example of a learning-based point cloud geometry encoding block method, the process may include: accessing a second set of distribution parameters before encoding the second feature map; and updating the second feature map by transforming it based on the second set of distribution parameters.

[0014] For some implementations of this example of a learning-based point cloud geometry encoding block method, the process may include aggregating the second feature map using a second neural network before encoding the second feature map.

[0015] For some implementations of the learning-based point cloud geometry encoding block method in this example, the second neural network is selected from a group consisting of sparse convolutional neural networks (CNN) and multiple perceptron layers (MLP).

[0016] For some implementations of this example of a learning-based point cloud geometry encoding block method, aggregating the second feature map may include using a residual network (ResNet) architecture.

[0017] Some implementations of the example of the learning-based point cloud geometry encoding block method may also include: determining a fourth feature map by aggregating the first feature map using a neural network in parallel with transforming the first feature map into the second feature map; and updating the second feature map by concatenating the fourth feature map to the second feature map.

[0018] For some implementations of this example of a learning-based point cloud geometry encoding block method, aggregating the first feature map using the neural network may include using a residual network (ResNet) architecture.

[0019] For some implementations of the learning-based point cloud geometry encoding block method in this example, the process includes aggregating the first feature map using a third neural network before transforming the first feature map into the second feature map.

[0020] For some implementations of the learning-based point cloud geometry encoding block method in this example, the third neural network is selected from a group consisting of sparse convolutional neural networks (CNN) and multiple perceptron layers (MLP).

[0021] According to some embodiments, an example of a learning-based point cloud geometry encoding block device may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the preceding claims.

[0022] Examples of learning-based point cloud geometry decoder methods, according to some implementation schemes, may include: decoding a first feature map from a bitstream; accessing a set of distribution parameters; transforming the first feature map into a second feature map based on the set of distribution parameters; and reconstructing the point cloud based on the second feature map.

[0023] Some implementations of this example of a learning-based point cloud geometry decoder method may also include updating the first feature map by normalizing the elements of the first feature map.

[0024] For some implementations of this example learning-based point cloud geometry decoder method, normalizing the element of the first feature map may include: determining the corresponding length of each feature vector associated with an element of the first feature map; and dividing each element of the first feature map by the corresponding length.

[0025] For some implementations of this example learning-based point cloud geometry decoder method, normalizing the element of the first feature map may include: arranging each of one or more vectors by reshaping the associated feature channels in the first feature map; determining a corresponding length for each of the one or more vectors for each feature channel; and dividing each vector element of each vector by the corresponding length for each feature channel, wherein each vector element is an element of the element of the first feature map.

[0026] For some implementations of this example learning-based point cloud geometry decoder method, normalizing the element of the first feature map may include: arranging each of one or more vectors by reshaping the associated feature channels in the first feature map; determining the corresponding standard deviation of each of the one or more vectors for each feature channel; determining the corresponding average of each of the one or more vectors for each feature channel; updating each vector element of each vector by subtracting the corresponding average; and dividing each updated vector element of each vector by the corresponding standard deviation for each feature channel, wherein each vector element is an element of the element of the first feature map.

[0027] For some implementations of this example of a learning-based point cloud geometry decoder method, the set of distribution parameters is determined using backpropagation during the training period.

[0028] For some implementations of this example of a learning-based point cloud geometry decoder method, the set of distribution parameters is determined on a per-feature-channel basis.

[0029] For some implementations of this example learning-based point cloud geometry decoder method, the process before encoding the second feature map includes: accessing a second set of distribution parameters; and updating the second feature map by transforming it based on the second set of distribution parameters.

[0030] For some implementations of this example learning-based point cloud geometry decoder method, the process includes aggregating the second feature map using a second neural network before encoding the second feature map.

[0031] For some implementations of this example of a learning-based point cloud geometry decoder method, the second neural network is selected from a group consisting of sparse convolutional neural networks (CNNs) and multiple perceptron layers (MLPs).

[0032] For some implementations of this example of a learning-based point cloud geometry decoder method, aggregating the second feature map may include using a residual network (ResNet) architecture.

[0033] For some implementations of this example learning-based point cloud geometry decoder method, the process includes aggregating the first feature map using a third neural network before transforming the first feature map into the second feature map.

[0034] For some implementations of this example of a learning-based point cloud geometry decoder method, the third neural network is selected from a group consisting of sparse convolutional neural networks (CNNs) and multiple perceptron layers (MLPs).

[0035] An example of a learning-based point cloud geometry decoder device according to some implementation schemes may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0036] The implementation schemes described herein include methods for video encoding and decoding (collectively, “decoding”).

[0037] In additional embodiments, encoder and decoder devices are provided to perform the methods described herein. The encoder or decoder device may include a processor configured to perform the methods described herein. The device may include a computer-readable medium (e.g., a non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, the computer-readable medium (e.g., a non-transitory medium) stores video encoded using any of the methods described herein.

[0038] One or more embodiments of the present invention also provide a computer-readable storage medium storing thereon instructions for performing bidirectional optical flow, encoding or decoding video data according to any of the methods described above. Embodiments of the present invention also provide a computer-readable storage medium storing thereon a bitstream generated according to the methods described above. Embodiments of the present invention also provide a method and apparatus for transmitting a bitstream generated according to the methods described above. Embodiments of the present invention also provide a computer program product including instructions for performing any of the methods. Attached Figure Description

[0039] Figure 1A This is a system diagram illustrating an example communication system based on some implementation schemes.

[0040] Figure 1B This illustrates, based on some implementation schemes, that... Figure 1A The diagram shows a system diagram of an example wireless transmit / receive unit (WTRU) used in a communication system.

[0041] Figure 1C It is a functional block diagram of a system based on some implementation schemes.

[0042] Figure 2A This is a functional block diagram of a block-based video encoder (such as an encoder for general video decoding (VVC)) according to some implementation schemes.

[0043] Figure 2B This is a functional block diagram of a block-based video decoder (such as a decoder for VVC) according to some implementation schemes.

[0044] Figure 3 This is a functional block diagram illustrating an example encoder architecture for point cloud (PC) feature extraction according to some implementation schemes.

[0045] Figure 4 This is a functional block diagram illustrating, according to some implementation schemes, how the PC group size K can be determined for a given voxel quantization for PC reconstruction in a point cloud compression (PCC) framework.

[0046] Figure 5 This is a functional block diagram illustrating point-by-point feature extraction based on some implementation examples.

[0047] Figure 6 This is a function block diagram illustrating the example group characteristic distribution tensor based on some implementation schemes.

[0048] Figure 7 This is a functional block diagram illustrating the distribution of example group features with learnable transformation parameters according to some implementation schemes.

[0049] Figure 8 This is a functional block diagram illustrating example transformations of feature aggregation, enhancement, and dimension matching based on some implementation schemes.

[0050] Figure 9 This is a functional block diagram illustrating pointwise residual feature extraction and cluster-wise feature aggregation and enhancement based on some implementation schemes.

[0051] Figure 10 This is a functional block diagram illustrating an example encoder architecture for PC feature extraction according to some implementation schemes.

[0052] Figure 11 This is a functional block diagram illustrating an example decoder architecture based on some implementation schemes.

[0053] Figure 12A This is a functional block diagram illustrating the encoder and decoder in lossy geometry compression based on an autoencoder.

[0054] Figure 12B This is a functional block diagram illustrating an example application of the PCC framework in encoders and decoders in autoencoder-based lossy geometry compression, according to some implementation schemes.

[0055] Figure 13A This is a functional block diagram illustrating point analysis and point composition within the extensible PCC framework.

[0056] Figure 13B This is a functional block diagram illustrating example applications of the PCC framework in point analysis and point composition within a scalable PCC framework, based on some implementation schemes.

[0057] Figure 14A This is a function block diagram illustrating the Collection Abstraction (SA) process in the PointContextNet environment.

[0058] Figure 14B This is a functional block diagram illustrating an example application of the PCC framework in the Collection Abstraction (SA) process within the PointContextNet environment, based on some implementation schemes.

[0059] Figure 15A This is a function block diagram illustrating the Set Abstraction (SA) process in the Coordinate Refinement Module (CRM).

[0060] Figure 15B This is a functional block diagram illustrating an example application of the PCC framework in the Set Abstraction (SA) process within the Coordinate Refinement Module (CRM) according to some implementation schemes.

[0061] Figure 16 This is a flowchart illustrating an example process for point cloud feature extraction according to some implementation schemes.

[0062] Figure 17 This is a functional block diagram illustrating a depth-feature-based PCC pipeline according to some implementation schemes.

[0063] Figure 18 This is a functional block diagram illustrating the DDA-Net encoder architecture according to some implementation schemes.

[0064] Figure 19 This is a functional block diagram illustrating a probability estimator distribution network based on some implementation schemes.

[0065] Figure 20 This is a flowchart illustrating an example process for point cloud feature extraction according to some implementation schemes.

[0066] Figure 21 This is a flowchart illustrating an example process for point cloud feature extraction according to some implementation schemes.

[0067] Figure 22 This is a flowchart illustrating an example of a learning-based point cloud geometry process based on some implementation schemes.

[0068] Figure 23 This is a flowchart illustrating an example of a learning-based point cloud geometry process based on some implementation schemes.

[0069] The entities, connections, arrangements, etc., shown in and described in conjunction with the various figures are presented by way of example rather than limitation. Therefore, any and all statements or other indications concerning what a particular figure “depicts,” what a particular element or entity in a particular figure “is” or “has,” and any and all similar states that may be interpreted in isolation and outside the context as absolute and therefore limiting, may only be appropriately interpreted in a constructive manner with a clause such as “In at least one embodiment, …”. For the sake of brevity and clarity, this implicit leading clause is not repeated in the detailed description. Detailed Implementation

[0070] Example network for implementing the implementation scheme

[0071] In some of the implementations described herein, the wireless transmit / receive unit (WTRU) can be used, for example, to perform point cloud (PC) extraction.

[0072] Figure 1A This is a diagram illustrating an example communication system 100 that may implement one or more of the disclosed embodiments. Communication system 100 may be a multiple access system providing content such as voice, data, video, messaging, and broadcasting to multiple wireless users. Communication system 100 enables multiple wireless users to access such content through the sharing of system resources (including wireless bandwidth). For example, communication system 100 may employ one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tail Unique Word DFT Extended OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, and Filter Bank Multicarrier (FBMC), etc.

[0073] like Figure 1A As shown, the communication system 100 may include wireless transceiver units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106, Public Switched Telephone Network (PSTN) 108, Internet 110, and other networks 112. However, it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a wireless environment. As an example, WTRUs 102a, 102b, 102c, and 102d (any of which may be referred to as a "station" and / or "STA") may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain environments), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. Any of WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0074] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a and 114b can be any type of device configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks (such as CN 106, Internet 110, and / or other networks 112). By way of example, base stations 114a and 114b can be transceiver base stations (BTS), Node B, evolved Node B, home Node B, home evolved Node B, gNB, NR Node B, site controller, access point (AP), wireless router, etc. Although base stations 114a and 114b are each depicted as a single element, it should be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0075] Base station 114a may be part of RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies (which may be referred to as cells (not shown)). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage of radio services to a specific geographic area, which may be relatively fixed or changeable over time. The cell may be further divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In embodiments, base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in desired spatial directions.

[0076] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116, which can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Any suitable radio access technology (RAT) can be used to establish air interface 116.

[0077] More specifically, as noted above, communication system 100 can be a multiple access system and can employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, and SC-FDMA. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN 104 / 113 can implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can use Wideband CDMA (WCDMA) to establish the air interface 116. WCDMA may include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0078] In the implementation scheme, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as evolved UMTS terrestrial radio access (E-UTRA), which can use Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-APro) to establish air interface 116.

[0079] In the implementation scheme, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as NR radio access, which can use New Radio (NR) to establish air interface 116.

[0080] In the implementation scheme, base station 114a and WTRUs 102a, 102b, and 102c can implement multiple radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can, for example, use the dual connectivity (DC) principle together to implement LTE radio access and NR radio access. Therefore, the air interface used by WTRUs 102a, 102b, and 102c can be characterized by multiple types of radio access technologies and / or transmissions to / from multiple types of base stations (e.g., eNBs and gNBs).

[0081] In other implementations, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate GSM Evolution (EDGE), and GSM EDGE (GERAN).

[0082] Figure 1A Base station 114b can be, for example, a wireless router, a home node B, a home evolution node B, or an access point, and can utilize any suitable RAT to facilitate wireless connectivity in local areas such as commercial locations, homes, vehicles, campuses, industrial facilities, air corridors (e.g., for use by drones), and roads. In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d can utilize cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish picocells or femtocells. Figure 1A As shown, base station 114b may have a direct connection to Internet 110. Therefore, base station 114b may not need to access Internet 110 via CN 106.

[0083] RAN 104 / 113 can communicate with CN 106, which can be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRUs 102a, 102b, 102c, and 102d. Data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, and mobility requirements. CN 106 can provide call control, billing services, location-based services, prepaid calling, internet connectivity, video distribution, and / or perform advanced security functions such as user authentication. Although not explicitly stated... Figure 1AAs shown, but it should be understood that RAN 104 / 113 and / or CN 106 can communicate directly or indirectly with other RANs that use the same RAT as RAN 104 / 113 or a different RAT. For example, in addition to being connected to RAN 104 / 113 which can utilize NR radio technology, CN 106 can also communicate with another RAN (not shown) that uses GSM, UMTS, CDMA 2000, WiMAX, E-UTRA or WiFi radio technology.

[0084] CN 106 may also act as a gateway for WTRUs 102a, 102b, 102c, and 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as Transmit Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN 104 / 113 or a different RAT.

[0085] Some or all of the WTRUs 102a, 102b, 102c, and 102d in communication system 100 may include multi-mode capability (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example, Figure 1A The WTRU 102c shown can be configured to communicate with a base station 114a that can employ cellular-based radio technology and with a base station 114b that can employ IEEE 802 radio technology.

[0086] Figure 1B This is a system diagram illustrating the example WTRU 102. For example... Figure 1B As shown, WTRU 102 may include a processor 118, a transceiver 120, a transmitting / receiving element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a Global Positioning System (GPS) chipset 136, and / or other peripheral devices 138, etc. It should be understood that, while remaining consistent with the implementation, WTRU 102 may include any sub-combination of the foregoing elements.

[0087] Processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. Processor 118 may perform signal decoding, data processing, power control, input / output processing, and / or any other functionality that enables WTRU 102 to operate in a wireless environment. Processor 118 may be coupled to transceiver 120, which may be coupled to transmitting / receiving element 122. Although Figure 1B The processor 118 and transceiver 120 are depicted as separate components, but it should be understood that the processor 118 and transceiver 120 may be integrated together in an electronic package or chip.

[0088] Transmitting / receiving element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via air interface 116. For example, in one embodiment, transmitting / receiving element 122 may be an antenna configured to transmit and / or receive RF signals. In another embodiment, transmitting / receiving element 122 may be a transmitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, transmitting / receiving element 122 may be configured to transmit and / or receive both RF signals and optical signals. It should be understood that transmitting / receiving element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0089] Although the transmitting / receiving element 122 is in Figure 1B While depicted as a single element, WTRU 102 may include any number of transmitting / receiving elements 122. More specifically, WTRU 102 may employ MIMO technology. Thus, in one embodiment, WTRU 102 may include two or more transmitting / receiving elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals via air interface 116.

[0090] Transceiver 120 can be configured to modulate signals to be transmitted by transmitting / receiving element 122 and demodulate signals to be received by transmitting / receiving element 122. As noted above, WTRU 102 may have multi-mode capability. For example, transceiver 120 may therefore include multiple transceivers to enable WTRU 102 to communicate via multiple RATs (such as NR and IEEE 802.11).

[0091] The processor 118 of WTRU 102 can be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit) and can receive user input data therefrom. The processor 118 can also output user data to the speaker / microphone 124, keypad 126, and / or display / touchpad 128. Additionally, the processor 118 can access information from any type of suitable memory (such as non-removable memory 130 and / or removable memory 132) and store data in such suitable memory. Non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, and a secure digital storage (SD) card, etc. In other embodiments, processor 118 may access information from memory that is not physically located on WTRU 102 (such as on a server or home computer (not shown)) and store the data in that memory.

[0092] The processor 118 may receive power from the power supply 134 and may be configured to distribute and / or control power to other components in the WTRU 102. The power supply 134 may be any suitable device for powering the WTRU 102. For example, the power supply 134 may include one or more dry cell battery packs (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, and fuel cells, etc.

[0093] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to or instead of the information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It should be understood that, while remaining consistent with the implementation, the WTRU 102 may acquire location information using any suitable location determination method.

[0094] The processor 118 may also be coupled to other peripheral devices 138, which may include one or more software and / or hardware modules providing additional features, functionality, and / or wired or wireless connectivity. For example, peripheral device 138 may include an accelerometer, electronic compass, satellite transceiver, digital camera (for photos and / or video), Universal Serial Bus (USB) port, vibration device, television transceiver, hands-free headset, etc. Modules, FM radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, and activity trackers, etc. Peripheral devices 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors; geolocation sensors; altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors.

[0095] WTRU 102 may include a full-duplex radio for which the transmission and reception of some or all signals (e.g., associated with specific subframes for both UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference through signal processing via hardware (e.g., a choke) or via a processor (e.g., a separate processor (not shown) or via processor 118). In an embodiment, WTRU 102 may include a half-duplex radio for which the transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)) may be concurrent and / or simultaneous.

[0096] Despite WTRU in Figures 1A to 1B While described as a wireless terminal, it is conceivable that in some representative implementations, such a terminal may (e.g., temporarily or permanently) use a wired communication interface with a communication network.

[0097] In a representative implementation, the other network 112 may be a WLAN.

[0098] Given Figures 1A to 1B As described herein, one or more of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to mimic one or more or all of the functions described herein. For example, an emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0099] Simulation devices can be designed to perform one or more tests on other devices in a laboratory environment and / or a carrier network environment. For example, one or more simulation devices may perform one or more functions or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. One or more simulation devices may perform one or more functions or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. Simulation devices may be directly coupled to another device for testing purposes and / or may use over-the-air wireless communication to perform tests.

[0100] One or more emulation devices may perform one or more (including all) functions without being implemented / deployed as part of a wired and / or wireless communication network. For example, emulation devices may be used in test scenarios within a test laboratory and / or non-deployed (e.g., testing) wired and / or wireless communication networks to perform testing of one or more components. One or more emulation devices may be test rigs. Direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0101] Example System

[0102] The implementation described herein is not limited to implementation on WTRU. Such implementations can use other systems (such as, Figure 1C This is achieved through a system (such as a .). Figure 1C This is a block diagram illustrating examples of systems implementing various aspects and embodiments. System 150 may be embodied as a device including the various components described below and configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 150 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various embodiments, system 150 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 150 is configured to implement one or more aspects described in this document.

[0103] System 150 includes at least one processor 152 configured to execute instructions loaded therein for implementing various aspects, such as those described in this document. Processor 152 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 150 includes at least one memory 154 (e.g., a volatile memory device and / or a non-volatile memory device). System 150 includes a storage device 158 that may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 158 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0104] System 150 includes an encoder / decoder module 156 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 156 may include its own processor and memory. The encoder / decoder module 156 represents one or more modules that may be included in the device to perform encoding and / or decoding functions. It is well known that a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 156 may be implemented as a separate element of system 150, or it may be incorporated within processor 152 as a combination of hardware and software known to those skilled in the art.

[0105] Program code to be loaded onto processor 152 or encoder / decoder 156 to execute the various aspects described in this document may be stored in storage device 158 and subsequently loaded onto memory 154 for execution by processor 152. According to various embodiments, one or more of processor 152, memory 154, storage device 158, and encoder / decoder module 156 may store one or more items from various projects during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0106] In some embodiments, the memory within processor 152 and / or encoder / decoder module 156 is used to store instructions and provide working memory for processing during encoding or decoding. However, in other embodiments, external memory (e.g., processor 152 or encoder / decoder module 156) is used for one or more of these functions. This external memory may be memory 154 and / or storage device 158, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one implementation, a fast external dynamic volatile memory (such as RAM) is used as working memory for video decoding and decoding operations, such as MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Decoding; also known as H.265 and MPEG-H Part 2), or VVC (Universal Video Decoding, a new standard developed by the Joint Video Experts Group (JVET)).

[0107] Input to the components of system 150 can be provided through various input devices as indicated in box 1130. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster; (ii) a component input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1C Other examples not shown include composite video.

[0108] In various embodiments, the input device of block 1130 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as a selection signal, or limiting a signal band to a band), (ii) down-converting the selected signal, (iii) again limiting the band to a narrower band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various functions among these functions, including, for example, down-converting the received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or to baseband. In one set-top box implementation, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to the desired frequency band. Various implementations rearrange the order of the aforementioned (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as, for example, inserting amplifiers and analog-to-digital converters. In various implementations, the RF section includes an antenna.

[0109] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting system 150 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, if necessary, within a separate input processing IC or within processor 152. Similarly, various aspects of USB or HDMI interface processing may be implemented, if necessary, within a separate interface IC or within processor 152. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 152 and encoder / decoder 156, which operate in conjunction with memory and storage elements to process the data stream as needed for presentation on the output device.

[0110] Various components of system 150 may be housed within an integrated housing, in which the various components may be interconnected and transmit data between the components using a suitable connection arrangement 1140 (e.g., an internal bus as known in the art, including inter-chip (I2C) bus, wiring, and printed circuit board).

[0111] System 150 includes a communication interface 160 that enables communication with other devices via a communication channel 162. The communication interface 160 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 162. The communication interface 160 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 162 may be implemented, for example, in a wired and / or wireless medium.

[0112] In various implementations, a wireless network (such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) is used to stream or otherwise provide data to system 150. The Wi-Fi signal in these implementations is received via a communication channel 162 and a communication interface 160 suitable for Wi-Fi communication. The communication channel 162 in these implementations is typically connected to an access point or router that provides access to external networks (including the Internet) to allow streaming applications and other over-the-top communications. Other implementations use a set-top box to provide streaming data to system 150, delivering data via an HDMI connection to input box 1130. Still other implementations use an RF connection to input box 1130 to provide streaming data to system 150. As mentioned above, various implementations provide data in a non-streaming manner. Additionally, various implementations use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0113] System 150 can provide output signals to various output devices, including display 170, speaker 172, and other peripheral devices 174. Display 170 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 170 can be used in televisions, tablets, laptops, cellular phones (mobile phones), or other devices. Display 170 can also be integrated with other components (e.g., as in a smartphone) or standalone (e.g., an external monitor for a laptop). In various examples of embodiments, other peripheral devices 174 include one or more of a standalone digital video disc (or digital versatile disc) (DVR, applicable to both terms), an optical disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 174 that provide functionality based on the output of system 150. For example, an optical disc player performs the function of playing the output of system 150.

[0114] In various embodiments, control signals are communicated between system 150 and display 170, speaker 172, or other peripheral devices 174 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to system 150 via dedicated connections through corresponding interfaces 164, 166, and 168. Alternatively, output devices may be connected to system 150 via communication interface 160 using communication channel 162. Display 170 and speaker 172 may be integrated into a single unit with other components of system 150 in electronic devices such as, for example, televisions. In various embodiments, display interface 164 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0115] For example, if the RF portion of input 1130 is part of a standalone set-top box, then display 170 and speaker 172 may alternatively be independent of one or more other components. In various embodiments where display 170 and speaker 172 are external components, output signals may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0116] These implementations can be executed by computer software implemented by processor 152, by hardware, or by a combination of hardware and software. As a non-limiting example, these implementations can be implemented by one or more integrated circuits. As a non-limiting example, memory 154 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical storage devices, magnetic storage devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 152 can be of any type suitable for the technical environment and can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0117] Block-based video decoding

[0118] Similar to HEVC, VVC is built on a block-based hybrid video decoding framework. Figure 2A A block diagram of a block-based hybrid video coding system 200 is given. Variations of the encoder 200 are envisioned, but for clarity, the encoder 200 is described below without describing all anticipated variations.

[0119] Before being encoded, the video sequence may undergo pre-coding processing (204), such as applying color transformations to the input color image (e.g., a transformation from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of the input image components to make the signal distribution more resilient to compression (e.g., using histogram equalization of one color component). Metadata may be associated with the pre-processing and attached to the bitstream.

[0120] The input video signal, including the image to be encoded, is partitioned 202 (206) and processed block by block, for example, in units of CUs. Different CUs can have different sizes. In VTM-1.0, CUs can be up to 128×128 pixels. However, unlike HEVC, which partitions blocks solely based on quadtrees, in VTM-1.0, the decode tree unit (CTU) is split into CUs to accommodate the local characteristics of variations based on quadtrees / binaries / tritrees. Additionally, the concept of multiple partition unit types in HEVC is removed, so the separation of CUs, prediction units (PUs), and transform units (TUs) no longer exists in VVC-1.0; instead, each CU is always used as the basic unit for both prediction and transform without further partitioning. In a multi-type tree structure, the CTU is first partitioned by a quadtree structure. Then, each quadtree leaf node can be further partitioned by binary and ternary tree structures. Different partitioning types can be used, such as quad partitioning, vertical binary partitioning, horizontal binary partitioning, vertical ternary partitioning, and horizontal ternary partitioning.

[0121] exist Figure 2A In the encoder, spatial prediction (208) and / or temporal prediction (210) can be performed. Spatial prediction (or “intra-frame prediction”) uses pixels from samples (referred to as reference samples) of neighboring blocks decoded in the same video picture / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also referred to as “inter-frame prediction” or “motion-compensated prediction”) uses reconstructed pixels from decoded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU can be signaled by one or more motion vectors (MVs) indicating the amount and direction of motion between the current CU and its temporal reference. Moreover, if multiple reference pictures are supported, a reference picture index can be additionally transmitted to identify which reference picture in the reference picture repository (212) the temporal prediction signal comes from.

[0122] The mode decision block (214) in the encoder selects the optimal prediction mode, for example, based on a rate distortion optimization method. This selection may be made after performing spatial and / or temporal predictions. Intra-frame / inter-frame decisions may be indicated, for example, by a prediction mode flag. The prediction block is subtracted from the current video block (216) to generate a prediction residual. The prediction residual is decorrelated using a transform (218) and quantization (220). (For some blocks, the encoder may bypass both the transform and quantization, in which case the residual can be directly decoded without applying the transform or quantization process.) The quantized residual coefficients are inversely quantized (222) and inversely transformed (224) to form a reconstructed residual, which can then be added back to the prediction block (226) to form the reconstructed signal of the CU. Further loop filtering, such as deblocking / SAO (sample adaptive offset) filtering, may be applied (228) to the reconstructed CU to reduce coding artifacts before they are placed in the reference picture repository (212) and used for decoding future video blocks. In order to form the output video bitstream 230, the decoding mode (inter-frame or intra-frame), prediction mode information, motion information and quantized residual coefficients are transmitted to the entropy decoding unit (108) for further compression and packing to form the bitstream.

[0123] Figure 2B A block diagram of a block-based video decoder 250 is given. In decoder 250, the bitstream is decoded by decoder elements as described below. Video decoder 250 typically performs decoding processing, which is the reverse of encoding processing, such as... Figure 2A The encoder 200 typically also performs video decoding as part of the encoding of video data.

[0124] Specifically, the decoder's input includes a video bitstream 252, which can be generated by the video encoder 200. The video bitstream 252 is first unpacked and entropy decoded at the entropy decoding unit 254 to obtain transform coefficients, motion vectors, and other decoding information. Picture partitioning information indicates how the picture is partitioned. Therefore, the decoder can partition (256) the picture based on the decoded picture partitioning information. Decoding mode and prediction information are passed to the spatial prediction unit 258 (if it is intra-frame decoding) or the temporal prediction unit 260 (if it is inter-frame decoding) to form a prediction block. The residual transform coefficients are passed to the inverse quantization unit 262 and the inverse transform unit 264 to reconstruct the residual block. The prediction block and the residual block are then added together at 266 to generate a reconstructed block. The reconstructed block may further undergo loop filtering 268 before being stored in the reference picture repository 270 for predicting future video blocks.

[0125] The decoded image 272 may also undergo post-decoding processing (274), such as inverse color transformation (e.g., a transformation from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in pre-encoding processing (204). Post-decoding processing may utilize metadata exported in pre-encoding processing and signaled in the bitstream. The decoded processed video may be transmitted to display device 276. Display device 276 may be a separate device from decoder 250, or decoder 250 and display device 276 may be components of the same device.

[0126] The various methods and other aspects described in this disclosure can be used to modify modules of the video encoder 200 or decoder 250. Furthermore, the systems and methods disclosed herein are not limited to VVC or HEVC and can be applied to, for example, other standards and recommendations (whether pre-existing or future-developed) and any extensions to such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this disclosure may be used alone or in combination.

[0127] This disclosure discusses point cloud compression and processing, which may include tools for compressing, analyzing, interpolating, representing, and understanding point cloud signals.

[0128] Point cloud data can consume significant amounts of network bandwidth, for example, in vehicles connected via 5G networks and in immersive communications (VR / AR / MR). Efficient representation formats can be used for point cloud communication. Specifically, raw point cloud data can be organized and processed for modeling and sensing of things such as the world, environment, or scene. Compression of the raw point cloud can be used for data storage and transmission.

[0129] Furthermore, point clouds can represent sequential scans of a scene that may contain multiple moving objects. Such point clouds are called dynamic point clouds, in contrast to static point clouds that can be captured from static scenes and / or static objects. Dynamic point clouds can be organized into frames, where different frames are captured at different times. Processing and compression of dynamic point clouds can be performed in real-time or with minimal latency.

[0130] The automotive industry (including, for example, autonomous vehicles) can utilize point clouds. Autonomous vehicles "detect" their environment to make driving decisions based on their surroundings. Typically, LiDAR sensors generate (dynamic) point clouds for use by perception engines. Furthermore, these point clouds are often dynamic, have a high capture frequency, are sparse, not necessarily colored, and are not visible to the human eye. Such point clouds can include other properties, such as reflectivity provided by LiDAR, which can indicate the material of the sensed object and can be used to make decisions.

[0131] Virtual reality (VR) and immersive worlds have become hot topics and are seen by many as the future of 2D flat video. Viewers can be immersed in an all-around environment, unlike standard television, where viewers can only see the virtual world in front of them. The sense of immersion has several levels depending on the viewer's degree of freedom within the environment. Point cloud formats can be used to distribute VR world and environmental data. Such point clouds can be static or dynamic and typically have an average size, such as fewer than a few million points at a time.

[0132] Point clouds can also be used for a variety of other purposes, such as scanning cultural heritage objects and / or buildings, where objects such as statues or buildings are scanned in 3D. Spatial configuration data of objects can be shared without the need to transport or access the actual objects or buildings. Moreover, this data can be used to preserve knowledge of an object even if it has been damaged (e.g., a temple destroyed by an earthquake). Such point clouds are typically static, colored, and very large.

[0133] Another use case is in topographic maps and cartography using 3D representations, where the map is not limited to a plane and can include topographic relief. For example, Google Maps may use grids instead of point clouds for its 3D maps. However, point clouds can be a suitable data format for 3D maps, and such point clouds are typically static, colored, and very large.

[0134] World modeling and sensing via point clouds allows machines to record and use spatial configuration data about the 3D world around them, which can be used in the applications discussed above.

[0135] 3D point cloud data consists of discrete samples of the surfaces of objects or scenes. To adequately represent the real world with these point samples, a massive number of points can be used. For example, a typical VR immersive scene includes millions of points, while a point cloud can often contain hundreds of millions. Therefore, processing such large-scale point clouds is computationally expensive, especially for consumer devices with potentially limited computing power, such as smartphones, tablets, and car navigation systems.

[0136] Any processing or inference of point clouds can utilize efficient storage techniques. To store and process the input point cloud at an affordable computational cost, it can be downsampled, where the downsampled point cloud summarizes the geometry of the input point cloud while having far fewer points. The downsampled point cloud is then fed into subsequent machine tasks for further processing. However, further reductions in storage space can be achieved by converting the original point cloud data (raw or downsampled) into a bitstream via entropy decoding techniques used for lossless compression.

[0137] Besides lossless decoding, many scenarios use lossy decoding to significantly improve compression ratios while maintaining induced distortion at a specific quality level. To achieve decoding with lower loss, efficient point feature extractors can be used to improve the accuracy of reconstruction within a given resource budget.

[0138] Several articles have demonstrated interest in applying sparse convolutions, including: Benjamin Graham's *Sparse 3D Convolutional Neural Networks*, ARXIV preprint, arXiv:1505.02890 (2015); Baoyuan Liu et al.'s *Sparse Convolutional Neural Networks*, PROC. OF THE IEEE CONFERENCE ON COMPUTER VISION AND PATTERNRECOGNITION, 806-814 (2015); Benjamin Graham et al.'s *3D Semantic Segmentation with Submanifold Sparse Convolutional Networks*, IEEE Conference on Computer Vision and Pattern Recognition, 9224-9232 (2018); and Christopher Choy et al.'s 4D... Spatio-Temporal Convnets: Minkowski Convolutional Neural Networks (4D Spatio-Temporal Convolutional Networks: Minkowski Convolutional Neural Networks), Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3075-3084 (2019). Along with these articles, point feature extraction from point cloud (PC) data has been used in AI-based PC analysis, such as classification, segmentation, registration, and compression. Point cloud compression (PCC) in such applications may involve a trade-off between complexity (e.g., computational cost or storage consumption) and performance (e.g., accuracy) for PC reconstruction, employing a balanced architecture between the encoder and decoder.

[0139] Local group analysis allows for the extraction of more representative features, but the method should not be overly complex due to the encoding and decoding time in the PCC framework. Furthermore, since storage capacity may be limited, the point feature extractor should allow for deeper architectures without significantly increasing the feature dimensionality.

[0140] Recent AI-based end-to-end frameworks and deep entropy models for point cloud compression (PCC) have focused primarily on the application of sparse voxel convolutions, with less emphasis on point-based feature extraction from point cloud data. Point-wise feature analysis can play a greater role as the bit depth of the input data increases. Furthermore, the geometric representation of the point cloud influences the ability to efficiently decompress highly abstract features to lower bit rates without incurring significant computational costs. As such point cloud datasets continue to grow, efficient AI-based feature extractors for PCC can be utilized.

[0141] The paper "Pointnet: Deep Learning on Point Sets for 3D Classification and Segmentation" by Charles R. Qi et al. (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 652-660 (2017)) discusses the PointNet architecture for point-based feature extractors. The PointNet architecture consists of a series of point-wise fully connected multilayer perceptron (MLP) layers with specific feature dimensions, followed by pooling of all points. The PointNet architecture lacks feature details, such as local geometric information.

[0142] The PointNet++ architecture described in Charles R. Qi et al.'s *Pointnet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space* (Advances in Neural Information Processing Systems, 30 (2017)) introduces a set abstraction layer within this architecture. This architecture utilizes local geometric information in a hierarchical manner; however, each sampling process requires sampling (e.g., farthest point sampling) followed by grouping functions (e.g., ball lookup). Furthermore, a small PointNet needs to be run for each set abstraction layer, which may incur some computational cost.

[0143] The PointNet and PointNet++ methods described above extract features from the locations of given discrete points. To generalize the problem, points from 3D space are randomly sampled, and then the function is approximated to give the probability of a point being occupied at any given coordinate in 3D space. In Lars Mescheder et al.'s paper "Occupancy Networks: Learning 3D Reconstruction in Function Space" (IEEE / CVF Conference Proceedings on Computer Vision and Pattern Recognition, 4460-4470 (2019)), the occupancy network attempts to learn this non-discrete function with the help of conditional batch normalization parameters fed to the occupancy probability generator at each corresponding step. To better extract fine details from points and surfaces, the paper *Convolutional Occupancy Networks* (EUROPEAN CONFERENCE ON COMPUTER VISION, 523-540 (2020)) by Songyou Peng et al. mentions that occupancy networks are further improved by adding U-Net-like convolutional layers before fully connected layers. However, these methods are too complex and may be difficult to deploy within the PCC framework. The paper *Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework* (ARXIV preprint, arXiv:2202.07123 (2022)) by Xu Ma et al. discusses the PointMLP architecture. The paper *Surface Representation for Point Clouds* (ARXIV preprint, arXiv:2205.05740 (2022)) by Haoxi Ran et al. discusses the RepSurf architecture. These architectures are introduced to improve the ensemble abstraction layer. The PointMLP architecture introduces a hierarchical, multi-level architecture with an affine local geometry extractor. The RepSurf architecture adds local information to point cloud data, such as triangles and umbrella orientations. All of these methods require costly multi-stage sampling and grouping processes.

[0144] The methods described above are designed for point cloud classification or segmentation problems. Due to the trade-off between accuracy and complexity, they are not entirely suitable for the PCC framework. Furthermore, the optimized loss function (rate and distortion) for PCC differs from the optimized loss function (cross-entropy) used for classification problems.

[0145] The paper "Deep Autoencoder-Based Lossy Geometry Compression for Point Clouds" by Wei Yan et al. (ARXIV preprint, arXiv:1905.03691 (2019)) ("Yan") introduces an autoencoder-based PCC network. An MLP with aggregation layers is abstracted to obtain the global features that are ultimately fed to the entropy encoder. On the decoder side, the decoded feature codes are input into the MLP decoder, and the point cloud is reconstructed. Based on the PointNet network, Yan applies an example point feature extractor within an end-to-end PCC framework.

[0146] The '482 application describes, for example, the PointContextNet algorithm, while the '894 application describes, for example, the Coordinate Refinement Module (CRM) algorithm. Both the '482 and '894 applications use their respective example algorithms to perform deep entropy decoding within, for example, an end-to-end (AI-based) PCC network, to apply, for example, a series of set abstraction (SA) layers with layer (costly) sampling and grouping that utilizes hierarchical point feature analysis.

[0147] The '869 application, for example, introduces a scalable PCC framework by implementing a feature extractor using sparse convolutional networks (in both the encoder and decoder). For instance, the feature extractor in this end-to-end framework handles local point group information by integrating a K-nearest neighbor-oriented analysis for each point. The microarchitecture used in some examples can be enhanced in the domain of feature extraction.

[0148] According to some implementations, this application introduces a series of point cloud (PC) feature extraction processes that take 3D points or features as input, and in some implementations, these processes extract cluster-wise feature points at the encoder output. While such an architecture is suitable for any point cloud feature extraction for various purposes, such as point cloud classification, point cloud model segmentation, point cloud flow analysis, and autonomous driving applications, this architecture is adapted for an AI-based point cloud compression (PCC) framework. The overall process and architectural details based on some examples are described first, followed by an explanation of how these processes are applied within the PCC framework.

[0149] The point feature extractor described below, according to some embodiments, is applicable to all example PCC frameworks described above. For the architecture of Yan and '869 applications, the architecture and process of this application enhance the representation of point features on both the encoder and decoder sides. For the example architectures of '482 and '894 applications, the example architecture and process of this application, according to some embodiments, simplifies the complexity of computing local geometric information.

[0150] Figure 3 This is a functional block diagram illustrating an example encoder architecture for feature extraction from point clouds (PCs) according to some implementation schemes. Features are multidimensional vectors representing the outputs of neural network layers. In point cloud applications, these vectors are typically extracted point-by-point by an MLP network. Figure 3 The diagram illustrates a general example architecture 300 of this application according to some implementation schemes. One version of the feature extractor is a single point bounding box 306. A more advanced version queries group points 304, extracts features from all points 306, aggregates by group points, and runs group-by-group feature extraction 318 to generate group features 320. Both versions of the example architecture can perform cost-effective feature extraction, but may lack accuracy and detailed feature representation.

[0151] Features, such as pointwise or groupwise features, can be represented as high-dimensional vectors. A feature is a tensor of three-dimensional points embedded in a higher-dimensional space. For some implementations, performing an embedding machine learning process may be a necessary step. For some implementations, a feature is a set of three-dimensional points in a 3D point cloud model of an environment or scene. In some implementations, features can be generated through a machine learning process. In some implementations, feature tensors can be used as... Figure 3 The point cloud input 302 is used for point cluster query 304. For some implementations, the feedback can be expressed using a multi-level architecture. A multi-level architecture can be one example among many examples of a learning-based point cloud compression (PCC) framework. For some implementations, the feature is the data processed through a learning-based PCC framework.

[0152] This application focuses on higher-level feature aggregation to better (e.g., more accurately) represent PC features. In some embodiments, a depth distribution-aware point feature extractor 306 may be used. In some embodiments, point features 308 may be input to a transformation of the group feature distribution. In some embodiments, a transformation 310 is performed on the group feature distribution, and the transformed features are aggregated and matched 312. Furthermore, local and global features based on depth residuals 314, 316 may be combined to further enhance feature extraction. The order of these processes 310, 312, 314, 316 may be cascaded more deeply to further improve the quality of feature representation. The dashed lines in the aggregation / enhancement box 316 indicate branches for additional outputs of the enhancement process. Enhancement is used if a new series continues via feedback loops 316, 310. Figure 9 More details about this branch are illustrated in the example.

[0153] For some implementations, transform feature aggregation matching (dashed process 312) is another layer that can be added to blend group features. For some implementations, the group feature distribution transformation process 310 can directly receive point features instead of embedded 3D points via pointwise feature extraction 306. For some implementations, point group query (box (a)) can use an optimal number of group points to obtain better (e.g., more accurate) reconstruction quality. For some implementations, Figure 3 The feedback path can be derived from group feature aggregation and augmentation boxes ( Figure 3 The output of 316) is passed to the pointwise residual network ( Figure 3 The input is 314). This is possible because... Figure 3 Boxes 310 and 314 in the diagram use the same feature dimension as input.

[0154] Group queries and analysis can classify the shape of a local neighborhood around a query location. Utilizing such local neighborhoods / groups to process features allows for leveraging the hierarchical structure of point clouds and typically performs more accurate feature extraction using PC.

[0155] For PC classification or segmentation, downsampling is often accompanied by group queries. This operation can be applied to PCC frameworks, such as end-to-end compression architectures, where PC quantization results in point-wise downsampling. For some implementations, any grouping algorithm can be applied, such as ball queries or the K-nearest neighbor (KNN) algorithm. Although ball queries extract more accurate geometric details, KNN may be an efficient and more straightforward process for PCC frameworks.

[0156] Figure 4 This is a functional block diagram illustrating, according to some implementation schemes, how the PC group size K can be determined for a given voxel quantization for PC reconstruction in a point cloud compression (PCC) framework. Figure 4The example process 400 shown can efficiently select the group size K based on the quantization step size and the density of occupied voxels 404 in the grid. Figure 4 The grid 402 on the left is the original voxel representation of the 4-bit PC model. Figure 4 The left side 402 shows the original (occupied) point cloud voxels before quantization. Figure 4 The right side 408 shows the quantized voxel with query point 410. Both occupied point 404 and query point 410 are considered to reside at the center of the corresponding voxel. By using quantization 406 with a step size of, for example, 4, the original 4×4×4 voxel mesh is quantized to 1×1×1 voxels, which in Figure 4 As shown in grid 408 on the right. All occupied points (circles) in the original grid 402 are represented by individual voxel points (stars) in the quantized grid 408. In some implementations, quantization or voxel quantization 406 can be used to perform grouping, and features can be extracted from the resulting groups.

[0157] The average number of original points at each quantized grid point is the size of the neighbors processed for a given quantization step. In other words, the total number of points in the point cloud model, K. avg It is the average number of original points. Divide by the average number of quantization points

[0158]

[0159] in and It is the number of original points in the i-th point cloud model, and K is the number of quantized points in the i-th point cloud model. The sum can be calculated across S point cloud models selected from the point cloud training dataset. For some implementations, the point cloud training dataset may be, for example, one million point cloud models, and only, for example, a few thousand point cloud models are used to calculate the average. The obtained K avg The trade-off between computational cost and reconstruction accuracy is quantified. For some implementations, the best K neighbors are the average number of occupied voxels merged into the quantized voxels (the number of original points divided by the number of quantized points).

[0160] In some implementations, multiple training models with different K values ​​can be pre-trained. The number of points in a specific point cloud, K. pc This can be expressed as shown in Equation 2:

[0161]

[0162] With K pcA trained network with the closest K can be used for inference. In this example context, inference refers to reconstructing (which may include decoding) a detailed point cloud from a quantized point cloud model. For some implementations, a quantized point cloud is a simplified point cloud with fewer points than the original point cloud before quantization. For some implementations, the quantized point cloud model can be generated using training data within a point cloud compression (PCC) framework. Inference can then be performed against the training data.

[0163] Figure 5 This is a functional block diagram illustrating example point-by-point feature extraction according to some implementation schemes. The methods and processes of this application seek to extract meaningful PC features via neural networks. Along with the cluster points from each query point, three-dimensional (N×K) locations 502 are embedded into the desired feature dimension D. N is the number of query points, and K is the size of the cluster of output features F 508. Such a process 500 can be used as a connection to the following methods and processes. Figure 5 As illustrated, the feature dimension gradually increases to the desired size and is then progressively connected to the next module. In some implementations, the unit microarchitecture can combine batch normalization (BN) layers between fully connected (FC) layers and activation functions (ACT). FC layers contain neurons that apply a linear transformation to the input vector via a weight matrix. A nonlinear transformation is then applied via ACT. Typically, BN layers, positioned between FC and ACT, normalize the weights to improve training performance. Fully connected layers, batch normalization layers, and activation layers can be used to configure convolutional neural network (CNN) or multilayer perceptron (MLP) architectures. See also... Figure 3 and Figure 5 For some implementation schemes, the group feature output can have (N×K) features.

[0164] Local groups of point clouds can be used for point-based feature extraction. However, hierarchical methods using ensemble abstraction (SA) layers employ sampling functions such as farthest point sampling (FPS), which may not be differentiable depending on the intended use. Furthermore, each SA layer may require both grouping and another pass through the PointNet network, which is expensive for the PCC framework. To avoid potential factors that could degrade accuracy and cost-efficiency, a fully differentiable and unique method emphasizing the shape of local geometric information can be created. In this sense, grouping methods can analyze the group distribution in each feature dimension to better distinguish local groups in an efficient manner.

[0165] For some implementations, the distribution-aware feature representation in the PCC framework represents how the point group around the quantized point will be processed or interpreted. For some implementations, quantization can be performed from the original voxel representation and / or the quantized voxel representation. Such distribution-aware features can be used to reconstruct the point cloud. For some implementations, the point cloud compression (PCC) framework can be expressed as a learning-based point cloud geometry or an artificial intelligence (AI)-based point cloud compression framework. For some implementations, the geometry processing block can be a processing block within a learning-based PCC framework.

[0166] Figure 6 This is a function block diagram illustrating an example group characteristic distribution tensor based on some implementation schemes. The detailed calculation process of the group distribution is as follows. Figure 6 As depicted, the input pointwise feature tensor F608 has N groups 604 multiplied by the feature size dimension D, and each group consists of K points 602. For some implementations, Figure 6 Project F within Project 602 11 Here is an example of an eigenvector. For some implementations, eigenvectors can be row-wise or column-wise. The length of an eigenvector is a norm calculated using the elements of the eigenvector. The feature F(F) at this point... P ) can be a set of point-by-point row features [F 11 F 12 ...F jk ...F NK ] T For each group j, the average eigenvalue is calculated using Equation 3:

[0167]

[0168] For each feature row F jk Subtract the corresponding average eigenvalue μ(F) j To form the re-centered features 606 and 612, as shown in Equation 4:

[0169] ΔF jk =F jk -μ(F j ),j∈{1,…,N}and k∈{1,…,K} (4)

[0170] All these elements together, 614, are represented as 610 as the modified tensor ΔF618. For each column f of the feature ΔF618... i 616, Calculate the standard deviation σ(f) of the entire column. i Furthermore, for each column f of feature ΔF iThe index i is a member of {1,…,D}, where D is the size of the feature dimension. Furthermore, computation 620 can be performed on group 622, which together form a group 624, to compute the feature distribution F'628, which may have a series of columns {1,…,D}626.

[0171] For some implementations, the feature map can be a set of channels, for example, C channels. Figure 6 The columns in items 616 and 626 are examples of feature maps with D channels (e.g., in this example, D = C). For some implementations, examples of reshaped vectors are... Figure 6 f in i and f i For some implementations, the length of the reshaped feature vector is a norm calculated using the elements of the reshaped feature vector. For some implementations, updating each vector element in a series of vector elements by subtracting the corresponding average result is as shown in Equation 4 for ΔF. jk As shown. In Figure 6 In this context, the term "calculate characteristic distribution" involves dividing an element by its corresponding standard deviation.

[0172] Figure 7 This is a functional block diagram illustrating an example group feature distribution with learnable transformation parameters according to some embodiments. For some embodiments, γ (gamma) and β (beta) are learned through training. In some embodiments, the distribution parameters may include transformation parameters. In some embodiments, the distribution parameters may be predetermined, for example, through training a neural network. For some embodiments, the distribution parameters (e.g., γ (gamma) and β (beta)) may be determined during a training period using example backpropagation techniques. Such a training period can be used to train the neural network. For some embodiments, the feature tensors F′702, 704 (see...) Figure 7 The group distribution is constructed using process 700, where each column of the feature is split into a one-dimensional vector f′. i 706, 708, 710. Each column forms a bell-shaped curve centered on the group mean or query point. These bell curves represent the group distribution for each feature dimension (D). Figure 7 Feature map F′704 in the example is an example of a feature map with D channels. For some implementations, these group distributions 712, 714, 716 can be individually deformed via transformation parameters γ and β, allowing the feature representation to further distinguish the shape of the corresponding feature dimensions during point cloud (PC) reconstruction. For some implementations, γ and β are examples of distribution parameters. For some implementations, the term "distribution parameter" can be considered as including the transformation parameters. Figure 7 As illustrated below and in Equation 5, f′ i It is a one-dimensional feature fi The (final) distribution:

[0173] f′ i =γ i [f i / (σ(f i )+∈)]+β i (5)

[0174] Where, σ(f i ) is a one-dimensional feature f in feature ΔF. i The standard deviation of column i. γ i Let be the transformation parameters of column i, and let β be the transformation parameters. i This is the offset of the bell curve of column i. In some implementations, the standard deviation can be calculated over the entire feature element. Equation 5 calculates the standard deviation for the 1D element. For some implementations, the vector elements of the feature map can be point-wise vectors. Such vector elements can be normalized by dividing the vector element by its length. For some implementations, the reshaped vector element can be normalized by dividing the reshaped vector element by its length. Equation 5 shows the channel-wise vector. For some implementations, Equation 5 will reshape the vector f... i Divide by σ()+∈. In some implementations, this can be done on the entire feature map element instead of on f. i The element calculation term σ(). Furthermore, Equation 5 can describe the characteristic ΔF matrix on a column-by-column basis. Figure 6 The relationship between the output tensor F′ matrix. For some implementations, γ i These are learnable scalar coefficients. In some implementations, γ i A specific column i can be divided into multiple coefficients. In some implementations, each of the N groups in the column can have a separate transformation parameter γ. i and β i This provides a better approximation of the original point cloud data. The (final) output tensor F′ is listed in Equation 6:

[0175] F′=[f′1 f′2 … f′ i … f′ D (6)

[0176] For some implementations, the transformation parameters γ and β can be partitioned not only by feature dimension but also by group, which can further emphasize the local distribution at a small additional cost. In this case, the number of parameters increases from D to (K×D), where D is the size of the feature dimension and K is the size of the group of the output features F. For some implementations, a density coefficient can be introduced to additionally weigh the importance of each transformation parameter in the tensor. For some implementations, slices can be partitioned into groups. One-dimensional feature elements are represented as f.i These elements f i The fi term can be further subdivided into local group elements. If the point cloud consists of N local groups, the fi term can be further subdivided into N element groups. For some implementations, the feature map can be a concatenation of feature elements per slice, which can be computed by normalizing corresponding slices from another feature map.

[0177] In pointwise features, each point is a member of a group. For each group member point, a micro-distribution can be computed using all group points centered at the group mean. These processes are repeated for all points in the point cloud scene. In some implementations, this process can generate updated pointwise features, which consist of floating-point matrix forms. This process can be separated by feature dimensions (which can be matrix columns or channels). For each set of point cloud data, the distribution of channel elements may vary in, for example, range, magnitude, and mean. In some implementations, these distributions can be manipulated using transformation parameters that can be learned during the training process. These distribution curves can be transformed so that the feature extractor can better distinguish or emphasize the features of each point.

[0178] Figure 8 This is a functional block diagram illustrating feature aggregation, enhancement, and dimension matching based on some implementation schemes. For example... Figure 8 As shown in example process 800, global and local representations can be mixed, and features F′802 can be further enhanced. Local features can be aggregated, and then global features can be used to enhance the extended tensor 804. For some implementations, dimension matching 810 can be performed, and the updated features F″808 can be output. The dimensions of F′ and F″ are the same. Therefore, the enhanced features can be combined with group feature distribution transformation and pointwise residual networks (respectively...). Figure 3 Used together with 310 and 314). For some implementations, the term "pooling" refers to the function of aggregating multiple point features into a single point feature by averaging or taking the maximum value within the feature. Figure 8 In this model, the aggregated feature F′804 is the result of a pooling operation. For some implementations, the pooling operation may depend on, for example, average pooling or max pooling. For some implementations, a smoothing filter may be applied to the feature map. For example, Figure 8 Extended feature F′ in AE This is an example of the output of such a smoothing filter.

[0179] It enables efficient analysis of fully differentiable local groups. For example... Figure 8 As described, the output feature F′ can be used independently in some implementations, or in some implementations it can be used with group-wise set and extended features (F′). AE806 is cascaded. A matching layer may follow, which matches the feature dimensions back to the input size. For some implementations, the output 812 of the matching box 810 may reflect the transformation of the dimension of the cascaded output (which is (N×K)×(2D)) to the dimension of the input (which is (N×K)×D).

[0180] The PointNet architecture is a cost-effective PC feature extractor. Due to its simplicity, it can be used within PCC frameworks. However, for some applications, the PointNet architecture lacks sufficient detail, especially within PCC frameworks, such as for lower bit-rate compression. To overcome this potential problem, according to some implementations, deeper networks with a combination of global and local features that all pass through the network can be used. Residual networks are, for example, networks that learn residuals. Residual networks can be used to design deeper neural networks. In some contexts, "deeper network" is used as a more general term compared to residual networks.

[0181] Figure 9 This is a functional block diagram illustrating pointwise residual feature extraction and cluster-wise feature aggregation and enhancement based on some implementation schemes. For example... Figure 9 As shown, there is an architecture 900 that can be used with deeper networks. To avoid degradation in deeper networks, a ResNet-like design is used. For some implementations, residual network blocks 922 (such as...) Figure 9 The example shown can be based on the ResNet architecture. Kaiming He's paper Deep Residual Learning for Image Recognition (ARXIV preprint, arXiv:1512.03385 (2015)) ("He") describes an example ResNet architecture. (See, for example, the right side of page 4) Figure 3 )exist Figure 9 In the example implementation shown, depending on the implementation, the image input of the ResNet architecture is replaced by point cloud input 902. The convolutional layers of the ResNet architecture are replaced by fully connected (FC) layers. To match the feature dimensions of the input and output, the input to the fully connected layer (which can be pointwise features 904) is scaled down. Therefore, the input to the FC layer 916 in the next row is scaled down. In some implementations, the input to the previous row (closer to...) is scaled down. Figure 9 The input to the second FC layer 912 (located in the center of the line) is reduced. In some implementations, this reduction is a 2x reduction for both the inputs to the FC layers in the previous and next rows. This 2x reduction can be performed to offset the increase made in AUG block 926. In some implementations, the input to the previous row (closer to the center of the line) is reduced. Figure 9The input of the first FC layer 906 (left side) is reduced. Both global and local features are preserved through the "aggregate and enhance" process. Unlike image input, 3D point cloud structures are irregular. Therefore, convolutional layers are replaced by fully connected (FC) layers. The FC layer links the output of the previous process box with the input of the next process box. In some implementations, the FC layer outputs a linear transformation of the input. Each FC layer 906, 912, 916 is followed by batch normalization (BN) layers 908, 914, 918 and activation functions (ACT) 910, 920. In some implementations, BN layers can be used to train the network and help the network converge faster. The activation function (ACT) can be, for example, a rectified linear unit (ReLU) function where negative values ​​are replaced by zero. In some implementations, the BN layer can be omitted, so that the output of the FC layer is fed into the ACT layer. In order to connect several processes in these processes sequentially and to be compatible with the previously introduced distribution transformation tensor, the shape of the input can be matched with the output tensor. For the residual path ( Figure 9 Top path in the middle) and shortcut path ( Figure 9 In the bottom path, both the top and bottom paths are fully connected (FC) layers, and the feature dimension is reduced (divided by 2). The two outputs of the top and bottom paths are added together by a plus sign ("⊕"), and then an activation function is applied. For some implementations, each output of the top and bottom BN is a matrix (tensor) of floating-point values. Since these matrices have the same dimension (size), the plus sign ("⊕") indicates element-wise matrix addition. For some implementations, the top path determines the residual value added to the version of the input value from the bottom path. For some implementations, the bottom path can be a shortcut path without FC and BN layers if the dimension from the pointwise feature input and the output of the top path's BN layer to the plus sign ("⊕") matches. For some implementations, such scenarios may occur if, for example, an AUG function and a sequential cascaded feedback path are not performed.

[0182] So far, pointwise features have been processed. Each group is aggregated (AGG) 924, and a groupwise feature 928 is created. For some implementations, the groupwise features are output 932. In some implementations, the sequential block can be repeated using the outputs of a combined expansion (EXP) 930 and enhancement (AUG) 926 process. During enhancement, the output features of the residual network (before the AGG box) and the expanded groupwise features (output of the EXP box) are concatenated. This concatenated feature is passed to the beginning to repeat the sequential block. For some implementations, Figure 9 The aggregation (AGG) box is similar to Figure 8 The aggregation process. In some implementations, Figure 9 The expanded (EXP) box is similar to Figure 8 The expansion process. For some implementation schemes, Figure 9 The enhanced (AUG) box is similar to Figure 8 The cascading process.

[0183] For some implementation plans Figure 9 The architecture shown can be plugged into Figure 3 In the architecture, Figure 9 The residual network frame was used for point-to-point networks ( Figure 3 (e) Figure 9 The AGG, AUG, and EXP boxes are used for group feature aggregation and augmentation boxes. Figure 3 The box (f)). For some implementations, Figure 3 The feedback path can be derived from group feature aggregation and augmentation boxes ( Figure 3 The output of the box (f) is passed to the pointwise residual network ( Figure 3 The input to box (e) in the middle. If Figure 9 The architecture shown is inserted into Figure 3 In the architecture, then it can be used Figure 3 Such configurations of the feedback path in, where Figure 9 The sequential cascade feedback path is Figure 3 The feedback path in.

[0184] In some implementations, the deep residual process can work in conjunction with and advantageously with a distributed sensing process. In some implementations, the distributed sensing process can be... Figure 3 Boxes 310 and 312. In some embodiments, the distributed sensing process can be... Figure 3 Box 310. In some implementations, the depth residual process can be... Figure 3 Boxes 314 and 316. In each stage, the features of the two processes can be chained together (see...). Figure 3 ) or in parallel (see Figure 10 Rich representations are propagated through deep networks. In other words, deep feature extraction processes can be sequentially connected while preserving local geometric information without causing degradation. Furthermore, maintaining compatibility between input / output dimensions facilitates the creation of variant designs for different purposes.

[0185] Figure 10 This is a functional block diagram illustrating an example encoder architecture for PC feature extraction according to some implementation schemes. The overall architecture connects several processes in series; however, as... Figure 10 As shown, some implementations may use a parallel architecture 1000. In some implementations, the point cloud 1002 is the input to a point group query 1004, which outputs to point-by-point feature extraction 1006.

[0186] and Figure 3 In comparison, one difference is that, Figure 10The distribution transformations 1008 and 1010 in the model use independent parallel paths (path 1 for 1008 and 1010, and path 2 for 1012 and 1014), and then utilize features from the residual module to enhance these outputs 1016. The dimension of output 1016 is then matched with (matched to) a model similar to... Figure 8 The microarchitecture introduced in the process is matched to generate group features 1020. To complete the deep phase, a final aggregation is performed (below the dashed line in box 1016), and then group-by-group feature extraction 1018 is performed. For some implementations, the group size K can be 1, in which case the group features may be referred to as point features.

[0187] Figure 11 This is a functional block diagram illustrating an example decoder architecture based on some implementation schemes. For an end-to-end PCC framework, the decoder is used in conjunction with the encoder. For some implementation schemes, such as... Figure 11 As described, another decoder architecture 1100 is applicable. In some implementations, the first pointwise extractor may increase the feature dimension. Several processes described above may occur between the first and second pointwise extractors. The second pointwise extractor may reduce the feature dimension to 3D to reconstruct the finally decompressed point cloud. For some implementations, there is no aggregation / grouping in the decoder, and the dimension of the decoder input data 1102 is N groups multiplied by the feature dimension D (N×D).

[0188] In some implementations, the decoder architecture can obtain group feature data. The decoder process can extract 1104 group-wise features from the group feature data. Then, the group size can be matched / expanded 1106 to the point size. The expanded point-wise features can be passed to a group feature distribution transformation process 1108, which can perform a transformation on the group feature distribution. The transformed features can be input into a point-wise residual network 1110, which can generate residual-based feature data. The decoder process can run point-wise feature extraction 1112 to extract the reconstructed point cloud 1114.

[0189] Compared to PC classification or segmentation, PC feature extraction in AI-based PCC architectures is a relatively new area. In the Yan and '869 applications, PC feature extractors, for example, define an MLP layer followed by an aggregation step. While PCC frameworks seek a cost-effective architecture, there is still room for improvement in representing PC features. For some implementations, the aforementioned feature extractors can be used within the PCC framework, improving compression performance. Enhanced feature extractors can be used for other tasks, such as segmentation and point cloud classification.

[0190] Figure 12A This is a functional block diagram illustrating an example application of the PCC framework in the encoder and decoder of autoencoder-based lossy geometry compression. For example... Figure 12A As shown, Yan uses an end-to-end PCC architecture 1200. The encoder 1204 and decoder 1220 are designed with MLPs 1206, 1222 and a pooling layer 1208.

[0191] Yan discusses four modules: a PointNet-based encoder, a uniform quantizer, an entropy estimation block, and a nonlinear synthesis transform module. Yan uses an autoencoder as the compression platform. As mentioned on page 4323, Yan's first column:

[0192] First, the input point cloud is downsampled through a sampling layer S to create point clouds with different point densities. Then, the downsampled point set is passed through an autoencoder-based encoder-decoder. This encoder-decoder consists of an encoder E, a quantizer Q, and a decoder D. The encoder E takes the unordered point set as input and produces a compressed representation, while the decoder D takes the quantized representation produced by Q and produces a reconstructed point cloud.

[0193] like Figure 12A As shown, the downsampled point cloud is used as input points 1202. These points serve as a multi-layer point set undergoing (max) pooling during the encoding process. The encoder output, as latent code 1210, is passed through entropy encoder 1212 to generate a synthetic representation, which is Figure 12A The bitstream 1214 shown on the right is located between the entropy encoder 1212 and the entropy decoder 1216. This synthesized representation is passed through the entropy decoder 1216 to generate quantization code 1218. The quantization code 1218 is passed through the decoder to reconstruct a multi-layer point set, which is the output point cloud 1224.

[0194] Figure 12B This is a functional block diagram illustrating example applications of the PCC framework in encoders and decoders in autoencoder-based lossy geometry compression, according to some implementation schemes. For some implementations of the encoder / decoder architecture 1250, Figure 12A The encoder and decoder on the left can be Figure 3 or Figure 10 Architectural alternatives, such as Figure 12B As shown. For example, in some implementations, Figure 12A The encoder can be such as Figure 12B The distributed sensing process 1254, the deep residual process 1256, and the feedback path substitution are shown. Similarly, for some implementation schemes, Figure 12A The decoder can be such as Figure 12BThe distribution sensing process 1268, deep residual process 1270, and feedback path substitution are shown in the diagram. Therefore, higher quality latent codes 1258 and higher quality quantized codes 1266 can be generated. Such codes can contain more representative PC features with a smaller increase in computational cost. For some implementations, Figure 12A The encoder was Figure 10 The encoder architecture replacement. For Figure 12B Input point 1252, Figure 12A The input point is entered into the point group query ( Figure 10 Point cloud replacement in bounding box 1004). Cluster feature extraction ( Figure 10 The group feature output of the bounding box (1018) is Figure 12A The potential code. In some implementations, the potential code 1258 is the input to the entropy encoder 1260, which outputs a bitstream 1262. In some implementations, the bitstream 1262 is the input to the entropy decoder 1264, which outputs a quantized code 1266. In some implementations, Figure 12A The decoder was Figure 11 The decoder architecture replacement, in which Figure 12A The quantization code is input into the group-by-group feature extraction ( Figure 11 The group features of the bounding box (1104). For Figure 12B Output point 1272, point-by-point feature extraction ( Figure 11 The reconstructed point cloud output of the box 1112) is replaced Figure 12A The output point.

[0195] Figure 13A Another example from '869 application is shown in the figure. Figure 13A This is a functional block diagram illustrating an example application of the PCC framework in point analysis and point synthesis within a scalable PCC framework. The '869 application introduced a scalable end-to-end PCC framework. This application focuses on feature analysis and synthesis, rather than point analysis and synthesis. Similarly, in this framework, the MLP and aggregation layer are combined in both the "Residual-to-Feature Transformer (Point Analysis in the Encoder)" box and the "Feature-to-Residual Transformer (In the Decoder)" box. For some implementations, the geometry processing block within the learning-based PCC framework can be, for example... Figure 13A One or more boxes in the box, such as box 1304, box 1308, box 1310, box 1312, box 1316, box 1318, box 1320 and / or box 1322.

[0196] '869 application discusses a lossy point cloud compression scheme to encode the geometry of point clouds using deep neural networks. For such a scheme 1300, a coarse version of the input point cloud 1302 is encoded 1304 into a first bitstream 1306 (from '869 application). Figure 5 The BS0), and the residual data (fine geometric details) are encoded 1312 into a second bitstream 1314 (from '869 application). Figure 5 The pointwise features of BS1). The residual data can be generated through point analysis 1308 and feature analysis 1310.

[0197] On the decoding side, the coarse point cloud (PC1 of Figure 14 in '869 application) is decoded from the first bitstream (BS0 of Figure 14 in '869 application) 1316. The residual data (R' of Figure 14 in '869 application) is decoded from the pointwise features (F' of Figure 14 in '869 application) 1318, and is added to the coarse point cloud 1320, 1322 to retrieve the decoded version 1324 (PC0 of Figure 14 in '869 application) of the original input point cloud.

[0198] Figure 13B This is a functional block diagram illustrating example applications of the PCC framework in point analysis and point synthesis within a scalable PCC framework, according to some implementations. In some implementations, input point 1352 may be fed into an octree encoder 1354 and a distribution-aware process 1358. The octree encoder 1354 outputs a basic bitstream 1356, which is then fed into an octree decoder 1368. For end-to-end PCC compression 1350, the choice of group size K may affect the level of compression enhancement and performance. Furthermore, a high-quality representation of PC features can improve reconstruction performance, especially at lower bit rates. For example, Figure 13A The point analysis process can be Figure 3 or Figure 10 The proposed architecture alternatives, and have add or remove processes (such as, Figure 3 Box 312 and Figure 10 The additional flexibility of box 1010 in the [section / framework]. For some implementations, Figure 13A The point synthesis process can be Figure 11 The proposed architectural alternatives. For example, in some implementations, Figure 13A The point analysis process can be as follows: Figure 13B The distributed sensing process 1358, the deep residual process 1360, and the feedback path substitution are shown. Similarly, for some implementation schemes, Figure 13A The point synthesis process can be like Figure 13B The distributed sensing process 1374, the deep residual process 1376, and the feedback path substitution are shown. For some implementation schemes, Figure 13A Point analysis was Figure 10 The encoder architecture replacement, in which Figure 13A The input point is the input to the point group query ( Figure 10 The point cloud in the bounding box (1004) and the group-by-group feature extraction ( Figure 10The group feature output of the bounding box (1018) is to Figure 13A The input for feature analysis. For some implementation schemes, Figure 13A The point synthesis was Figure 11 The decoder architecture replacement, in which Figure 13A The feature synthesis output is input to the group-wise feature extraction ( Figure 11 The group features in the box 1104) and the point-by-point feature extraction ( Figure 11 The reconstructed point cloud output of the bounding box (1112) is Figure 13A The output point.

[0199] In some implementations, the deep residual process 1360 outputs to the feature analysis process 1362, which in turn outputs to the entropy encoder 1364. The output of the entropy encoder is an enhanced bitstream 1366, which in turn is the input to the entropy decoder 1370. The entropy decoder 1370 outputs to the feature synthesis process 1372. In some implementations, the feature synthesis process 1372 takes inputs from the octree decoder 1368 and the entropy decoder 137 and outputs to the distribution-aware process 1374. The output of the deep residual process 1376 is a set of output points 1378.

[0200] Figure 14A This is a functional block diagram illustrating an example application of the PCC framework in the Collection Abstraction (SA) process within the PointContextNet environment. Figure 14A Example method 1400 from application '482 is described, which uses an AI-based octree structure entropy model.

[0201] Application '482 discusses retrieving point clouds based on tree-structured compression and retrieving points in the neighborhood of nodes in a tree structure. Two features are computed 1404 based on the retrieved data and their positions 1402. Application '482 then fuses these two features with one or more known features of the node, and finally determines the occupancy of the current node based on the encoded bitstream and the predicted occupancy symbol distribution 1408.

[0202] Figure 14B This is a functional block diagram illustrating an example application of the PCC framework in the Collection Abstraction (SA) process within the PointContextNet environment, based on some implementation schemes. Figure 3 or Figure 10 The architecture in the middle can be replaced Figure 14A The SA process / module. For example, in some implementations, Figure 14A The SA process 1404 can be as follows Figure 14B The distributed sensing process 1454, the deep residual process 1456, and the feedback path substitution are shown. For some implementation schemes, Figure 14AThe Set Abstraction (SA) process 1404 was... Figure 10 The encoder architecture replacement, in which Figure 14A The output of the fill point context box 1402 is the input to the point group query ( Figure 10 The point cloud in 1004) and the group-by-group feature extraction ( Figure 10 The group feature output of (1018) is Figure 14A SA feature 1406.

[0203] For some implementations, example process 1450 may populate the point context 1452 into the distribution-aware process 1454. The output of the deep residual process 1456 may be an SA feature 1458, which may in turn be the input to the occupancy probability prediction 1460.

[0204] This architecture extracts cost-effective features with high-level representations through feature distribution transformation and a deep residual architecture within the octree entropy model of the PCC framework. In addition to the SA layer, several of the aforementioned processes (such as...) Figure 3 Processes 310, 312, 314, 316, and 318 can be used to replace the FC layer in both the PointContextNet and CRM architectures.

[0205] Figure 15A This is a functional block diagram illustrating an example application of the PCC framework in the Set Abstraction (SA) process within the Coordinate Refinement Module (CRM). Application '894 discusses coordinate refinement and upsampling of quantized and reconstructed point cloud data. Neighboring points of the decoded point cloud can be determined through an AI-based coordinate refinement process 1500. Based on the properties of one of those neighboring points, neural network techniques can be used to determine refinement features. These refinement features can be used to predict the refinement of the decoded point cloud 1508.

[0206] Figure 15B This is a functional block diagram illustrating an example application of the PCC framework in the Set Abstraction (SA) process within the Coordinate Refinement Module (CRM) according to some implementation schemes. Figure 3 or Figure 10 The architecture in the middle can be replaced Figure 15A The SA process / module. For example, in some implementations, Figure 15A The SA process 1506 can be as follows Figure 15B The distributed sensing process 1554, the deep residual process 1556, and the feedback path substitution are shown. For some implementation schemes, Figure 15A The Set Abstraction (SA) process 1504 was... Figure 10 The encoder architecture replacement, in which Figure 15A The output of the fill point context box 1502 is the input to the point group query ( Figure 10The point cloud in 1004) and the group-by-group feature extraction ( Figure 10 The group feature output of (1018) is Figure 15A SA characteristics.

[0207] In some implementations of process 1550, the fill-in point context 1552 is the input to the distribution-aware process 1554. In some implementations, the output of the depth residual process 1556 is the SA feature 1558, which can in turn be the input to the offset prediction 1560.

[0208] Compared to the methods in Yan and the '869 application, the methods in the '482 and '894 applications employ a more advanced feature extractor with a set abstraction (SA) module and further enhancements with multi-resolution (MRG) or multi-scale (MSG) grouping. The SA layer is followed by a series of fully connected (FC) layers. While these architectures can leverage hierarchical and multi-level approaches to extract well-represented PC features, they can incur high computational costs, which may be less advantageous for some practical implementations.

[0209] This application describes a depth distribution-aware point feature extractor for point cloud data. This architecture can be used in AI-based point cloud compression (PCC) frameworks, as well as other architectures. The PCC framework strikes a good trade-off between reconstruction accuracy and computational cost. A per-channel feature distribution transformation process can be used to facilitate this objective. A depth residual-based network with repeated mixing of global and local information can be used for further enhancement. The two processes described above can be used in both the encoder and decoder of a given PCC framework.

[0210] For some implementations, the input and output of the feature extractor described in this application can be pointwise features. Therefore, the feature extractor described herein can be inserted... Figure 12A , Figure 13A , Figure 14A , Figure 15A In the framework shown, for example, respectively as Figure 12B , Figure 13B , Figure 14B and Figure 15B exemplified.

[0211] Figure 16This is a flowchart illustrating an example process for point cloud feature extraction according to some embodiments. For some embodiments, example process 1600 may include: querying a local point group for each point in a point cloud having a selected group size 1602. For some embodiments, the example process may also include: extracting a first pointwise feature 1604. For some embodiments, the example process may also include: performing a first and a second processing of a feedback process 1606. For some embodiments, the example feedback process of this example process may include: transforming the first pointwise feature 1608 into a second pointwise per-channel distribution feature based on a set of transformation parameters. For some embodiments, the example feedback process of this example process may also include: extracting a third pointwise feature 1610 from the second pointwise feature via a deep network. For some embodiments, the example feedback process of this example process may also include: aggregating the third pointwise feature based on the local point group 1612 to form a first groupwise feature. For some embodiments, the example feedback process of this example process may also include: enhancing the third pointwise feature 1614 using an extended version of the first groupwise feature. For some implementations, the example feedback process of this example process may also include: obtaining 1616 next-level first pointwise features as the first pointwise features for the second processing of the feedback process.

[0212] Deep feature decoding for PCC

[0213] Learning-based PCC methods can be divided into two main categories: deep octree-based PCC and deep feature-based PCC. In octree-based PCC, voxel occupancy is directly entropy-decoded into a bitstream. Learning-based methods are then used to predict the probabilities of these voxel occupancy. In deep feature-based PCC, geometric features are quantized and compressed end-to-end.

[0214] Figure 17 This is a functional block diagram illustrating a depth-feature-based PCC pipeline according to some implementation schemes. Figure 17 The gray dashed arrows in the diagram illustrate the data flow of the general depth-feature-based PCC framework 1700. Encoder E1 1704 extracts intermediate features from the input point cloud X 1702. Encoder E2 1708 further compresses the intermediate features to optimize compression using entropy encoder E3 1710. On the decoder, entropy decoder D3 1714 decodes the features from bitstream 1712. Decoder D2 1716 processes the intermediate features, and decoder D1 1720 reconstructs the final decoded point cloud. 1722.

[0215] To extract unique features of PCC, the Deep Distributed Aware Network (DDA-Net) can be combined with a general depth feature-based PCC pipeline, such as... Figure 17 As shown, DDA-Net blocks 1706 and 1718 are inserted between the E1 and E2 encoders on the encoder side for the bitstream and between the D1 and D2 decoders on the decoder side for the bitstream. The DDA-Net blocks take the intermediate features generated by the E1 encoder (or D2 decoder) as input and manipulate the distribution to further distinguish them for point cloud compression (PCC). The modified features are then input to the E2 encoder (or D1 decoder).

[0216] The DDA-Net block can be applied to GRASP-Net, a PCC framework that combines point-based and voxel-based architectures. GRASP-Net is described in Pang, J. et al.'s GRASP-Net: Geometric Residual Analysis and Synthesis for Point Cloud Compression (PROC. OF THE 1ST INTERNATIONAL WORKSHOP ON ADVANCES IN POINT CLOUD COMPRESSION, PROCESSING, AND ANALYSIS), pp. 11-19 (2022) ("Pang"). Specifically, the E1 encoder and E2 encoder correspond to PointNet (point analysis in Pang) and CNN-based downsampling (feature analysis in Pang), respectively. On the decoder, D1 and D2 correspond to the MLP layer (point synthesis in Pang) and CNN-based upsampling (feature synthesis in Pang). Input / output pairs and They share the same dimensions. DDA-Net manipulates feature distributions that are closer to the input / output point cloud than those of the E3 encoder and D3 decoder.

[0217] Distribution-aware feature manipulation

[0218] Given a set of pointwise features Where l is the point index, D is the feature dimension, and F... l The distribution of the feature elements in the vector F is modeled as an iid Gaussian. Therefore, the vector F l The joint distribution can be parameterized by a multivariate Gaussian, which has The mean μ and standard deviation σ in F are being sought. A method for manipulating F is being sought. l The network of distributions of each feature element in the network To promote more descriptive features.

[0219] Implementing standardization processes The eigenvalues ​​are normalized. The mean eigenvalue is σ. The standard deviation is σ. The partitioning is element-wise. The Gaussian model is modified by transforming (scaling and shifting) into a deformable curve.

[0220] DDA-Net architecture

[0221] Figure 18 This is a functional block diagram illustrating the DDA-Net encoder architecture according to some implementation schemes. Figure 18 The diagram shows a Deep Distributed Aware Network (DDA-Net), where the feature map F in Equation 7... p 1802 as its input:

[0222]

[0223] Where NK is the size of the feature map.

[0224] Initial feature map F p The data are input into probability distribution estimation functions 1804 and 1812, which output characteristics of the modified distribution. Execute residual network functions 1806 and 1814 in parallel to output These two output features are concatenated 1808 and 1816, and then fed into MLPs 1810 and 1818 to obtain features with a further modified distribution. One objective of the MLP is to match feature dimensions to facilitate concatenation. The "mix and then propagate" process can be repeated several times. Empirically, four iterations maximize the trade-off between reconstruction performance and memory efficiency. Residual connections can be added before the final iteration to achieve more stable learning. (Symbol F) g 1820 represents the group characteristic.

[0225] Probability distribution estimation

[0226] Local point clusters around each query point can be collected to gather local geometric information. The input "point features" is shown in Equation 8:

[0227]

[0228] Where N is the number of groups, and K is the number of points in a group. These points in the group are then pooled into a feature, which becomes the "group feature" F. g Part of it. For some implementations, the term "group" can be viewed as points after downsampling each local group (a set of points near the point). The point features before downsampling include all the local points being queried.

[0229] Figure 19 This is a functional block diagram illustrating a probability estimator distribution network based on some implementation schemes. For example... Figure 19 As illustrated on the left, local grouping and NK point feature embedding follow the process described in Pang's GRASP-Net architecture. Figure 19 In the example PCC framework 1900, these local groupings and NK point features are input to the E1 encoder 1902.

[0230] In probability distribution estimation, F p It is embedded into a D / 2 dimension with shared MLP 1904 to obtain F. This feature F can be viewed as a set of point features. For each group j, the corresponding mean characteristic is calculated as shown in Equation 9:

[0231]

[0232] For each feature row F jk Subtract the corresponding group average To obtain the re-centered feature element, as shown in Equation 10:

[0233]

[0234] A reshaped feature vector (channel-wise vector) is a set of feature elements. An example of a reshaped vector is... Figure 6 f in i or f i Calculate the average for these feature elements. Subtract this average from the above feature elements. Calculate the standard deviation vector for each column of ΔF. Standardized point features can be expressed as equal to

[0235] Formula 11:

[0236]

[0237] Where ∈ represents a small value used for numerical stability. The set F′ of values ​​for j and k. jk The characteristic F′ of the 1906 standardization is formed.

[0238] Each standardized i-th column f i This forms a bell-shaped curve centered on the group mean or query point. The aim is to achieve this through learnable parameter transformations. and This makes the distributions of these groups deformable. Equation 12 is used to calculate the point characteristics F″ of the 1908 distribution transformation. jk :

[0239] F″ k =γ T ·F′ jk +β (12)

[0240] For each group j, a pooling operation is performed, resulting in the group characteristics in Equation 13:

[0241]

[0242] And as Figure 19 As shown. For deeper networks, output features can be further propagated to the input of the next level. In this case, pooling... It is expanded by 1910 to match the size of feature F″ and is represented as F″′. These features are concatenated by 1912, and another output point feature is generated. 1914. See also Figure 19 .

[0243] Residual Networks

[0244] In parallel with probability distribution estimation, pointwise residual networks can be generated for each level. For example, see [link to relevant documentation]. Figure 9 Based on He's ResNet architecture, the residual network is designed to adapt to the PCC framework. To adapt the network to point cloud input, convolutional layers are replaced with fully connected (FC) layers. Figure 9 As shown, the feature dimension D can be reduced by half through a fully connected (FC) layer. Similar to the probability distribution estimation process, the residual network also has branching outputs. and For deeper iterations, you can use otherwise, It can be used in the final aggregation stage. See also Figure 18 For the DDA-Net output, the group features are concatenated during the final stage. The final MLP extracts feature dimensions to match the sparse CNN, which in... Figure 17 It is shown as encoder E2.

[0245] Figure 20 This is a flowchart illustrating an example process for point cloud feature extraction according to some implementations. For some implementations, the example learning-based point cloud geometry encoder process may include: accessing a first feature map, wherein the first feature map is input to the encoder, and wherein the first feature map is generated by a first set of neural network layers. For some implementations, the example process may further include: normalizing the elements of the first feature map to generate a second feature map. For some implementations, the example process may further include: accessing a set of distribution parameters. For some implementations, the example process may further include: transforming the second feature map into a third feature map based on the set of distribution parameters. For some implementations, the example process may further include: aggregating the third feature map into a fourth feature map.

[0246] Figure 21This is a flowchart illustrating an example process for point cloud feature extraction according to some implementations. For some implementations, an additional example of a learned point cloud geometry encoder process may include: accessing a first feature map, wherein the first feature map is input to the encoder, and wherein the first feature map is generated by a first set of neural network layers. For some implementations, this additional example process may also include: normalizing the elements of the first feature map to generate a second feature map. For some implementations, this additional example process may also include: accessing a set of distribution parameters. For some implementations, this additional example process may also include: transforming the second feature map into a third feature map based on the set of distribution parameters. For some implementations, this additional example process may also include: aggregating the third feature map into a fourth feature map. For some implementations, this additional example process may also include: expanding the fourth feature map to the size of the third feature map. For some implementations, this additional example process may also include: using the third feature map to enhance the expanded feature map. For some implementations, this additional example process may also include: repeating the learning-based point cloud geometry encoder once or multiple times, wherein the enhanced feature map is used as the first feature map, and wherein the next set of distribution parameters is used as the distribution parameters.

[0247] Figure 22 This is a flowchart illustrating an example of a learning-based point cloud geometry process according to some implementations. Some implementations of example process 2200 may include: accessing 2202 a first feature map, wherein the first feature map has C channels and is the input to the processing block, and wherein the first feature map is generated by a first set of neural network layers. For some implementations, the example process may also include: accessing 2204 a set of distribution parameters. For some implementations, the example process may also include: transforming 2206 the first feature map into a second feature map based on the set of distribution parameters. For some implementations, the example process may also include: encoding 2208 the second feature map into a bitstream.

[0248] Figure 23 This is a flowchart illustrating an example of a learning-based point cloud geometry process according to some implementations. Some implementations of example process 2300 may include: decoding a first feature map from a bitstream 2302. For some implementations, the example process may also include: accessing a set of distribution parameters 2304. For some implementations, the example process may also include: transforming the first feature map 2306 into a second feature map based on the set of distribution parameters. For some implementations, the example process may also include: reconstructing the point cloud 2308 from the second feature map.

[0249] While the methods and systems according to some embodiments are generally discussed in the context of extended reality (XR), some embodiments can be applied to any XR context, such as, for example, virtual reality (VR) / mixed reality (MR) / augmented reality (AR) contexts. Furthermore, although the term "head-mounted display (HMD)" is used herein according to some embodiments, some embodiments can be applied to wearable devices (which may or may not be attached to the head) capable of implementing, for example, XR, VR, AR, and / or MR (for some embodiments).

[0250] Examples of learning-based point cloud geometry processing blocks, according to some implementation schemes, may include: accessing a first feature map, wherein the first feature map has C channels and is the input to the processing block, and wherein the first feature map is generated by a first set of neural network layers; accessing a set of distribution parameters; transforming the first feature map into a second feature map based on the set of distribution parameters; and encoding the second feature map into a bitstream.

[0251] Some implementations of the example-based point cloud geometry encoding block method may also include updating the first feature map by normalizing the elements of the first feature map.

[0252] For some implementations of the example learning-based point cloud geometry encoding block method, normalizing the element of the first feature map may include: determining the corresponding length of each feature vector associated with an element of the first feature map; and dividing each element of the first feature map by the corresponding length.

[0253] For some implementations of the example learning-based point cloud geometry encoding block method, normalizing the element of the first feature map may include: arranging each of one or more vectors by reshaping the associated feature channels in the first feature map; determining a corresponding length for each of the one or more vectors for each feature channel; and dividing each vector element of each vector by the corresponding length for each feature channel, wherein each vector element is an element of the element of the first feature map.

[0254] For some implementations of the example learning-based point cloud geometry encoding block method, normalizing the element of the first feature map may include: arranging each of one or more vectors by reshaping the associated feature channels in the first feature map; determining the corresponding standard deviation of each of the one or more vectors for each feature channel; determining the corresponding average of each of the one or more vectors for each feature channel; updating each vector element of each vector by subtracting the corresponding average; and dividing each updated vector element of each vector by the corresponding standard deviation for each feature channel, wherein each vector element is an element of the element of the first feature map.

[0255] For some implementations of this example of a learning-based point cloud geometry encoding block method, the set of distribution parameters is determined using backpropagation during the training period.

[0256] For some implementations of this example of a learning-based point cloud geometry encoding block method, the set of distribution parameters is determined on a per-feature-channel basis.

[0257] Some implementations of the example of the learning-based point cloud geometry encoding block method may also include updating the second feature map by performing downsampling using a function of average pooling or max pooling.

[0258] Some implementations of the example of the learning-based point cloud geometry encoding block method may also include: determining a third feature map by filtering the second feature map using a smoothing filter; and updating the second feature map by concatenating the third feature map to the second feature map.

[0259] For some implementations of this example of a learning-based point cloud geometry encoding block method, the process may include: accessing a second set of distribution parameters before encoding the second feature map; and updating the second feature map by transforming it based on the second set of distribution parameters.

[0260] For some implementations of this example of a learning-based point cloud geometry encoding block method, the process may include aggregating the second feature map using a second neural network before encoding the second feature map.

[0261] For some implementations of the learning-based point cloud geometry encoding block method in this example, the second neural network is selected from a group consisting of sparse convolutional neural networks (CNN) and multiple perceptron layers (MLP).

[0262] For some implementations of this example of a learning-based point cloud geometry encoding block method, aggregating the second feature map may include using a residual network (ResNet) architecture.

[0263] Some implementations of the example of the learning-based point cloud geometry encoding block method may also include: determining a fourth feature map by aggregating the first feature map using a neural network in parallel with transforming the first feature map into the second feature map; and updating the second feature map by concatenating the fourth feature map to the second feature map.

[0264] For some implementations of this example of a learning-based point cloud geometry encoding block method, aggregating the first feature map using the neural network may include using a residual network (ResNet) architecture.

[0265] For some implementations of the learning-based point cloud geometry encoding block method in this example, the process includes aggregating the first feature map using a third neural network before transforming the first feature map into the second feature map.

[0266] For some implementations of the learning-based point cloud geometry encoding block method in this example, the third neural network is selected from a group consisting of sparse convolutional neural networks (CNN) and multiple perceptron layers (MLP).

[0267] According to some embodiments, an example of a learning-based point cloud geometry encoding block device may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the preceding claims.

[0268] Examples of learning-based point cloud geometry decoder methods, according to some implementation schemes, may include: decoding a first feature map from a bitstream; accessing a set of distribution parameters; transforming the first feature map into a second feature map based on the set of distribution parameters; and reconstructing the point cloud based on the second feature map.

[0269] Some implementations of this example of a learning-based point cloud geometry decoder method may also include updating the first feature map by normalizing the elements of the first feature map.

[0270] For some implementations of this example learning-based point cloud geometry decoder method, normalizing the element of the first feature map may include: determining the corresponding length of each feature vector associated with an element of the first feature map; and dividing each element of the first feature map by the corresponding length.

[0271] For some implementations of this example learning-based point cloud geometry decoder method, normalizing the element of the first feature map may include: arranging each of one or more vectors by reshaping the associated feature channels in the first feature map; determining a corresponding length for each of the one or more vectors for each feature channel; and dividing each vector element of each vector by the corresponding length for each feature channel, wherein each vector element is an element of the element of the first feature map.

[0272] For some implementations of this example learning-based point cloud geometry decoder method, normalizing the element of the first feature map may include: arranging each of one or more vectors by reshaping the associated feature channels in the first feature map; determining the corresponding standard deviation of each of the one or more vectors for each feature channel; determining the corresponding average of each of the one or more vectors for each feature channel; updating each vector element of each vector by subtracting the corresponding average; and dividing each updated vector element of each vector by the corresponding standard deviation for each feature channel, wherein each vector element is an element of the element of the first feature map.

[0273] For some implementations of this example of a learning-based point cloud geometry decoder method, the set of distribution parameters is determined using backpropagation during the training period.

[0274] For some implementations of this example of a learning-based point cloud geometry decoder method, the set of distribution parameters is determined on a per-feature-channel basis.

[0275] For some implementations of this example learning-based point cloud geometry decoder method, the process before encoding the second feature map includes: accessing a second set of distribution parameters; and updating the second feature map by transforming it based on the second set of distribution parameters.

[0276] For some implementations of this example learning-based point cloud geometry decoder method, the process includes aggregating the second feature map using a second neural network before encoding the second feature map.

[0277] For some implementations of this example of a learning-based point cloud geometry decoder method, the second neural network is selected from a group consisting of sparse convolutional neural networks (CNNs) and multiple perceptron layers (MLPs).

[0278] For some implementations of this example of a learning-based point cloud geometry decoder method, aggregating the second feature map may include using a residual network (ResNet) architecture.

[0279] For some implementations of this example learning-based point cloud geometry decoder method, the process includes aggregating the first feature map using a third neural network before transforming the first feature map into the second feature map.

[0280] For some implementations of this example of a learning-based point cloud geometry decoder method, the third neural network is selected from a group consisting of sparse convolutional neural networks (CNNs) and multiple perceptron layers (MLPs).

[0281] An example of a learning-based point cloud geometry decoder device according to some implementation schemes may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0282] Example methods according to some implementations may include: querying a local point cluster for each point in a point cloud having a selected cluster size; extracting a first point-by-point feature; performing a first and a second processing of a feedback process, the feedback process including: transforming the first point-by-point feature into a second point-by-point per-channel distribution feature based on a set of transformation parameters; extracting a third point-by-point feature from the second point-by-point feature via a deep network; aggregating the third point-by-point feature based on the local point cluster to form a first cluster feature; enhancing the third point-by-point feature using an extended version of the first cluster feature; and obtaining a next-level first point-by-point feature as the first point-by-point feature for the second processing of the feedback process.

[0283] For some implementations of this example method, the set of transformation parameters is a set of learnable transformation parameters.

[0284] For some implementations, the example method may further include: extending the first group-by-group feature to generate the extended version of the first group-by-group feature.

[0285] In some implementations of this example method, the next-level first pointwise feature can be the enhanced third pointwise feature.

[0286] For some implementations, the example method may further include: aggregating the second point-by-point feature based on the local point group to form a second point-by-group feature; enhancing the second point-by-point feature with an extended version of the second point-by-group feature to form an updated second point-by-point feature; obtaining a fourth point-by-point feature; and matching the updated second point-by-point feature with the fourth point-by-point feature.

[0287] For some implementations, the example method may also include: extending the third pointwise feature to generate the extended version of the second groupwise feature.

[0288] For some implementations of this example method, the potential code may include this third pointwise feature.

[0289] For some implementations of this example method, the set abstract feature may include the second group-by-group feature.

[0290] Example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to: query a local point group for each point in a point cloud having a selected group size; extract a first point-wise feature; and perform a first processing and a second processing of a feedback process, the instructions being further operable when executed by the processor to perform the feedback process and cause the apparatus to: transform the first point-wise feature into a second point-wise per-channel distribution feature based on a set of transformation parameters; extract a third point-wise feature from the second point-wise feature via a deep network; aggregate the third point-wise feature based on the local point group to form a first group-wise feature; enhance the third point-wise feature using an extended version of the first group-wise feature; and obtain a next-level first point-wise feature as the first point-wise feature for the second processing of the feedback process.

[0291] Additional example methods according to some implementations may include: querying a local point group for each point in a point cloud having a selected group size; extracting a first point-by-point feature; and performing a first and a second processing of a feedback process, the feedback process including: extracting a second point-by-point feature from the first point-by-point feature via a residual-based network; aggregating the second point-by-point feature based on the local point group to form a first point-by-group feature; transforming the first point-by-point feature into a third point-by-point per-channel distribution feature based on a set of transformation parameters; extracting a fourth point-by-point feature from the third point-by-point feature via a deep network; aggregating the fourth point-by-point feature based on the local point group to form a second point-by-group feature; enhancing the second point-by-point feature with an extended version of the first point-by-group feature to form a next-level first point-by-point feature, thereby serving as the first point-by-point feature for the second processing of the feedback process; and enhancing the fourth point-by-point feature with an extended version of the second point-by-group feature to form a next-level third point-by-point feature, thereby serving as the third point-by-point feature for the second processing of the feedback process.

[0292] For some implementations of this additional example method, the set of transformation parameters can be a set of learnable transformation parameters.

[0293] For some implementations, the additional example method may also include: extending the first group-by-group feature to generate the extended version of the first group-by-group feature.

[0294] For some implementations, the additional example method may also include: extending the second group-by-group feature to generate the extended version of the second group-by-group feature.

[0295] For some implementations, the additional example method may also include: extracting a fifth pointwise feature; transforming the fifth pointwise feature into a sixth pointwise per-channel feature distribution based on a second set of transformation parameters; and extracting a seventh pointwise feature from the sixth pointwise feature via a second residual-based network.

[0296] For some implementations of this additional example method, the second set of transformation parameters may be a second set of learnable transformation parameters.

[0297] For some implementations, this additional example method may also include: outputting the seventh point-by-point feature.

[0298] For some implementations of this additional example method, the potential code may include at least one of the first pointwise feature and the third pointwise feature.

[0299] For some implementations of this additional example method, the set of abstract features may include at least one of the first pointwise feature and the third pointwise feature.

[0300] Additional example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to: query a local point group for each point in a point cloud having a selected group size; extract a first point-wise feature; perform a first processing and a second processing of a feedback process, the instructions being further operable when executed by the processor to perform the feedback process and cause the apparatus to: extract a second point-wise feature from the first point-wise feature via a residual-based network; and aggregate the second point-wise feature based on the local point group to form a first group-wise feature; The first pointwise feature is transformed into a third pointwise per-channel distribution feature based on a set of transformation parameters; a fourth pointwise feature is extracted from the third pointwise feature via a deep network; the fourth pointwise feature is aggregated based on the local point cluster to form a second pointwise cluster feature; the second pointwise feature is enhanced using an extended version of the first pointwise cluster feature to form a next-level first pointwise feature, which serves as the first pointwise feature for the second processing in the feedback process; and the fourth pointwise feature is enhanced using an extended version of the second pointwise cluster feature to form a next-level third pointwise feature, which serves as the third pointwise feature for the second processing in the feedback process.

[0301] Other example methods according to some implementation schemes may include: querying a local point group for each point in a point cloud having a selected group size; extracting a first point-by-point feature; transforming the first point-by-point feature into a second point-by-point per-channel distribution feature based on a set of transformation parameters; extracting a third point-by-point feature from the second point-by-point feature via a deep network; aggregating the third point-by-point feature based on the local point group to form a first group-by-group feature; and extracting a second group-by-group feature from an expanded version of the first group-by-group feature.

[0302] In some implementations of this additional example method, the set of transformation parameters can be a set of learnable transformation parameters.

[0303] For some implementations, this additional example method may also include: extending the first group-by-group feature to generate the extended version of the first group-by-group feature.

[0304] For some implementations, this additional example method may also include: outputting the second group-by-group feature.

[0305] For some implementations, this additional example method may also include: extracting a fourth point-by-point feature from the first point-by-point feature via a residual-based network; aggregating the fourth point-by-point feature based on the local point group to form a third group-by-group feature; and extracting the fourth group-by-group feature from an extended version of the third group-by-group feature.

[0306] For some implementations, this additional example method may also include: extending the third group-by-group feature to generate the extended version of the third group-by-group feature.

[0307] For some implementations of this additional example method, the potential code may include the second group-by-group feature.

[0308] For some implementations of this additional example method, the set abstract feature may include the second group-by-group feature.

[0309] Other example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to: query a local point group for each point in a point cloud having a selected group size; extract a first point-wise feature; transform the first point-wise feature into a second point-wise per-channel distribution feature based on a set of transformation parameters; extract a third point-wise feature from the second point-wise feature via a deep network; aggregate the third point-wise feature based on the local point group to form a first group-wise feature; and extract a second group-wise feature from an extended version of the first group-wise feature.

[0310] In some embodiments of this additional example device, the instructions, when executed by the processor, may further operate such that the device: extracts a fourth point-by-point feature from the first point-by-point feature via a residual-based network; aggregates the fourth point-by-point feature based on the local point group to form a third group-by-group feature; and extracts the fourth group-by-group feature from an extended version of the third group-by-group feature.

[0311] Additional example methods according to some implementation schemes may include: extracting a first group feature; expanding the first group feature into a first point feature; transforming the first point feature into a second point per channel distribution feature based on a set of transformation parameters; extracting a third point feature from the second point feature via a deep network; and extracting a fourth point feature from the third point feature.

[0312] Additional example apparatus according to some implementations may include: one or more processors; and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform any of the methods listed above.

[0313] Example signals according to some implementation schemes may include: a bit stream, which is formed by performing any of the methods listed above.

[0314] According to some examples of implementation schemes, a computer-readable storage medium may store thereon instructions for encoding or decoding a point cloud according to any of the methods listed above.

[0315] According to some implementation examples, a learning-based point cloud geometry processing block method may include: accessing a first feature map, wherein the first feature map is the input to the processing block, and wherein the first feature map is generated by a first set of neural network layers; normalizing the elements of the first feature map to generate a second feature map; accessing a set of distribution parameters; transforming the second feature map into a third feature map based on the set of distribution parameters; and aggregating the third feature map into a fourth feature map.

[0316] For some implementations of the learning-based point cloud geometry processing block method in this example, the first feature map is a point-by-point feature map of local group queries.

[0317] For some implementations of the learning-based point cloud geometry processing block method in this example, the second feature map is a cascade of feature elements per slice computed by normalizing the corresponding slices from the first feature map.

[0318] For some implementations of this example of a learning-based point cloud geometry processing block method, the set of distribution parameters is obtained through training.

[0319] For some implementations of this example of a learning-based point cloud geometry processing block method, the set of distribution parameters is defined and applied per slice.

[0320] For some implementations of this example of a learning-based point cloud geometry processing block method, the fourth feature map is a cascade of per-local-group feature aggregations to generate a group-by-group feature map.

[0321] In some implementations of this example of a learning-based point cloud geometry processing block method, the fourth feature map is encoded into a bitstream.

[0322] Some implementations of the example learning-based point cloud geometry processing block method may further include: expanding the fourth feature map to the size of the third feature map; using the third feature map to enhance the expanded feature map; and repeating the learning-based point cloud geometry encoder method according to claim 34 once or more, wherein the enhanced feature map is used as the first feature map, and wherein the next set of distribution parameters is used as the distribution parameters.

[0323] For some implementations of this example of a learning-based point cloud geometry processing block method, the next set of distribution parameters is obtained through training.

[0324] For some implementations of this example learning-based point cloud geometry processing block method, the next set of distribution parameters is defined for each repetition of the learning-based point cloud geometry encoder method described in the preceding claims.

[0325] For some implementations of this example of a learning-based point cloud geometry processing block method, the third feature map is a point-by-point feature map.

[0326] For some implementations of the learning-based point cloud geometry processing block method in this example, the fourth feature map is a group-wise feature map.

[0327] For some implementations of this example of a learning-based point cloud geometry processing block method, the third feature map is a fine feature map.

[0328] For some implementations of this example of a learning-based point cloud geometry processing block method, the fourth feature map is a coarse feature map.

[0329] Some implementations of this example of a learning-based point cloud geometry processing block method may also include: connecting a residual network in parallel to the learning-based point cloud geometry encoder.

[0330] Some implementations of this example of a learning-based point cloud geometry processing block method may also include concatenating a residual network to the learning-based point cloud geometry encoder.

[0331] Some implementations of this example of a block-based approach to processing point cloud geometry may also include: inputting the fourth feature map into a second set of neural network layers; and encoding the output of the second set of neural network layers into a bitstream.

[0332] According to some implementation examples, a learning-based point cloud geometry processing block device may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the methods listed above.

[0333] According to some implementation examples, a learning-based point cloud geometry decoder method may include: accessing a first feature map, wherein the first feature map is input and wherein the first feature map is decoded from a bitstream; expanding the first feature map into a second feature map; normalizing the elements of the second feature map to generate a third feature map; accessing a set of distribution parameters; and transforming the third feature map into a fourth feature map based on the set of distribution parameters.

[0334] For some implementations of the learning-based point cloud geometry decoder method in this example, expanding the first feature map into the second feature map may include pooling the first feature map into the second feature map.

[0335] For some implementations of this example of a learning-based point cloud geometry decoder method, the first feature map is a decoded group-wise feature map.

[0336] In some implementations of this example of a learning-based point cloud geometry decoder method, the second feature map is generated by the first set of neural network layers.

[0337] For some implementations of this example of a learning-based point cloud geometry decoder method, the second feature map is expanded to the size of a point-by-point feature map.

[0338] In some implementations of this example learning-based point cloud geometry decoder method, the third feature map is a concatenation of feature elements per slice, and the third feature map is computed by normalizing the corresponding slices from the second feature map.

[0339] For some implementations of the learning-based point cloud geometry decoder method in this example, at least one corresponding slice in the corresponding slice is further divided into two or more groups.

[0340] For some implementations of this example of a learning-based point cloud geometry decoder method, the set of distribution parameters is obtained through training.

[0341] In some implementations of this example of a learning-based point cloud geometry decoder method, the fourth feature map is fed into a second set of neural network layers to reconstruct the point cloud.

[0342] Some implementations of the example learning-based point cloud geometry decoder method may also include: embedding the fourth feature map via a set of neural network layers; repeating the learning-based point cloud geometry decoder method once or multiple times, wherein the embedded feature map is used as the second feature map, and wherein a next set of distribution parameters is used as the distribution parameters.

[0343] For some implementations of this example of a learning-based point cloud geometry decoder method, the next set of distribution parameters is obtained through training.

[0344] For some implementations of this example learning-based point cloud geometry decoder method, the next set of distribution parameters is defined for each repetition of the learning-based point cloud geometry decoder method.

[0345] Some implementations of the example learning-based point cloud geometry decoder method may further include: aggregating the fourth feature map to a fifth feature map; expanding the fifth feature map to the size of the fourth feature map; using the fourth feature map to enhance the expanded feature map; repeating the learning-based point cloud geometry decoder method according to claim 84 once or more, wherein the enhanced feature map is used as the second feature map, and wherein the next set of distribution parameters is used as the distribution parameters.

[0346] For some implementations of this example of a learning-based point cloud geometry decoder method, the next set of distribution parameters is obtained through training.

[0347] For some implementations of this example learning-based point cloud geometry decoder method, the next set of distribution parameters is defined for each repetition of the learning-based point cloud geometry decoder method according to claim 84.

[0348] For some implementations of the learning-based point cloud geometry decoder method in this example, the fourth feature map is a point-by-point feature map.

[0349] For some implementations of this example of a learning-based point cloud geometry decoder method, the fifth feature map is a group-wise feature map.

[0350] For some implementations of this example of a learning-based point cloud geometry decoder method, the fourth feature map is a fine feature map.

[0351] For some implementations of this example of a learning-based point cloud geometry decoder method, the fifth feature map is a coarse feature map.

[0352] Some implementations of this example of a learning-based point cloud geometry decoder method may also include: connecting a residual network in parallel to the learning-based point cloud geometry decoder.

[0353] Some implementations of this example of a learning-based point cloud geometry decoder method may also include concatenating a residual network into the learning-based point cloud geometry decoder.

[0354] According to some implementation examples, a learning-based point cloud geometry decoder device may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform one of the methods listed above.

[0355] For some implementations of this example of a learning-based point cloud geometry processing block method, at least one of the distribution parameters is predetermined.

[0356] Example methods according to some implementations may include: querying a local point group for each point in a point cloud having a selected group size; extracting a first point-by-point feature; and performing a process including: transforming the first point-by-point feature into a second point-by-point feature according to a per-channel distribution specified by a set of transformation parameters; extracting a third point-by-point feature from the second point-by-point feature via a neural network layer; aggregating the third point-by-point feature based on the local point group to form a first group-by-group feature; enhancing the third point-by-point feature using an extended version of the first group-by-group feature; and obtaining a next-level first point-by-point feature as the first point-by-point feature for use in a next-level process.

[0357] This disclosure describes various aspects, including tools, features, implementation schemes, models, methods, etc. Many of these aspects are described in detail, and at least individual characteristics are shown, generally in a manner that may sound limited. However, this is for clarity in the description and does not limit the content or scope of the disclosure of those aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can also be combined and interchanged with those described in previous submissions.

[0358] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically shown, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of specific implementations. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having a bitstream generated according to any of the methods stored thereon.

[0359] In this disclosure, the terms “reconstructed” and “decoded” are used interchangeably, as are the terms “pixel” and “sample”, and the terms “image,” “picture,” and “frame.” Generally, but not necessarily, the term “reconstructed” is used on the encoder side, while “decoded” is used on the decoder side.

[0360] The terms HDR (High Dynamic Range) and SDR (Standard Dynamic Range) generally convey specific dynamic range values ​​to those skilled in the art. However, it is also intended to employ additional embodiments in which references to HDR are understood to mean "higher dynamic range," and references to SDR are understood to mean "lower dynamic range." Such additional embodiments are not bound by any specific value of dynamic range that may generally be associated with the terms "high dynamic range" and "standard dynamic range."

[0361] This document describes various methods, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or purpose of specific steps and / or actions may be modified or combined. Additionally, terms such as "first," "second," etc., may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply an ordering of the modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and may occur, for example, before, during, or in overlapping time periods of the second decoding.

[0362] For example, various numerical values ​​may be used in this disclosure. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0363] The embodiments described herein can be executed by a processor or other hardware or by computer software implemented by a combination of hardware and software. As a non-limiting example, these embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be of any type suitable for the technical environment and can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0364] Various embodiments involve decoding. As used in this disclosure, “decoding” can encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also include, or alternatively include, processes performed by a decoder of the various embodiments described in this disclosure, such as extracting an image from a tiled (packed) image, determining an upsampling filter to use, then upsampling the image, and flipping the image back to its intended orientation.

[0365] As another example, in one implementation, "decoding" refers only to entropy decoding; in another implementation, "decoding" refers only to differential decoding; and in yet another implementation, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to a broader decoding process will be determined based on the specific context of the description.

[0366] Various embodiments involve encoding. In a manner similar to the discussion above regarding “decoding,” the term “encoding,” as used herein, can encompass all or part of a process performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy coding. In various embodiments, such processes also include, or alternatively include, processes performed by the encoders of the various embodiments described herein.

[0367] As another example, in one implementation, "encoding" refers only to entropy encoding; in another, "encoding" refers only to differential encoding; and in yet another, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to specifically refer to a subset of operations or generally to a broader encoding process will be determined based on the specific context of the description.

[0368] When the accompanying drawings are presented as flowcharts, it should be understood that block diagrams of the corresponding devices are also provided. Similarly, when the accompanying drawings are presented as block diagrams, it should be understood that flowcharts of the corresponding methods / processes are also provided.

[0369] Various implementations refer to rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often taking into account computational complexity constraints. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. Different approaches exist to solve the rate-distortion optimization problem. For example, these methods may be based on extensive testing of all encoding options (including all considered modes or decoding parameter values) and a complete evaluation of their decoding costs and the associated distortion of the reconstructed signal after decoding and decoding. Faster methods can also be used to reduce encoding complexity, particularly for the computation of approximate distortion based on prediction or prediction of the residual signal rather than the reconstructed residual signal. A hybrid of these two approaches can also be used, such as by using approximate distortion only for some possible encoding options and full distortion for others. Other methods evaluate only a subset of possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of both decoding costs and associated distortion.

[0370] The specific embodiments and aspects described herein may be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even if discussed only in the context of a single form of specific embodiment (e.g., discussed only as a method), specific embodiments of the discussed features may also be implemented in other forms (e.g., apparatus or program). Apparatus may be implemented, for example, in suitable hardware, software, and firmware. Methods may be implemented, for example, in a processor that generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

[0371] References to “an implementation scheme” or “implementation scheme” or “a specific implementation” or “a specific implementation”, and their other variations, mean that the specific features, structures, characteristics, etc., described in connection with the implementation scheme are included in at least one implementation scheme. Therefore, the appearance of the phrase “in an implementation scheme” or “in an implementation scheme” or “in a specific implementation” or “in a specific implementation”, and any other variations appearing throughout this disclosure, do not necessarily refer to the same implementation scheme.

[0372] Additionally, this disclosure may refer to "determining" each piece of information. Determining information may include, for example, one or more of estimated information, calculated information, predicted information, or information retrieved from memory.

[0373] Furthermore, this disclosure may refer to “accessing” each piece of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or more of these.

[0374] Additionally, this disclosure may refer to "receiving" each piece of information. Like "access," "receiving" is intended as a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) or more. Furthermore, "receiving" typically participates in one or more of the following operations: such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0375] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one of” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such phrases are intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to as many items as possible listed.

[0376] Moreover, as used herein, the term "signaling" specifically refers to the corresponding decoder indicating something. For example, in some implementations, the encoder signals.

[0377] Region-based filter parameter selection chooses a specific parameter from a set of parameters used for artifact removal filtering. Thus, in one implementation, the same parameter is used on both the encoder and decoder sides. Therefore, for example, the encoder can send a specific parameter (explicit signaling) to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without sending it (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various implementations by avoiding sending any actual functionality. It should be understood that signaling can be implemented in various ways. For example, in various implementations, information is signaled to the corresponding decoder using one or more syntax elements, flags, etc. Although the verb form of the term "signal" is mentioned above, the term can also be used herein as the noun "signal".

[0378] Specific implementations may generate various signals, formatting these signals to carry, for example, information that can be stored or transmitted. This information may include, for example, instructions for performing a method or data generated by one of the specific implementations. For example, the signal may be formatted to carry a bit stream of the implementation. Such signals may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and using the encoded data stream to modulate a carrier wave. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via various wired or wireless links. The signal may be stored on a processor-readable medium.

[0379] Several embodiments are described. Features of these embodiments may be provided individually or in any combination across various claim classes and types. Furthermore, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination across various claim classes and types:

[0380] • Adjust the residual at the encoder according to any of the implementation schemes discussed.

[0381] • A bitstream or signal including one or more of the syntax elements or their variants.

[0382] • A bitstream or signal comprising a syntax for transmitting information generated according to any of the embodiments described.

[0383] • Insert a syntax element into the signaling that enables the decoder to adjust the residual in a manner corresponding to that used by the encoder.

[0384] • Creating and / or sending and / or receiving and / or decoding bitstreams or signals comprising one or more of the stated syntax elements or variations thereof.

[0385] • Creating and / or sending and / or receiving and / or decoding according to any one of the embodiments described.

[0386] • The method, process, apparatus, medium for storing instructions, medium for storing data, or signal according to any one of the embodiments described above.

[0387] • A television, set-top box, cellular phone, tablet computer or other electronic device that performs adaptive filter parameters according to any of the embodiments described.

[0388] • A television, set-top box, cellular phone, tablet computer, or other electronic device that performs adaptive filter parameters according to any of the embodiments and displays the resulting image (e.g., using a monitor, screen, or other type of display).

[0389] • (For example, using a tuner) Selecting a channel to receive a signal including an encoded image and performing adaptive filtering of the filter parameters according to any of the embodiments of the present invention on a television, set-top box, cellular phone, tablet computer or other electronic device.

[0390] • A television, set-top box, cellular phone, tablet computer, or other electronic device that receives signals including encoded images over the air (e.g., using an antenna) and performs adaptive filtering of filter parameters according to any of the embodiments described.

[0391] It should be noted that the various hardware elements of one or more embodiments in the described implementation scheme are referred to as “modules” that perform (i.e., execute, implement, etc.) the various functions described herein in conjunction with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that a person skilled in the art would consider suitable for a given specific implementation. Each of the modules may also include executable instructions for performing one or more functions described as being performed by the respective module, and it should be noted that these instructions may take the form of or include the following instructions: hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may be stored in any suitable one or more non-transitory computer-readable media (such as, commonly referred to as RAM, ROM, etc.).

[0392] Although features and elements have been described above in specific combinations, those skilled in the art will understand that each feature or element may be used alone or in any combination with other features and elements. Furthermore, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as internal hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROM disks and digital versatile optical discs (DVDs)). A processor associated with the software may be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A learning-based point cloud geometry processing block method, the method comprising: Access the first feature map, wherein the first feature map has C channels and is an input to the processing block, and wherein the first feature map is generated by a first set of neural network layers; access a set of distribution parameters; transforming the first feature map into a second feature map based on the set of distribution parameters; as well as The second feature map is encoded into a bitstream.

2. The method according to claim 1, further comprising: The first feature map is updated by normalizing vector elements of the first feature map.

3. The method of claim 2, wherein normalizing the vector elements of the first feature map comprises: determining a respective length of each feature vector associated with one of the vector elements of the first feature map; as well as Each of the vector elements of the first feature map is divided by the corresponding length.

4. The method of claim 2, wherein normalizing the vector elements of the first feature map comprises: arranging the vector by reshaping each associated feature channel in the first feature map; determining the corresponding length of each reshaped vector for each feature channel; as well as dividing each of the vector elements of each reshaped vector by the corresponding length for each feature channel, Each vector element is one of the elements of the first feature map.

5. The method of claim 2, wherein normalizing the vector elements of the first feature map comprises: arranging the vector by reshaping each associated feature channel in the first feature map; determining the corresponding standard deviation of each reshaped vector for each feature channel; as well as determining a corresponding mean value of each reshaped vector for each feature channel; and updating each vector element of each vector by subtracting the corresponding mean value; and Divide each updated vector element of each vector by the corresponding standard deviation for each feature channel, Each vector element is one of the elements of the first feature map.

6. The method according to any one of claims 1 to 5, wherein the set of distribution parameters is determined during a training period using a back-propagation technique.

7. A method according to any one of claims 1 to 5, wherein the set of distribution parameters is determined on a per-feature channel basis.

8. The method according to any one of claims 1 to 7, further comprising: The second feature map is updated by performing downsampling using an average pooling or maximum pooling function.

9. The method according to any one of claims 1 to 7, further comprising: determining a third feature map by filtering the second feature map using a smoothing filter; as well as The second feature map is updated by concatenating the third feature map to the second feature map.

10. The method according to any one of claims 1 to 9, wherein before encoding the second feature map, performing a process comprising: Access the second set of distribution parameters; as well as The second feature map is updated by transforming the second feature map based on the second set of distribution parameters.

11. The method of any one of claims 1 to 10, wherein before encoding the second feature map, performing a process includes aggregating the second feature map using a second neural network.

12. The method of claim 11, wherein the second neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

13. The method of any one of claims 11 to 12, wherein aggregating the second feature map comprises using a residual network (ResNet) architecture.

14. The method according to any one of claims 1 to 13, further comprising: determining a fourth feature map by aggregating the first feature map using a neural network in parallel with transforming the first feature map into the second feature map; as well as The second feature map is updated by concatenating the fourth feature map to the second feature map.

15. The method of claim 14, wherein aggregating the first feature map using the neural network comprises using a residual network (ResNet) architecture.

16. The method of any one of claims 1 to 15, wherein before transforming the first feature map into the second feature map, performing a process includes aggregating the first feature map using a third neural network.

17. The method of claim 16, wherein the third neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

18. A device, comprising: processor; and A non-transitory computer readable medium storing instructions which, when executed by the processor, are operable to cause the apparatus to perform any one of claims 1 to 17 19. A method for a point cloud geometry decoder based on learning, the method comprising: decoding a first feature map from a bitstream; access a set of distribution parameters; transforming the first feature map into a second feature map based on the set of distribution parameters; as well as The point cloud is reconstructed according to the second feature map.

20. The method according to claim 19, further comprising: The first feature map is updated by normalizing elements of the first feature map.

21. The method of claim 20, wherein normalizing the elements of the first feature map comprises: determining a respective length of each feature vector associated with one of the elements of the first feature map; as well as Each element of the first feature map is divided by the corresponding length.

22. The method of claim 20, wherein normalizing the elements of the first feature map comprises: arranging each of the one or more vectors by reshaping an associated feature channel in the first feature map; determining, for each feature channel, a respective length of each of the one or more vectors; and Divide each vector element of each vector by the corresponding length for each feature channel, Each vector element is one of the elements of the first feature map.

23. The method of claim 20, wherein normalizing the elements of the first feature map comprises: arranging each of the one or more vectors by reshaping an associated feature channel in the first feature map; determining, for each feature channel, a respective standard deviation of each of the one or more vectors; as well as determining, for each feature channel, a respective mean value for each of the one or more vectors; and updating each vector element of each vector by subtracting the corresponding mean value; and Divide each updated vector element of each vector by the corresponding standard deviation for each feature channel, Each vector element is one of the elements of the first feature map.

24. A method according to any one of claims 19 to 23, wherein the set of distribution parameters is determined during a training period using a back-propagation technique.

25. A method according to any one of claims 19 to 23, wherein the set of distribution parameters is determined on a per feature channel basis.

26. The method according to any one of claims 19 to 27, wherein before encoding the second feature map, performing a process comprising: Access the second set of distribution parameters; as well as The second feature map is updated by transforming the second feature map based on the second set of distribution parameters.

27. The method of any one of claims 19 to 26, wherein before encoding the second feature map, performing a process includes aggregating the second feature map using a second neural network.

28. The method of claim 27, wherein the second neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

29. The method of any one of claims 27 to 28, wherein aggregating the second feature map comprises using a residual network (ResNet) architecture.

30. The method of any one of claims 19 to 29, wherein before transforming the first feature map to the second feature map, performing a process includes aggregating the first feature map using a third neural network.

31. The method of claim 30, wherein the third neural network is selected from the group consisting of a sparse convolutional neural network (CNN) and a multi-perceptron layer (MLP).

32. An apparatus, comprising: processor; and A non-transitory computer readable medium storing instructions which, when executed by the processor, are operable to cause the apparatus to perform any one of claims 19 to 31.

33. A method comprising: querying a local point cluster for each point in the point cloud with a selected cluster size; Extracting the first point-wise feature; as well as A first processing and a second processing of a feedback process are performed, wherein the feedback process includes: transforming the first point-by-point feature into a second point-by-point per-channel distribution feature based on a set of transformation parameters; Extracting a third point-by-point feature from the second point-by-point feature via a deep network; aggregating the third point-by-point features based on the local point groups to form first group-by-group features; enhancing the third point-wise feature using an extended version of the first group-wise feature; and A next-level first point-by-point feature is obtained as the first point-by-point feature for the second processing of the feedback process.

34. The method of claim 33, wherein the set of transformation parameters is a learnable set of transformation parameters.

35. The method according to any one of claims 33 to 34, further comprising: The first group-wise feature is expanded to generate the expanded version of the first group-wise feature.

36. The method according to any one of claims 33 to 35, wherein the next-level first point-by-point feature is an enhanced third point-by-point feature.

37. The method according to any one of claims 33 to 36, further comprising: aggregating the second point-by-point features based on the local point group to form a second group-by-group feature; enhancing the second point-by-point feature using an extended version of the second group-by-group feature to form an updated second point-by-point feature; Obtain the fourth point-wise feature; as well as The updated second point-by-point feature is matched with the fourth point-by-point feature.

38. The method of claim 37, further comprising: The third point-wise feature is expanded to generate the expanded version of the second group-wise feature.

39. A method according to any one of claims 33 to 38, wherein the latent code comprises the third point-by-point feature.

40. The method of any one of claims 33 to 39, wherein a collective abstract feature comprises the second group-by-group feature.

41. An apparatus, comprising: processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to: querying a local point cluster for each point in the point cloud with a selected cluster size; Extracting the first point-wise feature; as well as performing a first process and a second process of a feedback process, the instructions, when executed by the processor, may further operate to perform the feedback process and cause the device to: transforming the first point-by-point feature into a second point-by-point per-channel distribution feature based on a set of transformation parameters; Extracting a third point-by-point feature from the second point-by-point feature via a deep network; aggregating the third point-by-point features based on the local point groups to form first group-by-group features; enhancing the third point-wise feature using an extended version of the first group-wise feature; as well as A next-level first point-by-point feature is obtained as the first point-by-point feature for the second processing of the feedback process.

42. A method comprising: querying a local point cluster for each point in the point cloud with a selected cluster size; Extracting the first point-wise feature; as well as A first processing and a second processing of a feedback process are performed, wherein the feedback process includes: extracting second point-by-point features from the first point-by-point features via a residual-based network; aggregating the second point-by-point features based on the local point group to form a first group-by-group feature; transforming the first point-by-point feature into a third point-by-point per-channel distribution feature based on a set of transformation parameters; Extracting a fourth point-by-point feature from the third point-by-point feature via a deep network; aggregating the fourth point-by-point features based on the local point groups to form second group-by-group features; enhancing the second point-by-point features using an extended version of the first group-by-group features to form next-level first point-by-point features, thereby serving as the first point-by-point features for the second processing of the feedback process; and The fourth point-by-point feature is enhanced by using the extended version of the second group-by-group feature to form a next-level third point-by-point feature, so as to be used as the third point-by-point feature for the second processing of the feedback process.

43. The method of claim 42, wherein the set of transformation parameters is a set of learnable transformation parameters.

44. The method according to any one of claims 42 to 43, further comprising: The first group-wise feature is expanded to generate the expanded version of the first group-wise feature.

45. The method according to any one of claims 42 to 44, further comprising: The second group-wise feature is expanded to generate the expanded version of the second group-wise feature.

46. ​​The method according to any one of claims 42 to 45, further comprising: Extracting the fifth point-wise feature; transforming the fifth point-by-point feature into a sixth point-by-point per-channel feature distribution based on a second set of transformation parameters; as well as A seventh point-by-point feature is extracted from the sixth point-by-point feature via a second residual-based network.

47. The method of claim 46, wherein the second set of transformation parameters is a second set of learnable transformation parameters.

48. The method of claim 46, further comprising: The seventh point-by-point feature is output.

49. The method of any one of claims 42 to 48, wherein a latent code comprises at least one of the first point-by-point feature and the third point-by-point feature.

50. The method of any one of claims 42 to 49, wherein a collective abstract feature comprises at least one of the first point-by-point feature and the third point-by-point feature.

51. An apparatus, comprising: processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to: querying a local point cluster for each point in the point cloud with a selected cluster size; Extracting the first point-wise feature; as well as performing a first process and a second process of a feedback process, the instructions, when executed by the processor, may further operate to perform the feedback process and cause the device to: extracting second point-by-point features from the first point-by-point features via a residual-based network; aggregating the second point-by-point features based on the local point group to form a first group-by-group feature; transforming the first point-by-point feature into a third point-by-point per-channel distribution feature based on a set of transformation parameters; Extracting a fourth point-by-point feature from the third point-by-point feature via a deep network; aggregating the fourth point-by-point features based on the local point groups to form second group-by-group features; enhancing the second point-by-point features by using an extended version of the first group-by-group features to form next-level first point-by-point features, thereby serving as the first point-by-point features for the second processing of the feedback process; as well as The fourth point-by-point feature is enhanced by using the extended version of the second group-by-group feature to form a next-level third point-by-point feature, so as to be used as the third point-by-point feature for the second processing of the feedback process.

52. A method comprising: querying a local point cluster for each point in the point cloud with a selected cluster size; Extracting the first point-wise feature; transforming the first point-by-point feature into a second point-by-point per-channel distribution feature based on a set of transformation parameters; Extracting a third point-by-point feature from the second point-by-point feature via a deep network; aggregating the third point-by-point features based on the local point groups to form first group-by-group features; as well as A second group-wise feature is extracted from an extended version of the first group-wise feature.

53. The method of claim 52, wherein the set of transformation parameters is a set of learnable transformation parameters.

54. The method according to any one of claims 52 to 53, further comprising: The first group-wise feature is expanded to generate the expanded version of the first group-wise feature.

55. The method according to any one of claims 52 to 54, further comprising: The second group-by-group features are output.

56. The method according to any one of claims 52 to 55, further comprising: extracting fourth point-by-point features from the first point-by-point features via a residual-based network; aggregating the fourth point-by-point features based on the local point groups to form third group-by-group features; and A fourth group-wise feature is extracted from the extended version of the third group-wise feature.

57. The method according to any one of claims 52 to 56, further comprising: The third group-wise feature is expanded to generate the expanded version of the third group-wise feature.

58. A method according to any one of claims 52 to 57, wherein the latent code comprises the second group-by-group feature.

59. The method of any one of claims 52 to 58, wherein a collective abstract feature comprises the second group-by-group feature.

60. An apparatus, comprising: processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to: querying a local point cluster for each point in the point cloud with a selected cluster size; Extracting the first point-wise feature; transforming the first point-by-point feature into a second point-by-point per-channel distribution feature based on a set of transformation parameters; Extracting a third point-by-point feature from the second point-by-point feature via a deep network; aggregating the third point-by-point features based on the local point groups to form first group-by-group features; as well as A second group-wise feature is extracted from an extended version of the first group-wise feature.

61. The device of claim 60, wherein the instructions, when executed by the processor, are further operable to cause the device to: extracting fourth point-by-point features from the first point-by-point features via a residual-based network; aggregating fourth point-by-point features based on the local point groups to form third group-by-group features; and A fourth group-wise feature is extracted from the extended version of the third group-wise feature.

62. A method comprising: Extracting first group-by-group features; Expanding the first group-wise feature into a first point-wise feature; transforming the first point-by-point feature into a second point-by-point per-channel distribution feature based on a set of transformation parameters; extracting a third point-by-point feature from the second point-by-point feature via a deep network; and A fourth point-by-point feature is extracted from the third point-by-point feature.

63. An apparatus, comprising: one or more processors; and at least one memory coupled to the one or more processors, Wherein the one or more processors are configured to perform any one of claims 33 to 40, 42 to 50, 52 to 59 and 62.

64. A signal comprising a bit stream, the signal being formed by executing any one of claims 33 to 40, 42 to 50, 52 to 59 and 62.

65. A computer readable storage medium having stored thereon instructions for encoding or decoding a point cloud according to any one of claims 33 to 40, 42 to 50, 52 to 59 and 62.

66. A learning-based point cloud geometry processing block method, the method comprising: Access the first feature map, wherein the first feature map is an input to the processing block, and wherein the first feature map is generated by a first set of neural network layers; Normalizing elements of the first feature map to generate a second feature map; access a set of distribution parameters; transforming the second feature map into a third feature map based on the set of distribution parameters; as well as The third feature map is aggregated into a fourth feature map.

67. The method of claim 66, wherein the first feature map is a point-by-point feature map of a local cluster query.

68. A method according to any one of claims 66 to 67, wherein the second feature map is a cascade of per-slice feature elements computed by normalizing corresponding slices from the first feature map.

69. The method according to any one of claims 66 to 68, wherein the set of distribution parameters is obtained by training.

70. The method of any one of claims 66 to 69, wherein the set of distribution parameters is defined and applied per slice.

71. A method according to any one of claims 66 to 70, wherein the fourth feature map is a cascade of per-local group feature aggregations to generate a group-by-group feature map.

72. A method according to any one of claims 66 to 71, wherein the fourth feature map is encoded into a bitstream.

73. The method of any one of claims 66 to 72, further comprising: Expanding the fourth feature map to a size of the third feature map; enhancing the extended feature map using the third feature map; as well as Repeating the learning-based point cloud geometry encoder method according to claim 34 one or more times, wherein the enhanced feature map is used as the first feature map, and Wherein the next set of distribution parameters are used as the distribution parameters.

74. The method of claim 73, wherein the next set of distribution parameters is obtained through training.

75. A method according to any one of claims 73 to 74, wherein the next set of distribution parameters is defined for each repetition of the learning-based point cloud geometry encoder method according to claim 34.

76. A method according to any one of claims 73 to 75, wherein the third feature map is a point-by-point feature map.

77. A method according to any one of claims 73 to 75, wherein the fourth feature map is a group-by-group feature map.

78. A method according to any one of claims 73 to 75, wherein the third feature map is a fine feature map.

79. A method according to any one of claims 73 to 75, wherein the fourth feature map is a coarse feature map.

80. The method of any one of claims 66 to 79, further comprising: A residual network is connected in parallel to the learning-based point cloud geometry encoder.

81. The method of any one of claims 66 to 79, further comprising: The residual network is concatenated to the learning-based point cloud geometry encoder.

82. The method of any one of claims 66 to 81, further comprising: Inputting the fourth feature map into a second set of neural network layers; as well as Encoding outputs of the second set of neural network layers into a bitstream.

83. An apparatus, comprising: processor; and A non-transitory computer readable medium storing instructions which, when executed by the processor, are operable to cause the apparatus to perform any one of claims 66 to 82.

84. A learning-based point cloud geometry decoder method, the method comprising: Access the first feature map, where the first feature map is the input, and wherein the first feature map is decoded from a bitstream; Expanding the first feature map into a second feature map; Normalizing elements of the second feature map to generate a third feature map; access a set of distribution parameters; as well as The third feature map is transformed into a fourth feature map based on the set of distribution parameters.

85. The method of claim 84, wherein expanding the first feature map into the second feature map comprises pooling the first feature map into the second feature map.

86. A method according to any one of claims 84 to 85, wherein the first feature map is a decoded group-by-group feature map.

87. A method according to any one of claims 84 to 86, wherein the second feature map is generated by a first set of neural network layers.

88. A method according to any one of claims 84 to 87, wherein the second feature map is expanded to the size of a point-by-point feature map.

89. The method according to any one of claims 84 to 88, wherein the third feature map is a concatenation of per-slice feature elements, and Wherein the third feature map is calculated by normalizing the corresponding slice from the second feature map.

90. The method of claim 89, wherein at least one of the corresponding slices is further divided into two or more groups.

91. A method according to any one of claims 84 to 90, wherein the set of distribution parameters is obtained through training.

92. A method according to any one of claims 84 to 91, wherein the fourth feature map is input into a second set of neural network layers to reconstruct a point cloud.

93. The method of any one of claims 84 to 92, further comprising: embedding the fourth feature map via a set of neural network layers; Repeating the learning-based point cloud geometry decoder method according to claim 84 one or more times, wherein the embedded feature map is used as the second feature map, and Wherein the next set of distribution parameters are used as the distribution parameters.

94. The method of claim 93, wherein the next set of distribution parameters is obtained through training.

95. A method according to any one of claims 92 to 94, wherein the next set of distribution parameters is defined for each repetition of the learning-based point cloud geometry decoder method according to claim 84.

96. The method of any one of claims 84 to 92, further comprising: aggregating the fourth feature map into a fifth feature map; Expanding the fifth feature map to a size of the fourth feature map; enhancing the extended feature map using the fourth feature map; Repeating the learning-based point cloud geometry decoder method according to claim 84 one or more times, wherein the enhanced feature map is used as the second feature map, and Wherein the next set of distribution parameters are used as the distribution parameters.

97. The method of claim 96, wherein the next set of distribution parameters is obtained through training.

98. A method according to any one of claims 96 to 97, wherein the next set of distribution parameters is defined for each repetition of the learning-based point cloud geometry decoder method according to claim 84.

99. A method according to any one of claims 96 to 98, wherein the fourth feature map is a point-by-point feature map.

100. The method of any one of claims 96 to 98, wherein the fifth feature map is a group-by-group feature map.

101. The method of any one of claims 96 to 98, wherein the fourth feature map is a fine feature map.

102. A method according to any one of claims 96 to 98, wherein the fifth feature map is a coarse feature map.

103. The method of any one of claims 84 to 102, further comprising: A residual network is connected in parallel to the learning-based point cloud geometry decoder.

104. The method of any one of claims 84 to 103, further comprising: The residual network is concatenated to the learning-based point cloud geometry decoder.

105. An apparatus, the apparatus comprising: processor; and A non-transitory computer readable medium storing instructions which, when executed by the processor, are operable to cause the apparatus to perform any one of claims 84 to 104.

106. The method of claim 76, wherein at least one of the distribution parameters is predetermined.

107. A method comprising: querying a local point cluster for each point in the point cloud with a selected cluster size; Extracting the first point-wise feature; as well as The execution process includes: transforming the first point-wise features into second point-wise features according to a per-channel distribution specified by a set of transformation parameters; extracting a third point-by-point feature from the second point-by-point feature via a neural network layer; aggregating the third point-by-point features based on the local point groups to form first group-by-group features; enhancing the third point-wise feature using an extended version of the first group-wise feature; and The next-level first point-by-point features are obtained as the first point-by-point features used for the next-level process.

Images