Learning-Based Bit-Wise Octree Entropy Coding Compression and Processing in Light Detection and Ranging (LIDAR) and Other Systems

Abstract

The device may use a tree-based point cloud decoder to access occupancy bit sets of neighboring voxels at a current level of detail (LoD) that have been previously coded or decoded. The device may use a neural network module to compute features based on the accessed voxels. The device may access a deep feature vector from a previous LoD. The device may use another neural network module to combine two features to obtain a new combined feature. The device may concatenate the new combined feature with known features of the current voxel to be coded or decoded to construct a new global feature. The device may use yet another neural network module to predict a probability distribution of occupancies of voxels at the current LoD based on the global feature. One or more of the neural network modules may be convolution-based neural network modules.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 388,462, filed July 12, 2022, the contents of which are incorporated herein by reference. [Background technology]

[0002] Point cloud compression and processing advances the compression, analysis, interpolation, representation, and understanding of point cloud signals. The point cloud data format is a universal data format across several processing domains, from autonomous driving, robotics, augmented reality (AR) / virtual reality (VR), civil engineering, computer graphics, to the animation / film industry. Self-driving cars are being equipped with three-dimensional (3D) light detection and ranging (LiDAR) sensors, and affordable LiDAR sensors are becoming available from various manufacturers. With advances in sensing technology, 3D point cloud data is becoming more practical than ever and is expected to become the ultimate enabler for the described applications.

[0003] Point cloud data is also expected to consume a large portion of network traffic between vehicles connected via immersive communications, such as fifth-generation (5G) networks and VR / AR-enabled communications. Efficient representation formats are necessary for understanding and communicating point clouds. In particular, raw point cloud data needs to be properly organized and processed for world modeling and sensing purposes. Compression of raw point clouds is essential when data storage and transmission are required in relevant scenarios. Summary of the Invention

[0004] The embodiments and examples provided herein improve binary voxel-based octree coding methods through an improved method for context modeling. Given a current voxel to be coded / decoded, its occupancy probability can be estimated based on associated binary occupancy contexts at the current and parent level of detail (LoD). However, the improved method provided herein operates at the parent LoD resolution to process the current and parent binary contexts. Under this proposed model, costly upsampling operations are not required, resulting in reduced computational and memory complexity compared to existing learning-based approaches.

[0005] In examples provided herein, a device, which may be a wireless device such as a wireless transmit / receive unit (WTRU), may be configured to access occupancy bit sets of encoded or decoded neighboring voxels in a current LoD using a tree-based point cloud decoder. Further, the device may compute a first feature based on the accessed occupancy bit sets of neighboring voxels in the current LoD using a first neural network module. In one example, the first neural network module may be a first convolution-based neural network module.

[0006] The device can also access second features. In one example, the second features can be deep feature vectors from a previous LoD. Furthermore, the device can combine the first and second features to obtain a new, combined third feature.

[0007] Additionally, the device can concatenate the new, combined third feature with known features of the current voxel to be encoded or decoded to construct a new, comprehensive fourth feature. Additionally, the device can use a second neural network module to predict a probability distribution of the occupancy of the voxel at the current LoD based on the comprehensive fourth feature. In one example, the second neural network module can be a second convolution-based neural network module. In another example, the second neural network module can be a fully connected (FC) module. In a further example, the second neural network module can be part of a multi-layer perceptron (MLP)-based module.

[0008] Additionally, the device may derive a fifth feature based on the second feature using a third neural network module. In one example, the third neural network module may be a third convolution-based neural network module. In additional or alternative examples, the fifth feature may be combined with the first feature to derive a third feature.

[0009] In a further example, the device may access a sixth feature. The sixth feature may, in one example, be based on bits from encoded or decoded voxels at the current LoD. As another example, the fifth feature may be obtained by combining the second feature and the sixth feature using a second neural network module or a third neural network model. Also, the fourth feature may be further constructed based on the sixth feature combined with the third feature. Additionally or alternatively, the fourth feature may be further constructed based on the second feature combined with the third feature.

[0010] In a further example, the prediction of the probability distribution of the occupancy of a voxel in the current LoD may be further based on context information about the coded or decoded current voxel. In another example, the second feature may be based on the first point cloud of the previous LoD. Also, in one example, occupancy bits of neighboring voxels in the current LoD may be received in the bitstream.

[0011] In yet a further example, the device can reconstruct an octree structure based on the probability distribution of voxel occupancies at the current LoD. Further, the device can reconstruct a second point cloud based on leaf nodes of the reconstructed octree structure. Further, the device can then transmit the bits of the second point cloud. [Brief explanation of the drawings]

[0012] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings, in which like reference numerals indicate similar elements and in which: [Figure 1A] 1 is a system diagram illustrating an example communication system in which one or more disclosed embodiments may be implemented. [Figure 1B] 1B is a system diagram illustrating an exemplary wireless transmit / receive unit (WTRU) that may be used within the communication system illustrated in FIG. 1A, according to one embodiment. [Figure 1C] 1A is a system diagram illustrating an example radio access network (RAN) and an example core network (CN) that may be used within the communication system illustrated in FIG. 1A, according to one embodiment. [Figure 1D] 1B is a system diagram illustrating a further exemplary RAN and a further exemplary CN that may be used within the communication system illustrated in FIG. 1A, according to one embodiment. [Figure 2] FIG. 1 is a diagram illustrating an exemplary point cloud level of detail (LoD) configuration. [Figure 3] FIG. 1 is an architecture diagram illustrating a proposed exemplary basic architecture. [Figure 4] FIG. 1 is an architecture diagram illustrating a proposed exemplary basic architecture deployed. [Figure 5A] FIG. 10 is an operation diagram illustrating an example of a prediction operation between bits. [Figure 5B] FIG. 10 is an operation diagram illustrating an example of a prediction operation between bits. [Figure 6] FIG. 10 is a flow chart diagram of an example of predicting voxel occupancy using occupancy bits of coded or decoded neighboring voxels. [Figure 7] FIG. 10 is a flowchart diagram of another example of predicting voxel occupancy using occupancy bits of coded or decoded neighboring voxels. [Figure 8] FIG. 11 is an architecture diagram illustrating an example of the proposed architecture with residual connections from previous level features. [Figure 9] FIG. 1 is an architecture diagram illustrating an example of the proposed architecture with residual connections from features of the previous bit. [Figure 10] FIG. 1 is an architecture diagram illustrating an example of a proposed basic architecture, including exemplary network details. [Figure 11] FIG. 10 is an extraction diagram illustrating an example of feature extraction from all previously decoded bits. DETAILED DESCRIPTION OF THE INVENTION

[0013] 1A is a diagram illustrating an example communication system 100 in which one or more disclosed embodiments may be implemented. Communication system 100 may be a multiple-access system that provides content, such as voice, data, video, messaging, broadcasts, etc., to multiple wireless users. Communication system 100 may enable multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication system 100 may use one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word discrete Fourier transform spread OFDM (ZT-UW-DFT-S-OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multicarrier (FBMC), etc.

[0014] 1A, communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a radio access network (RAN) 104, a core network (CN) 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, although it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a station (STA), may be configured to transmit and / or receive wireless signals and may include user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a mobile phone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and application (e.g., remote surgery), an industrial device and application (e.g., robots and / or other wireless devices operating in an industrial and / or automated processing chain context), a consumer electronics device, a device operating in a commercial and / or industrial wireless network, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be referred to interchangeably as a UE.

[0015] The communications system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communications networks, such as the CN 106, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNode B (eNB), a Home Node B, a Home eNode B, a next generation Node B (gNode B (gNB), etc.), a new radio (NR) Node B, a site controller, an access point (AP), a wireless router, etc. Although the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0016] The base station 114a may be part of the RAN 104, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage for a particular geographic area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In one embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers per sector of the cell, for example, using beamforming to transmit and / or receive signals in desired spatial directions.

[0017] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0018] More specifically, as noted above, the communications system 100 may be a multiple-access system, but may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base station 114a and the WTRUs 102a, 102b, 102c of the RAN 104 may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 116 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink (DL) Packet Access (HSDPA) and / or High-Speed Uplink (UL) Packet Access (HSUPA).

[0019] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0020] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR radio access, which may establish the air interface 116 using NR.

[0021] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may jointly implement LTE radio access and NR radio access, e.g., using dual connectivity (DC) principles. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to and from multiple types of base stations (e.g., eNBs and gNBs).

[0022] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a wireless technology such as IEEE 802.11 (i.e., Wireless Fidelity, WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access, WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), or the like.

[0023] 1A may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point and may utilize any suitable RAT to facilitate wireless connectivity in a local area such as a business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, etc. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may establish a picocell or a femtocell using a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). As shown in FIG. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not need to access the Internet 110 through the CN 106.

[0024] The RAN 104 may communicate with the CN 106, which may be any type of network configured to provide voice, data, application, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have various quality of service (QoS) requirements, such as different throughput, latency, error tolerance, reliability, data throughput, mobility, etc. The CN 106 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions such as user authentication. Although not shown in FIG. 1A , it will be understood that the RAN 104 and / or CN 106 may communicate directly or indirectly with other RANs that use the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which may utilize NR radio technology, the CN 106 may also communicate with another RAN (not shown) employing GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0025] The CN 106 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), the user datagram protocol (UDP), and / or the internet protocol (IP) of the TCP / IP Internet protocol suite. The networks 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the network 112 may include another CN connected to one or more RANs, which may use the same RAT as the RAN 104 or a different RAT.

[0026] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links.) For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with a base station 114a, which may employ a cellular-based wireless technology, and a base station 114b, which may employ an IEEE 802.2 wireless technology.

[0027] 1B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0028] The processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. While FIG. 1B depicts the processor 118 and the transceiver 120 as separate components, it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0029] The transmit / receive element 122 may be configured to transmit or receive signals to or from a base station (e.g., base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR signals, UV signals, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF signals and light signals. It will be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0030] 1B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0031] The transceiver 120 may be configured to modulate signals transmitted by the transmit / receive element 122 and demodulate signals received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as, for example, NR and IEEE 802.11.

[0032] The processor 118 of the WTRU 102 may be coupled to and may receive user-entered data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. Additionally, the processor 118 may access information from and store data in any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In other embodiments, the processor 118 may access information from and store data in memory that is not physically located on the WTRU 102, such as on a server or home computer (not shown).

[0033] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control the power to other components in the WTRU 102. The power source 134 may be any suitable device for providing power to the WTRU 102. For example, the power source 134 may include one or more dry batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0034] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or instead of, information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0035] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, etc. The peripherals 138 may include one or more sensors. The sensor may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, a direction sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, a humidity sensor, and the like.

[0036] The WTRU 102 may include a full-duplex radio where transmission and reception of some or all of the signals (e.g., associated with a particular subframe on both the UL (e.g., for transmission) and DL (e.g., for reception)) may be simultaneous and / or together. The full-duplex radio may include an interference management unit to reduce and or substantially eliminate self-interference through either hardware (e.g., a choke) or signal processing via a processor (e.g., via a separate processor (not shown) or processor 118). In one embodiment, the WTRU 102 may include a half-duplex radio where transmission and reception of some or all of the signals (e.g., associated with a particular subframe on either the UL (e.g., for transmission) or DL (e.g., for reception)) may be simultaneous and / or together.

[0037] 1C is a system diagram illustrating the RAN 104 and the CN 106, according to one embodiment. As noted above, the RAN 104 may employ E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also communicate with the CN 106.

[0038] The RAN 104 may include eNodeBs 160a, 160b, and 160c, although it will be understood that the RAN 104 may include any number of eNodeBs while remaining consistent with an embodiment. The eNodeBs 160a, 160b, and 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, and 102c over the air interface 116. In an embodiment, the eNodeBs 160a, 160b, and 160c may implement MIMO technology. Thus, the eNodeB 160a may, for example, use multiple antennas to transmit wireless signals to and / or receive wireless signals from the WTRU 102a.

[0039] Each of the eNodeBs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, etc. As shown in FIG. 1C, the eNodeBs 160a, 160b, 160c may communicate with one another via an X2 interface.

[0040] 1C may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (PGW) 166. Although the foregoing elements are depicted as part of the CN 106, it will be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0041] The MME 162 may be connected to each of the eNodeBs 162a, 162b, 162c in the RAN 104 via an S1 interface and may function as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, activating / deactivating bearers, selecting a particular serving gateway during initial attach of the WTRUs 102a, 102b, 102c, etc. The MME 162 may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies such as GSM and / or WCDMA.

[0042] The SGW 164 may be connected to each of the eNodeBs 160a, 160b, 160c in the RAN 104 via an S1 interface. The SGW 164 may generally route and forward user data packets to and from the WTRUs 102a, 102b, 102c. The SGW 164 may perform other functions, such as anchoring the user plane during inter-eNodeB handovers, triggering paging when DL data is available to the WTRUs 102a, 102b, 102c, and managing and storing the context of the WTRUs 102a, 102b, 102c.

[0043] The SGW 164 may be connected to a PGW 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0044] The CN 106 may facilitate communications with other networks. For example, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional landline communications devices. For example, the CN 106 may include or communicate with an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that serves as an interface between the CN 106 and the PSTN 108. Additionally, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers.

[0045] Although the WTRU is illustrated in FIGS. 1A-1D as a wireless terminal, it is contemplated that in certain representative embodiments, such a terminal may use a wired communication interface (e.g., temporarily or permanently) with the communication network.

[0046] In a representative embodiment, the other network 112 may be a WLAN.

[0047] A WLAN in infrastructure Basic Service Set (BSS) mode may have an access point (AP) of the BSS and one or more stations (STAs) associated with the AP. The AP may have access to or interface with a distribution system (DS) or another type of wired / wireless network that carries traffic within and / or outside the BSS. Traffic originating from outside the BSS to a STA may arrive through the AP and be delivered to the STA. Traffic originating from a STA to a destination outside the BSS may be sent to the AP for delivery to the respective destination. Traffic between STAs within the BSS may be sent, for example, through the AP, where the source STA may send traffic to the AP, and the AP may deliver the traffic to the destination STA. Traffic between STAs within the BSS may be considered and / or referred to as peer-to-peer traffic. Peer-to-peer traffic may be sent between (e.g., directly between) a source STA and a destination STA using a direct link setup (DLS). In certain representative embodiments, the DLS may use 802.11e DLS or 802.11z tunneled DLS (TDLS). A WLAN using an Independent BSS (IBSS) mode may not have an AP, and STAs within or using the IBSS (e.g., all of the STAs) may communicate directly with each other. The IBSS mode of communication may be referred to herein as an "ad hoc" communication mode.

[0048] When using the 802.11ac infrastructure mode of operation or a similar mode of operation, an AP may transmit beacons on a fixed channel, such as a primary channel. The primary channel may be a fixed width (e.g., a 20 MHz wide bandwidth) or a dynamically configured width. The primary channel may be the operating channel of the BSS, but may be used by STAs to establish a connection with the AP. In certain representative embodiments, Carrier Sense Multiple Access with Collision Avoidance (CSMA / CA) may be implemented, for example, in an 802.11 system. With CSMA / CA, STAs (e.g., all STAs), including the AP, may sense the primary channel. If the primary channel is sensed / detected and / or determined to be busy by a particular STA, the particular STA may back off. One STA (e.g., only one station) may transmit in a given BSS at any given time.

[0049] High Throughput (HT) STAs may use 40 MHz wide channels for communication, which may be formed, for example, through a combination of a primary 20 MHz channel and adjacent or non-adjacent 20 MHz channels.

[0050] A Very High Throughput (VHT) STA may support channels with widths of 20 MHz, 40 MHz, 80 MHz, and / or 160 MHz. A 40 MHz and / or 80 MHz channel may be formed by combining multiple contiguous 20 MHz channels. A 160 MHz channel may be formed by combining eight contiguous 20 MHz channels or by combining two non-contiguous 80 MHz channels, which may be referred to as an 80+80 configuration. For the 80+80 configuration, after channel encoding, the data may pass through a segment parser that may separate the data into two streams. Inverse Fast Fourier Transform (IFFT) processing and time-domain processing may be performed separately on each stream. The streams may be mapped to two 80 MHz channels, and the data may be transmitted by the transmitting STA. At the receiver of the receiving STA, the operations described above for the 80+80 configuration may be reversed, and the combined data may be sent to Medium Access Control (MAC).

[0051] Sub-1 GHz operating modes are supported by 802.11af and 802.11ah. Channel operating bandwidths and carriers are reduced in 802.11af and 802.11ah compared to those used in 802.11n and 802.11ac. 802.11af supports bandwidths of 5 MHz, 10 MHz, and 20 MHz in the television (TV) white space (TVWS) spectrum, while 802.11ah supports bandwidths of 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz using non-TVWS spectrum. According to representative embodiments, 802.11ah may support Meter Type Control / Machine-Type Communications (MTC), such as MTC devices, in macro coverage areas. MTC devices may have limited capabilities, including, for example, support for (e.g., only) certain and / or limited bandwidths. An MTC device may include a battery with a battery life above a threshold (eg, to maintain a very long battery life).

[0052] WLAN systems that can support multiple channels and channel bandwidths, such as 802.11n, 802.11ac, 802.11af, and 802.11ah, include a channel that can be designated as a primary channel. The primary channel can have a bandwidth equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel can be configured and / or limited by the STAs among all STAs operating in the BSS that support the minimum bandwidth operating mode. In an 802.11ah embodiment, the primary channel can be 1 MHz wide for STAs (e.g., MTC-type devices) that support (e.g., only) the 1 MHz mode, even if the AP and other STAs in the BSS support 2 MHz, 4 MHz, 8 MHz, 16 MHz, and / or other channel bandwidth operating modes. Carrier sensing and / or Network Allocation Vector (NAV) configuration can depend on the status of the primary channel. For example, if a STA (that only supports 1 MHz mode of operation) transmitting to an AP has a busy primary channel, all of the available frequency bands may be considered busy even if most of the available frequency bands are idle.

[0053] In the United States, the available frequency band that can be used by 802.11ah is 902MHz to 928MHz. In South Korea, the available frequency band is 917.5MHz to 923.5MHz. In Japan, the available frequency band is 916.5MHz to 927.5MHz. The total bandwidth available for 802.11ah is 6MHz to 26MHz depending on the country code.

[0054] 1D is a system diagram illustrating the RAN 104 and the CN 106, according to one embodiment. As noted above, the RAN 104 may use NR radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also communicate with the CN 106.

[0055] The RAN 104 may include gNBs 180a, 180b, and 180c, although it will be understood that the RAN 104 may include any number of gNBs while remaining consistent with an embodiment. The gNBs 180a, 180b, and 180c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, and 102c over the air interface 116. In an embodiment, the gNBs 180a, 180b, and 180c may implement MIMO technology. For example, the gNBs 180a, 180b may transmit signals to and / or receive signals from the gNBs 180a, 180b, and 180c using beamforming. Thus, the gNB 180a may transmit and / or receive wireless signals to and / or from the WTRU 102a using, for example, multiple antennas. In one embodiment, the gNBs 180a, 180b, 180c may implement carrier aggregation technology. For example, the gNB 180a may transmit multiple component carriers to the WTRU 102a (not shown). A subset of these component carriers may be on unlicensed spectrum, while the remaining component carriers may be on licensed spectrum. In one embodiment, the gNBs 180a, 180b, 180c may implement Coordinated Multi-Point (CoMP) technology. For example, the WTRU 102a may receive coordinated transmissions from the gNBs 180a and 180b (and / or 180c).

[0056] The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using transmissions associated with scalable numerology. For example, the OFDM symbol spacing and / or OFDM subcarrier spacing may vary for different transmissions, different cells, and / or different portions of the wireless transmission spectrum. The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using subframes or transmission time intervals (TTIs) of varying or scalable lengths (e.g., including varying numbers of OFDM symbols and / or varying lengths of absolute time).

[0057] The gNBs 180a, 180b, 180c may be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and / or a non-standalone configuration. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c without accessing another RAN (e.g., eNodeBs 160a, 160b, 160c, etc.). In a standalone configuration, the WTRUs 102a, 102b, 102c may utilize one or more of the gNBs 180a, 180b, 180c as mobility anchor points. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using signals in unlicensed bands. In a non-standalone configuration, the WTRUs 102a, 102b, 102c may communicate with and connect to gNBs 180a, 180b, 180c while also communicating with and connecting to another RAN, such as eNodeBs 160a, 160b, 160c. For example, the WTRUs 102a, 102b, 102c may implement DC principles to communicate with one or more gNBs 180a, 180b, 180c and one or more eNodeBs 160a, 160b, 160c substantially simultaneously. In a non-standalone configuration, the eNodeBs 160a, 160b, 160c may act as mobility anchors for the WTRUs 102a, 102b, 102c, and the gNBs 180a, 180b, 180c may provide additional coverage and / or throughput for serving the WTRUs 102a, 102b, 102c.

[0058] Each of the gNBs 180a, 180b, 180c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, support for network slicing, DC, interworking between NR and E-UTRA, routing of user plane data towards User Plane Functions (UPFs) 184a, 184b, routing of control plane information towards Access and Mobility Management Functions (AMFs) 182a, 182b, etc. As shown in FIG. 1D , the gNBs 180a, 180b, 180c may communicate with each other via an Xn interface.

[0059] 1D may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one Session Management Function (SMF) 183a, 183b, and possibly a Data Network (DN) 185a, 185b. While the foregoing elements are depicted as part of the CN 106, it will be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0060] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 104 via an N2 interface and may function as a control node. For example, the AMF 182a, 182b may be responsible for user authentication of the WTRUs 102a, 102b, 102c, support for network slicing (e.g., handling different protocol data unit (PDU) sessions with different requirements), selection of a particular SMF 183a, 183b, management of registration areas, termination of non-access stratum (NAS) signaling, mobility management, etc. The network slicing may be used by the AMF 182a, 182b to customize the CN support for the WTRUs 102a, 102b, 102c based on the type of service utilizing the WTRUs 102a, 102b, 102c. For example, different network slices may be established for different use cases, such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced massive mobile broadband (eMBB) access, services for MTC access, etc. The AMFs 182a, 182b may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies, such as WiFi.

[0061] The SMFs 183a, 183b may be connected to the AMFs 182a, 182b in the CN 106 via an N11 interface. The SMFs 183a, 183b may also be connected to the UPFs 184a, 184b in the CN 106 via an N4 interface. The SMFs 183a, 183b may select and control the UPFs 184a, 184b and configure the routing of traffic through the UPFs 184a, 184b. The SMFs 183a, 183b may perform other functions, such as managing and assigning UE IP addresses, managing PDU sessions, controlling policy enforcement and QoS, providing DL data notification, etc. The PDU session type may be IP-based, non-IP-based, Ethernet-based, etc.

[0062] The UPFs 184a, 184b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 104 via an N3 interface, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks such as the Internet 110 to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The UPFs 184, 184b may perform other functions such as routing and forwarding packets, enforcing user plane policy, supporting multi-homed PDU sessions, handling user plane QoS, buffering DL packets, providing mobility anchoring, etc.

[0063] The CN 106 may facilitate communication with other networks. For example, the CN 106 may include or communicate with an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that acts as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers. In one embodiment, the WTRUs 102a, 102b, 102c may be connected to the local DNs 185a, 185b through the UPFs 184a, 184b via an N3 interface to the UPFs 184a, 184b and an N6 interface between the UPFs 184a, 184b and the DNs 185a, 185b.

[0064] 1A-1D and the corresponding description thereof, one or more or all of the functions described herein with respect to one or more of the WTRUs 102a-102d, base stations 114a-114b, eNodeBs 160a-160c, MME 162, SGW 164, PGW 166, gNBs 180a-180c, AMFs 182a-182b, UPFs 184a-184b, SMFs 183a-183b, DNs 185a-185b, and / or any other devices described herein may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more or all of the functions described herein. For example, the emulation devices may be used to test other devices and / or simulate network and / or WTRU functions.

[0065] The emulation devices may be designed to implement one or more tests of other devices in a lab environment and / or an operator network environment. For example, one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices in the communication network. One or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation devices may be directly coupled to another device for the purpose of testing and / or performing tests using over-the-air wireless communication.

[0066] One or more emulation devices may perform one or more functions, inclusive, while not being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in test scenarios in a test lab and / or in an undeployed (e.g., test) wired and / or wireless communication network to implement testing of one or more components. One or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (which may include, e.g., one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0067] Point cloud compression and processing advances the compression, analysis, interpolation, representation, and understanding of point cloud signals. The point cloud data format is a universal data format across several processing domains, from autonomous driving, robotics, augmented reality (AR) / virtual reality (VR), civil engineering, computer graphics, to the animation / film industry. Self-driving cars are being equipped with three-dimensional (3D) light detection and ranging (LiDAR) sensors, and affordable LiDAR sensors are becoming available from various manufacturers. With advances in sensing technology, 3D point cloud data is becoming more practical than ever and is expected to become the ultimate enabler for the described applications.

[0068] Point cloud data is also expected to consume a large portion of network traffic between vehicles connected via immersive communications, such as fifth-generation (5G) networks and VR / AR-enabled communications. Efficient representation formats are necessary for understanding and communicating point clouds. In particular, raw point cloud data needs to be properly organized and processed for world modeling and sensing purposes. Compression of raw point clouds is essential when data storage and transmission are required in relevant scenarios.

[0069] Furthermore, point clouds may represent successive scans of the same scene containing multiple moving objects. Such a scene representation is called a dynamic point cloud, as opposed to a static point cloud captured from a static scene or static objects. Dynamic point clouds are typically organized into frames, with different frames captured at different times. Dynamic point clouds may require real-time or low-latency processing and compression.

[0070] The automotive industry and autonomous vehicles are among the areas where point clouds can be used in several point cloud data use cases. Autonomous vehicles should be able to explore their environment and make good driving decisions based on the reality in their immediate vicinity. Typical sensors such as LiDAR generate point clouds, which can be dynamic point clouds, that are used by perception engines. These point clouds are not intended to be seen by the human eye; they are usually sparse, not necessarily color-coded, and are frequently populated and dynamic. These point clouds can have other attributes, such as reflectivity provided by LiDAR, that can indicate the material of the detected object and help make decisions.

[0071] VR and immersive worlds have been a hot topic recently, with many seeing them as the future of two-dimensional (2D) flat video. The basic idea is to immerse the viewer in the environment around them, as opposed to standard TV, where the viewer can only see the virtual world in front of them. There are several levels of immersion, depending on the viewer's degrees of freedom within the environment. Point clouds are a good candidate format for delivering virtual reality worlds. Point clouds can be static or dynamic, and are typically of average size, often no more than a few million points at a time.

[0072] Point clouds can also be used for various purposes such as cultural heritage / architecture, where objects such as statues or buildings are scanned in 3D to share the spatial configuration of the object without sending or visiting the object. They can also ensure preservation of knowledge about the object in case it may be destroyed, for example, a temple may be destroyed by an earthquake. Such point clouds are usually static, color-coded, and large.

[0073] Another use case is in topography and cartography, where the use of 3D representations, maps, etc. is not limited to flat surfaces but can include relief. Mapping applications publicly available on the Internet are currently good examples of 3D maps, but often use meshes instead of point clouds. Nevertheless, point clouds can be the preferred data format for three-dimensional maps, and such point clouds are usually static, color-coded, and large.

[0074] World modeling and sensing via point clouds is an essential technology that enables machines to gain knowledge about the 3D world around them and is crucial to the applications mentioned above. The examples and embodiments provided herein may be applied with the foregoing in mind.

[0075] Three-dimensional point cloud data are essentially discrete samples on the surface of an object or scene. To fully represent the real world with point samples, a vast number of points is required in practice. For example, a typical VR immersive scene contains millions of points, while a point cloud typically contains hundreds of millions of points. Therefore, processing such large-scale point clouds is computationally expensive, especially for consumer devices that may have limited computing power, such as smartphones, tablets, WTRUs, and automobile navigation systems.

[0076] The first step for any processing or inference on point clouds is to have an efficient storage method. To store and process the input point cloud at an affordable computational cost, one solution is to first downsample the point cloud; the downsampled point cloud summarizes the geometry of the input point cloud while having very few points. The downsampled point cloud can then be fed to a subsequent machine task for further consumption. However, further reduction in storage space can be achieved by converting the raw point cloud data (original or downsampled) into a bitstream through entropy encoding techniques for lossless compression. A better entropy model results in a smaller bitstream and therefore more efficient compression. Note that the entropy model can also be paired with downstream tasks, allowing the entropy encoder to maintain task-specific information while compressing.

[0077] In addition to lossless coding, many scenarios require lossy coding to significantly improve compression ratios while maintaining the induced distortion at a certain quality level. The separation of occupied voxels continues until the final octree depth level. The leaves of the octree ultimately represent the point cloud.

[0078] At the encoder side, the octree nodes, i.e., node values, are typically sent to an entropy coder to generate a bitstream. A decoder can then use the decoded octree node values to reconstruct the octree structure and finally reconstruct the point cloud based on the leaf nodes of the octree structure. To efficiently entropy code the octree nodes, a probability distribution model is typically utilized to assign shorter symbols to octree node values that are more likely to occur.

[0079] Point cloud compression is a critical issue in many practical applications, such as autonomous driving, AR / VR, etc. Examples and embodiments provided herein relate specifically to lossless compression of voxelized point cloud data using novel context modeling techniques. Such techniques can provide solutions to problems in point cloud compression.

[0080] Point cloud compression can be performed via octree encoding. Point clouds can be represented via an octree decomposition tree. In an octree decomposition tree, the root node covers the entire space within a bounding box. This space is divided equally in all directions, i.e., the x-axis, y-axis, and z-axis, resulting in eight voxels. For each voxel, if there is at least one occupied point, the voxel is marked as occupied and represented by a value of "1", otherwise it is marked as empty and represented by a value of "0". The voxel division then continues until a pre-specified condition is met.

[0081] An example of octant-based octree coding is provided herein. A common approach to encoding an octree is by encoding each occupied voxel with an 8-bit value that indicates the occupancy of its respective octant. In this way, the root voxel node can be first encoded with an 8-bit value. Then, for each occupied voxel at the next level or level of detail (LoD), its 8-bit occupancy symbol can be encoded, and then the next LoD can be moved on. This type of octree coding algorithm that encodes 8-bit occupancy symbols is sometimes referred to as an octant-based method.

[0082] An example of binary voxel-based encoding is provided herein. An alternative perspective for encoding the octree is by directly encoding binary occupancy bits. At each LoD, a sequence of occupancy bits representing voxels in the current LoD can be encoded, and then the encoder moves on to the next LoD. This type of approach is sometimes called a binary voxel-based method. The examples and embodiments provided herein include this type of approach.

[0083] By comparing the two types of encoding methods, it can be seen that in the octant-based method, the encoding of a current voxel is actually the encoding of the occupancy symbols of its child voxels. In contrast, in the binary voxel-based method, the encoding of a current voxel is actually the encoding of its own binary occupancy bits. In the following, some examples and embodiments will examine these two types of approaches in detail.

[0084] This specification provides examples and embodiments of octant-based octree coding. Further, this specification provides examples of learning-based octree coding for point clouds. Deep entropy models may refer to a category of learning-based methods that use neural network modules to formulate context models and attempt to predict probability distributions of 8-bit occupancy symbols.

[0085] One deep entropy model is known as OctSqueeze. This model utilizes a hierarchical hierarchy of ancestor nodes, including parent and grandparent nodes. Three multilayer perceptron (MLP)-based modules are used to estimate the probability distribution of the occupancy symbol of the current octree node. Another deep entropy model is known as VoxelContextNet. Unlike OctSqueeze, which uses ancestor nodes, VoxelContextNet utilizes a technique that first analyzes the local surface shape and then uses spatially neighboring voxels to predict the probability distribution.

[0086] A previous approach proposed a self-supervised compression model consisting of an adaptive entropy coder operating on a tree-structured conditional entropy model. Information from local neighborhoods as well as the global topology is utilized through an octree structure. Another approach proposed a two-branch hybrid method to digest a given point cloud and estimate a probability distribution. One branch is based on 3D convolutions similarly implemented in VoxelContextNet, while the other branch uses point-based operators such as PointNet++.

[0087] This document provides examples and embodiments of non-learning-based octant coding for point clouds. The most representative non-learning-based octant coding technique is the Motion Picture Experts Group (MPEG) geometry-based point cloud compression (G-PCC). For a current node with an occupancy symbol to be coded, G-PCC strives to use its neighbors' occupancy patterns, such as the 26 face / edge / vertex neighbors and the child neighbors of the three coded face-neighbors, to serve as the coding context for the current occupancy. G-PCC involves a highly complex handcrafted construction and context merging mechanism based on principles such as rotation invariance, plane detection, and angle plane detection. In this way, an optimal probability model for arithmetic coding is obtained. In addition to improving compression performance, G-PCC also allows the use of more flexible tree structures, such as quadtrees and binary trees (QTBTs).

[0088] This specification provides examples and embodiments of binary voxel-based octree coding. An alternative perspective for coding the octree is to directly consider each point as a voxel grid at a certain bit depth, rather than encoding the 8-bit occupancy symbols of the octree nodes. The occupancy of the voxel grid is then encoded with a binary symbol.

[0089] In a related example, the occupancy bits of neighboring voxels in the same LoD can be used as context to predict the occupancy probability of a current voxel. Note that the occupancy bits of neighboring voxels together represent the occupancy state, indicating whether each individual neighboring voxel is occupied or not. The probability is then predicted via a neural network module, followed by an adaptive arithmetic coding device that encodes the occupancy bits. Furthermore, recent studies such as SparsePCGCv2 have proposed another architecture that predicts the occupancy probability of a current voxel using the occupancy bits of neighboring voxels in two consecutive LoDs (one parent LoD and one same LoD as context). All of these methods either operate only on the current (higher) LoD or do not operate on two consecutive LoDs, and then require upsampling on the previous LoD to align the information with the current LoD.

[0090] Unlike these existing approaches, the examples and embodiments herein provide an occupancy prediction model that operates at the resolution of the previous (parent) LoD rather than at the same LoD as the current voxel whose occupancy is predicted. As a result, the method described herein enjoys reduced computational complexity compared to existing approaches.

[0091] Examples and embodiments of a modified binary voxel-based octree encoding scheme are provided herein. To predict the binary occupancy probability of each child node / bit at a particular octree level, the examples and embodiments provided herein propose to do all processing at the parent level, thus reducing computational and memory complexity.

[0092] The system overview of the modified binary voxel-based octree coding embodiments and examples provided herein may include a hierarchical coding structure. The examples provided herein hierarchically compress the octree by directly encoding the binary occupancy of the voxels.

[0093] 2 is a block diagram illustrating an exemplary point cloud LoD construction. Given an input point cloud with a bit depth of n, the examples provided herein hierarchically encode and decode this input point cloud as illustrated in the example shown in block diagram 200. In the example shown in block diagram 200, black indicates that the voxel is occupied (a value of "1" is placed in the voxel), and gray and white indicate that the voxel is empty (a value of "0" is placed in the voxel).

[0094] On the encoder side, the coarsest voxel representation may first be constructed as a first point cloud (PC), such as PC1220. PC1220 is coded first and transmitted as a first bitstream BS1. BS1 may include voxels in PC1220, such as black voxel 221, gray voxel 222, gray voxel 223, and black voxel 224. Then, PC2240 of the next LoD may be constructed. By comparing PC2240 with PC1220, it can be determined that only the black and gray voxels need to be coded to code PC2240, since the white voxels are guaranteed to be empty by seeing PC1220 with empty gray voxel 222 and empty gray voxel 223. Therefore, the black and gray voxels of PC2240 are coded and transmitted as a second bitstream BS2. Specifically, black voxel 241, gray voxel 242, gray voxel 243, black voxel 244, black voxel 246, gray voxel 247, gray voxel 248, and black voxel 249 may be encoded and transmitted in BS2. BS2 may include voxels in PC2 240.

[0095] Next, a finer LoD, PC3 260, can be constructed. Again, only black and gray voxels can be coded to represent PC3 260, resulting in a third bitstream BS3, by comparing PC3 260 with PC2 240. The white voxels of PC3 260 are guaranteed to be empty by checking the white and gray voxels 242, 243, 247, 248 of PC2 240. Specifically, black voxel 261, gray voxel 262, gray voxel 263, black voxel 264, black voxel 266, gray voxel 267, gray voxel 268, black voxel 269, black voxel 271, gray voxel 272, gray voxel 273, black voxel 274, black voxel 276, gray voxel 277, gray voxel 278, and black voxel 279 may be encoded and transmitted over BS3. This procedure may be repeated until the finest bit depth of the point cloud is reached. For example, the procedure may proceed beyond PC3 260 to the next LoD.

[0096] Similarly, on the decoder side, the decoder first reconstructs the coarsest LoD of PC1 220 by decoding the first bitstream BS1. By referencing the decoded PC1 220, the decoder understands that only the black and gray voxels in PC2 240 are included in the second bitstream BS2. Therefore, the decoder decodes BS2 and assigns the decoded bits to the black and gray voxels of PC2 240 for reconstruction. Similarly, the third bitstream BS3 is also decoded, and the decoded bits are assigned to the black and gray voxels of PC3 260 for reconstruction. This procedure can be repeated until the finest bit depth of the point cloud is reached.

[0097] This specification provides examples and embodiments of voxel grid coding using deep context-based entropy coding. The occupancy bits of voxels within each LoD can be coded. To achieve this, this specification provides a learning-based entropy model that takes the entire point cloud in the previous LoD and the decoded voxels in the current LoD and outputs binary occupancy probabilities in the current LoD. Occupancy probability predictions can be made only for children of occupied nodes from the previous LoD. Furthermore, predictions can be made bit-by-bit sequentially, starting with the occupancy probability prediction of the first octant child. The ground truth occupancy of this first child in the current LoD and the ground truth occupancy from the previous LoD can then be used to predict the occupancy probability of the second octant child, and so on.

[0098] The entropy model proposed herein aims to predict occupancy probabilities at each LoD using feature descriptors that characterize the local surface. This feature extractor can be a convolution-based architecture or a point-based architecture, or a combination of both, as proposed in other approaches. The feature extractor provided herein operates on the currently available, i.e., decoded, ground truth occupancies at the current LoD to generate features that are combined with features from the previous LoD; this combined feature is ultimately used to predict the occupancy probability of voxels at the current LoD.

[0099] 3 is an architecture diagram illustrating a proposed example basic architecture. Specifically, architecture diagram 300 shows an example architecture with a convolution-based feature extractor for a particular child at a particular LoD.

[0100] 4 is an architecture diagram illustrating the proposed example basic architecture expanded. The example architecture diagram 400 shows an expanded version of the same architecture starting from the root LoD and the first two child nodes of the first LoD.

[0101] This document provides a basic architecture for the proposed examples and embodiments. At the beginning of decoding a new LoD, a conventional binary voxel-based octree coding architecture begins by upsampling the features of the last decoded LoD to the current LoD resolution, and then uses this upsampled LoD as the starting point. Furthermore, all preceding calculations are performed at the higher resolution of the new LoD. This procedure is necessary to achieve adequate resolution for the initial features, but it also incurs computational costs for the upsampling step and all preceding calculations.

[0102] In contrast to this conventional approach, the embodiments and examples provided herein propose not to upsample features from the last LoD but to maintain its resolution. In addition, to generate occupancy probabilities for child nodes in a new LoD, the embodiments and examples provided herein propose to consider them as additional channels, where each new channel (child) is predicted from the last LoD and the preceding channels (children).

[0103] 5A and 5B are operational diagrams illustrating an example of an inter-bit prediction operation. The spatial resolution is the same as the previous LoD during the encoding of the bits, as shown in operational diagram 500. After decoding the bits from the current LoD, the bits are placed in the appropriate 3D positions to obtain the current decoded LoD. A visual example of the operations for generating occupancy probabilities of child nodes in the new LoD is shown in operational diagram 500.

[0104] Specifically, in the basic version of the embodiment provided herein, to predict the occupancy probability of the current bit in the current LoD, features from the previous LoD and features from the previous bit are accessed and transformed by a 3D convolution module to generate an updated combined feature. Furthermore, the ground truth occupancy of the previous bit can also be accessed and transformed into a feature using another 3D convolution module. It should be emphasized that all features share the same location in 3D space, just with different channels. Finally, both features can be combined into a final global feature that describes information from the previous level and the previous bit. This final global feature, together with contextual information about the current bit, such as bit position, bit index, etc., can be fed to an MLP-based fully connected module to output the occupancy probability of the current bit.

[0105] Figure 3 includes an example of an architecture diagram for predicting the occupancy probability of a current bit in a current LoD. Specifically, in Figure 3, the current LoD can be represented by k, the previous LoD can be represented by k-1, while the current bit can be represented by i, and the previous bit can be represented by i-1. Therefore, the features from the previous LoD can be used to predict the occupancy probability of a current bit in a current LoD. k-1 and the feature from the previous bit can be represented as f k i-1 These features can be accessed and transformed by a 3D convolutional neural network (CNN) module 320. Furthermore, the ground truth occupancy of the previous bit can be expressed as O k i-1, which can be accessed and transformed by the 3D CNN module 340. Furthermore, both features can be combined into a global feature by the combine module 330, which can add features or concatenate input features. This global feature can be provided to the CNN module 360. Furthermore, the output of the CNN module 360 can be provided to the MLP module 370. The context information for the current bit can include the bit position, bit index, etc. This context information can be provided as C k i , which may also be fed to the MLP module 370. Furthermore, the MLP module 370 may k i The occupancy probability of the current bit may be output, which may be expressed as:

[0106] After the occupancies of all bits in the current LoD are obtained, they are placed in the appropriate positions in the 3D space to fully occupy the current LoD. Furthermore, the features of each bit are also properly placed to construct the complete features of the current LoD that can be used by the next LoD.

[0107] FIG. 4 includes an example of an expanded architecture diagram. Specifically, in FIG. 4, the features of the previous level before level 0 may be initialized to a default vector of all ones, and the features of the previous bit before bit 1 in level 0 may be initialized to a default vector of all ones, and both occupancies may be accessed and transformed by CNN module 422. Furthermore, the occupancy of the previous bit before bit 1 in level 0 may also be initialized to a default occupancy of 1 and accessed and transformed by CNN 424. The outputs of CNN module 422 and CNN module 424 may be combined into a global feature by combination module 423. This global feature may be provided to CNN 426. Furthermore, the output of CNN module 426 may be represented as f0 and may be provided to MLP module 427 and output to CNN module 432 for use in predicting level 1, bit 1. Contextual information about the current bit may be provided by C0. 1, which may also be fed to the MLP module 427. Furthermore, the MLP module 427 may 1 The occupancy probability of the current level 0, bit 1 may be output, which may be represented as:

[0108] For level 1, bit 1, the CNN module 432 accesses and transforms f0 from level 0. Furthermore, the occupancy of the bit before bit 1 in level 1 is O1 0 , which may be initialized to an occupancy of all ones and may be accessed and transformed by CNN 434. The outputs of CNN module 432 and CNN module 434 may be combined into a global feature by combination module 433.

[0109] This global feature can be fed to the CNN module 436. Furthermore, the output of the CNN module 436 is f1 1 and may be fed to the MLP module 437 and output to the CNN module 442 for use in predicting level 1, bit 2. The context information for the current bit may be represented as C1 1 , which may also be fed to the MLP module 437. Furthermore, the MLP module 437 may 1 The occupancy probability of the current level 1, bit 1 may be output, which may be represented as:

[0110] For level 1, bit 2, the CNN module 442 calculates f0 from level 0, bit 1 and f1 from level 1, bit 1. 1 Furthermore, the occupancy of the previous bit 1 in level 1 is O1 1 The output of the CNN module 442 and the CNN module 444 may be combined into a global feature by a combination module 443. The global feature may be fed to the CNN module 446. Furthermore, the output of the CNN module 446 may be represented as f1 2and may be fed to the MLP module 447 and output to further CNN modules for use in predicting further bits, e.g., level 1, bit 3, etc.

[0111] Additionally or alternatively, the output of the CNN module 446 may be used in predicting further bits at further levels, such as level 2, bit 1, etc. Context information about the current bit is stored in C1 2 , which may also be fed to the MLP module 447. Furthermore, the MLP module 447 may 2 The occupancy probability of the current level 1, bit 2 may be output, which may be represented as:

[0112] 5A and 5B include an example of a prediction operation between bits. As shown in FIG. 5A, a previous level, level k-1, has occupied voxel 521, unoccupied voxel 522, unoccupied voxel 523, and occupied voxel 524. A current level, first bitstream, level k, bitstream 1, has occupied voxel 541, unoccupied voxel 542, unoccupied voxel 543, and occupied voxel 544. Level k, bitstream 1 may also be referred to as level k, bitstream 1, as shown in example operational diagram 500. Known voxels from level k-1 may be used to predict the occupancies of voxels 541, 542, 543, and 544 in bitstream 1, as described elsewhere herein. Similarly, the ground truth occupancies of previous voxels 541, 542, 543, and 544 can be used to predict voxels 546, 547, 548, and 549 in bitstream 2. Furthermore, the spatial resolution of each level k bitstream i is the same as the previous level k−1 during the encoding of the voxels.

[0113] Similarly, known voxels from bitstream 2 can be used to predict the occupancy of voxels in bitstream 3, which may also be referred to as bitstream 3. As shown, level k, bitstream 3, has unoccupied voxels 551, 552, 553, 554.

[0114] 5B, known voxels from bitstream 3 can be used to predict the occupancy of voxels in bitstream 4, i.e., bit 4. As shown, level k, bitstream 4 has occupied voxels 556, unoccupied voxels 557, unoccupied voxels 558, and occupied voxels 559.

[0115] In the example of Figures 5A and 5B, decoding all of the voxels from level k results in full occupancy of the currently decoded LoD at the end of decoding bitstream 4, with the voxels placed in their appropriate positions. For example, decoded level k is shown on the right side of Figure 5B, with the voxels placed in their appropriate positions. In one example, the positions may be 3D positions.

[0116] 6 is a flowchart diagram of an example of predicting voxel occupancy using occupancy bits of encoded or decoded neighboring voxels. As shown in example flowchart 600, a device may be configured to access occupancy bit sets of encoded or decoded neighboring voxels in a current LoD using a tree-based point cloud decoder (620). In one example, the device may be a wireless device. For example, the device may be a WTRU. In a further example, the device may be a decoder implemented by one or more processors. In another example, the device may be an encoder implemented by one or more processors.

[0117] Further, the device may be configured to compute first features based on the accessed occupancy bit sets of neighboring voxels in the current LoD using a first neural network module (630). In one example, the first neural network module may be a first convolution-based neural network module. The device may also be configured to access second features. In one example, the second features may be deep feature vectors from the previous LoD (640). In additional or alternative examples, the second features may be computed using a second neural network module or a third neural network module. In one example, the second neural network module may be a convolution-based module, such as a second convolution-based neural network module.

[0118] Further, the device may be configured to combine the first feature and the second feature to obtain a third feature (650). Additionally, the device may be configured to concatenate the third feature with one or more features of the current voxel to be encoded or decoded to form a fourth feature, the fourth feature being a global feature (660).

[0119] Further, the device may be configured to predict a probability distribution of voxel occupancies at the current LoD based on the fourth feature using a second neural network module (670). In one example, the second neural network module may be a fully connected (FC) module. In a further example, the second neural network module may be a convolution-based module, such as a second convolution-based neural network module. In another example, the second neural network module may be part of an MLP-based module.

[0120] In a further example, the device may use a third neural network module to derive a fifth feature based on the second feature. In a further example, the third neural network module may be a convolution-based module, such as a third convolution-based neural network module. In a further or alternative example, the device may use a second neural network model to derive the fifth feature. Also, in a further or alternative example, the fifth feature may be combined with the first feature to derive a third feature.

[0121] In a further example, the device can access a sixth feature. The sixth feature can be based on bits of encoded or decoded voxels at the current LoD. Furthermore, the fifth feature can be obtained, in another example, by combining the second feature and the sixth feature using a second neural network module or a third neural network model.

[0122] In a further example, the fourth feature may be further based on the sixth feature combined with the third feature. Additionally or alternatively, the fourth feature may be further based on the second feature combined with the third feature.

[0123] In a further example, the prediction of the probability distribution of the occupancy of a voxel in the current LoD may be further based on context information about the coded or decoded current voxel. In another example, the second feature may be based on the first point cloud of the previous LoD. Also, in one example, occupancy bits of neighboring voxels in the current LoD may be received in the bitstream.

[0124] In yet a further example, the device can reconstruct an octree structure based on the probability distribution of voxel occupancies at the current LoD. Further, the device can reconstruct a second point cloud based on leaf nodes of the reconstructed octree structure. Further, the device can then transmit the bits of the second point cloud.

[0125] 7 is a flowchart diagram of another example of predicting voxel occupancy using occupancy bits of coded or decoded neighboring voxels. As shown in the example flowchart 700, a device may be configured to access occupancy bit sets of coded or decoded neighboring voxels in a current LoD using a tree-based point cloud decoder (720). In one example, the device may be a wireless device. For example, the device may be a WTRU. In a further example, the device may be a decoder implemented by one or more processors.

[0126] The device may further be configured to compute a first feature based on accessed occupancy bit sets of neighboring voxels in the current LoD using a first convolution-based neural network module (730). The device may also be configured to access a second feature and a third feature. In one example, the second feature may be a deep feature vector from a previous LoD. In a further example, the third feature may be based on bits of encoded or decoded voxels in the current LoD (740). In one example, the second feature may be computed using a convolution-based neural network module. The device may further be configured to combine the second feature and the third feature to obtain a fourth feature using a second convolution-based neural network module (760).

[0127] Further, the device may be configured to combine the first feature and the fourth feature to obtain a fifth feature (770). Additionally, the device may be configured to concatenate the fifth feature with one or more features of the current voxel to be encoded or decoded to construct a sixth feature, the sixth feature being a global feature (780). Further, the device may be configured to predict a probability distribution of occupancies of voxels at the current LoD based on the sixth feature using a fully connected (FC) module (790).

[0128] In a further example, the second feature may be further combined with a fifth feature, which may itself be concatenated with one or more features of the current voxel to be encoded or decoded to form a sixth feature. As noted above, the second feature may be a deep feature vector from a previous LoD, and the fourth feature may be a global feature. Examples of second features that are further combined with the fifth feature, concatenated with the fifth feature, or both, are provided below with respect to FIG. 8.

[0129] In another example, the third feature may be further combined with a fifth feature, which itself may be concatenated with one or more features of the current voxel to be encoded or decoded to form a sixth feature. As noted above, the third feature may be a deep feature vector from an encoded or composited voxel at the current LoD, and the sixth feature may be a global feature. Examples of a third feature that is further combined with the fifth feature, concatenated with the fifth feature, or both, are provided below with respect to FIG. 9.

[0130] Figure 8 is an architecture diagram illustrating an example of the proposed architecture with residual connections from previous level features. In one example, the basic architecture can be improved by adding residual / skip connections from previous level features to the final global feature. This can help alleviate the vanishing gradient problem through residual connections.

[0131] An example including this architecture is shown in architecture diagram 800. For example, if a feature f k-1 and feature f from the previous bit k i-1 can be accessed and transformed by the CNN module 820. In some examples, one or both of these features can be deep features. Furthermore, the ground truth occupancy of the previous bit, O k i-1 can be accessed and transformed by the 3D CNN module 840. Furthermore, both features can be combined into a global feature by the combination module 830.

[0132] This global feature can be fed to the CNN module 860. The output of the CNN module 860 can then be combined with the features f from the previous LoD by a combination module 765 that describes the residual connections. k-1 This combination is achieved by adding two additional features:

[0133] The resulting output may be provided to the MLP module 870. k i may also be provided to the MLP module 870. As noted above, the context information for the current bit may include the bit position, the bit index, etc. Additionally, the MLP module 870 may provide the occupancy probability P k i can be output.

[0134] 9 is an architecture diagram illustrating an example of the proposed architecture with residual connections from the previous bit feature. In another example, a skip connection can be made from the previous bit feature to the last feature.

[0135] An example including this architecture is shown in architecture diagram 900. For example, if a feature f k-1 and feature f from the previous bit k i-1 can be accessed and transformed by the CNN module 920. In some examples, one or both of these features can be deep features. Furthermore, the ground truth occupancy of the previous bit, O k i-1 can be accessed and transformed by the 3D CNN module 940. Furthermore, both features can be combined into a global feature by the combination module 930.

[0136] This global feature can be fed to a CNN module 960. The output of the CNN module 960 can then be combined with the feature f from the previous bit by a combination module 965 that describes the residual connections. ki-1 This combination is achieved by adding two additional features:

[0137] The resulting output may be provided to the MLP module 970. k i may also be provided to the MLP module 970. As noted above, the context information for the current bit may include the bit position, the bit index, etc. Additionally, the MLP module 970 may provide the occupancy probability P k i can be output.

[0138] 10 is an architecture diagram illustrating one example of a proposed basic architecture, including example network details. Examples provided herein may use the architecture shown in architecture diagram 1000. For example, feature f from the previous LoD k-1 and feature f from the previous bit k i-1 may be accessed and transformed by a CNN module, which may include a convolutional layer 1021, a rectified linear unit (RelU) layer 1022, a convolutional layer 1023, a RelU layer 1024, a convolutional layer 1025, a RelU layer 1026, and a convolutional layer 1027. In one example, the CNN module may be similar to CNN module 320 of FIG. 3. In a particular example, the convolutional layer 1021 may have 64 dimensions as output features and a kernel size of 3. In a further example, the convolutional layer 1021 may also have 64 dimensions as input features. In another example, the convolutional layer 1021 may have 128 channels, and the kernel size may be 3×3×3. In a further example, a RelU layer, such as the RelU layer 1022, may be a nonlinear layer.

[0139] Specifically, in the example in Figure 10, the feature f k-1 and feature f from the previous bit k i-1may be fed to a convolutional layer 1021, the output of which may then be fed to a RelU layer 1022. The output of the RelU layer 1022 may be fed to a convolutional layer 1023, which may have a dimension of 32 for output features, input features, or both, and a kernel size of 3. The output of the convolutional layer 1023 may be fed to a RelU layer 1024, the output of which may then be fed to a convolutional layer 1025. The convolutional layer 1025 may also have a dimension of 32 for output features, input features, or both, and a kernel size of 3. The output of the convolutional layer 1025 may be fed to a RelU layer 1026, the output of which may then be fed to a convolutional layer 1027. The convolutional layer 1027 may have a dimension of 32 for output features, input features, or both, and a kernel size of 3.

[0140] Furthermore, the ground truth occupancy of the previous bit, O k i-1 may be accessed and transformed by a CNN, which may be a 3D CNN module. In one example, the CNN module may be similar to CNN module 340 of FIG. 3. The CNN module may include a convolutional layer 1041, a RelU layer 1042, a convolutional layer 1043, a RelU layer 1044, a convolutional layer 1045, a RelU layer 1046, and a convolutional layer 1027. In one example, the convolutional layer 1041 may have a dimension of 1 and a kernel size of 3. The output of the convolutional layer 1041 may be provided to the RelU layer 1042, whose output may then be provided to the convolutional layer 1043. The convolutional layer 1043 may also have a dimension of 32 for output features, input features, or both, and a kernel size of 3. The output of the convolutional layer 1043 may be provided to the RelU layer 1044, whose output may then be provided to the convolutional layer 1045. Convolutional layer 1045 may also have a dimension of 32 for output features, input features, or both, and a kernel size of 3. The output of convolutional layer 1045 may be fed to RelU layer 1046, whose output may then be fed to convolutional layer 1047. Convolutional layer 1047 may also have a dimension of 32 for output features, input features, or both, and a kernel size of 3.

[0141] Furthermore, the outputs of both convolutional layer 1027 and convolutional layer 1047 may be combined into global features. In one example, the outputs of convolutional layer 1027 and convolutional layer 1047 may be combined by a combination module. In one example, the combination module may be similar to combination module 330 of FIG. 3. In one example, the combination may be by concatenation. The global features may be fed into another CNN module. In one example, the CNN module may be similar to CNN module 360 of FIG. 3.

[0142] The CNN module may include a convolutional layer 1061, a RelU layer 1062, a convolutional layer 1063, a RelU layer 1064, a convolutional layer 1065, a RelU layer 1066, and a convolutional layer 1067. In one example, the convolutional layer 1061 may have a dimension of 64 and a kernel size of 3. The output of the convolutional layer 1061 may be provided to the RelU layer 1062, whose output may then be provided to the convolutional layer 1063. The convolutional layer 1063 may also have a dimension of 32 for output features, input features, or both, and a kernel size of 3. The output of the convolutional layer 1063 may be provided to the RelU layer 1064, whose output may then be provided to the convolutional layer 1065. The convolutional layer 1065 may also have a dimension of 32 for output features, input features, or both, and a kernel size of 3. The output of the convolutional layer 1065 may be fed to a RelU layer 1066, whose output may then be fed to a convolutional layer 1067. The convolutional layer 1067 may also have a dimension of 32 as the output features and a kernel size of 3.

[0143] Furthermore, the output of the convolutional layer 1067, and therefore the output of this CNN module, is the context information C k i Context information C may be provided to the MLP module along with the context information C. As noted above, the context information for the current bit may include a bit position, a bit index, etc. In one example, the MLP module may be similar to MLP module 370 of FIG. 3. Additionally, the MLP module may include an FC module layer 1071, a RelU layer 1072, an FC module layer 1073, a RelU layer 1074, and an FC module layer 1075.k i may have a dimension of 5, and the dimension of the output of the convolutional layer 1067 may be 32. Thus, the FC module layer 1071 may have an input dimension of 32 + 5. In one example, the FC module layer 1071 may have a joint dimension of 37, which may be the input dimension. Furthermore, 37 may be the output dimension of the FC module layer 1071. The output of the FC module layer 1071 may be provided to a RelU layer 1072, which may then be provided to an FC module layer 1073. In one example, the FC module layer 1073 may have a dimension of 16. The 16 dimension may be the input dimension, the output dimension, or both. The output of the FC module layer 1073 may be provided to a RelU layer 1074, which may then be provided to an FC module layer 1075. In one example, the FC module layer 1075 may have a dimension of 1. The dimension of 1 may be the input dimension, the output dimension, or both. In one example, a dimension of 1 is desirable for the output. Therefore, the FC module layer 1075 calculates the occupancy probability P k i As a result, the occupancy probability of the current bit P k i may be the output of the MLP module.

[0144] The layers provided for the modules in the above examples are for illustrative purposes: in operation, additional or fewer layers may be used in each module to suit the data, bitstream, and voxel context required for processing.

[0145] Examples and embodiments of shared and non-shared network parameters are provided herein. In one embodiment, the network parameters of all modules in the network are shared across all bits and all levels. In another embodiment, the network parameters of all modules are shared across levels but not for different bits. In yet another embodiment, the network parameters are never shared but are separate for each level and each bit.

[0146] Another exemplary improved embodiment can use features and ground truth of all previous bits in the same LoD instead of using only the most recently decoded bits. Such an approach can differ from the proposed basic architecture.

[0147] Figure 11 is an extraction diagram illustrating an example of feature extraction from all previously decoded bits. As shown in the example in extraction diagram 1100, all child bits of a voxel from the previous LoD have been decoded except for the last child bit, which is currently being decoded. As shown in Figure 11, the top left voxel has a bit of 1, the top right voxel has a bit of 0, the bottom left voxel has a bit of 0, and the bit of the bottom right voxel is still unknown as it is currently being decoded.

[0148] In the example shown in the top sub-diagram of FIG. 11 , the occupancies of all previous bits along with bitwise context information are concatenated into one single vector and then converted into features using either the point-based shared MLP or the voxel-based convolution module 1120. As shown in FIG. 11 , the top-left voxel has a bit occupancy of 1, a bit count of 0, and a bit position of 0,0. The bit position may represent row 0 and column 0 as the top-left position. In one example, the bit position may be a local position. Furthermore, the top-right voxel has a bit occupancy of 0, a bit count of 1, and a bit position of 0,1. The bit position may represent row 0 and column 1. Similarly, the bottom-left voxel has a bit occupancy of 0, a bit count of 2, and a bit position of 1,0 (which may represent row 1 and column 0).

[0149] In another example, shown in the center sub-diagram of Figure 11, a shared MLP or voxel-based convolution module 1170 operates in parallel on all bits with their context information to generate initial features, which are then max pooled 1180 into a single feature that summarizes the information of all previous bits. As shown in Figure 11, the information used in the center sub-diagram is the same as in the sub-diagrams above, but in the center sub-diagram, operations are performed in parallel on each decoded bit.

[0150] In yet another example, as shown in the bottom subfigure of Figure 11, the occupancies of all previous bits along with bitwise context information are placed into a 2x2x2 local 3D volume with k channels (depending on the available context), and then operated by a voxel-based convolution module 1190 to obtain features that summarize the information of all previous bits. As shown in Figure 11, the bit occupancies of voxels in the bottom subfigure correspond to the voxels on the left, and the bit of the bottom right voxel is shown as -1 because it is currently being decoded and is still unknown.

[0151] The decoded bits of a voxel are indicated by a 1, which indicates a bit that has been decoded, and a 0, which indicates a bit that is currently being decoded but is still unknown. Thus, the top left voxel, top right voxel, and bottom left voxel have been decoded and are therefore represented by a 1. Furthermore, the bottom right voxel has not yet been decoded or is currently being decoded and is therefore represented by a 0.

[0152] Similarly, the current bit of a voxel, or a voxel to be decoded, is indicated by a 0, which represents a bit that is not being decoded, for example, because it has already been decoded. Additionally, a 0 may represent a bit that is currently being decoded.

[0153] In the basic proposal, features from the previous LoD are directly used in the current LoD. In a different exemplary version, instead of directly using features from the previous LoD, new features can be constructed using the ground truth occupancies of the previous LoD. Because all operations are performed at a lower resolution and the bits in the current LoD are considered as channels in the previous LoD, the same 3D convolution module operating on the occupancies in the current LoD can be used.

[0154] In another example, the 3D convolution module can be replaced with a point-based module that is better suited to processing sparse point cloud data, such as LiDAR scans. In yet another example, the 3D convolution module can be replaced with a sparse 3D convolution module that is more computationally efficient and can also process sparse point clouds.

[0155] Further examples use advanced feature extraction modules: Basic convolutional modules, sparse convolutional modules, and point-based modules can be replaced with their advanced counterparts, such as ResNets, voxel transformers, and point transformers.

[0156] The proposed architecture is targeted at lossless octree entropy coding of point cloud data. However, this proposal can be easily modified to perform lossy octree coding by thresholding the resulting occupancy probability of each bit at the current LoD, while maintaining the same network architecture. Bits with occupancy probabilities above the threshold can be considered occupied, and the rest can be considered unoccupied. All remaining details can remain the same.

[0157] Although features and elements are described above in particular combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with the other features and elements. Furthermore, those skilled in the art will understand that the embodiments and examples provided herein include devices configured to perform the methods described herein. The embodiments and examples provided herein also include means for performing or means configured to perform the methods described herein. Additionally, the methods described herein can be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted via wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. 1. A method for use in a device, the method comprising: accessing the occupancy bit sets of neighboring voxels at the current level of detail (LoD) that have been coded or decoded using a tree-based point cloud decoder; using a first neural network module to calculate a first feature based on the accessed occupancy bit sets of neighboring voxels at the current LoD; accessing a second feature, the second feature being a deep feature vector from a previous LoD; combining the first feature and the second feature to obtain a third feature; concatenating the third feature with one or more features of the current voxel to be encoded or decoded to form a fourth feature, the fourth feature being a global feature; and and using a second neural network module to predict a probability distribution of voxel occupancy at the current LoD based on the fourth feature.

2. The method of claim 1 , further comprising using a third neural network module to derive fifth features based on the second features.

3. The method of claim 2 , wherein the third feature is obtained by combining the first feature and the fifth feature.

4. 3. The method of claim 2, further comprising accessing sixth features, the sixth features being based on bits from encoded or decoded voxels at the current LoD, and the fifth features being obtained by combining the second features and the sixth features using the second neural network module.

5. The method of claim 4 , wherein the fourth feature is further configured based on the sixth feature combined with the third feature.

6. The method of claim 2 , wherein the third neural network module is a convolution-based module.

7. The method of claim 1 , wherein the fourth feature is further constructed based on the second feature combined with the third feature.

8. The method of claim 1 , wherein the second neural network module is part of a multi-layer perceptron (MLP) based module.

9. The method of claim 1 , wherein the prediction is further based on context information about the current voxel to be coded or decoded.

10. The method of claim 1 , wherein the second features are based on the first point cloud of the previous LoD.

11. The method of claim 1 , wherein the occupancy bits of the neighboring voxels at the current LoD are those received in a bitstream.

12. reconstructing an octree structure based on the probability distribution of the occupancy of the voxels at the current LoD; reconstructing a second point cloud based on leaf nodes of the reconstructed octree structure; The method of claim 1 , further comprising: transmitting the bits of the second point cloud.

13. The method of claim 1 , wherein the first neural network module and the second neural network module are convolution-based modules.

14. The method of claim 1 , wherein the second neural network module is a fully connected (FC) module.

15. The method of claim 1 , wherein the device is a wireless device.

16. The method of claim 15 , wherein the wireless device is a wireless transmit / receive unit (WTRU).

17. A device, a processor; a transceiver operably coupled to the processor; the processor and the transceiver are configured to use a tree-based point cloud decoder to access occupancy bit sets of encoded or decoded neighboring voxels at a current level of detail (LoD); The processor is configured to use a first neural network module to calculate a first feature based on the accessed occupancy bit sets of neighboring voxels at the current LoD; the processor is configured to access second features, the second features being deep feature vectors from a previous LoD; the processor is configured to combine the first feature and the second feature to obtain a third feature; the processor is configured to concatenate the third feature with one or more features of the current voxel to be encoded or decoded to form a fourth feature, the fourth feature being a global feature; The device, wherein the processor is configured to use a second neural network module to predict a probability distribution of voxel occupancy at the current LoD based on the fourth feature.

18. 20. The device of claim 17, wherein the processor is further configured to derive a fifth feature based on the second feature using a third neural network module.

19. 20. The device of claim 18, wherein the third characteristic is obtained by combining the first characteristic and the fifth characteristic.

20. 20. The device of claim 18, wherein the processor and the transceiver are further configured to access a sixth feature, the sixth feature being based on bits from encoded or decoded voxels at the current LoD, and the fifth feature being obtained by combining the second feature and the sixth feature using the second neural network module.

21. 21. The device of claim 20, wherein the fourth feature is further configured based on the sixth feature combined with the third feature.

22. 20. The device of claim 18, wherein the third neural network module is a convolution-based module.

23. 18. The device of claim 17, wherein the fourth feature is further configured based on the second feature combined with the third feature.

24. 20. The device of claim 17, wherein the second neural network module is part of a multi-layer perceptron (MLP) based module.

25. The device of claim 17 , wherein the prediction is further based on context information about the current voxel to be encoded or decoded.

26. The device of claim 17 , wherein the second features are based on a first point cloud of the previous LoD.

27. The device of claim 17 , wherein the occupancy bits of the neighboring voxels at the current LoD are received in a bitstream.

28. 18. The device of claim 17, wherein the processor is further configured to reconstruct an octree structure based on the probability distribution of the occupancies of the voxels at the current LoD, the processor is further configured to reconstruct a second point cloud based on leaf nodes of the reconstructed octree structure, and the processor and the transceiver are further configured to transmit bits of the second point cloud.

29. 20. The device of claim 17, wherein the first neural network module and the second neural network module are convolution-based modules.

30. 20. The device of claim 17, wherein the third neural network module is a fully connected (FC) module.

31. The device of claim 17, wherein the device is a wireless device.

32. 32. The device of claim 31, wherein the wireless device is a wireless transmit / receive unit (WTRU).

Images