Micro patch convolutions for point cloud attribute compression

Abstract

Some embodiments of a method may include: obtaining information corresponding to a point cloud; selecting an examined point from a plurality of point cloud points corresponding to the information; determining a set of indices corresponding to a patch set for the examined point, wherein the patch set is a set of data points selected from the plurality of point cloud points, and wherein the patch set corresponds to the examined point; performing a patch convolution using the determined set of indices and a subset of the plurality of point cloud points; generating a latent space using an output of the patch convolution; and encoding the latent space data as an encoder bitstream.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims benefit of European Patent Application No. EP24305212, entitled "MICRO PATCH CONVOLUTIONS FOR POINT CLOUD ATTRIBUTE COMPRESSION” and filed February 9, 2025, which is hereby incorporated by reference in its entirety. CROSS-REFERENCE TO OTHER APPLICATIONS

[0002] The present application incorporates by reference in their entirety the following applications: European Patent Application Serial No. EP22306317, entitled "METHODS AND APPARATUSES FOR ENCODING AND DECODING A POINT CLOUD” and filed September 6,2022 ("‘317 application”); European Patent Application Serial No. EP23305352, entitled "METHODS AND APPARATUSES FOR ENCODING AND DECODING A POINT CLOUD” and filed March 15, 2023 ("‘352 application”); and European Patent Application Serial No. EP23306306, entitled "NORMALIZING FLOW SMALL ARCHITECTURES TO CODE POINT CLOUD ATTRIBUTES” and filed July 28, 2023 ("‘306 application”). BACKGROUND

[0003] The use of 3D applications is becoming more popular every day, and to be able to exploit said applications different data formats are being used. One of the main data formats is a point cloud. Point clouds (PCs) are a set of unordered points with coordinates x, y, z, corresponding to the location of the point in the space and its attributes (such as colors and normal vectors, among other things).

[0004] The use of this new type of data requires the creation of new compression methods to efficiently store and transmit data, especially since point clouds may have millions of points. Different methods to compress the attributes have been proposed, and among those methods, learning-based architectures are gaining strength. Those architectures are often extensions of learning-based methods in the 2D image domain. SUMMARY

[0005] Embodiments described herein include methods that are used in video encoding and decoding (collectively "coding”).

[0006] A first example method in accordance with some embodiments may include: obtaining information corresponding to a point cloud; selecting an examined point from a plurality of point cloud points corresponding to the information; determining a set of indices corresponding to a patch set for the examined point, wherein the patch set is a set of data points selected from the plurality of point cloud points, and wherein the patch set corresponds to the examined point; performing a patch convolution using the determined set of indices and a subset of the plurality of point cloud points; generating a latent space using an output of the patch convolution; and encoding latent space data as an encoder bitstream.

[0007] For some embodiments of the first example method, the patch convolution is a micropatch convolution.

[0008] For some embodiments of the first example method, generating the latent space comprises performing a first rectifier linear process (ReLU) on the output of the patch convolution.

[0009] For some embodiments of the first example method, generating the latent space further includes: performing a second patch convolution on an output of the first rectifier linear process; performing a second rectifier linear process on an output of the second patch convolution; and performing a downsampling on an output of the second rectifier linear process.

[0010] Some embodiments of the first example method may further include: determining a second set of indices corresponding to the patch set for the examined point, wherein the second set of indices comprises a downsampled set of indices generated based on the first set of indices, wherein generating the latent space comprises performing two encoding sub-processes, and wherein the first encoding sub-process corresponds to the first set of indices and the second encoding sub-process corresponds to the second set of indices.

[0011] For some embodiments of the first example method, determining the second set of indices is performed prior to performing the two encoding sub-processes.

[0012] For some embodiments of the first example method, determining the second set of indices is performed after performing at least one of the two encoding sub-processes.

[0013] For some embodiments of the first example method, encoding the latent space data as the encoder bitstream comprises using a variational auto-encoder (VAE).

[0014] For some embodiments of the first example method, encoding the latent space data as the encoder bitstream comprises using a normalizing flow (NF) process.

[0015] For some embodiments of the first example method, encoding the latent space data as the encoder bitstream comprises using a wavelet process.

[0016] For some embodiments of the first example method, encoding the latent space data as the encoder bitstream comprises using a micropatch convolution.

[0017] For some embodiments of the first example method, encoding the latent space data is performed as part of an artificial intelligence point cloud compression (AI-PCC) algorithm.

[0018] For some embodiments of the first example method, performing the patch convolution comprises performing a two-dimensional (2D) convolution.

[0019] Some embodiments of the first example method may further include determining the patch set for the examined point.

[0020] For some embodiments of the first example method, determining the patch set for the examined point includes: determining a neighborhood of points surrounding the examined point, and determining the patch set to be one or more points selected from the neighborhood of points.

[0021] For some embodiments of the first example method, determining the patch set further comprises adding one or more points to the neighborhood of points.

[0022] For some embodiments of the first example method, determining the set of indices corresponding to the patch set for the examined point includes: populating a first subset of the set of indices corresponding to the patch set using occupied points selected from the plurality of point cloud points; and populating a second subset of the set of indices corresponding to the patch set using a filling algorithm.

[0023] For some embodiments of the first example method, the filling algorithm is a mirroring operation performed on the first subset of the set of indices corresponding to the patch set.

[0024] For some embodiments of the first example method, the filling algorithm is a zero padding operation.

[0025] For some embodiments of the first example method, the filling algorithm is a nearest neighbor axis order filling operation performed on the first subset of the set of indices corresponding to the patch set.

[0026] For some embodiments of the first example method, the filling algorithm is a nearest neighbor averaging operation performed on nearest neighbors selected from the first subset of the set of indices corresponding to the patch set.

[0027] For some embodiments of the first example method, the filling algorithm is a global averaging operation performed on the first subset of the set of indices corresponding to the patch set.

[0028] For some embodiments of the first example method, the filling algorithm is an interpolation operation performed on nearest neighbors selected from the first subset of the set of indices corresponding to the patch set.

[0029] A first example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0030] A second example method in accordance with some embodiments may include: obtaining information corresponding to a point cloud; selecting an examined point from a plurality of point cloud points corresponding to the information; determining a set of indices corresponding to a patch set for the examined point, wherein the patch set is a set of data points selected from the plurality of point cloud points, and wherein the patch set corresponds to the examined point; generating a latent space using the determined set of indices and a subset of the plurality of point cloud points, wherein generating the latent space comprises performing a patch convolution using the determined set of indices and the subset of the plurality of point cloud points; and encoding latent space data as an encoder bitstream.

[0031] A second example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0032] A third example method in accordance with some embodiments may include: obtaining first information corresponding to a point cloud geometry; obtaining second information corresponding to a point cloud latent space; selecting an examined point from a plurality of point cloud points corresponding to the first information; determining a set of indices corresponding to a patch set for the examined point, wherein the patch set is a set of data points selected from the plurality of point cloud points, and wherein the patch set corresponds to the examined point; performing a patch convolution using the determined set of indices and the second information corresponding to the point cloud latent space; and generating a reconstructed point cloud using an output of the patch convolution.

[0033] Some embodiments of the third example method may further include decoding a decoder bitstream to generate the information corresponding to a point cloud geometry and the information corresponding to the point cloud latent space.

[0034] For some embodiments of the third example method, the patch convolution is a micropatch convolution.

[0035] For some embodiments of the third example method, generating the reconstructed point cloud comprises performing a rectifier linear process on the output of the patch convolution.

[0036] For some embodiments of the third example method, generating the reconstructed point cloud further includes: performing a second patch convolution on an output of the first rectifier linear process; performing a second rectifier linear process on an output of the second patch convolution; and performing an upsampling on an output of the second rectifier linear process.

[0037] Some embodiments of the third example method may further include: determining a second set of indices corresponding to the patch set for the examined point, wherein the second set of indices comprises an upsampled set of indices generated based on the first set of indices, wherein generating the reconstructed point cloud comprises performing two decoding sub-processes, and wherein the first decoding sub-process corresponds to the first set of indices and the second decoding sub-process corresponds to the second set of indices.

[0038] For some embodiments of the third example method, determining the second set of indices is performed prior to performing the two decoding sub-processes.

[0039] For some embodiments of the third example method, determining the second set of indices is performed after performing at least one of the two decoding sub-processes.

[0040] For some embodiments of the third example method, obtaining information corresponding to the point cloud latent space comprises using a variational auto-decoder (VAD).

[0041] For some embodiments of the third example method, obtaining information corresponding to the point cloud latent space comprises using a normalizing flow (NF) process.

[0042] For some embodiments of the third example method, obtaining information corresponding to the point cloud latent space comprises using a wavelet process.

[0043] For some embodiments of the third example method, obtaining information corresponding to the point cloud latent space comprises using a micropatch convolution.

[0044] For some embodiments of the third example method, obtaining information corresponding to the point cloud latent space is performed as part of an artificial intelligence point cloud compression (AI-PCC) algorithm.

[0045] For some embodiments of the third example method, performing the patch convolution comprises performing a two-dimensional (2D) convolution.

[0046] Some embodiments of the third example method may further include determining the patch set for the examined point.

[0047] For some embodiments of the third example method, determining the patch set for the examined point includes: determining a neighborhood of points surrounding the examined point, and determining the patch set to be one or more points selected from the neighborhood of points.

[0048] For some embodiments of the third example method, determining the patch set further includes adding one or more points to the neighborhood of points.

[0049] A third example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0050] A fourth example method in accordance with some embodiments may include: obtaining information corresponding to a point cloud geometry; obtaining information corresponding to a point cloud latent space; selecting an examined point from a plurality of point cloud points corresponding to the information; determining a set of indices corresponding to a patch set for the examined point, wherein the patch set is a set of data points selected from the plurality of point cloud points, and wherein the patch set corresponds to the examined point; and generating a reconstructed point cloud using the determined set of indices and the point cloud latent space, wherein generating the reconstructed point cloud comprises performing a patch convolution using the determined set of indices and the second information corresponding to the point cloud latent space.

[0051] A fourth example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0052] A fifth example method in accordance with some embodiments may include: downsampling a point cloud by reducing an N x N x N voxel to an N x N projection; copying occupied indices in the N x N x N point cloud to corresponding locations in the N x N projection; and populating unoccupied indices in the N x N projection using an algorithm.

[0053] A fifth example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above. -6-

[0054] A sixth example method in accordance with some embodiments may include: downsampling a point cloud by iteratively, for each point of a plurality of points of the point cloud: reducing an N x N x N voxel centered on the point to an N x N projection centered on the point, wherein the N x N projection comprises N x N index locations, by: copying neighboring points occupying indices in a corresponding N x N slice centered on the point and within the N x N x N voxel to corresponding index locations in the N x N projection; and populating unoccupied indices, if any, from the corresponding N x N slice into the N x N projection at corresponding index locations using an algorithm.

[0055] For some embodiments of the sixth example method, reducing the N x N x N voxel to the N x N projection further includes disregarding additional neighboring points outside of the N x N slice within the N x N x N voxel such that the N x N projection does not include such neighboring points.

[0056] A sixth example apparatus in accordance with some embodiments may include a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0057] A seventh example apparatus in accordance with some embodiments may include at least one processor configured to perform any one of the methods listed above.

[0058] An eighth example apparatus in accordance with some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any one of the methods listed above.

[0059] A ninth example apparatus in accordance with some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any one of the methods listed above.

[0060] An example signal in accordance with some embodiments may include an encoder bitstream generated according to any one of the methods listed above.

[0061] In additional embodiments, encoder and decoder apparatus are provided to perform the methods described herein. An encoder or decoder apparatus may include a processor configured to perform the methods described herein. The apparatus may include a computer-readable medium (e.g. a non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, a computer-readable medium (e.g. a non-transitory medium) stores a video encoded using any of the methods described herein.

[0062] One or more of the present embodiments also provide a computer readable storage medium having stored thereon instructions for performing bi-directional optical flow, encoding, or decoding video data -7 - according to any of the methods described above. The present embodiments also provide a computer readable storage medium having stored thereon a bitstream generated according to the methods described above. The present embodiments also provide a method and apparatus for transmitting the bitstream generated according to the methods described above. The present embodiments also provide a computer program product including instructions for performing any of the methods described. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] FIG. 1A is a system diagram illustrating an example communications system according to some embodiments.

[0064] FIG. 1B is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that may be used within the communications system illustrated in FIG. 1A according to some embodiments.

[0065] FIG. 1C is a system diagram illustrating an example set of interfaces for a system according to some embodiments.

[0066] FIG. 2A is a block diagram showing an example system according to some embodiments.

[0067] FIG. 2B is a system diagram illustrating an example set of interfaces for networking according to some embodiments.

[0068] FIG. 2C is a message bitfield diagram showing an example packet according to some embodiments.

[0069] FIG. 3 is a process diagram showing an example normalizing flow architecture to compress and uncompress point cloud data according to some embodiments.

[0070] FIG. 4A is a process diagram illustrating an example feature enhancement layer according to some embodiments.

[0071] FIG. 4B is a process diagram illustrating an example feature enhancement according to some embodiments.

[0072] FIG. 4C is a process diagram illustrating an example coupling layer according to some embodiments.

[0073] FIG. 4D is a process diagram illustrating an example transformation block for coupling layers according to some embodiments.

[0074] FIG. 5 is a flowchart illustrating an example process for encoding a point cloud according to some embodiments.

[0075] FIG. 6 is a flowchart illustrating an example process for reconstructing a point cloud according to some embodiments.

[0076] FIG. 7A is a schematic illustration showing an example of a 3D sparse point cloud being translated into a 3x3 patch according to some embodiments.

[0077] FIG. 7B is a schematic illustration showing an example 3D sparse point cloud to 3x3 patch translation showing padding with mirrored inliers according to some embodiments.

[0078] FIG. 7C is a schematic illustration showing an example 3D sparse point cloud to 3x3 patch translation showing padding with zeros according to some embodiments.

[0079] FIG. 7D is a schematic illustration showing an example 3D sparse point cloud to 3x3 patch translation showing padding with the nearest neighbor (considering the Y axis as being priority) according to some embodiments.

[0080] FIG. 7E is a schematic illustration showing an example 3D sparse point cloud to 3x3 patch translation showing padding with the nearest neighbor average according to some embodiments.

[0081] FIG. 7F is a schematic illustration showing an example 3D sparse point cloud to 3x3 patch translation showing padding with a global average according to some embodiments.

[0082] FIG. 7G is a schematic illustration showing an example 3D sparse point cloud to 3x3 patch translation showing padding with an interpolation of nearest neighbors according to some embodiments.

[0083] FIGs. 8A-8D are schematic illustrations showing a series progression for an example 3D sparse point cloud to 3x3 patch translation showing padding with mirrored inliers according to some embodiments.

[0084] FIG. 9 is a schematic illustration showing example convolutions for a 3D sparse point cloud and for a 3D sparse point cloud to 3x3 patch translation according to some embodiments.

[0085] FIGs. 10A-10B are system diagrams illustrating example sets of interfaces for a network according to some embodiments.

[0086] FIG. 11 is a process diagram illustrating an example encoder / decoder system according to some embodiments.

[0087] FIG. 12 is a process diagram illustrating an example patches generation block according to some embodiments.

[0088] FIG. 13 is a process diagram illustrating an example of average pooling according to some embodiments.

[0089] FIG. 14 is a process diagram illustrating an example variational auto-encoder / decoder architecture according to some embodiments.

[0090] FIG. 15 is a process diagram illustrating an example encoder using patch convolutions according to some embodiments.

[0091] FIG. 16 is a process diagram illustrating an example encoder using patch convolutions with on-the-fly patch generation according to some embodiments.

[0092] FIG. 17 is a process diagram illustrating an example decoder using patch convolutions according to some embodiments.

[0093] FIG. 18 is a flowchart illustrating an example encoder process according to some embodiments.

[0094] FIG. 19 is a flowchart illustrating an example decoder process according to some embodiments.

[0095] The entities, connections, arrangements, and the like that are depicted in—and described in connection with—the various figures are presented by way of example and not by way of limitation. As such, any and all statements or other indications as to what a particular figure "depicts,” what a particular element or entity in a particular figure "is” or "has,” and any and all similar statements—that may in isolation and out of context be read as absolute and therefore limiting—may only properly be read as being constructively preceded by a clause such as "In at least one embodiment, ..." For brevity and clarity of presentation, this implied leading clause is not repeated ad nauseum in the detailed description. DETAILED DESCRIPTION

[0096] FIG. 1A is a diagram illustrating an example communications system 100 in which one or more disclosed embodiments may be implemented. The communications system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0097] As shown in FIG. 1A, the communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104 / 113, a ON 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. Byway of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a "station” and / or a "STA”, may be configured to transmit and / or receive wireless signals and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fl device, an Internet of Things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and / or other wireless devices operating in an industrial and / or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and / or industrial wireless networks, and the like. Any of the WTRUs 102a, 102b, 102c and 102d may be interchangeably referred to as a UE.

[0098] The communications systems 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106, the Internet 110, and / or the other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0099] The base station 114a may be part of the RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for a wireless service to a specific geographical area that may be relatively fixed or that may change over time. The cell may further be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one for each sector of the cell. In an embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in desired spatial directions.

[0100] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0101] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104 / 113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 116 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink (DL) Packet Access (HSDPA) and / or High-Speed UL Packet Access (HSUPA).

[0102] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0103] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR Radio Access , which may establish the air interface 116 using New Radio (NR).

[0104] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement LTE radio access and NR radio access together, for instance using dual connectivity (DC) principles. Thus, the air interface utilized by WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., a eNB and a gNB).

[0105] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0106] The base station 114b in FIG. 1A may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a roadway, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR etc.) to establish a picocell or femtocell. As shown in FIG. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the CN 106.

[0107] The RAN 104 / 113 may be in communication with the CN 106, which may be any type of network configured to provide voice, data, applications, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have varying quality of service (QoS) requirements, such as differing throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 106 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions, such as user authentication. Although not shown in FIG. 1A, it will be appreciated that the RAN 104 / 113 and / or the CN 106 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which may be utilizing a NR radio technology, the CN 106 may also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0108] The CN 106 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or the other networks 112. The PSTN 108 may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and / or the internet protocol (IP) in the TCP / IP internet protocol suite. The networks 112 may include wired and / or wireless communications networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104 / 113 or a different RAT.

[0109] Some or all oftheWTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0110] FIG. 1B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138, among others. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0111] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. While FIG. 1B depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0112] The transmit / receive element 122 may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It will be appreciated that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0113] Although the transmit / receive element 122 is depicted in FIG. 1B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ -14- MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0114] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit / receive element 122 and to demodulate the signals that are received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11, for example.

[0115] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and / or the removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0116] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0117] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 116 from a base station (e.g., base stations 114a, 114b) and / or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable locationdetermination method while remaining consistent with an embodiment.

[0118] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a Virtual Reality and / or Augmented Reality (VR / AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, the sensors may be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0119] The WTRU 102 may include a full duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and downlink (e.g., for reception) may be concurrent and / or simultaneous. The full duplex radio may include an interference management unit to reduce and or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In an embodiment, the WTRU 102 may include a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).

[0120] Although the WTRU is described in FIGs. 1A-1B as a wireless terminal, it is contemplated that in certain representative embodiments that such a terminal may use (e.g., temporarily or permanently) wired communication interfaces with the communication network.

[0121] In representative embodiments, the other network 112 may be a WLAN.

[0122] In view of FIGs. 1 A-1 B, and the corresponding description, one or more, or all, of the functions described herein may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, the emulation devices may be used to test other devices and / or to simulate network and / or WTRU functions.

[0123] The emulation devices may be designed to implement one or more tests of other devices in a lab environment and / or in an operator network environment. For example, the one or more emulation devices may perform the one or more, or all, functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network in order to test other devices within the communication -16- network. The one or more emulation devices may perform the one or more, or all, functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for purposes of testing and / or may perform testing using over-the-air wireless communications.

[0124] The one or more emulation devices may perform the one or more, including all, functions while not being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in a testing scenario in a testing laboratory and / or a non-deployed (e.g., testing) wired and / or wireless communication network in order to implement testing of one or more components. The one or more emulation devices may be test equipment. Direct RF coupling and / or wireless communications via RF circuitry (e.g., which may include one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0125] FIG. 1C is a system diagram illustrating an example set of interfaces for a system according to some embodiments. An extended reality display device, together with its control electronics, may be implemented for some embodiments. System 150 can be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this document. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 150, singly or in combination, can be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 150 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 150 is configured to implement one or more of the aspects described in this document.

[0126] The system 150 includes at least one processor 152 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this document. Processor 152 may include embedded memory, input output interface, and various other circuitries as known in the art. The system 150 includes at least one memory 154 (e.g., a volatile memory device, and / or a non-volatile memory device). System 150 may include a storage device 158, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. The storage device 158 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and / or a network accessible storage device, as non-limiting examples.

[0127] System 150 includes an encoder / decoder module 156 configured, for example, to process data to provide an encoded video or decoded video, and the encoder / decoder module 156 can include its own processor and memory. The encoder / decoder module 156 represents module(s) that can be included in a device to perform the encoding and / or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 156 can be implemented as a separate element of system 150 or can be incorporated within processor 152 as a combination of hardware and software as known to those skilled in the art.

[0128] Program code to be loaded onto processor 152 or encoder / decoder 156 to perform the various aspects described in this document can be stored in storage device 158 and subsequently loaded onto memory 154 for execution by processor 152. In accordance with various embodiments, one or more of processor 152, memory 154, storage device 158, and encoder / decoder module 156 can store one or more of various items during the performance of the processes described in this document. Such stored items can include, but are not limited to, the input video, the decoded video, or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0129] In some embodiments, memory inside of the processor 152 and / or the encoder / decoder module 156 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device can be either the processor 152 or the encoder / decoder module 152) is used for one or more of these functions. The external memory can be the memory 154 and / or the storage device 158, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of, for example, a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations, such as for MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also referred to as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or WC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).

[0130] The input to the elements of system 150 can be provided through various input devices as indicated in block 172. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples, not shown in FIG. 1C, include composite video.

[0131] In various embodiments, the input devices of block 172 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements suitable for (I) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various of these functions, including, for example, downconverting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements can include inserting elements in between existing elements, such as, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[0132] Additionally, the USB and / or HDMI terminals can include respective interface processors for connecting system 150 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, can be implemented, for example, within a separate input processing IC or within processor 152 as necessary. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface ICs or within processor 152 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 152, and encoder / decoder 156 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.

[0133] Various elements of system 150 can be provided within an integrated housing, Within the integrated housing, the various elements can be interconnected and transmit data therebetween using suitable connection arrangement 174, for example, an internal bus as known in the art, including the InterIC (I2C) bus, wiring, and printed circuit boards.

[0134] The system 150 includes communication interface 160 that enables communication with other devices via communication channel 162. The communication interface 160 can include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 162. The communication interface 160 can include, but is not limited to, a modem or network card and the communication channel 162 can be implemented, for example, within a wired and / or a wireless medium.

[0135] Data is streamed, or otherwise provided, to the system 150, in various embodiments, using a wireless network such as a Wi-Fi network, for example IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal of these embodiments is received over the communications channel 162 and the communications interface 160 which are adapted for Wi-Fi communications. The communications channel 162 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 150 using a set-top box that delivers the data over the HDMI connection of the input block 172. Still other embodiments provide streamed data to the system 150 using the RF connection of the input block 172. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth network.

[0136] The system 150 can provide an output signal to various output devices, including a display 176, speakers 178, and other peripheral devices 180. The display 176 of various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 176 can be for a television, a tablet, a laptop, a cell phone (mobile phone), or other device. The display 176 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop). The other peripheral devices 180 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 180 that provide a function based on the output of the system 150. For example, a disk player performs the function of playing the output of the system 150.

[0137] In various embodiments, control signals are communicated between the system 150 and the display 176, speakers 178, or other peripheral devices 180 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communications protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to system 150 via dedicated connections through respective interfaces 164, 166, and 168. Alternatively, the output devices can be connected to system 150 using the communications channel 162 via the communications interface 160. The display 176 and speakers 178 can be integrated in a single unit with the other components of system 150 in an electronic device such as, for example, a television. In various embodiments, the display interface 164 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0138] The display 176 and speaker 178 can alternatively be separate from one or more of the other components, for example, if the RF portion of input 172 is part of a separate set-top box. In various embodiments in which the display 176 and speakers 178 are external components, the output signal can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0139] The system 150 may include one or more sensor devices 168. Examples of sensor devices that may be used include one or more GPS sensors, gyroscopic sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors may be used to determine information such as user's position and orientation. Where the system 150 is used as the control module for an extended reality display (such as control modules 124, 132), the user's position and orientation may be used in determining how to render image data such that the user perceives the correct portion of a virtual object or virtual scene from the correct point of view. In the case of head-mounted display devices, the position and orientation of the device itself may be used to determine the position and orientation of the user for the purpose of rendering virtual content. In the case of other display devices, such as a phone, a tablet, a computer monitor, or a television, other inputs may be used to determine the position and orientation of the user for the purpose of rendering content. For example, a user may select and / or adjust a desired viewpoint and / or viewing direction with the use of a touch screen, keypad or keyboard, trackball, joystick, or other input. Where the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation used for the purpose of rendering content may be selected and / or adjusted based on motion of the display device.

[0140] The embodiments can be carried out by computer software implemented by the processor 152 or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 154 can be of any type appropriate to the technical environment and can be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 152 can be of any type appropriate to the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0141] The use of 3D applications is becoming more popular every day. To be able to exploit such applications, different data formats are being used. One of the data formats is point clouds. Point clouds are a set of unordered points with coordinates (x, y, z) corresponding to a point location in space and its attributes (such as colors and normal vectors).

[0142] Use of these new types of data likely requires new compression methods to efficiently store and transmit data, especially since point clouds may have millions of points. Of the different methods that have been proposed, learning-based architectures are gaining strength. Those architectures are often extensions of learning-based methods explored in the 2D image domain.

[0143] The conference paper, L. Dihn, J. Sohl-Dickstein, and S. Bengio, Density Estimation Using Real NVP, arXiv: 1605.08803v3 (2016), is understood to explore the use of normalizing flows as a compression architecture in the 2D image compression domain. However, this method uses a squeeze layer that is not adapted for point cloud data structure. Normalizing flows are a type of architecture that produces a latent space from an input. The latent space is a representation of the input with different coefficients. The goal is to produce a latent space that is easier to compress than the original input.

[0144] FIG. 2A is a block diagram showing an example system according to some embodiments. FIG. 2A illustrates a block diagram of a system within which aspects of the present embodiments may be implemented, according to another embodiment. FIG. 2A shows one embodiment of an apparatus 200 for encoding or decoding a point cloud or attributes of a point cloud as described according to any one of the embodiments described herein. The apparatus may include a processor 210 and can be interconnected to a memory 220 through at least one port. Both processor 210 and memory 220 can also have one or more additional interconnections to external connections.

[0145] Processor 220 is also configured to code one or more attributes of a point cloud using an invertible neural network, using any one of the embodiments described herein. For instance, the processor 210 is configured using a computer program product comprising code instructions that implements any one of embodiments described herein.

[0146] FIG. 2B is a system diagram illustrating an example set of interfaces for networking according to some embodiments. In an embodiment, illustrated in FIG. 2B, in a transmission context between two remote devices A and B over a communication network NET, the device A comprises a processor in relation with memory RAM and ROM which are configured to implement a method for encoding a point cloud, as described with FIGs. 3-13 and the device B comprises a processor in relation with memory RAM and ROM which are configured to implement a method for decoding a point cloud as described in relation with FIGs 3-13. In accordance with an example, the network is a broadcast network, adapted to broadcast / transmit encoded point cloud from device A to decoding devices including the device B.

[0147] FIG. 2C is a message bitfield diagram showing an example packet according to some embodiments. FIG. 20 shows an example of the syntax of a signal transmitted over a packet-based transmission protocol. Each transmitted packet P comprises a header H and a payload PAYLOAD. In some embodiments, the payload PAYLOAD may comprise coded point cloud data according to any one of the embodiments described above. In a variant, the signal comprises a flag indicating a deep learning method for decoding the point cloud or for decoding one or more attributes of the point cloud.

[0148] FIG. 3 is a process diagram showing an example normalizing flow (NF) architecture to compress and uncompress point cloud data according to some embodiments. The inversibility of a normalizing flow architecture enables a normalizing flow architecture to differ from a lot of other architecture types. After a latent space is created, the original input may be reconstructed by applying the architecture in an inverted or reverse fashion. This property is very interesting considering that data compression may be naturally treated as an inversion problem. To be able to use such architectures, a squeezing operation is performed, efficiently trading the spatial size of the input for channels in order for the operations to be performed. An invertible neural network (flow) block was used in the ‘317 application and the journal article, Pinheiro, R. Borba, et al., NF-PCAC: Normalizing Flow Based Point Cloud Attribute Compression, 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) (2023) (“Pinheiro"). The architecture was adapted from the article Xie, Y., Cheng, K. L, and Chen, Q., Enhanced Invertible Encoding for Learned Image Compression, arXiv:2108.03690v1 (2021) (“Xie"). The architecture is shown in FIG. 3.

[0149] FIG. 3 illustrates encoding and decoding processes with a bitstream generated in-between each process. FIG 3 shows a normalizing flow architecture 300 adapted for compression of 3D point cloud color attributes. For some embodiments, a point cloud is inputted into the example architecture includes a feature enhancement block 302, an invertible neural network (INN) (or flow) block 304, a channel average block 306, and an attentive layer 308.

[0150] On the encoding side, the normalizing flow architecture 300 produces a latent space that is encoded by an entropy encoder 310 to produce a bitstream. In the example of FIG. 3, the entropy encoder 310 is a neural network-based encoder coupled with a hyperprior encoder 312. For the example configuration -23- shown in FIG. 3, the input to entropy encoder 310 receives the output of the attentive layer 308 and the previous version of the output of the hyperprior decoder 314. The hyperprior encoder 312 may include a neural network with sparse convolutions to transform the output of the attentive layer 308 into a bitstream with side information. This side information is used by the entropy encoder to output the main bitstream. For some embodiments, the goal of the hyperprior encoder 312 is to generate side information for use by the (main) entropy encoder. For some embodiments, the output of the hyperprior encoder 312 is passed through a hyperprior decoder 314, the output of which is used as an input into the entropy encoder 310. The output of the attentive layer 308 corresponds to the latent space of the original signal. The output of the hyperprior decoder 314, after passing through the hyperprior encoder 312, will provide context information to code the original latent space. In terms of data shape, for each coefficient in the latent space, the hyperprior decoder output 314 may provide a mean and a scale. Hence, the hyperprior may have double the size of the latent space that is outputted by the attentive layer 308.

[0151] On the decoder side, the bitstream is entropy-decoded using, for instance, a neural network-based entropy decoder 316 coupled with a hyperprior decoder 314 to provide a decoded latent space. The decoded latent is passed to the example normalizing flow architecture that includes an attentive layer 318, a channel copy layer 320, an invertible neural network 304, and a feature enhancement block 322 to produce a reconstructed point cloud.

[0152] For some embodiments, the feature enhancement layer 322 has the goal of extracting more nonlinear features from the original point cloud. In the architecture shown in FIG. 3, the feature enhancement layer 302 provides features to the INN block 304.

[0153] For some embodiments, the invertible neural network 322 includes three sets of a voxel shuffling layer, a 1x1 convolution, and a set of coupling layers. Because each of the layers has its own weight and bias, the three sets are not represented as a loop repeated three times. Since the data goes through a voxel shuffling layer in each set, the number of channels changes. As explained below, the size of the filters in the subsequent convolutions also changes. In the architecture of Xie, there is a sequence of 4 repetitions of the invertible block, which comprises a pixel shuffling layer, a 1x1 convolution, and 3 coupling layers.

[0154] In the architecture of FIG. 3, the number of invertible blocks is reduced in comparison with other architectures, such as example architectures shown, e.g., in the ‘317 application. This reduction is motivated by the evolution from a 2D architecture to a 3D domain. When adding a new dimension without reducing the number of repetitions of the invertible block, the number of coefficients explodes, and the use of the network may be affected negatively. The number of coupling layers in the invertible block also may be reduced for the same reason. For example, one squeeze operation may include one 1x1 sparse convolution followed by two coupling layers.

[0155] For the INN to have the desired effect and the convergence to be sped up, the voxel shuffling layer may be specifically designed for sparse 3D data. For some embodiments, the voxel shuffling layer has the goal of efficiently trading spatial dimension for channels without losing any information. For some embodiments, the 1x1 convolution of the invertible block has the goal of enhancing feature representation for the coupling layers. In the INN illustrated on FIG. 3, the 1x1 convolution is a sparse 1x1 convolution instead of the 1x1 convolution shown in Xie. A coupling layer may be a series of inversible transformations that are applied to an input tensor. In FIG. 3, for some embodiments, all the convolutions used in the coupling layers may be sparse 3D convolutions.

[0156] For some embodiments, the channel average layer is a layer in which the number of channels for the latent space is reduced by taking the average of all the channels in each spatial location. The voxel shuffling layer may be specifically designed for 3D sparse data. Without such a design for 3D sparse data, the channel average may take into account several zeros, thereby passing distorted coefficients to the attentive layer.

[0157] For some embodiments, the attentive layer has the goal of focusing on "more important” areas in the point cloud. For some embodiments, the attentive layer uses a sigmoid function as part of a weighting function for the encoder to allocate more bits to certain regions of the point cloud data. The attentive layer block illustrated in FIG. 3 is different from Xie because all the regular 2D convolutions of Xie are replaced with sparse 3D convolutions, which enables the process to handle the sparsity and the extra dimension of the point clouds.

[0158] The sparse nature and high number of points of a point cloud representation are typically not adapted to the use of regular 3D convolutions. Therefore, the architecture illustrated in FIG. 3 uses sparse convolutions. In some embodiments, a specifically designed 3D voxel shuffling layer is provided, which allows the system illustrated in FIG. 3 to converge faster.

[0159] For some embodiments, the layers of a network, which may become very large with over 270 million parameters, may not necessarily be used in compression. For some embodiments, a goal may be to enhance the performance of the architecture in the high bitrate domain while also reducing the number of coefficients.

[0160] FIG. 4A is a process diagram illustrating an example feature enhancement layer according to some embodiments. The feature enhancement layer has the goal to help extract more non-linear features from the original point cloud. An example of a feature enhancement layer 400 is illustrated in FIG. 4A: it is composed of a dense block architecture 402 followed by three 3D Sparse Convolutions 404, 406, 408 of kernel size 3x3x3 followed by another dense block architecture 410.

[0161] FIG. 4B is a process diagram illustrating an example feature enhancement according to some embodiments. An example of a sparse dense block 420 that may be used in the feature enhancement layer is illustrated in FIG. 4B. The dense block has the goal of preserving initial features along the convolutions by concatenating (cat) the output of previous convolutions in the output of current convolution. This block has been proven to enhance performance of learning-based architecture. In the architecture illustrated in FIG. 2, the feature enhancement layer provides better features for the INN in the core of the illustrated network. The feature enhancement layer is inspired by the one from the architecture of Xie, but where sparse 3D convolutions are used instead of the 2D regular convolutions of Xie.

[0162] FIG. 4C is a process diagram illustrating an example coupling layer according to some embodiments. A coupling layer 450 is a series of transformations that are applied to an input tensor and are completely inversible. The input tensor is split 452 into two parts, and each part goes through its own transformation according to a scheme illustrated on FIG. 4C. In FIG. 4C, the input tensor is split 452 into two parts: xi (454) and X2 (456), which are then transformed into yi (458) and y2 (460) before being concatenated together again to provide an output y (462). The transformations Gi, G2, Hi and H2are all composed of 3 sparse 3D convolutions each.

[0163] FIG. 4D is a process diagram illustrating an example transformation block for coupling layers according to some embodiments. FIG. 4D illustrates an example of a transformation block 480 for the coupling layers that may be used for the transformations G1, G2, Hi and H2 Inside a coupling layer, each one of the transformations is composed of 3 stages of 3D sparse convolutions of kernel 3x3x3 (482, 486, 490) and a LeakyReLU 484, 488. The LeakyReLU 484,488 is a deep learning block. The LeakyReLU block 484, 488 is similar to a ReLU block except that instead of passing only positive values, a LeakyReLU block 484, 488 also "leaks” negative values as well. For some embodiments, the amount of "leak” may be indicated with a parameter.

[0164] The arrangement of transformations guarantees the invertibility. In the embodiment illustrated on FIG. 2 and 4C, all the convolutions used in the coupling layers are sparse 3D convolutions.

[0165] The channel average layer is a layer where the number of channels for the latent space is reduced by taking the average of all the channels in each spatial location. The voxel shuffling layer specifically designed for 3D sparse data (which is described further below) is especially important here, because without it, the channel average would have several zeros taken into account, passing distorted coefficients to the attentive layer.

[0166] The attentive layer has the goal of helping the architecture to focus on more important areas in the point cloud. It uses a sigmoid function to give a weight to tell the encoder which regions of the point cloud would need more bits to be encoded. The attentive layer block illustrated on FIG. 2 is also modified by replacing all the regular 2D convolutions of Xie by sparse 3D convolutions to be able to handle the sparsity and the extra dimension of the point clouds.

[0167] The sparse nature and high number of points of the point cloud representation are not adapted to the use of regular 3D convolutions. Therefore, the architecture illustrated on FIG. 2 uses sparse convolutions.

[0168] FIG. 5 is a flowchart illustrating an example process for encoding a point cloud according to some embodiments. An embodiment for coding one or more attributes of the point cloud using an invertible neural network is illustrated on FIG. 5. FIG. 5 illustrates an example of a block diagram of a method 500 for encoding one or more attributes of the point cloud. At 502, the point cloud is provided as input to the encoding system. The encoding system comprises at least an invertible neural network configured for encoding at least one attribute of the point cloud. In some variants, the encoding system comprises geometry encoding module configured for encoding geometry of the point cloud. At 504, a latent representation of at least one attribute of the point cloud is obtained using at least the invertible neural network. At 506, the latent is encoded to produce a bitstream, for instance using a neural network-based entropy encoder.

[0169] FIG. 6 is a flowchart illustrating an example process for reconstructing a point cloud according to some embodiments. Another embodiment for coding one or more attributes of the point cloud using an invertible neural network is illustrated on FIG. 6. FIG. 6 illustrates an example of a block diagram of a method 600 for decoding one or more attributes of the point cloud. At 602, a bitstream is provided to the decoding system. The bitstream comprises at least coded data representative of at least one attribute of the point cloud. The decoding system comprises at least an invertible neural network configured for decoding at least one attribute of the point cloud. In some variants, the bitstream also comprises coded data representative of the geometry of the point cloud and the decoding system comprises geometry decoding module configured for decoding and reconstructing the geometry of the point cloud. At 604, a latent representation of at least one attribute of the point cloud is obtained by decoding the bitstream part representative of the at least one attribute, for instance using a neural network-based entropy decoder. At 606, the at least one attribute is reconstructed using at least the invertible neural network.

[0170] The use of 3D applications is becoming more popular every day, and to be able to exploit said applications different data formats are being used. One of the main data formats is a point cloud. Point clouds -27- (PCs) are a set of unordered points with coordinates x, y, z, corresponding to the location of the point in the space and its attributes (such as colors and normal vectors, among other things).

[0171] The use of this new type of data requires the creation of new compression methods to efficiently store and transmit data, especially since point clouds may have millions of points. Different methods to compress the attributes have been proposed, and among those methods, learning-based architectures are gaining strength. Those architectures are often extensions of learning-based methods in the 2D image domain.

[0172] The use of Variational Auto Encoders (VAEs) (such as Wang, J., et al., [AI-3DGC] Point Cloud Attribute Compression using Sparse Tensor-Representation, Moving Pictures Expert Group (2022) (“Wancf)) and Normalizing Flows (NFs) (such as Pinheiro, Rodrigo B., et al., NF-PCAC: Normalizing Flow Based Point Cloud Attribute Compression, 2023 IEEE International Conf, on Acoustics, Speech, and Signal Processing (ICASSP) (2023) (“Pinhero")) as compression architectures have been explored in the point cloud domain. However, these methods are understood to be not readily interpretable, to have some performance issues for high bitrates in the VAE case, and to have performance issues for model size and complexity in the NF case. Also, none of these methods are able to match the performance of traditional methods.

[0173] Based on experiments, what makes the attributes of point clouds so difficult to code is their support. The sparse nature of the data stops the network from learning good context information to learn how to code point clouds. Even architectures that work very well in image compression, suffer in performance when translated to PCs.

[0174] One of the biggest differences between an image and a point cloud is the support, which is sparse and 3D in the point cloud case. Such 3D sparse support is translated into 2D dense support, such as an image. This translation is done in a small space to minimize distortions that may be introduced. A 3D sparse convolution is translated into a regular 2D convolution. Instead of processing a point cloud in three dimensions, the inherent support of a point cloud is modified to resemble more that of an image.

[0175] Since the convolutions used in the networks are usually 3x3x3 or 5x5x5, the context for a voxel may be calculated using only a small support. A 3D point cloud may be reduced to 2D patches by producing micro-patches for each occupied voxel in the point cloud. This reduction is done to perform the convolution and to calculate the features, but the support of the point cloud continues to be 3D. Introduced herein is a micropatch. A micropatch is a 2D square supported by a 3D plane onto which principal components of a 3D voxel are projected. A micropatch summarizes the context of the center pixel of a 3D voxel and may be used by a traditional 2D convolutional systems.

[0176] Included within are descriptions of how to compute such micropatches and how to adapt a deep architecture to use a micropatch for point cloud attributes compression (especially point colors). Micropatches also may be used for the positions of the points.

[0177] The capabilities of sparse convolutions may be mixed with the capabilities of regular 2D convolutions. The capabilities of sparse convolutions may be used to perform downsampling of point clouds and pooling of features. The capabilities of regular 2D convolutions may be used to learn the weights that define the tensor to be compressed.

[0178] For some embodiments, the patches may be pre-calculated before the input of the network and used to produce the indices of each micropatch. For some embodiments, the patches may be calculated online in the network. For some embodiments, the patches are calculated on-the-fly while the network is running.

[0179] Since the calculation of the patches may become computationally heavy, the patches may be calculated beforehand for some embodiments. By doing so, a model may be trained without the need to recalculate the patches on every forward pass. The patches may be calculated using only the geometry of the model.

[0180] FIG. 7A is a schematic illustration showing an example of a 3D sparse point cloud being translated into a 3x3 patch according to some embodiments. FIG. 7A shows an example 3x3 patch 702 that describes the context of each point in a point cloud 700 using a 3D to plane approximation. The two points on the left side of the front slice are ignored for fabrication of the micropatch. They are shown in FIGs. 7A to 7G to demonstrate an example point cloud with points that are ignored and thereby approximated with the same micropatch as a point cloud that does not include such additional points for some embodiments.

[0181] Depending on the occupied neighbors in 3D, there are a limited number of possible combinations of projections. Also, in most cases, when too many points exist in the surrounding area, a principal component that supports a projection plane may be hard to find. If there are 2 or more support planes that fit well the current neighborhood, the neighborhood may be extended. For example, a search may be done in a 3x3x3 neighborhood, and 2 planes may be found that are tied. The size of the neighborhood may be increased to 5x5x5, and a comparison of the 2 planes that are tied may be done to find the best global one. The micropatches may be generated from the chosen plane. An approximation may be used, which tries to find the best possible reduced context.

[0182] For each point Pijk of a point cloud, a sparse tensor that has a size (shape) of Nx3 may be transformed into a tensor of size Nx3x3x3. N is the number of points in the point cloud. With the addition of 2 extra dimensions in the tensor, a 3D point cloud may be treated as a regular 2D convolution with a small patch. In this case, the 3D point cloud may be processed by regular 2D convolutions of the generated local patches. For some embodiments, the situation may be treated as a batch of N micropatches of size 3x3.

[0183] A 2D convolution with a kernel size of Cout x Cin x 3 x 3 may be performed. Cin is the number of channels in the current tensor. Cout is the number of channels of the desired output tensor. For a first layer of an RGB scenario, Cin is 3.

[0184] Stated differently, for example, a point cloud may have a size of N x Cin in which Cin corresponds to RGB. Micropatches are generated, and the point cloud becomes N x Cin x 3 x 3. A convolution is performed, in which the kernel is of size Cin x Cout x 3x3. A result is obtained which has a size of X x Cout x 1 x 1. Since there is no padding, the convolution is calculated only in the center point.

[0185] Without a padding operation, a tensor of size N x Cout x 1 x 1 is recovered. Information that was given by the context may be translated into new channels of the network. As a result, a feature map may be created similar to creating one in the 2D image domain. Since a 3x3 patch always has the same support shape, the weights of this 2D convolution are always trained with some data. In a sparse convolution, some of the voxels are empty. As a result, some of the weights of the kernel are not always used during the training stage. For a dense 2D convolution, since the 3x3 micropatch is filled and dense, there is always some data to use for training. Patch production

[0186] Different methods may be used to produce 3x3 patches for each point of a point cloud. To find a plane for each point, the closest neighbors are used. A smaller point cloud in a 3x3x3 cube may be built around the current point. With that smaller point cloud, different methods may be used to find the plane with the largest support.

[0187] For some embodiments, a neighborhood of points may be represented via Open3D using RANSAC. The inputs for such a method may be: (1) the distance threshold for a point to be considered an inlier, (2) the number of points that are sampled to estimate a plane, and (3) the number of iterations to find the best plane. The best plane is the one with the most inliers. See Point Cloud, Open3D, version 0.18.0, available at www<dot>open3d<dot>org / docs / release / tutorial / geometry / pointcloud<dot>html.

[0188] The threshold distance may be set to < 0.1. Such a threshold means that a plane may have a maximum of 9 inliers. With a plane equation and the inlier points that are part of the plane, a 3x3 patch is filled. If there are not enough inliers, the point in the 3x3 patch is filled with its closest neighbor.

[0189] The maximum number of inliers using the Open3D method is 9, which would perfectly fill a patch. The center point is an inlier to the plane. FIG. 7A has 9 inlier points (indices 0 to 8, with the center point being index 4). FIG. 7A is an example in which the calculated plane is the plane with Z = 0.

[0190] However, the scenario of FIG. 7A is not usually the case. Different strategies may be used to fill the remaining points. In the examples shown in FIGs. 7A to 7G, the point in the center has a value of 4, and the values of the surrounding points are equal to their corresponding index. For the right side of each of FIGs. 7A to 7G, the numbers that are in the 3x3 grids are index numbers that point to the original point cloud.

[0191] In FIGs. 7B to 7G, there are 7 inliers and 2 spots to fill. The inliers have indices of 0, 1, 2, 3, 4, 5, and 6. The spots to fill in FIGs. 7B to 7G are labeled with indices 7 and 8 in corresponding locations in FIG. 7A. Padding with inliers

[0192] FIG. 7B is a schematic illustration showing an example 3D sparse point cloud to 3x3 patch translation showing padding with mirrored inliers according to some embodiments. In the example translation from 3D sparse point cloud 710 to 3x3 patch 712, the padding may be done in a mirrored fashion with the points that are already available in a patch. The micropatch is traversed from left to right, top to bottom. So, the filling order for this example is 0->1->2->3->4->5->6, and the last two points will be filled. The micropatch is traversed backwards: 6->5->4->...->0. This method is a padding method that is done in image processing. The following website shows an example calculation:      PyTorch, pytorch<dot>org / docs / stable / generated / torch.nn.ReflectionPad2d<dot>html. Padding with zeros

[0193] FIG. 7C is a schematic illustration showing an example 3D sparse point cloud to 3x3 patch translation showing padding with zeros according to some embodiments. For some embodiments, one or more points of a point cloud may be ignored. The ignored points may be filled with zeros. Doing so may negatively impact the learning process because information that is not part of the actual point cloud is inserted into the learning process.

[0194] In the example translation from 3D sparse point cloud 720 to 3x3 patch 722 of FIG. 7C, locations "7” and "8” (which are not labeled in FIG. 7C) are ignored and padded with zeros in the output patch. Location "7” is the lower center location of the larger cube shown on the left side of FIG. 7C. Location "8” is the lower right location of the larger cube shown on the left side of FIG. 7C. The zero padding is shown on the right side of FIG. 70, with locations [lower row center] and [lower row right] of the patch padded with zeros. Padding with the nearest neighbor

[0195] FIG. 7D is a schematic illustration showing an example 3D sparse point cloud to 3x3 patch translation showing padding with the nearest neighbor (considering the y-axis as being priority) according to some embodiments. For FIGs. 7A to 7G, the y-axis is up and down. For some embodiments, the nearest neighbor or closest point to the examined point in the plane is chosen. In the example translation from 3D sparse point cloud 730 to 3x3 patch 732 of FIG. 7D, the patch is filled with the chosen point. Respecting an axis order to fill, choose the nearest neighbor. The empty spaces in the micropatch are filled with the closest neighbor while respecting the axis order. If there are 2 closest neighbors, the neighbor present in the y-axis is chosen, for example.

[0196] The "4” and "5” shown in the grid on the right side of FIG. 7D relate to the closest neighbors to the corresponding position on the patch. For the [lower row right] location, the closest neighbor has an index of 5, so a "5” is stuffed into the open location. For the [lower row center] location, the closest neighbors have indices of 4 and 6. Since the y-axis has priority, the index of "4” is chosen. Padding with the neighbor average

[0197] FIG. 7E is a schematic illustration showing an example 3D sparse point cloud to 3x3 patch translation showing padding with the nearest neighbor average according to some embodiments. For some embodiments, the patch is filled with the average of the closest neighbors.

[0198] In the example translation from 3D sparse point cloud 740 to 3x3 patch 742 of FIG. 7E, the locations [lower row center] and [lower row right] of the patch are padded with the average of the closest neighbors in the patch. For this example, the average is 5 in both scenarios. For the [lower row center] location, the closest neighbors have indices of 4 and 6, so the average is 5. For the [lower row right] location, the closest neighbor has an index of 5, so the average is 5. Padding with the global average

[0199] FIG. 7F is a schematic illustration showing an example 3D sparse point cloud to 3x3 patch translation showing padding with a global average according to some embodiments. After filling the patch with all the inliers, the average value of the inliers is determined, and the rest of the patch is filled with this calculated average.

[0200] In the example translation from 3D sparse point cloud 750 to 3x3 patch 752 of FIG. 7F, the locations [lower row center] and [lower row right] of the patch are padded with the average of the inliers. For this example, the average of the inliers is 3. The global average is shown in Eq. 1: iii          0+1+2+3+4+5+6  21 o global average =-----------= — = 3                  (1) Padding with an interpolation of neighbors

[0201] FIG. 7G is a schematic illustration showing an example 3D sparse point cloud to 3x3 patch translation showing padding with an interpolation of nearest neighbors according to some embodiments.

[0202] In the example translation from 3D sparse point cloud 760 to 3x3 patch 762 of FIG. 7G, the nearest neighbors with respect to their distances are interpolated to fill the remaining points. The voxel directly under the center is calculated by Eq. 2: (44+¼¾ = 3.91                               (2)

[0203] The voxel on the lower right is shown in Eq. 3: interpolation value =    1 =3.9                     (3)

[0204] Eq. 4 shows a general equation for the interpolated value. The equation depends on the distribution of the points in the point cloud: interpolation value =----disuc.n) '                             size(N)                                       ' ' For each non-filled patch point c, all n neighbors that are part of the N group of nearest neighbors are summed and divided by their relative distance to the point C. This numerator value is divided by the number of neighbors in the group, N.

[0205] By using methods in which a calculation is not needed to fill in the remaining points, the patch may be pre-determined for an entire dataset. The indices of the points that are used to fill each patch may be saved in an auxiliary file. For example, methods that fill in remaining points in a patch without a calculation may use inliers, zeros, and / or nearest neighbors. If a calculation is needed to fill in the remaining points, patch filling may need to be done on-the-fly during a training / inference process.

[0206] For some embodiments, if it is impossible to find a unique plane equation using a 3x3x3 neighborhood, then the neighborhood may be increased in size until a unique valid plane is found. For example, the neighborhood size may be increased to 5x5x5, 7x7x7, or larger. An example of when it is impossible to find a unique plane equation (and thereby there are many possible plans) occurs when all points are occupied.

[0207] FIGs. 8A-8D are schematic illustrations showing a series progression for an example 3D sparse point cloud to 3x3 patch translation showing padding with mirrored inliers according to some embodiments. FIG. 8A shows a starting position of an example mirror padding process. The point cloud 800 is populated for indices 0 and 3. A mirror line is drawn as a diagonal line from the lower left corner to the upper right corner of the patch grid 802. For some embodiments, mirror padding only works if at least one side neighbor and one corner neighbor are available.

[0208] FIG. 8B continues the example started in FIG. 8A. The [lower row right] and [lower row center] locations are filled with the corresponding mirror locations across the mirror line. The point cloud 820 populated points for indices 0 and 3 are filled in the [lower row right] and [lower row center] locations, respectively. A mirror line is drawn as a diagonal line from the lower left corner to the upper right corner of the patch grid 822.

[0209] FIG. 8C continues the example started in FIGs. 8A and 8B. A second mirror line is drawn from the upper left corner to the lower right corner of the patch grid. The [upper row center] and [middle row right] locations are filled with the corresponding mirror locations across the mirror line. The point cloud 840 populated point for index 3 is filled in the [upper row center] and [middle row right] locations. A mirror line is drawn as a diagonal line from the upper left corner to the lower right corner of the patch grid 842.

[0210] FIG. 8D continues the example started in FIGs. 8A-8C. A third mirror line is drawn horizontally across the middle row of the patch grid. The [upper row right] and [lower row left] locations are filled with the corresponding mirror locations across the mirror line. The point cloud 860 populated point for index 0 is filled in the [upper row right] and [lower row left] locations. A mirror line is drawn as a straight line from the left center to the right center of the patch grid 862.

[0211] For some embodiments, a vertical mirror line may be drawn through the middle column of the patch grid. This vertical mirror line may be used to fill empty locations in the patch grid with populated point cloud points and / or already filled points in the patch grid.

[0212] FIG. 9 is a schematic illustration showing example convolutions for a 3D sparse point cloud and for a 3D sparse point cloud to 3x3 patch translation according to some embodiments. The top portion of FIG. 9 shows a convolution process 900 for a center point 902 of a portion of a 3D point cloud 904. Each populated point of the sparse 3D point cloud 904 is multiplied by a 3D kernel 906 of corresponding weights. The results of these multiplications are summed and used as the convolution result 908 for the center point.

[0213] The solid blocks 910 in the 3D kernel 906 represent weights that are trained. These solid blocks 910 of the 3D kernel 906 are multiplied by an existing voxel in the point cloud. White or clear blocks 912 in the 3D kernel 906 represent locations that did not contribute to the convolution result.

[0214] The bottom portion of FIG. 9 shows a convolution process 916 for the center point of a patch grid. A patch projection 914 is performed on the sparse 3D point cloud 904 to generate the 2D patch 918. Each point of the patch grid 918 is multiplied by a corresponding grid of weights 920 (which may be called a 2D kernel for some embodiments). The results of these multiplications are summed and used as the convolution result 922 for the center point of the patch. This convolution result 922 is then inserted into the 3D location for the center point of the original point cloud 924.

[0215] The 2D kernel is shown as a set of solid squares to emphasize that each location in the 2D kernel contributes to the convolution result.

[0216] FIGs. 10A-10B are system diagrams illustrating example sets of interfaces for a network according to some embodiments. FIG. 10A shows a first set of interfaces to a network. For some embodiments, a network 1000 may receive input coordinates 1002 and input features 1004 as inputs. FIG. 10B shows a second set of interfaces 1052,1054,1056 to a network 1050. For some embodiments, a network 1050 may receive input coordinates 1052, input features 1054, and input indexes 1056 as inputs. The inputs 1002, 1004,1052,1054,1056 to FIGs. 10A and 10B show components that may be associated with a point cloud for some embodiments. The input coordinates are the coordinates of the point cloud (the positions x, y, z), and the input features are the colors, RGB. Comparing FIGs. 10A and 10B reveals the need to add an input to the current networks used for point cloud compression to be able to use micropatch convolution. An example of a network is shown in FIG. 11.

[0217] FIG. 11 is a process diagram illustrating an example encoder / decoder system according to some embodiments. Compare FIG. 11 with FIG. 13. FIG. 13 adds some details regarding patches that are described below. FIG. 11 shows an example process 1100 in which an input point cloud 1102 undergoes encoding 1104. The encoder output goes through a Q (quantization) block 1108 and then inputted to an arithmetic encoder (AE) 1112. For learning-based algorithms, the quantization block 1108 may be a rounding operation. Auto-encoding refers to the entire architecture shown in FIG. 11. The encoder output also goes through a hyper encoder 1106. The hyper encoder output goes through a Q block 1110 and then through a second arithmetic encoder (AE) 1114. The second arithmetic encoder 1114 is fed with the factorized entropy model parameters 1118 and the output of the Q block 1110 to produce the hyperprior bitstream. The factorized entropy model 1118 generates probabilities for the arithmetic encoder (AE) 1114 to code the quantized latent. The hyperprior bitstream is inputted into the second arithmetic decoder (AD) 1122 to -35- reconstruct the latents that were produced after the Q block 1110. The output of the second arithmetic decoder 1122 is inputted into a hyper decoder 1124. The hyper decoder 1124 generates a set of entropy parameters 1116. These entropy parameters 1116 are inputted into the first arithmetic encoder 1112. The outputs of the first arithmetic encoder 1112 and the second arithmetic encoder 1114 form first and second bitstreams, respectively.

[0218] On the decoder side, the second bitstream is inputted into a second arithmetic decoder 1122. Factorized entropy model parameters are also inputted into the second arithmetic decoder 1122, which outputs to a hyper decoder 1124. The hyper decoder 1124 generates a set of entropy parameters 1116. These entropy parameters 1116 are inputted into the first arithmetic decoder 1120 alongside the first bitstream. The output of the first arithmetic decoder 1120 is inputted into the first decoder 1126. The output of the first decoder 1126 is a reconstructed point cloud 1128.

[0219] FIG. 12 is a process diagram illustrating an example patches generation block according to some embodiments. For some embodiments, the patches are determined (calculated) for a point cloud (PC) at different points of a model. For example, if a downsampling happens due to stride in the convolutions, the patches are recalculated with the newdownsampled point cloud points. In the learning-based architectures, the term "stride” means a convolution is performed on some points, not all of them. For example, for a Stride 2, the convolution is performed on point 0. Point 1 is skipped, and the convolution is performed on point 2.

[0220] FIG. 12 shows an example process block 1200 that may be used to produce the patches 1202, 1206, 1210 in a pre-processing step of an architecture. A downsampling method 1204, 1208 matches the level of downsampling used in the architecture. Each of the "produce patches” blocks 1202,1206,1210 may be any type of patch production. In FIG. 12, the input coordinates are coordinates of a point cloud.

[0221] For the architecture to work, the amount of downsampling performed during production of the patches needs to match the corresponding architecture layer.

[0222] The patches for a point cloud may be pre-calculated after each downsampling 1204, 1208, such as after a 2x2x2 downsample, after a 4x4x4 downsample, and after an 8x8x8 downsample. This correspondence means that there are 4 sets of indices for each point in the point cloud. FIG. 12 shows an example of a preprocessing block that produces such patches. For some embodiments, there may be more downsampling performed, and the downsampling may be of different sizes, such as 3 x 3 x 3 and 5 x 5 x 5, among other scenarios.

[0223] FIG. 13 is a process diagram illustrating an example of average pooling according to some embodiments. For this example process 1300, each quadrant 1302, 1304, 1306, 1308 in the 4x4 grid is averaged to generate the 2x2 grid 1310. For example, the values in the upper right quadrant 1302 of the 4x4 grid (28,184,70, and 38) are averaged to generate the value (80) in the upper right box of the 2x2 grid 1310.

[0224] For some embodiments, a point cloud may be downsampled using an AveragePooling process that is used by a MinkowskiEngine process. The MinkowskiEnginge takes into account the sparsity of the point cloud and the 3D nature of the data. The AveragePooling process calculates the average of a defined neighborhood.

[0225] For an original point cloud, the neighbors in the 3x3x3 bounding box around the point are calculated with an auxiliary vector that has the relative positions of those neighbors: neighborhood_0 = [[1,1, 0], 0] , [1,1, 1], 1] , [1,1,-1], [1,0, 0], [1,-1, 0], [1,0, 1], [1,-1, 1], [1,0,-1], [1,-1,-1], [0,1, 0] , [0,0, 0] , [0,1, 1], [0,0, 1], [0,1,-1] , [0,0,-1] , [0,-1, 0] , [-1,1, 0] , [0,-1, 1] , [-1,1, 1] , [0,-1,-1] , [-1,1,-1] , [-1,0, 0], [-1,-1, [-1,0, 1], [-1,-1, [-1,0,-1], [-1,-1,-

[0226] For a downsampled point cloud (by 2), the elements of the vector are multiplied by 2 to determine the new neighborhood. Downsampling by other amounts uses a similar process.

[0227] For the example nelghborhood_0 vector shown, the vector size is: (TV x 9) + (N' x 9) + (N" x 9) + (N'” x 9) in which N is the number of points in the point cloud and each apostrophe (') represents one downsample. For example, N' is the number of points in the 2x2x2 downsampled version of the point cloud. The example nelghborhood_0 vector is used in the code listing shown later in the application. For this example, the size of the neighborhood is 2x2x2.

[0228] The neighborhood_0 vector is used to calculate the neighborhood in the first produce patches block. The nelghborhood_0 vector is multiplied by 2, and the result is used for the second block. This process is repeated for the third and following blocks. Architecture Description

[0229] FIG. 14 is a process diagram illustrating an example variational auto-encoder / decoder architecture according to some embodiments. See the description of FIG. 11 for a description of most of the components shown in FIG. 14. Many different types of architecture (especially an architecture known to be efficient for image compression) may be used with micropatch convolution. For example, a variational auto-encoder (VAE)-based architecture may be used to demonstrate the use of the micropatch convolution.

[0230] In FIG. 14, the patch calculation is performed before the encoding is done. The patch calculation (labeled as a "Patches” block) provides indices for the patched convolution to be performed. These indices are inputs into the encoder. FIG. 14 shows a VAE-based architecture 1400. The "Patches” blocks 1402,1404 produce the different indices used for each level of a patch convolution. For some embodiments, a "Patches” block is doing a 3D point cloud to 2D patch generation process, such as the examples seen in FIGs. 7A to 7G. The 3x3 micropatches shown in FIGs. 7A to 7G are examples. The micropatches may be 5x5, for example.

[0231] On the encoder side, a point cloud 1404 is an input into a patch generation block 1402. The output of the patch generation block 1402 is an input into the encoder 1406. The encoder output goes through a Q (quantization) block 1408 and then inputted to an arithmetic encoder (AE) 1410. The encoder output also goes through a hyper encoder 1412. The hyper encoder output goes through a Q block 1414 and then through a second arithmetic encoder (AE) 1416. The second arithmetic encoder 1416 is fed with the factorized entropy model parameters 1418 and the output of the Q block 1416 to produce the hyperprior bitstream. The hyperprior bitstream is inputted into the second arithmetic decoder (AD) 1420 to reconstruct the latents that were produced after the Q block 1414. The output of the second arithmetic decoder 1420 is inputted into a hyper decoder 1422. The hyper decoder 1422 generates a set of entropy parameters 1424. These entropy parameters 1424 are inputted into the first arithmetic encoder 1410. The outputs of the first arithmetic encoder 1410 and the second arithmetic encoder 1416 form first and second bitstreams, respectively.

[0232] On the decoder side, a reconstructed point cloud geometry 1432 is an input into a patch generation block 1404. The output of the patch generation block 1404 is an input into the decoder 1428. The architecture shown in FIG. 14 covers only the colors (RGB) of a point cloud. The geometry already in the decoder side is used to be able to perform the calculations discussed herein.

[0233] On the decoder side, the second bitstream is inputted into a second arithmetic decoder 1420. Factorized entropy model parameters are also inputted into the second arithmetic decoder 1420, which outputs to a hyper decoder 1422. The hyper decoder 1422 generates a set of entropy parameters 1424. These entropy parameters 1424 are inputted into the first arithmetic decoder 1426 alongside the first bitstream. The output of the first arithmetic decoder 1426 is an input into the decoder 1428. The output of the decoder 1428 is the reconstructed point cloud 1430.

[0234] FIG. 15 is a process diagram illustrating an example encoder using patch convolutions according to some embodiments. The patch production block shown in FIG. 15 may be implemented as shown in FIG. 12 for some embodiments. FIG. 15 shows an example implementation of the 3 blocks in the upper left corner of FIG. 15: an input point cloud, a patch production block (a "Patches” block in FIG. 14), and an encoder.

[0235] The large block in the center of the example process 1500 of FIG. 15 is an encoder 1502. FIG. 15 shows an example set of operations that may be performed for each set of patch indices generated. For example, an input point cloud and a set of indices may be inputted into a first patch convolution 1506 followed by a first LeakyReLU process 1508. A LeakyReLU process is a deep learning block for some embodiments. The output of the first LeakyReLU process 1508 may be an input into a second patch convolution process 1510 with a set of indices as additional inputs. The output of the second patch convolution process 1510 may go through a second LeakyReLU process 1512 and an average downsampling process 1514. This encoder sub-block that includes two patch convolution processes 1506, 1510, 1516, 1520, 1526, 1530, two LeakyReLU processes 1508,1512,1518,1522,1528,1532, and an average downsampling process 1514, 1524,1534 may be repeated for each set of indices generated by the patch production block. The output of the encoder 1502 is a latent space. For some embodiments, one or more of the LeakyReLU blocks are replaced with a rectifier linear unit (ReLU) block. For some embodiments, the indices are generated by a patch production process 1504.

[0236] FIG. 16 is a process diagram illustrating an example encoder using patch convolutions with on-the-fly patch generation according to some embodiments. FIG. 16 is similar to FIG. 15 except that patch production / generation 1602,1604,1606 is done “on-the-fly" for each encoder sub-block corresponding to a set of indices. In FIG. 15, the patches may be pre-determined prior to performing encoding for some embodiments, while in FIG. 16, the patches are determined on-the-fly for each encoder sub-block corresponding to a set of indices. Again, the output of the encoder is a latent space.

[0237] Each sub-block of the encoder 1600 may include two patch convolution processes 1608, 1612, 1618, 1622, 1628, 1632, two LeakyReLU processes 1610, 1614, 1620, 1624, 1630, 1634, and an average downsampling process 1616,1626,1636.

[0238] FIG. 17 is a process diagram illustrating an example decoder using patch convolutions according to some embodiments. FIG. 17 is a decoder process 1700 corresponding to the encoder process of FIG. 15. Also, FIG. 17 corresponds to the four components in the lower left portion of FIG. 14: reconstructed point cloud geometry, patch production (the "Patches” block for the decoder in FIG. 14), decoder, and reconstructed point cloud. In FIG. 17, the point cloud geometry is inputted into the patch production block 1702 to generate the sets of indices. The latent space is inputted into the decoder 1704. Each set of indices in FIG. 17 goes through a decoder sub-block that is similar to the encoder sub-block described above except that upsampling is performed after the first patch convolution and LeakyReLU combination and average downsampling is not performed. For some embodiments, each decoder sub-block may include the sequence of a first patch convolution 1706, 1716, 1726, a first LeakyReLU 1708, 1718, 1728, an upsampling 1710, 1720,1730, a second LeakyReLU 1712,1722,1732, and a second patch convolution 1714,1724,1734.

[0239] For some embodiments, the upsampling may also be performed by a Minkowski Engine framework for consistency throughout a network. For some embodiments, the goal of patch convolution is to make the network smaller going from 3D convolutions to 2D convolutions. Another goal is to have the same support for all the convolutions, thereby getting rid of the effect of sparsity of the point cloud in the convolutions.

[0240] The hyperprior process may be done either using patch convolutions or with regular sparse convolutions. If patch convolutions are used in the hyperprior part of the network, more indices may be calculated in the preprocessing. More indices are used because the hyperprior process also performs downsampling of the point cloud. Training

[0241] A network may be trained with any static point cloud dataset. In the training stage of a network, the quantization block is replaced for the sake of the back propagation needed to train the network.

[0242] Using a simulated dense 2D environment with micropatches, the dataset may be enriched using any image dataset available. This technique solves the problem of a lack of a dataset for 3D data.

[0243] The quantization process may be simulated by a random normal noise between [-0.5, 0.5], The entropy encoder and entropy decoder are not converted to models because they are lossless.

[0244] The network may also be trained and may function in 2 different modes according to the channel space used. RGB / YUV

[0245] For an RGB channel, the values may be divided by 255 to put them between 0 and 1. For a YUV channel, the values may be transformed into luma and chroma channels. Also, the network may be trained in other color spaces. Signaling / Syntax

[0246] Current activities in MPEG AI-PCC include work on defining some Al models for compression and decompression of point clouds geometry and photometry. The MPEG group currently handles photometry separately from geometry. A micropatch-based approach may be used for feature extraction instead of using sparse convolutions throughout the model. This approach may be applied to any type of architecture.

[0247] Using a micropatch-based architecture means that the architecture uses 2D convolutions (such as the ones used for image compression). A signaling flag may be present in the bitstream to indicate which deep decoder architecture to use. This flag or another flag may be used to indicate which type(s) of convolution is / are used. Further configuration data may indicate the color space and the channel mode used. Table 1 shows example signaling that may be used in a bitstream for a point cloud learning-based architecture. The PHOTOMETRY_CODEC_ARCHITECTURE configuration parameter / signal may be used to indicate the use of a variable auto-encoder (VAE), normalizing flow, or a wavelet architecture. Configuration Parameter Data Values Size PHOTOM ETRY_CODEC_ARCH ITECTU RE 00 -> use Variational Auto-Encoder 01 -> use Normalizing Flow 10 -> use Wavelet 2 bit flag USE_MICROPATCH 0 - > use sparse convolutions 1 -> use micropatch convolutions 1 bit flag COLOR_SPACE 0-> RGB 1->YUV 1 bit flag Table 1. 2D Patch Convolution

[0248] The code listing below shows an example code listing for a 2D patch convolution process. import torch from torch import Tensor from torch.nn.common_types import _size_2_t class ProducePatches(torch.nn.Module) : def __init__(self): super () .__init__() def forward (self, attributes, indexes) : feats                                                                               : attributes[indexes . view ( (len (attributes) *9,) ) ] . view (len(attributes) ,3,3,attri butes. shape[1]) . permute (dims=(0,3,1, 2) ) return feats class PatchConv2D(torch.nn.Conv2d) : def __init__(self, in_channels: int, out_channels: int, kernel_size: _size_2_t, stride: _size_2_t = 1, padding: _size_2_t = 0, dilation: _size_2_t = 1, groups: int = 1, bias: bool = True, padding_mode: str = 'zeros'): super().__init__(in_channels, out_channels, kernel_size, stride, padding, dilation, groups, bias, padding_mode) self.patches = ProducePatches() def forward (self, input: Tensor, indexes: Tensor) -> Tensor: input = self .patches (input, indexes) return super () . forward (input) . squeeze () Patch Construction

[0249] An example patch construction algorithm is shown below. After all point clouds are collected in a dataset, the dataset may be processed. neighborhood_0 =      [[1,1, 0], [1,0, 0], [1,-1, 0] , [0,1, 0], [0,0, 0], [0,-1, 0] , [-1,1, 0], [-1,0, 0], [-1,-1, 0], [1,1, 1], [1,0, 1], [1,-1, 1] , [0,1, 1], [0,0, 1], [0,-1, 1] , [-1,1, 1], [-1,0, 1], [-1,-1, 1], [1,1,-1],[1,0,-1],[1,-1,-l],[0,l,-l],[0,0,-l],[0,-l,-l],[-l,l,-l],[-l,0,-1],[-1,-1,-1]] neighborhood_down2 = [[2,2, 0], [2,0, 0], [2,-2, 0] , [0,2, 0], [0,0, 0], [0,-2, 0] , [-2,2, 0], [-2,0, 0] , [-2,-2, 0], [2,2, 2], [2,0, 2], [2,-2, 2], [0,2, 2], [0,0, 2], [0,-2, 2], [-2,2, 2], [-2,0, 2], [-2,-2, 2], [2,2,-2], [2,0,-2], [2,-2,-2] , [0,2,-2], [0,0,-2], [0,-2,-2] , [-2,2,-2], [-2,0,-2] , [-2,-2,-2]] neighborhood_down4 = [[4,4, 0], [4,0, 0], [4,-4, 0] , [0,4, 0], [0,0, 0], [0,-4, 0] , [-4,4, 0], [-4,0, 0] , [-4,-4, 0], [4,4, 4], [4,0, 4], [4,-4, 4], [0,4, 4], [0,0, 4], [0,-4, 4], [-4,4, 4], [-4,0, 4], [-4,-4, 4], [4,4,-4],[4,0,-4],[4,-4,-4],[0,4,-4],[0,0,-4],[0,-4,-4],[-4,4,-4],[-4,0,-4],[-4,-4,-4]] neighborhood_down8 = [[8,8, 0], [8,0, 0], [8,-8, 0] , [0,8, 0], [0,0, 0], [0,-8, 0] , [-8,8, 0], [-8,0, 0] , [-8,-8, 0], [8,8, 8], [8,0, 8], [8,-8, 8], [0,8, 8], [0,0, 8], [0,-8, 8], [-8,8, 8], [-8,0, 8], [-8,-8, 8], [8,8,-8],[8,0,-8],[8,-8,-8],[0,8,-8],[0,0,-8],[0,-8,-8],[-8,8,-8],[-8,0,-8],[-8,-8,-8]] neighborhood_all = [neighborhood_0, neighborhood_down2, neighborhood_down4, neighborhood_down8] downsample = ME.MinkowskiAvgPooling(kernel_size=2,stride=2, dimension=3) . to ( "cuda") for dataset in experiments["datasets"]: filenames = np.array(glob(os.path.join(dataset, "* . ply") )) for file in tgdm( filenames) : pc_data = o3d. io .read_point_cloud( file) pc_name = file. split (" / ") [-1] . split (".") [0] coords_initial = np.asarray(pc_data. points) .tolistO colors_initial = (np.asarray(pc_data. colors )*255) .tolistO coords,              feats              =             ME . utils . sparse_collate ([torch.Tensor (coords_initial)] , [torch . Tensor (colors_initial) / 25 5] ) ST = ME.SparseTensor(features=feats.cuda(), coordinates=coords.cuda() ) for n_down, neighborhood in enumerate(neighborhood_all): if os.path. exists (dataset+"patches / "+pc_name+"_"+str(n_down)+".npy") : continue patch_indexes = np.zeros ((len(ST.C),9)) #Fill DICT with the data for fast search dict_idx = {} for idx, coord in enumerate(ST.C[:,1:]): dict_idx[tuple(coord .cpu() .numpy())]=idx for enu, coord in enumerate(ST.C[:,1:]): idx_list = [] new_pc_coords = [] for neigh in neighborhood: try: idx = dict_idx[tuple(np. array(coord.cpu() )-np. array(neigh) )] except: continue idx_list.append(idx) new_pc_coords. append(list(np.array(coord.cpu() )-np. array(neigh)) ) new_pc = o3d.geometry.PointCloud() new_pc.points = o3d.utility.Vector3dVector(new_pc_coords) try: plane_model, inliers = new_pc.segment_plane(distanoe_threshold=0.01, ransac_n=3, num_iterations=10 0) except: inliers = [i for i in range(len(idx_list))] if len(inliers)==0: inliers = [i for i in range(len(idx_list))] while len (inliers)<5: inliers+=inliers patch_indexes[enu] [0:1en (inliers) ]=np. array(idx_list) [inliers] inliers.reverse () patch_indexes[enu] [len (inliers) :]=np. array (idx_list) [inliers[l:10-len(inliers) ]] np. save(dataset+"patches / "+pc_name+"_"+str(n_down)+".npy", patch_indexes. astype(np.uint32)) ST = downsample(ST)

[0250] This application introduces the concept of micro patch for discreet 3D surface unfolding. A micropatch convolution may be used for sparse point cloud coding. FIGs. 7B to 7G show examples using indexes to produce the patch. A patch approximation may be used to analyze point clouds. Patterns and contexts may be learned faster because a convolution learns in the same type of support, which is a 2D support.

[0251] Micropatch convolution may be used with any type of architecture for point cloud attribute compression, such as Variational Auto-Encoder (VAE), Normalizing Flow (NF), or wavelet, among other types. Micropatch convolution may be used for point cloud attribute compression.

[0252] The architecture is not as heavy compared to a full 3D architecture. Convolution kernels are reduced in size from 3x3x3 to 3x3. Compared to sparse convolutions, the computation used to produce the micropatches may cause a gain in complexity, because the patch may be precalculated. However, a preprocessing method with TLS is introduced to accelerate principal component plane detection and context projection.

[0253] This application may apply to the artificial intelligence point cloud compression (AI-PCC) coding standard extension based on deep learning.

[0254] FIG. 18 is a flowchart illustrating an example encoder process according to some embodiments. For some embodiments, an example process 1800 may include obtaining 1802 information corresponding to a point cloud. For some embodiments, the example process 1800 may further include selecting 1804 an examined point from a plurality of point cloud points corresponding to the information. For some embodiments, the example process 1800 may further include determining 1806 a set of indices corresponding to a patch set for the examined point, wherein the patch set is a set of data points selected from the plurality of point cloud points, and wherein the patch set corresponds to the examined point. For some embodiments, the example process 1800 may further include performing 1808 a patch convolution using the determined set of indices and a subset of the plurality of point cloud points. For some embodiments, the example process 1800 may further include generating 1810 a latent space using an output of the patch -43- convolution. For some embodiments, the example process 1800 may further include encoding 1812 latent space data as an encoder bitstream.

[0255] FIG. 19 is a flowchart illustrating an example decoder process according to some embodiments. For some embodiments, an example process 1900 may include obtaining 1902 first information corresponding to a point cloud geometry. For some embodiments, the example process 1900 may further include obtaining 1904 second information corresponding to a point cloud latent space. For some embodiments, the example process 1900 may further include selecting 1906 an examined point from a plurality of point cloud points corresponding to the first information. For some embodiments, the example process 1900 may further include determining 1908 a set of indices corresponding to a patch set for the examined point, wherein the patch set is a set of data points selected from the plurality of point cloud points, and wherein the patch set corresponds to the examined point. For some embodiments, the example process 1900 may further include performing 1910 a patch convolution using the determined set of indices and the second information corresponding to the point cloud latent space. For some embodiments, the example process 1900 may further include generating 1912 a reconstructed point cloud using an output of the patch convolution.

[0256] While the methods and systems in accordance with some embodiments are generally discussed in context of extended reality (XR), some embodiments may be applied to any XR contexts such as, e.g., virtual reality (VR) / mixed reality (MR) / augmented reality (AR) contexts. Also, although the term "head mounted display (HMD)” is used herein in accordance with some embodiments, some embodiments may be applied to a wearable device (which may or may not be attached to the head) capable of, e.g., XR, VR, AR, and / or MR for some embodiments.

[0257] A first example method in accordance with some embodiments may include: obtaining information corresponding to a point cloud; selecting an examined point from a plurality of point cloud points corresponding to the information; determining a set of indices corresponding to a patch set for the examined point, wherein the patch set is a set of data points selected from the plurality of point cloud points, and wherein the patch set corresponds to the examined point; performing a patch convolution using the determined set of indices and a subset of the plurality of point cloud points; generating a latent space using an output of the patch convolution; and encoding latent space data as an encoder bitstream.

[0258] For some embodiments of the first example method, the patch convolution is a micropatch convolution.

[0259] For some embodiments of the first example method, generating the latent space includes performing a first rectifier linear process (ReLU) on the output of the patch convolution. -44-

[0260] For some embodiments of the first example method, generating the latent space further includes: performing a second patch convolution on an output of the first rectifier linear process; performing a second rectifier linear process on an output of the second patch convolution; and performing a downsampling on an output of the second rectifier linear process.

[0261] Some embodiments of the first example method may further include: determining a second set of indices corresponding to the patch set for the examined point, wherein the second set of indices includes a downsampled set of indices generated based on the first set of indices, wherein generating the latent space includes performing two encoding sub-processes, and wherein the first encoding sub-process corresponds to the first set of indices and the second encoding sub-process corresponds to the second set of indices.

[0262] For some embodiments of the first example method, determining the second set of indices is performed prior to performing the two encoding sub-processes.

[0263] For some embodiments of the first example method, determining the second set of indices is performed after performing at least one of the two encoding sub-processes.

[0264] For some embodiments of the first example method, encoding the latent space data as the encoder bitstream includes using a variational auto-encoder (VAE).

[0265] For some embodiments of the first example method, encoding the latent space data as the encoder bitstream includes using a normalizing flow (NF) process.

[0266] For some embodiments of the first example method, encoding the latent space data as the encoder bitstream includes using a wavelet process.

[0267] For some embodiments of the first example method, encoding the latent space data as the encoder bitstream includes using a micropatch convolution.

[0268] For some embodiments of the first example method, encoding the latent space data is performed as part of an artificial intelligence point cloud compression (AI-PCC) algorithm.

[0269] For some embodiments of the first example method, performing the patch convolution includes performing a two-dimensional (2D) convolution.

[0270] Some embodiments of the first example method may further include determining the patch set for the examined point.

[0271] For some embodiments of the first example method, determining the patch set for the examined point includes: determining a neighborhood of points surrounding the examined point, and determining the patch set to be one or more points selected from the neighborhood of points.

[0272] For some embodiments of the first example method, determining the patch set further includes adding one or more points to the neighborhood of points.

[0273] For some embodiments of the first example method, determining the set of indices corresponding to the patch set for the examined point includes: populating a first subset of the set of indices corresponding to the patch set using occupied points selected from the plurality of point cloud points; and populating a second subset of the set of indices corresponding to the patch set using a filling algorithm.

[0274] For some embodiments of the first example method, the filling algorithm is a mirroring operation performed on the first subset of the set of indices corresponding to the patch set.

[0275] For some embodiments of the first example method, the filling algorithm is a zero padding operation.

[0276] For some embodiments of the first example method, the filling algorithm is a nearest neighbor axis order filling operation performed on the first subset of the set of indices corresponding to the patch set.

[0277] For some embodiments of the first example method, the filling algorithm is a nearest neighbor averaging operation performed on nearest neighbors selected from the first subset of the set of indices corresponding to the patch set.

[0278] For some embodiments of the first example method, the filling algorithm is a global averaging operation performed on the first subset of the set of indices corresponding to the patch set.

[0279] For some embodiments of the first example method, the filling algorithm is an interpolation operation performed on nearest neighbors selected from the first subset of the set of indices corresponding to the patch set.

[0280] A first example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0281] A second example method in accordance with some embodiments may include: obtaining information corresponding to a point cloud; selecting an examined point from a plurality of point cloud points corresponding to the information; determining a set of indices corresponding to a patch set for the examined point, wherein the patch set is a set of data points selected from the plurality of point cloud points, and wherein the patch set corresponds to the examined point; generating a latent space using the determined set of indices and a subset of the plurality of point cloud points, wherein generating the latent space includes performing a patch convolution using the determined set of indices and the subset of the plurality of point cloud points; and encoding latent space data as an encoder bitstream.

[0282] A second example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0283] A third example method in accordance with some embodiments may include: obtaining first information corresponding to a point cloud geometry; obtaining second information corresponding to a point cloud latent space; selecting an examined point from a plurality of point cloud points corresponding to the first information; determining a set of indices corresponding to a patch set for the examined point, wherein the patch set is a set of data points selected from the plurality of point cloud points, and wherein the patch set corresponds to the examined point; performing a patch convolution using the determined set of indices and the second information corresponding to the point cloud latent space; and generating a reconstructed point cloud using an output of the patch convolution.

[0284] Some embodiments of the third example method may further include decoding a decoder bitstream to generate the information corresponding to a point cloud geometry and the information corresponding to the point cloud latent space.

[0285] For some embodiments of the third example method, the patch convolution is a micropatch convolution.

[0286] For some embodiments of the third example method, generating the reconstructed point cloud includes performing a rectifier linear process on the output of the patch convolution.

[0287] For some embodiments of the third example method, generating the reconstructed point cloud further includes: performing a second patch convolution on an output of the first rectifier linear process; performing a second rectifier linear process on an output of the second patch convolution; and performing an upsampling on an output of the second rectifier linear process.

[0288] Some embodiments of the third example method may further include: determining a second set of indices corresponding to the patch set for the examined point, wherein the second set of indices includes an upsampled set of indices generated based on the first set of indices, wherein generating the reconstructed point cloud includes performing two decoding sub-processes, and wherein the first decoding sub-process corresponds to the first set of indices and the second decoding sub-process corresponds to the second set of indices.

[0289] For some embodiments of the third example method, determining the second set of indices is performed prior to performing the two decoding sub-processes.

[0290] For some embodiments of the third example method, determining the second set of indices is performed after performing at least one of the two decoding sub-processes.

[0291] For some embodiments of the third example method, obtaining information corresponding to the point cloud latent space includes using a variational auto-decoder (VAD).

[0292] For some embodiments of the third example method, obtaining information corresponding to the point cloud latent space includes using a normalizing flow (NF) process.

[0293] For some embodiments of the third example method, obtaining information corresponding to the point cloud latent space includes using a wavelet process.

[0294] For some embodiments of the third example method, obtaining information corresponding to the point cloud latent space includes using a micropatch convolution.

[0295] For some embodiments of the third example method, obtaining information corresponding to the point cloud latent space is performed as part of an artificial intelligence point cloud compression (AI-PCC) algorithm.

[0296] For some embodiments of the third example method, performing the patch convolution includes performing a two-dimensional (2D) convolution.

[0297] Some embodiments of the third example method may further include determining the patch set for the examined point.

[0298] For some embodiments of the third example method, determining the patch set for the examined point includes: determining a neighborhood of points surrounding the examined point, and determining the patch set to be one or more points selected from the neighborhood of points.

[0299] For some embodiments of the third example method, determining the patch set further includes adding one or more points to the neighborhood of points.

[0300] A third example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0301] A fourth example method in accordance with some embodiments may include: obtaining information corresponding to a point cloud geometry; obtaining information corresponding to a point cloud latent space; selecting an examined point from a plurality of point cloud points corresponding to the -48- information; determining a set of indices corresponding to a patch set for the examined point, wherein the patch set is a set of data points selected from the plurality of point cloud points, and wherein the patch set corresponds to the examined point; and generating a reconstructed point cloud using the determined set of indices and the point cloud latent space, wherein generating the reconstructed point cloud includes performing a patch convolution using the determined set of indices and the second information corresponding to the point cloud latent space.

[0302] A fourth example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0303] A fifth example method in accordance with some embodiments may include: downsampling a point cloud by reducing an N x N x N voxel to an N x N projection; copying occupied indices in the N x N x N point cloud to corresponding locations in the N x N projection; and populating unoccupied indices in the N x N projection using an algorithm.

[0304] A fifth example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0305] A sixth example method in accordance with some embodiments may include: downsampling a point cloud by iteratively, for each point of a plurality of points of the point cloud: reducing an N x N x N voxel centered on the point to an N x N projection centered on the point, wherein the N x N projection includes N x N index locations, by: copying neighboring points occupying indices in a corresponding N x N slice centered on the point and within the N x N x N voxel to corresponding index locations in the N x N projection; and populating unoccupied indices, if any, from the corresponding N x N slice into the N x N projection at corresponding index locations using an algorithm.

[0306] For some embodiments of the sixth example method, reducing the N x N x N voxel to the N x N projection further includes disregarding additional neighboring points outside of the N x N slice within the N x N x N voxel such that the N x N projection does not include such neighboring points.

[0307] A sixth example apparatus in accordance with some embodiments may include a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0308] A seventh example apparatus in accordance with some embodiments may include at least one processor configured to perform any one of the methods listed above.

[0309] An eighth example apparatus in accordance with some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any one of the methods listed above.

[0310] A ninth example apparatus in accordance with some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any one of the methods listed above.

[0311] An example signal in accordance with some embodiments may include an encoder bitstream generated according to any one of the methods listed above.

[0312] This disclosure describes a variety of aspects, including tools, features, embodiments, models, approaches, etc. Many of these aspects are described with specificity and, at least to show the individual characteristics, are often described in a manner that may sound limiting. However, this is for purposes of clarity in description, and does not limit the disclosure or scope of those aspects. Indeed, all of the different aspects can be combined and interchanged to provide further aspects. Moreover, the aspects can be combined and interchanged with aspects described in earlier filings as well.

[0313] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are illustrated specifically, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of the implementations. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects can be implemented as a method, an apparatus, a computer readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods described, and / or a computer readable storage medium having stored thereon a bitstream generated according to any of the methods described.

[0314] In the present disclosure, the terms "reconstructed” and "decoded” may be used interchangeably, the terms "pixel” and "sample” may be used interchangeably, the terms "image,” "picture” and "frame” may be used interchangeably. Usually, but not necessarily, the term "reconstructed” is used at the encoder side while "decoded” is used at the decoder side.

[0315] The terms HDR (high dynamic range) and SDR (standard dynamic range) often convey specific values of dynamic range to those of ordinary skill in the art. However, additional embodiments are also intended in which a reference to HDR is understood to mean "higher dynamic range” and a reference to SDR is understood to mean "lower dynamic range.” Such additional embodiments are not constrained by any specific values of dynamic range that might often be associated with the terms "high dynamic range” and "standard dynamic range.”

[0316] Various methods are described herein, and each of the methods includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as "first”, "second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., such as, for example, a "first decoding” and a "second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0317] Various numeric values may be used in the present disclosure, for example. The specific values are for example purposes and the aspects described are not limited to these specific values.

[0318] Embodiments described herein may be carried out by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The processor can be of any type appropriate to the technical environment and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as nonlimiting examples.

[0319] Various implementations involve decoding. "Decoding”, as used in this disclosure, can encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. In various embodiments, such processes also, or alternatively, include processes performed by a decoder of various implementations described in this disclosure, for example, extracting a picture from a tiled (packed) picture, determining an upsampling filter to use and then upsampling a picture, and flipping a picture back to its intended orientation.

[0320] As further examples, in one embodiment "decoding” refers only to entropy decoding, in another embodiment "decoding” refers only to differential decoding, and in another embodiment “decoding” refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process” is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the specific descriptions.

[0321] Various implementations involve encoding. In an analogous way to the above discussion about "decoding”, "encoding” as used in this disclosure can encompass all or part of the processes performed, for example, on an input video sequence in order to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, for example, partitioning, differential encoding, transformation, quantization, and entropy encoding. In various embodiments, such processes also, or alternatively, include processes performed by an encoder of various implementations described in this disclosure.

[0322] As further examples, in one embodiment "encoding” refers only to entropy encoding, in another embodiment "encoding” refers only to differential encoding, and in another embodiment “encoding” refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process” is intended to refer specifically to a subset of operations or generally to the broader encoding process will be clear based on the context of the specific descriptions.

[0323] When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of a corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of a corresponding method / process.

[0324] Various embodiments refer to rate distortion optimization. In particular, during the encoding process, the balance or trade-off between the rate and distortion is usually considered, often given the constraints of computational complexity. The rate distortion optimization is usually formulated as minimizing a rate distortion function, which is a weighted sum of the rate and of the distortion. There are different approaches to solve the rate distortion optimization problem. For example, the approaches may be based on an extensive testing of all encoding options, including all considered modes or coding parameters values, with a complete evaluation of their coding cost and related distortion of the reconstructed signal after coding and decoding. Faster approaches may also be used, to save encoding complexity, in particular with computation of an approximated distortion based on the prediction or the prediction residual signal, not the reconstructed one. A mix of these two approaches can also be used, such as by using an approximated distortion for only some of the possible encoding options, and a complete distortion for other encoding options. Other approaches only evaluate a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a complete evaluation of both the coding cost and related distortion.

[0325] The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of -52- features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants ("PDAs”), and other devices that facilitate communication of information between end-users.

[0326] Reference to "one embodiment” or "an embodiment” or "one implementation” or "an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase "in one embodiment” or "in an embodiment” or "in one implementation” or "in an implementation”, as well any other variations, appearing in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0327] Additionally, this disclosure may refer to "determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0328] Further, this disclosure may refer to "accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0329] Additionally, this disclosure may refer to "receiving” various pieces of information. Receiving is, as with "accessing”, intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, "receiving” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0330] It is to be appreciated that the use of any of the following “ / ”, "and / or”, and "at least one of, for example, in the cases of “A / B”, "A and / or B” and "at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of "A, B, and / or C” and "at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the -53- first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended for as many items as are listed.

[0331] Also, as used herein, the word "signal” refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments the encoder signals a particular one of a plurality of parameters for region-based filter parameter selection for de-artifact filtering. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various embodiments. It is to be appreciated that signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word "signal”, the word "signal” can also be used herein as a noun.

[0332] Implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0333] We describe a number of embodiments. Features of these embodiments can be provided alone or in any combination, across various claim categories and types. Further, embodiments can include one or more of the following features, devices, or aspects, alone or in any combination, across various claim categories and types: • Adapting residues at an encoder according to any of the embodiments discussed. • A bitstream or signal that includes one or more of the described syntax elements, or variations thereof. • A bitstream or signal that includes syntax conveying information generated according to any of the embodiments described. • Inserting in the signaling syntax elements that enable the decoder to adapt residues in a manner corresponding to that used by an encoder. • Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements, or variations thereof. • Creating and / or transmitting and / or receiving and / or decoding according to any of the embodiments described. • A method, process, apparatus, medium storing instructions, medium storing data, or signal according to any of the embodiments described. • A TV, set-top box, cell phone, tablet, or other electronic device that performs adaptation of filter parameters according to any of the embodiments described. • A TV, set-top box, cell phone, tablet, or other electronic device that performs adaptation of filter parameters according to any of the embodiments described, and that displays (e.g. using a monitor, screen, or other type of display) a resulting image. • A TV, set-top box, cell phone, tablet, or other electronic device that selects (e.g. using a tuner) a channel to receive a signal including an encoded image, and performs adaptation of filter parameters according to any of the embodiments described. • A TV, set-top box, cell phone, tablet, or other electronic device that receives (e.g. using an antenna) a signal over the air that includes an encoded image, and performs adaptation of filter parameters according to any of the embodiments described.

[0334] Note that various hardware elements of one or more of the described embodiments are referred to as "modules” that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions could take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.

[0335] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or 5 processor. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any 10 host computer.

Claims

1. A method comprising:obtaining information corresponding to a point cloud;selecting an examined point from a plurality of point cloud points corresponding to the information;determining a set of indices corresponding to a patch set for the examined point,wherein the patch set is a set of data points selected from the plurality of point cloud points, andwherein the patch set corresponds to the examined point;performing a patch convolution using the determined set of indices and a subset of the plurality of point cloud points;generating a latent space using an output of the patch convolution; andencoding latent space data as an encoder bitstream.

2. The method of claim 1, wherein the patch convolution is a micropatch convolution.

3. The method of any one of claims 1-2, wherein generating the latent space comprises performing a first rectifier linear process (ReLU) on the output of the patch convolution.

4. The method of claim 3, wherein generating the latent space further comprises:performing a second patch convolution on an output of the first rectifier linear process;performing a second rectifier linear process on an output of the second patch convolution; and performing a downsampling on an output of the second rectifier linear process.

5. The method of any one of claims 1-4, further comprising:determining a second set of indices corresponding to the patch set for the examined point,wherein the second set of indices comprises a downsampled set of indices generated based on the first set of indices,wherein generating the latent space comprises performing two encoding sub-processes, andwherein the first encoding sub-process corresponds to the first set of indices and the second encoding sub-process corresponds to the second set of indices.

6. The method of claim 5, wherein determining the second set of indices is performed prior to performing the two encoding sub-processes.

7. The method of claim 5, wherein determining the second set of indices is performed after performing at least one of the two encoding sub-processes.

8. The method of any one of claims 1-7, wherein encoding the latent space data as the encoder bitstream comprises using a variational auto-encoder (VAE).

9. The method of any one of claims 1-7, wherein encoding the latent space data as the encoder bitstream comprises using a normalizing flow (NF) process.

10. The method of any one of claims 1-7, wherein encoding the latent space data as the encoder bitstream comprises using a wavelet process.

11. The method of any one of claims 1-7, wherein encoding the latent space data as the encoder bitstream comprises using a micropatch convolution.

12. The method of any one of claims 1-11, wherein encoding the latent space data is performed as part of an artificial intelligence point cloud compression (AI-PCC) algorithm.

13. The method of any one of claims 1-12, wherein performing the patch convolution comprises performing a two-dimensional (2D) convolution.

14. The method of any one of claims 1-13, further comprising determining the patch set for the examined point.

15. The method of claim 14, wherein determining the patch set for the examined point comprises: determining a neighborhood of points surrounding the examined point, and determining the patch set to be one or more points selected from the neighborhood of points.

16. The method of any one of claims 14-15, wherein determining the patch set further comprises adding one or more points to the neighborhood of points.

17. The method of any one of claims 1-16, wherein determining the set of indices corresponding to the patch set for the examined point comprises:populating a first subset of the set of indices corresponding to the patch set using occupied points selected from the plurality of point cloud points; andpopulating a second subset of the set of indices corresponding to the patch set using a filling algorithm.

18. The method of claim 17, wherein the filling algorithm is a mirroring operation performed on the first subset of the set of indices corresponding to the patch set.

19. The method of claim 17, wherein the filling algorithm is a zero padding operation.

20. The method of claim 17, wherein the filling algorithm is a nearest neighbor axis order filling operation performed on the first subset of the set of indices corresponding to the patch set.

21. The method of claim 17, wherein the filling algorithm is a nearest neighbor averaging operation performed on nearest neighbors selected from the first subset of the set of indices corresponding to the patch set.

22. The method of claim 17, wherein the filling algorithm is a global averaging operation performed on the first subset of the set of indices corresponding to the patch set.

23. The method of claim 17, wherein the filling algorithm is an interpolation operation performed on nearest neighbors selected from the first subset of the set of indices corresponding to the patch set.

24. An apparatus comprising:a processor; anda non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform the method of any one of claims 1 through 23.

25. A method comprising:obtaining information corresponding to a point cloud;selecting an examined point from a plurality of point cloud points corresponding to the information; determining a set of indices corresponding to a patch set for the examined point,wherein the patch set is a set of data points selected from the plurality of point cloud points, andwherein the patch set corresponds to the examined point;generating a latent space using the determined set of indices and a subset of the plurality of point cloud points,wherein generating the latent space comprises performing a patch convolution using the determined set of indices and the subset of the plurality of point cloud points; andencoding latent space data as an encoder bitstream.

26. An apparatus comprising:a processor; anda non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform the method of claim 25.

27. A method comprising:obtaining first information corresponding to a point cloud geometry;obtaining second information corresponding to a point cloud latent space;selecting an examined point from a plurality of point cloud points corresponding to the first information;determining a set of indices corresponding to a patch set for the examined point,wherein the patch set is a set of data points selected from the plurality of point cloud points, andwherein the patch set corresponds to the examined point;performing a patch convolution using the determined set of indices and the second information corresponding to the point cloud latent space; andgenerating a reconstructed point cloud using an output of the patch convolution.

28. The method of claim 27, further comprising decoding a decoder bitstream to generate the information corresponding to a point cloud geometry and the information corresponding to the point cloud latent space.

29. The method of any one of claims 27-28, wherein the patch convolution is a micropatch convolution.

30. The method of any one of claims 27-29, wherein generating the reconstructed point cloud comprises performing a rectifier linear process on the output of the patch convolution.

31. The method of claim 30, wherein generating the reconstructed point cloud further comprises: performing a second patch convolution on an output of the first rectifier linear process; performing a second rectifier linear process on an output of the second patch convolution; and performing an upsampling on an output of the second rectifier linear process.

32. The method of any one of claims 27-31, further comprising: determining a second set of indices corresponding to the patch set for the examined point, wherein the second set of indices comprises an upsampled set of indices generated based on the first set of indices,wherein generating the reconstructed point cloud comprises performing two decoding subprocesses, andwherein the first decoding sub-process corresponds to the first set of indices and the second decoding sub-process corresponds to the second set of indices.

33. The method of claim 32, wherein determining the second set of indices is performed prior to performing the two decoding sub-processes.

34. The method of claim 32, wherein determining the second set of indices is performed after performing at least one of the two decoding sub-processes.

35. The method of any one of claims 27-34, wherein obtaining information corresponding to the point cloud latent space comprises using a variational auto-decoder (VAD).

36. The method of any one of claims 27-34, wherein obtaining information corresponding to the point cloud latent space comprises using a normalizing flow (NF) process.

37. The method of any one of claims 27-34, wherein obtaining information corresponding to the point cloud latent space comprises using a wavelet process.

38. The method of any one of claims 27-34, wherein obtaining information corresponding to the point cloud latent space comprises using a micropatch convolution.

39. The method of any one of claims 27-38, wherein obtaining information corresponding to the point cloud latent space is performed as part of an artificial intelligence point cloud compression (AI-PCC) algorithm.

40. The method of any one of claims 27-39, wherein performing the patch convolution comprises performing a two-dimensional (2D) convolution.

41. The method of any one of claims 27-40, further comprising determining the patch set for the examined point.

42. The method of claim 41, wherein determining the patch set for the examined point comprises: determining a neighborhood of points surrounding the examined point, and determining the patch set to be one or more points selected from the neighborhood of points.

43. The method of any one of claims 41-42, wherein determining the patch set further comprises adding one or more points to the neighborhood of points.

44. An apparatus comprising:a processor; anda non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform the method of any one of claims 27 through 43.

45. A method comprising:obtaining information corresponding to a point cloud geometry;obtaining information corresponding to a point cloud latent space;selecting an examined point from a plurality of point cloud points corresponding to the information;determining a set of indices corresponding to a patch set for the examined point,wherein the patch set is a set of data points selected from the plurality of point cloud points, andwherein the patch set corresponds to the examined point; andgenerating a reconstructed point cloud using the determined set of indices and the point cloud latent space,wherein generating the reconstructed point cloud comprises performing a patch convolution using the determined set of indices and the second information corresponding to the point cloud latent space.

46. An apparatus comprising:a processor; anda non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform the method of claim 45.

47. A method comprising:downsampling a point cloud by reducing an N x N x N voxel to an N x N projection;copying occupied indices in the N x N x N point cloud to corresponding locations in the N x N projection; andpopulating unoccupied indices in the N x N projection using an algorithm.

48. An apparatus comprising:a processor; anda non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform the method of claim 47.

49. A method comprising:downsampling a point cloud by iteratively, for each point of a plurality of points of the point cloud: reducing an N x N x N voxel centered on the point to an N x N projection centered on the point, wherein the N x N projection comprises N x N index locations, by:copying neighboring points occupying indices in a corresponding N x N slice centered on the point and within the N x N x N voxel to corresponding index locations in the N x N projection; andpopulating unoccupied indices, if any, from the corresponding N x N slice into the N x N projection at corresponding index locations using an algorithm.

50. The method of claim 49, wherein reducing the N x N x N voxel to the N x N projection further comprises: disregarding additional neighboring points outside of the N x N slice within the N x N x N voxel such that the N x N projection does not include such neighboring points.

51. An apparatus comprising:a processor; anda non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform the method of any one of claims 49-50.

52. An apparatus comprising at least one processor configured to perform the method of any one of claims 1-23, 25, 27-43,45, 47, 49, and 50.

53. An apparatus comprising a computer-readable medium storing instructions for causing one or more processors to perform the method of any one of claims 1-23, 25, 27-43,45, 47,49, and 50.

54. An apparatus comprising at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform the method of any one of claims 1-23, 25, 27-43, 45, 47, 49, and 50.

55. A signal including an encoder bitstream generated according to any one of claims 1-23 and 25.

Images