Position coding for point cloud compression

Abstract

A point cloud encoder is configured to receive a frame of point cloud data and encode coordinates of a geometry of the frame of point cloud data using a deep learning network. The deep learning network includes one or more layers configured to generate first features of the coordinates of the geometry, and at least one positional encoding layer configured to generate second features of the coordinates of the geometry and combine the first features and the second features to generate higher-dimensional features. The point cloud encoder can output an output tensor including the encoded coordinates and the higher-dimensional features.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 606,470, filed March 15, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to point cloud compression, including point cloud encoding and decoding. Background Technology

[0003] A point cloud is a collection of points in three-dimensional space. These points can correspond to points on objects within that space. Therefore, point clouds can be used to represent the physical content of three-dimensional space. Point clouds have practical applications in a wide variety of situations. For example, point clouds can be used in the context of autonomous vehicles to represent the position of objects on a road. In another example, point clouds can be used in the context of representing the physical content of an environment to locate virtual objects in augmented reality (AR) or mixed reality (MR) applications. Point cloud compression is the process of encoding and decoding point clouds. Encoding point clouds reduces the amount of data required to store and transmit them. Summary of the Invention

[0004] Generally, this disclosure describes techniques for point cloud compression, including techniques that utilize artificial intelligence (AI) and neural network techniques when performing compression. Specifically, this disclosure describes techniques and apparatuses for using one or more positional encoding layers in an encoder and decoder network for AI-based point cloud compression. The positional encoding layers described herein can be used for both geometric compression (e.g., the coordinates of the point cloud) and attribute compression (e.g., the color, normal vector, reflectivity, or other attributes of the point).

[0005] In the context of geometric encoding, an encoder network may include several downsampling (also known as reduction) stages that progressively reduce the number of points. The coordinates of these downsampled points, along with associated feature attributes, are output by an AI-based encoder. This disclosure describes a technique for adding one or more positional encoding layers to an encoder network. The positional encoding layers create additional higher-dimensional features from the currently processed coordinates. These features are higher-dimensional relative to other feature tensors generated by the encoder or decoder network. These additional higher-dimensional feature attributes are combined (e.g., concatenated) to feature tensors generated by other layers of the encoder network. By increasing the dimensionality of the feature tensors generated by an encoder network with positional encoding layers, the techniques of this disclosure improve the efficiency of compression.

[0006] In one example, this disclosure describes an apparatus configured to encode a point cloud, the apparatus comprising: one or more memories configured to store point cloud data; and processing circuitry communicating with the one or more memories, the processing circuitry being configured to: receive a point cloud data frame; encode coordinates of a geometry of the point cloud data frame using a deep learning network, wherein the deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometry, and wherein the deep learning network further includes at least one position encoding layer configured to generate a second feature of the coordinates of the geometry and combine the first feature and the second feature to generate a higher-dimensional feature; and output an output tensor comprising the encoded coordinates and the higher-dimensional feature.

[0007] In one example, this disclosure describes a method for encoding a point cloud, the method comprising: receiving a point cloud data frame; encoding coordinates of a geometric structure of the point cloud data frame using a deep learning network, wherein the deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometric structure, and wherein the deep learning network further includes at least one location encoding layer configured to generate a second feature of the coordinates of the geometric structure and to combine the first feature with the second feature to generate a higher-dimensional feature; and outputting an output tensor comprising the encoded coordinates and the higher-dimensional feature.

[0008] In another example, this disclosure describes an apparatus configured to decode coded point cloud data, the apparatus comprising: one or more memories configured to store the coded point cloud data; and processing circuitry communicating with the one or more memories, the processing circuitry being configured to: receive a frame of the coded point cloud data, wherein the coded point cloud data includes an input tensor including coded coordinates and corresponding features; decode coordinates of a geometric structure of the frame of the coded point cloud data using a deep learning network and the input tensor to generate decoded coordinates, wherein the deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometric structure, and wherein the deep learning network further includes at least one position encoding layer configured to generate a second feature of the coordinates of the geometric structure and combine the first feature and the second feature to generate a higher-dimensional feature, and wherein the deep learning network further includes one or more classifier layers configured to classify the higher-dimensional feature to generate the decoded coordinates; and output the decoded coordinates in the decoded point cloud.

[0009] In another example, this disclosure describes a method for decoding a point cloud, the method comprising: receiving a frame of encoded point cloud data, wherein the encoded point cloud data includes an input tensor including encoded coordinates and corresponding features; decoding coordinates of a geometric structure of the frame of the encoded point cloud data using a deep learning network and the input tensor to generate decoded coordinates, wherein the deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometry, and wherein the deep learning network further includes at least one position encoding layer configured to generate a second feature of the coordinates of the geometry and combine the first feature and the second feature to generate a higher-dimensional feature, and wherein the deep learning network further includes one or more classifier layers configured to classify the higher-dimensional feature to generate the decoded coordinates; and outputting the decoded coordinates in the decoded point cloud.

[0010] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0011] Figure 1 This is a block diagram illustrating an example encoding and decoding system that can perform the techniques of this disclosure.

[0012] Figure 2 This is a block diagram illustrating an example point cloud encoder that can perform the techniques of this disclosure.

[0013] Figure 3 This is a block diagram illustrating an example point cloud decoder that can perform the techniques of this disclosure.

[0014] Figure 4 This is a block diagram illustrating an example system model for AI-based point cloud compression that can implement the techniques of this disclosure.

[0015] Figure 5 This is an example of the technology according to this disclosure. Figure 4 A block diagram of an example encoder network.

[0016] Figure 6 This is an example of the technology according to this disclosure. Figure 4 A block diagram of an example decoder network.

[0017] Figure 7 This is a block diagram illustrating an example encoder network using position encoding according to the technology of this disclosure.

[0018] Figure 8 This is a block diagram illustrating an example decoder network using position encoding according to the technology of this disclosure.

[0019] Figure 9This is a block diagram illustrating an example system model for AI-based point cloud compression using position encoding, which exemplifies the techniques of this disclosure.

[0020] Figure 10 This is a block diagram illustrating an example location coding layer according to the technology of this disclosure.

[0021] Figure 11A This is a graph showing a test of the compression efficiency of a feature bitstream encoded according to the technology disclosed herein.

[0022] Figure 11B This is a graph showing a test of the compression efficiency of the overall bitstream encoded according to the technology disclosed herein.

[0023] Figure 12 This is a flowchart illustrating an example encoding technique of this disclosure.

[0024] Figure 13 This is a flowchart illustrating an example decoding technique of this disclosure.

[0025] Figure 14 This is a conceptual diagram illustrating an example ranging system that can be used with one or more techniques disclosed herein.

[0026] Figure 15 This is a conceptual diagram illustrating an example of a vehicle-based scenario in which one or more technologies of this disclosure may be used.

[0027] Figure 16 This is a conceptual diagram illustrating an example extended reality system in which one or more techniques of this disclosure may be used.

[0028] Figure 17 This is a conceptual diagram illustrating an example mobile device system in which one or more technologies of this disclosure may be used. Detailed Implementation

[0029] Point clouds (PCs) are 3D data representations used for tasks such as virtual reality (VR) and mixed reality (MR), autonomous driving, and cultural heritage preservation. PCs are 3D coordinates in 3D space represented by structures called geometry. x, y, z A point cloud represents a set of points. Each point can also be associated with multiple attributes such as color, normal vector, and reflectivity. Depending on the target application and the method of acquiring the point cloud, a point cloud can be classified as a point cloud scene or a point cloud object.

[0030] Static point cloud compression (PC) is a single object, while dynamic PC is a time-varying PC, where each instance of a dynamic PC is a static PC. These PCs can have a large number of points, especially at high precision or large-scale capture (e.g., millions of points per frame, at up to 60 frames per second (FPS)). Therefore, efficient point cloud compression (PCC) may be particularly important for enabling practical use in VR, MR, and automotive applications.

[0031] The Moving Picture Experts Group (MPEG) has approved two PCC standards. One standard is called Geometry-Based Point Cloud Compression (G-PCC), and the other is called Video-Based Point Cloud Compression (V-PCC). G-PCC includes octree geometry decoding as a general geometry decoding tool and predictive geometry decoding (tree-based) tools for LiDAR-based point clouds. Some examples of G-PCC may include methods based on triangular meshes or trisoups to approximate the surface of a 3D model. V-PCC encodes dynamic point clouds by projecting 3D points onto a 2D plane and then using a video codec (e.g., High Efficiency Video Decoding (HEVC)) to encode each frame over time. MPEG is also working on an AI-based Point Cloud Compression (AI-PCC, also known as AI-3DGC or AI-GC) standard.

[0032] In some examples, AI-based point cloud compression techniques may include the use of end-to-end deep learning solutions. Because point clouds possess both geometric structure and attributes, there are existing proposals for point cloud geometry compression, point cloud attribute compression, and joint point cloud geometry and attribute compression. In a general example, point cloud compression of dense, dynamic point clouds can be performed using a deep learning network comprising an encoding unit and a decoding unit. The encoding unit can extract features from the point cloud geometry into a reduced point cloud geometry (e.g., cubic or 8x reduction) with corresponding feature embeddings. The reduced geometry and corresponding features can then be sent to the decoding unit. The decoding unit can then reconstruct the original point cloud geometry hierarchically from the reduced representation using progressive rescaling.

[0033] Positional encoding is a deep learning technique used in Natural Language Processing (NLP) and Large Language Modeling (LLM). It's a technique used to provide a model with positional information (such as the order of words in a sentence) when processing sequences of text. Positional encoding has also been used in vision tasks, where the pixel locations of an image are fed to a neural network to understand the spatial relationships between pixels. Positional encoding generally involves applying mathematical representations to locations to obtain higher-dimensional feature representations. These features can be incorporated into the input data as additional channels or features, allowing the model to recognize the location and orientation of objects or patterns in an image.

[0034] This disclosure describes a technique for adding one or more positional encoding layers to encoder and decoder networks for AI-based point cloud compression. The positional encoding layer creates additional higher-dimensional features from the currently processed coordinates. These additional higher-dimensional feature attributes are combined (e.g., concatenated) to a feature tensor generated by other layers of the encoder and decoder networks. By increasing the dimensionality of the feature tensor generated by the network with positional encoding layers, the techniques of this disclosure improve the efficiency of point cloud compression.

[0035] In one example, a point cloud encoder is configured to receive a point cloud data frame and encode the coordinates of a geometric structure of that point cloud data frame using a deep learning network. The deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometry, and at least one positional encoding layer configured to generate a second feature of the coordinates of the geometry and combine the first and second features to generate a higher-dimensional feature. The point cloud encoder can output an output tensor including the encoded coordinates and the higher-dimensional feature.

[0036] In the corresponding example, the point cloud decoder can receive frames of encoded point cloud data, wherein the encoded point cloud data includes an input tensor comprising encoded coordinates and corresponding features. The point cloud decoder can use a deep learning network and the input tensor to decode the coordinates of the geometry of the frame of encoded point cloud data to generate decoded coordinates. The deep learning network includes: one or more layers configured to generate first features of the geometry coordinates; at least one positional encoding layer configured to generate second features of the geometry coordinates and combine the first and second features to generate higher-dimensional features; and one or more classifier layers configured to classify the higher-dimensional features to generate decoded coordinates. The point cloud decoder can output decoded coordinates from the decoded point cloud.

[0037] In the following examples, the terms "location encoding" and "location embedding" are used interchangeably.

[0038] Figure 1 This is a block diagram illustrating an example encoding and decoding system 100 capable of implementing the techniques of this disclosure. The techniques of this disclosure generally relate to AI-based point cloud data decoding (encoding and / or decoding), i.e., supporting point cloud compression. Generally, point cloud data includes any data used for processing point clouds. This decoding can be efficient in compressing and / or decompressing point cloud data.

[0039] like Figure 1 As shown, system 100 includes a source device 102 and a destination device 116. The source device 102 provides encoded point cloud data for decoding by the destination device 116. Specifically, in Figure 1In this example, source device 102 provides point cloud data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 can include any of a wide range of devices, including desktop computers, laptops, tablets, set-top boxes, mobile phones (such as smartphones), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, land or sea vehicles, spacecraft, aircraft, robots, LiDAR devices, satellites, etc. In some cases, source device 102 and destination device 116 may be equipped for wireless communication.

[0040] exist Figure 1 In the example, source device 102 includes a data source 104, a memory 106, a point cloud encoder 200, and an output interface 108. Destination device 116 includes an input interface 122, a point cloud decoder 300, a memory 120, and a data consumer 118. According to this disclosure, the point cloud encoder 200 of source device 102 and the point cloud decoder 300 of destination device 116 can be configured to apply the techniques of this disclosure related to AI-based point cloud encoding and encoding using positional encoding techniques. Therefore, source device 102 represents an example of an encoding device, while destination device 116 represents an example of a decoding device. In other examples, source device 102 and destination device 116 may include other components or arrangements. For example, source device 102 may receive data (e.g., point cloud data) from an internal or external source. Similarly, destination device 116 may interface with an external data consumer, rather than including the data consumer in the same device.

[0041] like Figure 1 The system 100 shown is merely an example. In general, other digital encoding and / or decoding devices can perform the techniques of this disclosure related to AI-based point cloud encoding and coding. Source device 102 and destination device 116 are merely examples of such devices, where source device 102 generates decoded data for transmission to destination device 116. This disclosure refers to a “decoding” device as a device that performs the decoding (e.g., encoding and / or decoding) of data. Thus, point cloud encoder 200 and point cloud decoder 300 represent examples of decoding devices, specifically, encoder and decoder, respectively. In some examples, source device 102 and destination device 116 can operate in a substantially symmetrical manner, such that each of source device 102 and destination device 116 includes both encoding and decoding components. Therefore, system 100 can support one-way or two-way transmission between source device 102 and destination device 116, for example, for streaming, playback, broadcasting, telephone, navigation, and other applications.

[0042] Generally, data source 104 represents the source of data (i.e., raw, unencoded point cloud data) and can provide a series of sequential “frames” of data to point cloud encoder 200, which encodes the data in these frames. Data source 104 of source device 102 may include point cloud capture devices such as any of a variety of cameras or sensors (e.g., a 3D scanner or a light detection and ranging (LIDAR) device, one or more cameras), archives containing previously captured data, and / or data feed interfaces receiving data from data content providers. Alternatively or additionally, point cloud data may be computer-generated from a scanner, camera, sensor, or other data source. For example, data source 104 may generate computer graphics-based data as source data, or produce a combination of real-time data, archived data, and computer-generated data. In each case, point cloud encoder 200 encodes the captured data, pre-captured data, or computer-generated data. Point cloud encoder 200 may rearrange frames from the received order (sometimes referred to as “display order”) to a decoding order for decoding. Point cloud encoder 200 may generate one or more bitstreams including encoded data. Then, the source device 102 can output encoded data to a computer-readable medium 110 via the output interface 108 for reception and / or retrieval by, for example, the input interface 122 of the destination device 116.

[0043] The memory 106 of the source device 102 and the memory 120 of the destination device 116 may represent general-purpose memory. In some examples, memory 106 and memory 120 may store raw data, such as raw data from data source 104 and raw decoded data from point cloud decoder 300. Additionally or alternatively, memory 106 and memory 120 may store software instructions executable, for example, by point cloud encoder 200 and point cloud decoder 300. Although memory 106 and memory 120 are shown separately from point cloud encoder 200 and point cloud decoder 300 in this example, it should be understood that point cloud encoder 200 and point cloud decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memory 106 and memory 120 may store encoded data, such as output from point cloud encoder 200 and input to point cloud decoder 300. In some examples, portions of memory 106 and memory 120 may be allocated as one or more buffers, for example, for storing raw decoded data and / or encoded data. For example, memory 106 and memory 120 can store data representing point clouds.

[0044] Computer-readable medium 110 may represent any type of medium or device capable of transmitting encoded data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium enabling source device 102 to transmit encoded data directly to destination device 116 in real time, for example, via a radio frequency network or a computer-based network. Output interface 108 may modulate the transmitted signal including the encoded data, and input interface 122 may demodulate the received transmitted signal according to a communication standard, such as a wireless communication protocol. The communication medium may include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network such as a local area network, a wide area network, or a global network (such as the Internet). The communication medium may include a router, a switch, a base station, or any other equipment that may be useful for facilitating communication from source device 102 to destination device 116.

[0045] In some examples, source device 102 can output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded data.

[0046] In some examples, source device 102 may output encoded data to file server 114 or another intermediate storage device that may store the encoded data generated by source device 102. Destination device 116 may access the stored data from file server 114 via streaming or downloading. File server 114 may be any type of server device capable of storing encoded data and sending it to destination device 116. File server 114 may represent a web server (e.g., for a website), a file transfer protocol (FTP) server, a content delivery network device, or a network attached storage (NAS) device. Destination device 116 may access the encoded data from file server 114 via any standard data connection, including an internet connection. This standard data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both suitable for accessing encoded data stored on file server 114. File server 114 and input interface 122 may be configured to operate according to a streaming protocol, a downloading protocol, or a combination thereof.

[0047] Output interface 108 and input interface 122 may represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 may be configured to transmit data (such as encoded data) according to cellular communication standards (such as 4G, 4G-LTE (Long Term Evolution), Advanced LTE, 5G, etc.). In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 may be configured to comply with specifications such as IEEE 802.11, IEEE 802.15 (e.g., ZigBee). ™ ),Bluetooth ™ Data (such as encoded data) is transmitted using other wireless standards, such as standard methods. In some examples, source device 102 and / or destination device 116 may include corresponding system-on-chip (SoC) devices. For example, source device 102 may include an SoC device for performing functions belonging to point cloud encoder 200 and / or output interface 108, and destination device 116 may include an SoC device for performing functions belonging to point cloud decoder 300 and / or input interface 122.

[0048] The technology disclosed herein can be applied to support encoding and decoding of any of a variety of applications, such as communication between autonomous vehicles, communication between scanners, cameras, sensors and processing devices such as local or remote servers, geographic mapping or other applications.

[0049] The input interface 122 of the destination device 116 receives an encoded bitstream from a computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, etc.). The encoded bitstream may include signaling information defined by the point cloud encoder 200 and also used by the point cloud decoder 300, such as syntax elements having values ​​describing the characteristics and / or processing of the decoding unit (e.g., slice, picture, picture group, sequence, etc.). The data consumer 118 uses the decoded data. For example, the data consumer 118 may use the decoded data to determine the location of a physical object. In some examples, the data consumer 118 may include a display for presenting an image based on the point cloud.

[0050] The point cloud encoder 200 and point cloud decoder 300 can each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic components, software, hardware, firmware, or any combination thereof. When the technology is partially implemented in software, the device may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the technology of this disclosure. Each of the point cloud encoder 200 and point cloud decoder 300 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (encoder-decoder) in the respective device. Devices including the point cloud encoder 200 and / or point cloud decoder 300 may include one or more integrated circuits, microprocessors, and / or other types of devices.

[0051] This disclosure may generally relate to the decoding (e.g., encoding and decoding) of images, thus including the process of encoding or decoding data. Encoded bitstreams typically include a series of values ​​for syntax elements representing decoding decisions (e.g., decoding modes).

[0052] This disclosure may generally relate to "signaling" certain information (such as syntax elements). The term "signaling notification" generally refers to communication of the values ​​of syntax elements and / or other data used to decode encoded data. That is, the point cloud encoder 200 may signal the values ​​of syntax elements in the bitstream. Generally speaking, signaling notification means generating values ​​in the bitstream. As noted above, the source device 102 may transmit the bitstream to the destination device 116 substantially in real time or not in real time (such as when syntax elements are stored in storage device 112 for later retrieval by the destination device 116).

[0053] ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) has studied and continues to study the potential need for standardization of point cloud decoding techniques with compression capabilities that significantly surpass current methods. The group is working collaboratively on this exploration through a team called the 3D Graphics Team (3DG) to evaluate compression technology designs proposed by their experts in the field.

[0054] The Moving Picture Experts Group (MPEG) has approved two PCC standards. One standard is called Geometry-Based Point Cloud Compression (G-PCC), and the other is called Video-Based Point Cloud Compression (V-PCC). G-PCC includes octree geometry decoding as a general geometry decoding tool and predictive geometry decoding (tree-based) tools for LiDAR-based point clouds. Some examples of G-PCC may include methods based on triangular meshes or trisoups to approximate the surface of a 3D model. V-PCC encodes dynamic point clouds by projecting 3D points onto a 2D plane and then using a video codec (e.g., High Efficiency Video Decoding (HEVC)) to encode each frame over time. MPEG is also working on an AI-based Point Cloud Compression (AI-PCC, also known as AI-3DGC or AI-GC) standard.

[0055] In some examples, AI-based point cloud compression techniques may include the use of end-to-end deep learning solutions. Because point clouds possess both geometric structure and attributes, there are existing proposals for point cloud geometry compression, point cloud attribute compression, and joint point cloud geometry and attribute compression. In a general example, point cloud compression of dense, dynamic point clouds can be performed using a deep learning network comprising an encoding unit and a decoding unit. The encoding unit can extract features from the point cloud geometry into a reduced point cloud geometry (e.g., triple reduction) with corresponding feature embeddings. The reduced geometry and corresponding features can be sent to the decoding unit. The decoding unit can then use progressive rescaling to hierarchically reconstruct the original point cloud geometry from the reduced representation.

[0056] Positional encoding is a deep learning technique used in Natural Language Processing (NLP) and Large Language Modeling (LLM). It's a technique used to provide a model with positional information (such as the order of words in a sentence) when processing sequences of text. Positional encoding has also been used in vision tasks, where the pixel locations of an image are fed to a neural network to understand the spatial relationships between pixels. Positional encoding generally involves applying mathematical representations to locations to obtain higher-dimensional feature representations. These features can be incorporated into the input data as additional channels or features, allowing the model to recognize the location and orientation of objects or patterns in an image.

[0057] This disclosure describes a technique for adding one or more positional encoding layers to encoder and decoder networks for AI-based point cloud compression. The positional encoding layer creates additional higher-dimensional features from the currently processed coordinates. These features are higher-dimensional relative to other feature tensors generated by the encoder or decoder network. These additional higher-dimensional feature properties are combined (e.g., concatenated) to feature tensors generated by other layers of the encoder and decoder networks. By increasing the dimensionality of the feature tensors generated by the network with positional encoding layers, the techniques of this disclosure improve the efficiency of point cloud compression.

[0058] In one example, a point cloud encoder 200, as explained in more detail below, may be configured to receive a point cloud data frame and encode the coordinates of a geometric structure of that point cloud data frame using a deep learning network. The deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometry, and at least one positional encoding layer configured to generate a second feature of the coordinates of the geometry and combine the first and second features to generate higher-dimensional features. The point cloud encoder 200 may output an output tensor comprising the encoded coordinates and the higher-dimensional features.

[0059] In the corresponding example, the point cloud decoder 300 can receive frames of encoded point cloud data, wherein the encoded point cloud data includes an input tensor that includes encoded coordinates and corresponding features. The point cloud decoder can use a deep learning network and the input tensor to decode the coordinates of the geometry of the frame of encoded point cloud data to generate decoded coordinates. The deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometry, at least one position encoding layer configured to generate a second feature of the coordinates of the geometry and combine the first feature and the second feature to generate a higher-dimensional feature, and one or more classifier layers configured to classify the higher-dimensional feature to generate the decoded coordinates. The point cloud decoder 300 can output the decoded coordinates from the decoded point cloud.

[0060] In the following examples, the terms "location encoding" and "location embedding" are used interchangeably.

[0061] A point cloud contains a set of points in 3D space and can have attributes associated with those points. Attributes can be color information such as R, G, B or Y, Cb, Cr, or reflectivity information, or other attributes. Point clouds can be captured by various cameras or sensors, such as LiDAR sensors and 3D scanners, and can also be computer-generated. Point cloud data is used in a variety of applications, including but not limited to architecture (modeling), graphics (3D models for visualization and animation), and the automotive industry (LiDAR sensors for navigation aids).

[0062] The 3D space occupied by point cloud data can be enclosed by virtual bounding boxes. The position of a point within the bounding box can be represented with a certain precision; therefore, the position of one or more points can be quantized based on this precision. At the smallest level, the bounding box is divided into voxels, which are the smallest spatial units represented by a unit cube. A voxel in the bounding box can be associated with zero, one, or more points. The bounding box can be divided into multiple cubic / cuboid regions, which can be called tiles. Each tile can be decoded into one or more slices. Dividing the bounding box into slices and tiles can be based on the number of points in each partition, or on other considerations (e.g., a specific region can be decoded into a tile). Slice regions can be further subdivided using splitting decisions similar to those in video codecs.

[0063] Figure 2 Provides a high-level overview of the Point Cloud Encoder 200. Figure 3 This provides a high-level overview of the point cloud decoder 300. The modules shown are logical and do not necessarily correspond one-to-one with the implemented hardware, firmware, and / or code. Figure 2 In the example, the point cloud encoder 200 may include a geometry encoding unit 250 and an attribute encoding unit 260. Generally, the geometry encoding unit 250 is configured to encode the positions (e.g., coordinates) of points in the point cloud frame to produce a geometry bitstream 203. The attribute encoding unit 260 is configured to encode the attributes of points in the point cloud frame to produce an attribute bitstream 205. As will be explained below, the attribute encoding unit 260 may also use the positions and the encoded geometry (e.g., reconstruction) from the geometry encoding unit 250 to encode the attributes.

[0064] like Figure 2 As shown in the example, the point cloud encoder 200 can obtain the set of point locations and attribute sets in a point cloud. The point cloud encoder 200 can obtain data from data source 104 ( Figure 1 The point cloud encoder 200 obtains a set of locations and a set of attributes for points in the point cloud. Locations may include the coordinates of the points in the point cloud (e.g., (x, y, z) coordinates). Attributes may include information related to the points in the point cloud, such as the color associated with the points, normal vectors, reflection information, or other attributes. The point cloud encoder 200 can generate an encoded geometric bitstream 203 containing the locations of the points in the point cloud. The point cloud encoder 200 can also generate an encoded attribute bitstream 205 containing the attribute set.

[0065] In some examples of AI-based point cloud compression, the geometry encoding unit 250 may be configured to, for example, reduce received geometry data by a certain amount along the X, Y, and / or Z axes, and encode the reduced geometry data. In some examples, the geometry encoding unit 250 may include a series of reduction and encoding task units as units, each task unit group reducing and encoding the geometry data by a certain amount, and then passing the reduced and encoded geometry data to subsequent task units. In other examples, the geometry encoding unit 250 may include a series of reduction stages for each type of geometry data, followed by a set of one or more encoding task units that encode the reduced geometry data.

[0066] In one example, the geometry encoding unit 250 can reduce the input geometry data (e.g., an octree) by factors of 8: factors of 2 along the X-axis, factors of 2 along the Y-axis, and factors of 2 along the Z-axis. This reduction can be performed by an artificial intelligence / machine learning (AI / ML) unit (such as a neural network). In some examples, the attribute encoding unit 260 can perform similar reduction on attribute values. Additional details regarding examples of such reductions are discussed below.

[0067] exist Figure 3 In the example, the point cloud decoder 300 may include a geometry decoding unit 350 and an attribute decoding unit 360. Generally, the geometry decoding unit 350 is configured to decode the geometry bitstream 203 to recover the positions of points in the point cloud frame. The attribute decoding unit 360 is configured to decode the attribute bitstream 205 to recover the attributes of the points in the point cloud frame. As will be explained below, the attribute decoding unit 360 may also use this position from the decoded geometry (e.g., reconstructed) from the geometry decoding unit 350 to encode the attribute.

[0068] In some examples, the geometry decoding unit 350 may decode a value indicating the amount of amplification to be applied to the geometry data (e.g., in the case of reduction performed by the point cloud encoder 200). Furthermore, the geometry decoding unit 350 may decode and amplify the geometry data, where the amplification corresponds to the decoded value. In some examples, the geometry decoding unit 350 may include a series of unit groups comprising decoding and amplification units, where the number of groups equals the decoded value representing the amplification. In some examples, the geometry decoding unit 350 may include one or more decoding units followed by a series of amplification unit groups, where the number of groups equals the decoded value representing the amplification. In some examples, the geometry decoding unit 350 may further reconstruct the point cloud geometry using the decoded and amplified point cloud geometry data.

[0069] Figure 2 and Figure 3Various units are illustrated to aid in understanding the operations performed by the point cloud encoder 200 and the point cloud decoder 300. Units can be implemented as fixed-function circuits, programmable circuits, or combinations thereof. A fixed-function circuit is a circuit that provides specific functionality and is pre-defined for the operations that can be performed. A programmable circuit is a circuit that can be programmed to perform various tasks and provides flexible functionality for the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. A fixed-function circuit can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more units in the unit may be different circuit blocks (fixed-function or programmable), and in some examples, one or more units in the unit may be integrated circuits.

[0070] Figure 4 The image shows an example system model for point cloud compression based on AI (e.g., deep learning). Figure 5 and Figure 6 More detailed examples of encoder network 412 and decoder network 464 are shown in the figure.

[0071] exist Figure 4 In the diagram, system model 400 illustrates both an AI-based point cloud encoder 410 and an AI-based point cloud decoder 460. The AI-based point cloud encoder 410 can be the point cloud encoder 200 (see...). Figure 1 and Figure 2 An example of this is the AI-based point cloud decoder 460. The AI-based point cloud decoder 300 (see...) Figure 1 and Figure 3 An example of this is shown. System model 400 is configured for intra-frame prediction (I-frame) encoding and decoding. However, the techniques disclosed herein can also be used for both unidirectional prediction (e.g., P-frames) and bidirectional prediction (e.g., B-frames) in inter-frame prediction encoding and decoding.

[0072] The AI-based point cloud encoder 410 includes an encoder network 412, an octree encoder 414, a quantizer (Q) 416, an arithmetic encoder (AE) 418, and an entropy model 430. At a high level, the encoder network 412 is configured to use a convolutional autoencoder. The encoder network 412 can accept sparse tensors (…). P ) as input. Sparse tensor P This represents the input point cloud frame and includes a set of coordinates ( C ) and their associated learning characteristics ( F ).Right now, P = [C, F] In this input, F It can be a uniform vector representing occupancy. Figure 4 In the example, encoder network 412 can be configured to reduce the point cloud by three times to generate a reduced tensor. P 3ds After reducing the tensor P 3ds Including reduced coordinates C 3ds and related features F 3ds .Right now, C 3ds Each coordinate in the table includes F 3ds The associated feature values ​​in the encoder network 412. Although the examples in this disclosure show a triple (8x) reduction in the encoder network 412 and a corresponding triple amplification in the decoder network 464, more or fewer reduction and amplification levels can be used.

[0073] Octree encoder 414 is configured to use lossless octree coding techniques (such as those used in G-PCC) to process the reduced coordinates. C 3ds Encoding is performed. For example, the octree encoder 414 can be based on reduced coordinates. C 3ds Generate an octree. At each node in the octree, signal occupancy (if not inferred) for one or more of its child nodes (up to eight nodes). Multiple neighborhoods are specified, including (a) nodes sharing a face with the current octree node, (b) nodes sharing a face, edge, or vertex with the current octree node, etc. Within each neighborhood, occupancy of the current node or its child nodes can be predicted using the occupancy of the node and / or its child nodes.

[0074] Assuming such features are in floating-point format, quantizer 416 can be configured to quantize the reduced learned features. F 3ds Once quantized, the arithmetic encoder 418 can be configured to process the quantized features using a learned entropy model 430. F 3d Perform arithmetic encoding.

[0075] The AI-based point cloud decoder 460 includes an octree decoder 462, a decoder network 464, an arithmetic decoder (AD) 466, and an entropy model 430. The octree decoder 462 receives the encoded octree from the encoded bitstream produced by the octree encoder 414 and recovers the reduced coordinates. C 3ds The arithmetic decoder 466 receives encoded features from the encoded bitstream. F 3ds The encoded features are then decoded using an entropy model 430. An arithmetic decoder 466 recovers the learned features. Acquired characteristics The hat symbol is used because the quantization and arithmetic decoding processes may introduce some loss.

[0076] Reduced coordinates C 3ds and learned characteristics With tensor The format is provided to the decoder network 464. That is... The decoder network reconstructs point clouds in 464 layers. For example, decoder network 464 can use three upsampling (also known as upsampling) stages, where binary classification is employed to reconstruct occupied voxels. For this purpose, decoder network 464 can use three binary cross-entropy loss functions.

[0077] The terms “reduction” and “downsampling” are used interchangeably in this disclosure. The terms “amplification” and “upsampling” are also used interchangeably in this disclosure. The entire framework can be viewed as a transmitting system, a compression pipeline, or a deep learning model. In the case of a transmitting system, the components preceding bitstream generation can be referred to as the forming part of a “transmitter” or “encoder,” while the components following bitstream generation can be referred to as the forming part of a “receiver” or “decoder.” In the case of a deep learning model, the encoder network 412 can be viewed as a multi-scale feature extractor, and the decoder network 464 as a progressively amplified network with hierarchical reconstruction of point clouds.

[0078] Figure 5 This is an example of the technology according to this disclosure. Figure 4 Here is a block diagram of an example encoder network 412. As described above, the input to the encoder network 412 includes coordinates. C and characteristics F sparse tensors P .exist Figure 5 In the middle, name Conv c×n 3 This refers to a convolution with c output channels and an n×n×n kernel size. As an example, a 500-layer convolutional layer (Conv 16×3) 3 This performs a convolution with a 3×3×3 kernel size and 16 output channels. ReLU stands for Rectified Linear Unit. IRB stands for Initial Residual Block. An IRB can include multiple convolutional layers. A convolutional layer with a downward arrow indicates quadratic reduction. For example, convolutional layer 510 performs a convolution with a 3×3×3 kernel size, 32 output channels, and quadratically reduces the input coordinates and corresponding features.

[0079] Generally speaking, (such as) Figure 5The 3D convolutions used are configured to process data with three spatial dimensions, such as the location of points in a point cloud (e.g., point cloud geometry). Each point can be represented by (x, y, z) coordinates. 3D convolutions use 3D filters (also known as 3D kernels). In 3D convolutions, the filter itself is a three-dimensional weight block. For example, a 3D convolution can have a 3×3×3 3D filter instead of a 3×3 2D filter used in a 2D convolution. This 3D filter slides (or convolves) across the 3D input volume.

[0080] 3D convolution operations involve computing the dot product between a 3D filter and the segments of the input volume covered by the convolution operation at each step. For each location, the filter is aligned with the 3D region of the input, the weights of the 3D core are multiplied element-wise with the corresponding input values, and these products are summed to produce the output value.

[0081] The output of 3D convolution is a 3D volume representing the convolutional features extracted from the input volume. This process can be repeated using multiple 3D filters to extract different types of features from the input. The stride in 3D convolution refers to the number of steps the filter moves along each of the three dimensions after each operation. A stride of 1 in all three dimensions means the filter moves one unit along each axis at a time.

[0082] In some examples, padding can also be applied to 3D convolutions, involving layers that add zeros around the input volume in all three dimensions. Padding helps control the size of the output volume and allows for more flexible architecture design.

[0083] ReLU is an activation function used in neural networks, particularly in hidden layers. If the input is positive, the ReLU function outputs the input directly; otherwise, it outputs zero. The ReLU function is piecewise linear, with non-linear breakpoints at zeros. Because it is non-linear, ReLU can be helpful in learning complex patterns in data.

[0084] In some examples, ReLU activation can lead to sparse representations, which can be advantageous in terms of computation and memory usage. Because the ReLU function outputs zero for all negative inputs, it effectively "disconnects" many neurons in the network. The gradient of the ReLU function is 1 for all positive inputs, which helps mitigate the vanishing gradient problem encountered with other activation functions. However, for inputs less than zero, the gradient is zero, which can lead to neurons that never start (a problem known as "dying ReLU").

[0085] Due to the "dying ReLU" problem, several variants of the ReLU function have been proposed, including the leaking ReLU function or parametric ReLU (PReLU). Leaking ReLU introduces a small positive gradient for negative inputs. PReLU is similar to leaking ReLU, but the coefficients of the negative portion are learned during training. In any example in this disclosure, leaking ReLU or PReLU can be used instead of ReLU.

[0086] Figure 5 The Initial Residual (Initial-ResNet) Block (IRB) in a neural network is a concept that uses residual connections (ResNet) to combine initial blocks for feature extraction. This neural network can facilitate deeper network architectures by solving the gradient vanishing problem. The hybrid design of IRBs leverages the advantages of both architectures to improve performance.

[0087] The initial block contains multiple parallel paths of convolution and pooling operations with filters of different sizes (e.g., 1×1, 3×3, 5×5). This design allows the IRB to capture and process information simultaneously at different scales or resolutions, enabling the network to adapt to various input patterns.

[0088] The residual connections (or shortcut connections) in the ResNet architecture add the block's input directly to its output, effectively allowing the network reference layer input of the IRB to learn the residual function. This technique helps combat the vanishing gradient problem by providing an alternative path for the gradient flow during backpropagation, enabling successful training of deeper networks.

[0089] The initial residual block integrates these two concepts by merging initial-pattern parallel convolutional paths within the block and then using residual connections to add the block's input to the block's output. The outputs from the parallel paths are typically concatenated and may then undergo additional processing (e.g., 1×1 convolution) before being added to the block's input via residual connections.

[0090] return Figure 5 The convolutional layer shown can be a sparse convolution. Sparse convolutions better utilize the sparsity of point clouds and allow for reduced complexity by applying computation only to occupied voxels (e.g., voxels comprising one or more points). Convolutional layer 500 performs with 16 output channels to draw from sparse tensors. P Input coordinates C Generate a set of initial features F A 3×3×3 convolution. The ReLU function 502 is a non-linear activation function that operates on the output of the convolutional layer 500.

[0091] like Figure 5 As shown, the encoder network 412 is configured to reduce the coordinates in the sparse tensor P. Figure 5An example of progressive reduction is shown, where three distinct layers (i.e., convolutional layer 510, convolutional layer 524, and convolutional layer 534) perform 2x reduction. Therefore, the input tensor... P The coordinates are reduced a total of three times, thus reducing the output tensor. P 3ds The number of coordinates in the coordinate system is reduced by 8x. That is, C 3 ds The coordinates are the input coordinates C One-eighth of it. F 3 ds It has C 3 ds The eigenvectors of each coordinate in the matrix.

[0092] Convolutional layer 510 operates on the output of ReLU 502 and performs a 2x reduction of the coordinates. Following convolutional layer 510 are ReLU 512 and IRB 514. As described above, the IRB block can include multiple convolutional layers and can be configured to generate additional features based on the coordinates. The general architecture of convolutional layer 500, ReLU layer 502, convolutional layer 510 (reduced), ReLU 512, and IRB 514 is described in... Figure 5 The architecture shown is repeated twice. That is, convolutional layer 520, ReLU layer 522, convolutional layer 524 (reduced), ReLU 526, and IRB 528 operate on the reduced coordinates and the corresponding features output by IRB 514. Similarly, convolutional layer 530, ReLU layer 532, convolutional layer 534 (reduced), ReLU 536, and IRB 538 operate on the reduced quadratic coordinates and the corresponding features output by IRB 528. The encoder network 412 may include an output reduced tensor. P 3ds The final output convolutional layer is 540.

[0093] Generally speaking, the encoder network 412 is configured with convolutional layers to handle data using geometric coordinates. C and eigenvectors F The sparse tensor is represented. The encoder network 412 learns the distribution of positively occupied voxels in the volume representation of the point cloud implicitly and embeds it into... F Features and coordinates C Gradually shrink to multiple scales.

[0094] Encoder network 412 can use sparse convolutions for low-complexity tensor processing. IRBs are used for efficient feature extraction. IRB blocks 514, 526, and 538 are positioned after each reduction process (e.g., convolutional layers 510, 524, and 534). Convolutional layers 510, 524, and 534 can be configured to perform reduction by applying convolutions with a stride of two to halve the scale of each geometric dimension (e.g., (x, y, z)). Convolutional layers 510, 524, and 534 can have similar structures but use different parameters in terms of kernel size and output feature length, such as... Figure 5 As shown. Compared to convolutional layer 524, convolutional layer 510 can use fewer channels to reduce computational cost, while convolutional layer 534 can use fewer channels to reduce the size of the output features.

[0095] Figure 6 This is an example of the technology according to this disclosure. Figure 4 Here is a block diagram of an example decoder network 464. The decoder network 464 is configured to perform the inverse process of the encoder network 412 to recover the original geometry (e.g., the coordinates of the points) of the encoded point cloud.

[0096] The input to the decoder network 464 is a sparse tensor generated by the encoder network 412. P 3ds ,in P 3d Including coordinates C 3ds and characteristics F 3ds .exist Figure 6 In the diagram, convolutional layers with upward arrows indicate secondary amplification. For example, convolutional layer 600 performs a convolution with a 3×3×3 kernel size, 64 output channels, and amplifies the input coordinates and corresponding features twice. The ReLU function 602 is a non-linear activation function that operates on the output of convolutional layer 600.

[0097] like Figure 6 As shown, the decoder network 464 is configured to amplify the sparse tensor P. 3ds The reduced coordinates in the equation. Figure 6 An example of progressive scaling is shown, where three different layers (i.e., convolutional layer 600, convolutional layer 612, and convolutional layer 624) perform 2x scaling. Therefore, the input tensor... P 3ds The coordinates are magnified a total of three times, thus producing a point cloud output. The number of coordinates in the point cloud is increased by 8x. The coordinates in the text are the input coordinates. C 3ds 8 times.

[0098] Convolutional layer 604 operates on the output of ReLU 602. Following convolutional layer 604 are ReLU 606 and IRB 608. As described above, the IRB block may include multiple convolutional layers and can be configured to generate additional features based on coordinates. Classifier 610 then operates on the output of IRB 608. Classifier 610 reconstructs geometric details (e.g., occupancy of voxels with points) by pruning pseudo-voxels and extracting occupied voxels according to a loss function L1 using binary classification. Classifier 610 may use hierarchical, coarse-to-fine refinement. Classifier 610 applies binary classification to determine whether the generated voxels are occupied and may use sparse convolutional layers to generate the probability of voxels being occupied after successive convolutions.

[0099] The general architecture of convolutional layer 600 (enlarged), ReLU layer 602, convolutional layer 604, ReLU 606, IRB 608, and classifier 610 is as follows: Figure 6 The architecture shown is repeated twice. That is, convolutional layer 600 (enlarged), ReLU layer 602, convolutional layer 604, ReLU 606, IRB 608, and classifier 610 operate on the reduced quadratic coordinates (due to the aforementioned enlargement) and corresponding features. Similarly, convolutional layer 612 (enlarged), ReLU layer 614, convolutional layer 616, ReLU 618, IRB 620, and classifier 622 (using a loss function) L 2 The operation is performed on the reduced coordinates (e.g., due to the magnification mentioned above) output by convolutional layer 612 and the corresponding features. This is followed by convolutional layer 624 (magnification), ReLU layer 626, convolutional layer 628, ReLU 630, IRB 632, and classifier 634 (using a loss function). L 3 The classifier 634 operates on the coordinates at the original scale (e.g., due to the magnification mentioned above) and corresponding features output by convolutional layer 624. The classifier 634 outputs the reconstructed point cloud. Generally speaking, in decoder network 464, reconstructions from previous lower scales are amplified by enhancing the feature properties of finer geometric refinement in the current layer.

[0100] Positional encoding is a technique used in deep learning models (especially in the field of natural language processing (NLP)) where the order of elements in a sequence (such as words in a sentence) is important. Positional encoding has been used to provide deep learning models with information about the position of elements within a sequence, which helps the model distinguish different elements based on their position.

[0101] In many deep learning architectures, such as transformers, sequences are typically processed in parallel, meaning the model doesn't inherently understand the order of elements. Positional encoding is introduced to address this issue. Positional encoding can be added to the input embedding of the sequence and learned along with other parameters during training. Positional encoding is also used in various other domains, such as computer vision tasks, and more broadly in transformer-based architectures.

[0102] In computer vision, convolutional neural networks (CNNs) can be used to process 2D images. CNNs are generally designed to process grid-like data, such as images where the spatial arrangement of pixels is important. However, unlike sequences in NLP, the concept of order is not inherent in images. To incorporate positional information, positional encoding can be added to the input image data. One approach is to use a grid of positional encodings concatenated to image features. Here, the (x,y) position of a pixel is used as the location for creating a higher-dimensional positional encoding.

[0103] For example, in image generation, the input might be a flattened version of an image that lacks spatial structure. Positional encoding can be added to represent the spatial layout of pixels. Similarly, in video analytics, each frame can be processed as a "position" in a sequence, and positional encoding helps the model understand the temporal order of the frames. Different types of positional encoding techniques are described below.

[0104] Sine and Cosine Position Coding: One type of position coding is sine and cosine position coding, which is based on the idea that it is feasible to represent position using sine and cosine functions of different frequencies. Each dimension of the position code corresponds to a different frequency, and the combination of these dimensions creates a unique pattern for each position.

[0105] In mathematics, position can be " pos "and dimensions" i The position code for "" is represented as:

[0106]

[0107]

[0108] in" pos " is the position in the sequence, " i "is a dimension index and" It is the dimension of the model embedding.

[0109] The idea behind this formula is that low-frequency terms capture long-range relationships between locations, while high-frequency terms capture short-range relationships. (Scaling factor) 10000 It is selected based on experience to ensure that the location code has different values ​​for different locations.

[0110] Besides sine and cosine positional encodings commonly used in transformer-based models, several other types of positional encodings and variations have been proposed in the field of deep learning, depending on the specific requirements of the task or the choice of architecture. Examples are described below.

[0111] Learned positional encoding: Learned positional encoding involves treating positional embeddings as trainable parameters within the model, allowing the network to adaptively learn the optimal representation of a position during training. This method is not limited to any particular mathematical function and can be customized for the task.

[0112] Linear positional encoding: Linear positional encoding assigns a simple linear function of positions to represent positional information. For example, a positional embedding might be a continuous sequence of integers representing the position of each element. This type of encoding is particularly simple but may fail to capture complex positional relationships.

[0113] Radial Basis Function (RBF) Position Encoding: RBF position encoding uses radial basis functions centered at each position in the sequence. The idea is to create a Gaussian-like function that varies in response to position. This effectively captures regional positional information.

[0114] Fermi-Dirac Function Position Encoding: Inspired by quantum mechanics, Fermi-Dirac function position encoding uses the Fermi-Dirac distribution function. The Fermi-Dirac function models positional relationships using a smooth, sigmoid curve, making it suitable for tasks involving gradual changes in positional importance.

[0115] Hybrid positional coding: Some models employ a combination of different positional coding methods. For example, some models might use sine and cosine positional coding for low-frequency information and linear or learned positional coding for high-frequency information. This hybrid approach aims to capture both short-range and long-range positional dependencies.

[0116] Relative positional encoding: Relative positional encoding is designed to capture the relative distances between elements in a sequence, rather than their absolute positions. It is commonly used in tasks such as machine translation, where models are configured to understand how elements are related to each other, regardless of their absolute positions in the input.

[0117] Image grid-like positional encoding: In computer vision tasks, especially when using transformers for image analysis, grid-like positional encoding is used to represent the spatial layout of image pixels. Each pixel in the grid is associated with a unique positional embedding.

[0118] The choice of location encoding depends on the specific task and the nature of the data. Some types may work better for certain tasks or data distributions, while others may be more computationally efficient.

[0119] After computing the positional encoding, the embedding can be added element-wise to the input embedding before feeding the sequence into the model. This addition ensures that the model has information about the position of the elements, allowing elements to be distinguished based on their order.

[0120] This disclosure describes techniques for adding one or more positional encoding layers (such as those previously described) to encoder networks (e.g., encoder network 412) and decoder networks (e.g., decoder network 464) for AI-based point cloud compression. As will be described below, the positional encoding layers create additional higher-dimensional features generated from the currently processed coordinates. These features are higher-dimensional relative to other feature tensors generated by the encoder or decoder networks. These additional higher-dimensional feature properties are combined (e.g., concatenated) to feature tensors generated by other layers of the encoder and decoder networks. By increasing the dimensionality of the feature tensors generated by the network with positional encoding layers, the techniques of this disclosure improve the efficiency of point cloud compression.

[0121] For example, tests have shown that adding a positional encoding layer to both the point cloud encoder and decoder used in example AI-based point cloud compression methods (e.g., PCGCv2) can improve compression results with up to 4% BD-rate (Bjontegaard Delta-rate) savings. Tests have also shown that the concept of positional encoding remains consistent even when applied to non-NLP domains, such as point cloud compression. The positional encoding techniques disclosed herein enhance the ability of AI-based point cloud compression models to handle sequentially important sequences, better ensuring that AI-based point cloud compression models capture meaningful relationships between elements based on their positions.

[0122] Figure 7 This is a block diagram illustrating an example encoder network 700 using position encoding according to the technology of this disclosure. The encoder network 700 includes convolutional layers, ReLU functions, and IRB layers... Figure 5 The encoder network 700 is the same as that shown in encoder network 412. Like encoder network 412, encoder network 700 can include more or fewer layers and have more or fewer convolutional layers for reduction in different combinations. In some examples, encoder network 700 may not have any layers for reduction.

[0123] At a higher level, encoder network 700 (which may be part of point cloud encoder 200) is configured to receive point cloud data frames (e.g., sparse tensors). P ) and use a deep learning network to obtain the coordinates of the geometric structure of the point cloud data frame. C Encode it. In Figure 7In this context, the deep learning network (i.e., encoder network 700) is a convolutional neural network. Encoder network 700 includes one or more layers (e.g., convolutional layers and / or IRB blocks) configured to generate first features of the coordinates of a geometric structure. Encoder network 700 includes at least one positional encoding layer (e.g., positional encoding layers 702, 704, 706, and / or 708) configured to generate second features of the coordinates of the geometric structure. As will be described below, the positional encoding layers may be configured to combine the first and second features to generate higher-dimensional features. Encoder network 700 may output coordinates ( C 3ds ) and higher-dimensional features ( F 3ds ) output tensor ( P 3ds ).

[0124] In one example, position coding layers 702, 704, 706, and 708 are implemented using learned position coding. In one example, learned position coding uses coordinates... This is achieved by mapping to higher-dimensional learned features (e.g., 128-dimensional).

[0125] Learning-based positional encoding layers are an alternative to traditional fixed positional encoding, which relies on predefined mathematical functions (e.g., sine and cosine functions) to represent positional information in a sequence. In learning-based methods, positional encoding is learned as part of the model training process. Instead of using fixed positional encoding, the model assigns learnable embeddings to each position in the sequence. These positional embeddings, like other weights in the network, are considered as additional parameters that the model updates during training.

[0126] During the forward pass of the training process, the model processes the input sequence, including the location embeddings. During training, the loss is computed, and gradients are backpropagated through the network, including the location embeddings. Over time, the location embeddings are adjusted to contain meaningful positional information that assists the model in capturing relationships between points based on point-based locations / coordinates. The advantage of learned location encoding is that it allows the model to adapt to specific tasks and data distributions. The following section discusses… Figure 10 More details are described regarding learned positional coding.

[0127] Figure 7 The example includes four positional encoding layers 702, 704, 706, and 708. Positional encoding layer 702 is located at the beginning of encoder network 700. Positional encoding layer 702 is configured to add positional codes to the tensor before processing. Positional encoding layer 702 can be viewed as input to the deep learning network (encoder network 700). PThe first positional coding layer that performs the operation. Although Figure 7 The example illustrates the use of four position coding layers, but the techniques disclosed herein can include any number of position coding layers, including as few as one position coding layer.

[0128] The encoder network 700 may include at least one position encoding layer (e.g., a second position encoding layer) that operates on the reduced coordinates. Figure 7 In the example, encoder network 700 includes three additional position encoding layers 704, 706, and 708 that operate on the reduced coordinates. Figure 7 The diagram shows positional encoding layers 704, 706, and 708 positioned between the shrunk convolutional layers (e.g., convolutional layer 510) and the IRB block (e.g., IRB 514). Positioning the positional encoding layers between the shrunk convolutional layers and the IRB allows positional encoding to be performed after the scale change but before most feature extraction / convolution is performed in the IRB, enabling such convolutions in the IRB to factor in positional embeddings.

[0129] Figure 7 The diagram shows positional encoding layers 704, 706, and 708 positioned between the ReLU function and the IRB block. In other examples, positional encoding layers 704, 706, and 708 may be positioned between the reduced convolutional layer and the ReLU function. In still other examples, positional encoding layers 704, 706, and 708 may be positioned between layers within the IRB. Furthermore, positional encoding layers 704, 706, and 708 do not need to be positioned at approximately the same location in each reduced segment of the encoder network 700, but can be placed at different locations in each reduced segment of the encoder network 700.

[0130] Therefore, in Figure 7 In the example, the encoder network 700 includes three layers (e.g., convolutional layer 510, convolutional layer 524, and convolutional layer 534) configured to progressively reduce the coordinates of the geometry to generate first reduced coordinates, second reduced coordinates, and third reduced coordinates. In this example, the output tensor ( P 3ds ) including the third reduced coordinate ( C 3ds ) and higher-dimensional features ( F 3ds The encoder network 700 includes inputs to the encoder network 700. PThe system comprises a first position encoding layer 702 that operates on the first reduced coordinate, a second position encoding layer 704 that operates on the first reduced coordinate, a third position encoding layer 706 that operates on the second reduced coordinate, and a fourth position encoding layer 708 that operates on the third reduced coordinate. As described above, at least one of the second position encoding layer 704, the third position encoding layer 706, or the fourth position encoding layer 708 is positioned between the layer configured to reduce the coordinate (e.g., convolutional layer 510) and the initial residual block (e.g., IRB 514).

[0131] Figure 8 This is a block diagram illustrating an example decoder network 800 using positional encoding according to the technology of this disclosure. The decoder network 800 includes convolutional layers, ReLU functions, IRB layers, and a classifier. Figure 6 The decoder network 800 is the same as that shown in decoder network 464. Like decoder network 464, decoder network 800 can include more or fewer layers and have more or fewer convolutional layers for amplification in different combinations. In the example where encoder network 700 does not perform reduction, decoder network 800 will not have any layers for amplification.

[0132] At a higher level, decoder network 800 (which may be part of point cloud decoder 300) is configured to perform the inverse process of encoder network 700. Decoder network 800 can receive frames of encoded point cloud data ( The encoded point cloud data includes an input tensor that includes encoded coordinates and corresponding features. The point cloud decoder 300 can use a deep learning network (e.g., decoder network 800) and the input tensor to decode the coordinates of the geometry of the frame of the encoded point cloud data to generate decoded coordinates.

[0133] The decoder network 800 includes one or more layers (e.g., convolutional layers and / or IRBs) configured to generate a first feature of the coordinates of a geometric structure, a second feature of the coordinates of the geometric structure, and at least one position encoding layer (e.g., position encoding layers 802, 804, 806, and / or 808) configured to generate a second feature of the coordinates of the geometric structure and combine the first feature with the second feature to generate a higher-dimensional feature. The decoder network 800 may also include one or more layers (e.g., classifiers 610, 622, and / or 634) configured to classify the higher-dimensional feature to generate decoded coordinates. A point cloud decoder 300 can decode the point cloud (…). The decoded coordinates are output in the output.

[0134] Similar to encoder network 700, the position coding layers 802, 804, 806, and 808 of decoder network 800 can be implemented using learned position coding. In one example, learned position coding uses coordinates configured to... This is achieved by mapping to higher-dimensional learned features (e.g., 128-dimensional).

[0135] Figure 8 Examples include four positional encoding layers 802, 804, 806, and 808. Positional encoding layer 802 is located at the beginning of the decoder network 800. Positional encoding layer 802 is configured to add positional codes to the input tensor before processing. Positional encoding layer 802 can be viewed as input to the deep learning network (decoder network 800). P The first positional coding layer that performs the operation. Although Figure 8 The example illustrates the use of four position coding layers, but the techniques disclosed herein can include any number of position coding layers, including as few as one position coding layer.

[0136] The decoder network 800 may include at least one position coding layer (e.g., a second position coding layer) that operates on magnified coordinates. Figure 8 In the example, the decoder network 800 includes three additional positional coding layers 804, 806, and 808 that operate on the magnified coordinates. Figure 8 The diagram shows positional encoding layers 804, 806, and 808 positioned between the amplified convolutional layers (e.g., convolutional layer 600) and the IRB block (e.g., IRB 608). Positioning the positional encoding layers between the amplified convolutional layers and the IRB allows positional encoding to be performed after the scale change but before most feature extraction / convolution is performed in the IRB, enabling such convolutions in the IRB to factor in positional embeddings.

[0137] Figure 8 The diagram shows positional coding layers 804, 806, and 808 positioned between the ReLU function and the IRB block. In other examples, positional coding layers 804, 806, and 808 may be positioned between the magnified convolutional layer and the ReLU function. In still other examples, positional coding layers 804, 806, and 808 may be positioned between layers within the IRB. Furthermore, positional coding layers 804, 806, and 808 do not need to be positioned at approximately the same location in each magnified segment of the decoder network 800, but can be placed at different locations in each magnified segment of the decoder network 800.

[0138] Therefore, in Figure 8In the example, the decoder network 800 includes three layers (e.g., convolutional layer 600, convolutional layer 612, and convolutional layer 624) configured to progressively magnify coordinates to generate first magnified coordinates, second magnified coordinates, and third magnified coordinates. The decoder network 800 includes a first positional encoding layer 802 operating on the decoder network 800, a second positional encoding layer 804 operating on the first magnified coordinates, a third positional encoding layer 806 operating on the second magnified coordinates, and a fourth positional encoding layer 808 operating on the third magnified coordinates. As described above, at least one of the second positional encoding layer 804, the third positional encoding layer 804, or the fourth positional encoding layer 808 is positioned between the layer configured to magnify the reduced coordinates (e.g., convolutional layer 600) and the initial residual block (e.g., IRB 608).

[0139] Figure 9 This is a block diagram of an example system model 900 for AI-based point cloud compression using position encoding, exemplifying the techniques of this disclosure. System model 900 shows both an AI-based point cloud encoder 910 and an AI-based point cloud decoder 960. The AI-based point cloud encoder 910 may be a point cloud encoder 200 (see...). Figure 1 and Figure 2 An example of this is the AI-based point cloud decoder 960. The AI-based point cloud decoder 960 could be the point cloud decoder 300 (see...). Figure 1 and Figure 3 An example of this is shown. System model 900 is configured for intra-frame prediction (I-frame) encoding and decoding. However, the techniques disclosed herein can also be used for both unidirectional prediction (e.g., P-frames) and bidirectional prediction (e.g., B-frames) in inter-frame prediction encoding and decoding.

[0140] Generally speaking, system model 900 is similar to the above-mentioned... Figure 4 The system operates in the same manner as described in system model 400. However, the point cloud encoder 910 uses an encoder network 700 with one or more positional encoding layers. Similarly, the point cloud decoder 960 uses a decoder network 800 with one or more positional encoding layers. The processes for octree encoding and decoding, as well as quantization and arithmetic decoding, remain the same.

[0141] For example, the point cloud encoder 910 can be configured to operate on the coordinates of the output tensor generated by the encoder network 700. C 3ds Octree encoding is performed on the point cloud (e.g., using octree encoder 414) to generate encoded coordinates. Point cloud encoder 910 can further quantize (e.g., using quantizer 416) the higher-dimensional features generated by encoder network 700. F 3dsThe output tensor can be arithmetically encoded using an entropy model (e.g., entropy model 430) (e.g., using an arithmetic encoder 418) to generate quantized higher-dimensional features. The point cloud encoder 910 can output coded coordinates and coded higher-dimensional features in a coded bitstream.

[0142] The point cloud decoder 960 can perform octree decoding on the encoded coordinates in the encoded bitstream (e.g., using the octree decoder 462) to recover the encoded coordinates in the input tensor. C 3ds The point cloud decoder 960 can further (e.g., using an arithmetic decoder 466) perform arithmetic decoding on the encoded features in the encoded bitstream to recover the input tensor. Corresponding features in ).

[0143] Figure 10 This is a block diagram illustrating an example position coding layer according to the technology of this disclosure. The learned position coding layer 1000 is an example of learned position coding that uses a 1×1×1 filter core (conv128×1) with 128 output channels. 3 Learned position coding layer 1000 receives tensors P As input. P Including coordinates C and corresponding features F o .feature F o Features are generated from one or more previous layers of an encoder or decoder network. That is, features F o It can come from anywhere within the encoder or decoder network. Similarly, coordinates... C These can be coordinates at any reduction or magnification level within the encoder or decoder network.

[0144] The learned position encoding layer 1000 includes a convolutional layer 1020 that performs a 1×1×1 convolution with a 128-dimensional output. The convolutional layer 1020 receives input coordinates... C Generate output features F e That is, convolutional layer 1020 is configured to... Coordinate mapping to higher-dimensional learned features (e.g., 128 dimension) F e In the middle. Cascade unit 1010 will feature F o With learned output features F eCascade to generate higher-dimensional features For example, if the input features F o It has a feature size of 64 and learned output features. F e With a feature size of 128, then higher-dimensional features It has a feature size of 192. In other examples, the learned positional encoding layer can encode the input features. F o With learned output features F e Multiplication to produce higher-dimensional features Then, the learned position encoding layer 1000 combines the input coordinates C with higher-dimensional features. Combine to generate output tensor ,in

[0145] The weights of convolutional layer 1020 can be learned during training. In practice, convolutional layer 1020 can behave similarly to a fully connected layer or a multilayer perceptron (MLP). Therefore, in other examples, the learned position encoding layer 1000 can use a fully connected layer or an MLP instead of convolutional layer 1020. In a fully connected layer, each neuron or node is connected to every neuron in the previous layer and every neuron in subsequent layers. This structure allows the layer to integrate information from all received inputs, enabling it to learn complex relationships in the data. An MLP may include at least three layers of nodes: an input layer, one or more hidden layers, and an output layer. Each node or neuron in a layer is connected with some weight to every node in subsequent layers, making the network fully connected. Nodes in the hidden and output layers typically use non-linear activation functions to process the input received from previous layers.

[0146] Figure 11A Chart 1050 shows a test of the compression efficiency of a feature bitstream encoded according to the technology disclosed herein. Figure 11B Graph 1100 shows a test of the compression efficiency of the overall bitstream encoded according to the technology disclosed herein. Figure 11A and Figure 11B The rate-distortion (RD) curves of the point cloud encoder output using the location encoding technique of this disclosure are shown. Figures 1050 and 1100 illustrate the significant improvement in the RD curves achieved by adding location encoding to the network. This translates to a BD-rate saving of approximately 4%.

[0147] Other examples

[0148] The following variations / flexibility can be combined in any way to add to any of the technologies described above.

[0149] In the above Figure 7 and Figure 8 In the example, the point cloud encoder network and the point cloud decoder network use a learning-based positional coding layer. However, any positional coding method can be employed. For example, positional coding can be any of the following: sine and cosine positional coding layers, linear positional coding layers, radial basis function positional coding layers, Fermi-Dirac positional coding layers, hybrid positional coding layers, relative positional coding layers, or image grid positional coding layers. Some tests have shown that sine and cosine positional coding has resulted in improved performance compared to techniques that do not use any positional coding.

[0150] exist Figure 7 and Figure 8 In the example, the learned position encoding uses 1×1×1 convolutional layers. However, a point cloud encoder or decoder can be configured with any number or type of layers to generate positional embedding features. This means that a point cloud encoder or decoder can be configured to use any neural network for learned position encoding. A common type of neural network used in position encoding is a fully connected layer.

[0151] In the example above, the feature length generated by positional encoding is 128. However, the positional embedding feature length may differ or vary in other examples, including feature lengths lower or higher than 128.

[0152] The above example uses a relationship to AI-based point cloud decoding techniques, with PCGCv2 as a baseline. However, the positional coding layer described above can be used with any point cloud compression method. Similarly, the techniques described above relate to intra-frame compression-based methods. The positional coding techniques of this disclosure have shown improved results compared to inter-frame compression-based methods, including one-way and two-way prediction.

[0153] As described above, the location or position of the location coding layer within the network / model can be changed, for example, based on the performance results of a specific application.

[0154] The examples above relate to point cloud encoders and decoders that perform point cloud geometry compression. However, the techniques disclosed herein are not limited to geometry compression, but can also be used for point cloud attribute compression or joint point cloud geometry and attribute compression methods. For example, a point cloud encoder can be configured to encode attributes of a point cloud data frame using a deep learning network. The deep learning network includes one or more layers configured to generate first features of the attributes, and the deep learning network further includes at least one positional encoding layer configured to generate second features from the attributes and combine the first and second features to generate higher-dimensional features. The point cloud decoder can perform the inverse process of attribute decoding.

[0155] Additionally, point cloud location encoding / embedding is not limited to point cloud compression tasks, but can also be used for any number of point cloud vision-related tasks, including classification, segmentation, etc.

[0156] In the example above, each positional encoding layer in the positional encoding layer using learned positional encoding can be trained independently and therefore can have individually determined and possibly unique weights for each of the 1×1×1 convolutions. That is, the encoder network or decoder network may include multiple positional encoding layers, wherein each of these positional encoding layers is configured to use a different set of weights.

[0157] In other examples, the location encoding layers can share the same weights. (See reference) Figure 7 and Figure 8 Instead of using eight distinct location coding layers, each with its own set of weights, the encoder and decoder networks can each use a single location coding layer with the same set of weights, and this layer can be reused multiple times. This is called weight sharing, and it reduces the number of parameters in the network and can produce better results.

[0158] The position encoding technique disclosed herein is not limited to using three-dimensional coordinates as input. That is, the encoder network or decoder network need not be configured to use only three-dimensional coordinates as input to learn high-dimensional position feature embeddings. Examples may include:

[0159] 1) For multi-frame dynamic point clouds using inter-frame compression, the input to the position coding layer can be four-dimensional (x, y, z, t), containing three-dimensional spatial coordinates and a fourth-dimensional time / sequence information (t). 4D position coding can be important for dynamic point cloud compression, but it can also be used for static point clouds;

[0160] 2) The encoder or decoder network can use higher-dimensional location information / features that can be input to the location encoding layer to obtain another location feature embedding, instead of just using three-dimensional coordinates.

[0161] In other examples, the encoder network or decoder network may use multiple types of location coding instead of a single type. This type of location coding layer in the encoder network or decoder network may be selectively activated or deactivated based on a specific type of use case / application and / or based on a specific type of point cloud. Therefore, the encoder network or decoder network may include multiple location coding layers, wherein these multiple location coding layers include at least two types of location coding.

[0162] Generally speaking, positional encoding layers typically do not incur any additional overhead in transmitting additional signals (e.g., syntax elements or metadata) within the bitstream. Positional encoding layers utilize the coordinates / positions of that tensor to generate higher-dimensional features.

[0163] The computational complexity of a positional coding layer depends on the type of positional coding employed. For learned positional coding, the computational complexity is based on the number of layers in the neural network, the number of parameters, and the number of multiplications. A positional coding layer implemented using a single convolutional layer with a core size of 1×1×1 is very small. Tests have shown that using a 1×1×1 core in a positional coding layer does not substantially change the encoding / decoding time.

[0164] for Figure 9 The network framework shown indicates that the point cloud encoder 910 is configured to signal the geometric bitstream, feature bitstream, and small signals used for arithmetic encoding. Because... Figure 9 The framework encodes coordinates and features separately, which may result in the loss of the correlation between coordinates and features. To address this, the tensor (containing coordinates and features) can be sorted about the coordinates after processing by the point cloud encoder 910 and before processing by the point cloud decoder 960.

[0165] In some examples, it may be desirable to design encoder network 700 and decoder network 800 with flexibility in the positional encoding layer. In this example, point cloud encoder 910 may be configured to signal one or more syntax elements to point cloud decoder 960 to indicate the use of positional encoding. The syntax elements may indicate one or more of the following parameters:

[0166] - The first syntax element, which indicates whether position encoding is enabled;

[0167] - A second syntax element that indicates the type of positional encoding used (e.g., learned positional encoding, sine and cosine positional encoding, etc.).

[0168] - A third syntax element that indicates the feature size of the positional encoding layer (e.g., the number of channels that the positional encoding is based on).

[0169] - Fourth syntax element, which indicates that the position of the positional coding layer is variable, and the position of the positional coding layer within the network;

[0170] - The fifth syntax element indicates the input type (e.g., coordinates, coordinates and box numbers, higher-dimensional features, etc.) of the location encoding layer that generates location embedding features.

[0171] Figure 12 This is a flowchart illustrating an example encoding technique of this disclosure. Figure 12 The technology can be derived from any AI-based point cloud encoder disclosed herein (including point cloud encoder 200). Figure 1 and Figure 2 ) and point cloud encoder 910 ( Figure 9)) Execution. Encoder network 700 ( Figure 7 The diagram illustrates an example of a deep learning network used in conjunction with the techniques disclosed herein.

[0172] In one example of this disclosure, a point cloud encoder 910 may be configured to receive a point cloud data frame (1200) and encode (1210) the coordinates of the geometry of the point cloud data frame using a deep learning network. The deep learning network (where encoder network 700) Figure 7 (This is just one example) includes one or more layers configured to generate coordinates of a first feature of a geometric structure. The deep learning network also includes at least one positional encoding layer configured to generate coordinates of a second feature of the geometric structure and to combine the first and second features to generate a higher-dimensional feature. To combine the first and second features to generate the higher-dimensional feature, the positional encoding layer may be configured to cascade the first and second features. In one example, the first feature has a feature size of 64 and the higher-dimensional feature has a feature size of 192. The point cloud encoder 910 is further configured to output an output tensor (1220) comprising the encoded coordinates and the higher-dimensional feature.

[0173] In some examples, the deep learning network includes one or more layers configured to reduce the coordinates of a geometry to generate reduced coordinates, and wherein the output tensor includes the reduced coordinates and higher-dimensional features. In a further example, at least one positional encoding layer of the deep learning network includes a first positional encoding layer that operates on the input of the deep learning network and a second positional encoding layer that operates on the reduced coordinates.

[0174] In another example, one or more layers of the deep learning network include three layers configured to progressively reduce the coordinates of the geometry to generate first reduced coordinates, second reduced coordinates, and a third reduced coordinate after three reductions. In this example, the output tensor includes the third reduced coordinates and higher-dimensional features. Further in this example, at least one positional encoding layer of the deep learning network includes a first positional encoding layer operating on the input of the deep learning network, a second positional encoding layer operating on the first reduced coordinates, a third positional encoding layer operating on the second reduced coordinates, and a fourth positional encoding layer operating on the third reduced coordinates. In one example, at least one of the second, third, or fourth positional encoding layers is positioned between the layer configured to reduce the coordinates of the geometry and the initial residual block.

[0175] In some examples, the at least one location coding layer includes a learned location coding layer. The learned location coding layer can be configured as a convolutional layer, a fully connected layer, or a multilayer perceptron.

[0176] In some examples, the deep learning network includes multiple positional encoding layers, where each positional encoding layer is configured to use a different set of weights. In other examples, the deep learning network includes multiple positional encoding layers, where each positional encoding layer is configured to use the same set of weights.

[0177] In other examples, the at least one positional coding layer includes one of a sine and cosine positional coding layer, a linear positional coding layer, a radial basis function positional coding layer, a Fermi-Dirac positional coding layer, a hybrid positional coding layer, a relative positional coding layer, or an image grid positional coding layer. Deep learning networks may include multiple positional coding layers, and these multiple positional coding layers may include at least two types of positional coding.

[0178] The point cloud encoder 910 can also be configured to encode attributes of point cloud data frames using a deep learning network. In this example, the deep learning network includes one or more layers configured to generate third features of the attributes, and the deep learning network also includes at least one location encoding layer configured to generate fourth features from the attributes and combine the third features with the fourth features to generate a second higher-dimensional feature.

[0179] The point cloud encoder 910 can also be configured to encode any combination of one or more syntax elements, the one or more syntax elements including: a first syntax element indicating whether positional encoding is enabled; a second syntax element indicating the positional encoding type of at least one positional encoding layer; a third syntax element indicating the feature size of at least one positional encoding layer; a fourth syntax element indicating the position of at least one positional encoding layer; and / or a fifth syntax element indicating the input type of at least one positional encoding layer.

[0180] The point cloud encoder 910 can also be configured to perform octree encoding on the coordinates of the output tensor to generate encoded coordinates, quantize higher-dimensional features to generate quantized higher-dimensional features, use an entropy model to perform arithmetic encoding on the quantized higher-dimensional features of the output tensor to generate encoded higher-dimensional features, and output encoded coordinates and encoded higher-dimensional features in an encoded bitstream.

[0181] Figure 13 This is a flowchart illustrating an example decoding technique of this disclosure. Figure 13 The technology can be derived from any AI-based point cloud decoder disclosed herein (including point cloud decoder 300). Figure 1 and Figure 3 ) and point cloud decoder 960 ( Figure 9 )) Execution. Decoder network 800 ( Figure 8The diagram illustrates an example of a deep learning network used in conjunction with the techniques disclosed herein.

[0182] In one example, the point cloud decoder 960 can be configured to receive frames of the encoded point cloud data, wherein the encoded point cloud data includes an input tensor that includes encoded coordinates and corresponding features (1300). The point cloud decoder 960 can also be configured to decode the coordinates of the geometry of the frame of the encoded point cloud data using a deep learning network and the input tensor to generate decoded coordinates (1310).

[0183] Deep learning network (of which the decoder network is 800) Figure 8 (This is just one example) may include one or more layers configured to generate a first feature of the coordinates of a geometric structure. The deep learning network also includes at least one position encoding layer configured to generate a second feature of the coordinates of the geometric structure and to combine the first and second features to generate a higher-dimensional feature. The deep learning network also includes one or more classifier layers configured to classify the higher-dimensional feature to generate decoded coordinates. To combine the first and second features to generate the higher-dimensional feature, the position encoding layer may be configured to cascade the first and second features. In one example, the first feature has a feature size of 64 and the higher-dimensional feature has a feature size of 192. The point cloud decoder 960 may also be configured to output decoded coordinates (1320) in the decoded point cloud.

[0184] In one example, the input tensor's coordinates include reduced coordinates and the deep learning network includes one or more layers configured to amplify the reduced coordinates to generate amplified coordinates.

[0185] In one example, at least one positional encoding layer of a deep learning network includes a first positional encoding layer that operates on the input of the deep learning network and a second positional encoding layer that operates on magnified coordinates.

[0186] In another example, one or more layers of the deep learning network include three layers configured to progressively enlarge the reduced coordinates to generate first enlarged coordinates, second enlarged coordinates, and third enlarged coordinates. In this example, at least one positional encoding layer of the deep learning network includes a first positional encoding layer that operates on the input of the deep learning network, a second positional encoding layer that operates on the first enlarged coordinates, a third positional encoding layer that operates on the second enlarged coordinates, and a fourth positional encoding layer that operates on the third enlarged coordinates. At least one of the second, third, or fourth positional encoding layers may be positioned between the layer configured to enlarge the coordinates and the initial residual block.

[0187] In some examples, the at least one location coding layer includes a learned location coding layer. The learned location coding layer can be configured as a convolutional layer, a fully connected layer, or a multilayer perceptron.

[0188] In some examples, the deep learning network includes multiple positional encoding layers, where each positional encoding layer is configured to use a different set of weights. In other examples, the deep learning network includes multiple positional encoding layers, where each positional encoding layer is configured to use the same set of weights.

[0189] In other examples, the at least one positional coding layer includes one of a sine and cosine positional coding layer, a linear positional coding layer, a radial basis function positional coding layer, a Fermi-Dirac positional coding layer, a hybrid positional coding layer, a relative positional coding layer, or an image grid positional coding layer. Deep learning networks may include multiple positional coding layers, and these multiple positional coding layers may include at least two types of positional coding.

[0190] In some examples, the point cloud decoder 960 may also be configured to use a deep learning network to decode attributes of frames of encoded point cloud data to generate decoded attributes. In this example, the deep learning network includes one or more layers configured to generate a third feature of the attribute. The deep learning network also includes at least one location encoding layer configured to generate a fourth feature from the attribute and combine the third and fourth features to generate a second higher-dimensional feature. The deep learning network also includes one or more layers configured to classify the higher-dimensional feature to generate the decoded attribute.

[0191] In another example, the point cloud decoder 960 may also be configured to decode one or more syntax elements, the one or more syntax elements including: a first syntax element indicating whether positional encoding is enabled; a second syntax element indicating the positional encoding type of at least one positional encoding layer; a third syntax element indicating the feature size of at least one positional encoding layer; a fourth syntax element indicating the position of at least one positional encoding layer; and / or a fifth syntax element indicating the input type of at least one positional encoding layer.

[0192] The point cloud decoder 960 can also be configured to perform octree decoding on the encoded coordinates in the encoded bitstream to recover the encoded coordinates in the input tensor, and to perform arithmetic decoding on the encoded features in the encoded bitstream to recover the corresponding features in the input tensor.

[0193] Figure 14This is a conceptual diagram illustrating an example ranging system 1400 that can be used in conjunction with one or more techniques of AI-based point cloud coding using location coding as disclosed herein. Figure 14 In one example, the ranging system 1400 includes an illuminator 1402 and a sensor 1404. The illuminator 1402 may emit light 1406. In some examples, the illuminator 1402 may emit light 1406 as one or more laser beams. Light 1406 may be of one or more wavelengths, such as infrared wavelengths or visible light wavelengths. In other examples, light 1406 is not a coherent laser. When light 1406 encounters an object (such as object 1408), light 1406 produces a return light 1410. The return light 1410 may include backscattered light and / or reflected light. The return light 1410 may pass through a lens 1411, which guides the return light 1410 to produce an image 1412 of object 1408 on the sensor 1404. The sensor 1404 generates a signal 1414 based on the image 1412. The image 1412 may include points (e.g., as shown by...). Figure 14 The set of points represented by the small dots in image 1412.

[0194] In some examples, illuminator 1402 and sensor 1404 may be mounted on a rotating structure, allowing illuminator 1402 and sensor 1404 to capture a 360-degree view of the environment (e.g., a rotating LIDAR sensor). In other examples, ranging system 1400 may include one or more optical components (e.g., mirrors, collimators, diffraction gratings, etc.) that enable illuminator 1402 and sensor 1404 to detect objects within a specific range (e.g., up to 360 degrees). Although Figure 14 The example shows only a single illuminator 1402 and sensor 1404, but the ranging system 1400 may include multiple sets of illuminators and sensors.

[0195] In some examples, illuminator 1402 generates a structured light pattern. In such examples, ranging system 1400 may include multiple sensors 1404 on which corresponding images of the structured light pattern are formed. Ranging system 1400 can use the differences between the images of the structured light pattern to determine the distance to object 1408 from which the structured light pattern backscatters. When object 1408 is relatively close to sensor 1404 (e.g., 0.2 meters to 2 meters), the structured light-based ranging system can have a high level of accuracy (e.g., sub-millimeter accuracy). This high level of accuracy can be useful in facial recognition applications such as unlocking mobile devices (e.g., mobile phones, tablets, etc.) and for security applications.

[0196] In some examples, the ranging system 1400 is a time-of-flight (ToF) based system. In some examples of the ToF-based ranging system 1400, an illuminator 1402 generates pulses of light. In other words, the illuminator 1402 can modulate the amplitude of the emitted light 1406. In such examples, a sensor 1404 detects the return light 1410 from the pulse of light 1406 generated by the illuminator 1402. The ranging system 1400 can then determine the distance to an object 1408 from which the light 1406 backscatters, based on the delay between the emission of the light 1406 and the detection of the light, and the known speed of light in air. In some examples, the illuminator 1402 can modulate the phase of the emitted light 1406 instead of (or in addition to) modulating the amplitude of the emitted light 1406. In such an example, sensor 1404 can detect the phase of the return light 1410 from object 1408, and use the speed of light and the time difference between when illuminator 1402 generates light 1406 at a particular phase and when sensor 1404 detects the return light 1410 at that particular phase to determine the distance to a point on object 1408.

[0197] In other examples, point clouds can be generated without using illuminator 1402. For example, in some examples, sensor 1404 of ranging system 1400 may include two or more optical cameras. In such examples, ranging system 1400 may use optical cameras to capture stereo images of the environment including object 1408. Ranging system 1400 may include point cloud generator 1416 that can calculate differences between locations in the stereo image. Ranging system 1400 can then use these differences to determine distances to locations shown in the stereo image. Based on these distances, point cloud generator 1416 can generate point clouds.

[0198] Sensor 1404 can also detect other properties of object 1408, such as color and reflectivity information. Figure 14 In the example, point cloud generator 1416 can generate a point cloud based on signal 1414 generated by sensor 1404. Ranging system 1400 and / or point cloud generator 1416 can form data source 104 ( Figure 1 As part of this disclosure, the point cloud generated by the ranging system 1400 can be encoded and / or decoded according to any technique disclosed herein for point cloud compression using position coding. Inter-frame prediction and residual prediction as described in this disclosure can reduce the size of the encoded data.

[0199] Figure 15 This is a conceptual diagram illustrating an example of a vehicle-based scene that demonstrates the use of one or more techniques for point cloud compression using location encoding as disclosed herein. Figure 15 In the example, vehicle 1500 includes a ranging system 1502. The ranging system 1502 can be used to measure distances via... Figure 14 The methods discussed are used to achieve this. Although Figure 15 As not shown in the example, vehicle 1500 may also include a data source (such as data source 104). Figure 1 )) and point cloud encoders (such as point cloud encoder 200 ( Figure 1 )).exist Figure 15 In the example, ranging system 1502 emits laser beams 1504, which are reflected from pedestrians 1506 or other objects on the road. The data source of vehicle 1500 can generate a point cloud based on the signal generated by ranging system 1502. A point cloud encoder of vehicle 1500 can encode the point cloud to generate a bitstream 1508, such as a geometric bitstream (…). Figure 2 ) and attribute bitstream ( Figure 2 Inter-frame prediction and residual prediction can reduce the size of the geometric bitstream. The bitstream 1508 can contain many fewer bits than the uncoded point cloud obtained by the point cloud encoder.

[0200] The output interface of vehicle 1500 (e.g., output interface 108) Figure 1 The vehicle 1500 can send bitstream 1508 to one or more other devices. Bitstream 1508 may include significantly fewer bits than the uncoded point cloud obtained by the point cloud encoder. Therefore, the vehicle 1500 may be able to send bitstream 1508 to other devices faster than uncoded point cloud data. Additionally, bitstream 1508 may require less data storage capacity on the device.

[0201] exist Figure 15 In the example, vehicle 1500 can send bitstream 1508 to another vehicle 1510. Vehicle 1510 may include a point cloud decoder, such as point cloud decoder 300. Figure 1 The point cloud decoder of vehicle 1510 can decode bitstream 1508 to reconstruct the point cloud. Vehicle 1510 can use the reconstructed point cloud for various purposes. For example, vehicle 1510 can determine, based on the reconstructed point cloud, that pedestrian 1506 is on the road ahead of vehicle 1500, and therefore begin to decelerate, for example, even before the driver of vehicle 1510 becomes aware of pedestrian 1506 on the road. Thus, in some examples, vehicle 1510 can perform autonomous navigation operations based on the reconstructed point cloud.

[0202] Additionally or alternatively, vehicle 1500 may send bit stream 1508 to server system 1512. Server system 1512 may use bit stream 1508 for various purposes. For example, server system 1512 may store bit stream 1508 for subsequent reconstruction of the point cloud. In this example, server system 1512 may use the point cloud together with other data (e.g., vehicle telemetry data generated by vehicle 1500) to train an autonomous driving system. In other examples, server system 1512 may store bit stream 1508 for subsequent reconstruction for accident forensic investigations.

[0203] Figure 16 This is a conceptual diagram illustrating an example extended reality system that may utilize one or more techniques of point cloud compression using location encoding as disclosed herein. Extended reality (XR) is a term used to cover a range of technologies including augmented reality (AR), mixed reality (MR), and virtual reality (VR). Figure 16 In the example, user 1600 is located at a first location 1602. User 1600 wears an XR headset 1604. Alternatively, user 1600 may use a mobile device (e.g., a mobile phone, tablet, etc.) with XR headset 1604. XR headset 1604 includes a depth sensor, such as a ranging system, which detects the position of points on object 1606 at location 1602. The data source of XR headset 1604 may use signals generated by the depth sensor to generate a point cloud representation of object 1606 at location 1602. XR headset 1604 may include a point cloud encoder (e.g., [missing information]) configured to encode the point cloud to generate a bitstream 1608. Figure 1 (Point cloud encoder 200). Inter-frame prediction and residual prediction as described in this disclosure can reduce the size of the bitstream 1608.

[0204] XR headset 1604 can send bitstream 1608 (e.g., via a network, such as the Internet) to XR headset 1610 worn by user 1612 at second location 1614. XR headset 1610 can decode bitstream 1608 to reconstruct a point cloud. XR headset 1610 can use the point cloud to generate an XR visualization (e.g., AR visualization, MR visualization, VR visualization) representing object 1606 at location 1602. Thus, in some examples, such as when XR headset 1610 generates a VR visualization, user 1612 may have a 3D immersive experience of location 1602. In some examples, XR headset 1610 can determine the location of virtual objects based on the reconstructed point cloud. For example, XR headset 1610 can determine, based on the reconstructed point cloud, that the environment (e.g., location 1602) includes flat surfaces, and then determine that virtual objects (e.g., cartoon characters) should be positioned on that flat surface. The XR headset 1610 can generate XR visualizations in which virtual objects are located at a defined position. For example, the XR headset 1610 can show a cartoon character sitting on a flat surface.

[0205] Figure 17 This is a conceptual diagram illustrating an example mobile device system that can use one or more techniques for point cloud compression using location encoding as disclosed herein. Figure 17 In the example, mobile device 1700, such as a mobile phone or tablet computer (e.g., a wireless communication device), includes a ranging system (such as a LiDAR system) that detects the location of points on object 1702 within the environment of mobile device 1700. The data source of mobile device 1700 can use signals generated by a depth detection sensor to generate a point cloud representation of object 1702. Mobile device 1700 may include a point cloud encoder configured to encode the point cloud to generate a bitstream 1704 (e.g., [missing information]). Figure 1 The point cloud encoder 170). Figure 17In the example, mobile device 1700 can send a bitstream to remote device 1706 (such as a server system or other mobile device). Inter-frame prediction and residual prediction as described in this disclosure can reduce the size of bitstream 1704. Remote device 1706 can decode bitstream 1704 to reconstruct a point cloud. Remote device 1706 can use the point cloud for various purposes. For example, remote device 1706 can use the point cloud to generate an environmental map of mobile device 1700. For example, remote device 1706 can generate a map of the interior of a building based on the reconstructed point cloud. In another example, remote device 1706 can generate an image (e.g., computer graphics) based on the point cloud. For example, remote device 1706 can use points in the point cloud as vertices of polygons and use the color attributes of the points as the basis for coloring the polygons. In some examples, remote device 1706 can use the reconstructed point cloud for facial recognition or other security applications.

[0206] The following numbered clauses illustrate one or more aspects of the devices and technologies described in this disclosure.

[0207] Clause 1. An apparatus configured to encode a point cloud, the apparatus comprising: one or more memories configured to store point cloud data; and processing circuitry communicating with the one or more memories, the processing circuitry configured to: receive a point cloud data frame; encode coordinates of a geometric structure of the point cloud data frame using a deep learning network, wherein the deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometry, and wherein the deep learning network further includes at least one position encoding layer configured to generate a second feature of the coordinates of the geometry and combine the first feature and the second feature to generate a higher-dimensional feature; and output an output tensor comprising the encoded coordinates and the higher-dimensional feature.

[0208] Clause 2. The apparatus of Clause 1, wherein the deep learning network includes one or more layers configured to reduce the coordinates of the geometry to generate reduced coordinates, and wherein the output tensor includes the reduced coordinates and the higher-dimensional features.

[0209] Clause 3. The apparatus according to Clause 2, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer that operates on the input of the deep learning network and a second position encoding layer that operates on the reduced coordinates.

[0210] Clause 4. An apparatus according to any one of Clauses 1 to 3, wherein the one or more layers of the deep learning network comprise three layers configured to progressively reduce the coordinates of the geometry to generate first reduced coordinates, second reduced coordinates, and third reduced coordinates, and wherein the output tensor comprises the third reduced coordinates after three reductions and the higher-dimensional features.

[0211] Clause 5. The apparatus according to Clause 4, wherein the at least one position encoding layer of the deep learning network includes a first position encoding layer operating on the input of the deep learning network, a second position encoding layer operating on the first reduced coordinates, a third position encoding layer operating on the second reduced coordinates, and a fourth position encoding layer operating on the third reduced coordinates.

[0212] Clause 6. The apparatus according to Clause 5, wherein at least one of the second position encoding layer, the third position encoding layer, or the fourth position encoding layer is positioned between the layer configured to reduce the coordinates of the geometry and the initial residual block.

[0213] Clause 7. The apparatus according to any one of Clauses 1 to 6, wherein the at least one position coding layer comprises a learned position coding layer.

[0214] Clause 8. The apparatus according to Clause 7, wherein the learned position encoding layer is configured as a convolutional layer, a fully connected layer, or a multilayer perceptron.

[0215] Clause 9. An apparatus according to any one of Clauses 1 to 8, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use a different set of weights.

[0216] Clause 10. An apparatus according to any one of Clauses 1 to 8, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use the same set of weights.

[0217] Clause 11. The apparatus according to any one of Clauses 1 to 10, wherein the at least one position coding layer comprises one of a sine and cosine position coding layer, a linear position coding layer, a radial basis function position coding layer, a Fermi-Dirac position coding layer, a hybrid position coding layer, a relative position coding layer, or an image grid position coding layer.

[0218] Clause 12. An apparatus according to any one of Clauses 1 to 11, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein the plurality of positional encoding layers comprises at least two types of positional encoding.

[0219] Clause 13. The apparatus according to any one of Clauses 1 to 12, wherein the position encoding layer is configured to cascade the first feature and the second feature in order to combine the first feature and the second feature to generate a higher-dimensional feature.

[0220] Clause 14. The apparatus according to Clause 13, wherein the first feature has a feature size of 64, and wherein the higher-dimensional feature has a feature size of 192.

[0221] Clause 15. The apparatus according to Clauses 1 to 14, wherein the processing circuitry is further configured to: encode an attribute of the point cloud data frame using the deep learning network, wherein the deep learning network includes one or more layers configured to generate a third feature of the attribute, and wherein the deep learning network further includes at least one location encoding layer configured to generate a fourth feature from the attribute and combine the third feature with the fourth feature to generate a second higher-dimensional feature.

[0222] Clause 16. The apparatus according to Clauses 1 to 15, wherein the processing circuitry is further configured to: encode one or more syntax elements, the one or more syntax elements comprising: a first syntax element indicating whether positional encoding is enabled; a second syntax element indicating the positional encoding type of the at least one positional encoding layer; a third syntax element indicating the feature size of the at least one positional encoding layer; a fourth syntax element indicating the position of the at least one positional encoding layer; or a fifth syntax element indicating the input type of the at least one positional encoding layer.

[0223] Clause 17. The apparatus according to any one of Clauses 1 to 16, wherein the processing circuitry is further configured to: perform octree encoding on the coordinates of the output tensor to generate the encoded coordinates; quantize the higher-dimensional features to generate quantized higher-dimensional features; perform arithmetic encoding on the quantized higher-dimensional features of the output tensor using an entropy model to generate encoded higher-dimensional features; and output the encoded coordinates and the encoded higher-dimensional features in an encoded bitstream.

[0224] Clause 18. The apparatus according to any one of Clauses 1 to 17, the apparatus further comprising a sensor configured to capture the point cloud data frame.

[0225] Clause 19. A method for encoding a point cloud, the method comprising: receiving a point cloud data frame; encoding coordinates of a geometric structure of the point cloud data frame using a deep learning network, wherein the deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometry, and wherein the deep learning network further includes at least one position encoding layer configured to generate a second feature of the coordinates of the geometry and to combine the first feature and the second feature to generate a higher-dimensional feature; and outputting an output tensor comprising the encoded coordinates and the higher-dimensional feature.

[0226] Clause 20. The method according to Clause 19, wherein the deep learning network includes one or more layers configured to reduce the coordinates of the geometry to generate reduced coordinates, and wherein the output tensor includes the reduced coordinates and the higher-dimensional features.

[0227] Clause 21. The method according to Clause 20, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer operating on the input of the deep learning network and a second position encoding layer operating on the reduced coordinates.

[0228] Clause 22. The method according to any one of Clauses 19 to 21, wherein the one or more layers of the deep learning network comprise three layers configured to progressively reduce the coordinates of the geometry to generate first reduced coordinates, second reduced coordinates, and third reduced coordinates, and wherein the output tensor comprises the third reduced coordinates after three reductions and the higher-dimensional features.

[0229] Clause 23. The method according to Clause 22, wherein the at least one position encoding layer of the deep learning network includes a first position encoding layer operating on the input of the deep learning network, a second position encoding layer operating on the first reduced coordinates, a third position encoding layer operating on the second reduced coordinates, and a fourth position encoding layer operating on the third reduced coordinates.

[0230] Clause 24. The method according to Clause 23, wherein at least one of the second position coding layer, the third position coding layer, or the fourth position coding layer is positioned between the layer configured to reduce the coordinates of the geometry and the initial residual block.

[0231] Clause 25. The method according to any one of Clauses 19 to 24, wherein the at least one position coding layer comprises a learned position coding layer.

[0232] Clause 26. The method according to Clause 25, wherein the learned position encoding layer is configured as a convolutional layer, a fully connected layer, or a multilayer perceptron.

[0233] Clause 27. The method according to any one of Clauses 19 to 26, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use a different set of weights.

[0234] Clause 28. The method according to any one of Clauses 19 to 26, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use the same set of weights.

[0235] Clause 29. The method according to any one of Clauses 19 to 28, wherein the at least one position coding layer comprises one of a sine and cosine position coding layer, a linear position coding layer, a radial basis function position coding layer, a Fermi-Dirac position coding layer, a hybrid position coding layer, a relative position coding layer, or an image grid position coding layer.

[0236] Clause 30. The method according to any one of Clauses 19 to 29, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein the plurality of positional encoding layers comprises at least two types of positional encoding.

[0237] Clause 31. The method according to any one of Clauses 19 to 30, wherein the position encoding layer is configured to cascade the first feature and the second feature in order to combine the first feature and the second feature to generate a higher-dimensional feature.

[0238] Clause 32. The method according to Clause 31, wherein the first feature has a feature size of 64, and wherein the higher-dimensional feature has a feature size of 192.

[0239] Clause 33. The method according to Clauses 19 to 32, further comprising: encoding an attribute of the point cloud data frame using the deep learning network, wherein the deep learning network includes one or more layers configured to generate a third feature of the attribute, and wherein the deep learning network further includes at least one location encoding layer configured to generate a fourth feature from the attribute and combine the third feature with the fourth feature to generate a second higher-dimensional feature.

[0240] Clause 34. The method according to Clauses 19 to 33, the method further comprising: encoding one or more syntax elements, the one or more syntax elements comprising: a first syntax element indicating whether positional encoding is enabled; a second syntax element indicating the positional encoding type of the at least one positional encoding layer; a third syntax element indicating the feature size of the at least one positional encoding layer; a fourth syntax element indicating the position of the at least one positional encoding layer; or a fifth syntax element indicating the input type of the at least one positional encoding layer.

[0241] Clause 35. The method according to any one of Clauses 19 to 34, the method further comprising: performing octree encoding on the coordinates of the output tensor to generate the encoded coordinates; quantizing the higher-dimensional features to generate quantized higher-dimensional features; arithmetic encoding the quantized higher-dimensional features of the output tensor using an entropy model to generate encoded higher-dimensional features; and outputting the encoded coordinates and the encoded higher-dimensional features in an encoded bitstream.

[0242] Clause 36. The method according to any one of Clauses 19 to 35, the method further comprising: capturing the point cloud data frame using a sensor.

[0243] Clause 37. An apparatus configured to decode coded point cloud data, the apparatus comprising: one or more memories configured to store the coded point cloud data; and processing circuitry communicating with the one or more memories, the processing circuitry configured to: receive frames of the coded point cloud data, wherein the coded point cloud data includes an input tensor including coded coordinates and corresponding features; decode coordinates of a geometric structure of the frames of the coded point cloud data using a deep learning network and the input tensor to generate decoded coordinates, wherein the deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometric structure, and wherein the deep learning network further includes at least one location encoding layer configured to generate a second feature of the coordinates of the geometric structure and combine the first feature and the second feature to generate a higher-dimensional feature, and wherein the deep learning network further includes one or more classifier layers configured to classify the higher-dimensional feature to generate the decoded coordinates; and output the decoded coordinates in the decoded point cloud.

[0244] Clause 38. The apparatus of Clause 37, wherein the coordinates of the input tensor include reduced coordinates, and wherein the deep learning network includes one or more layers configured to amplify the reduced coordinates to generate amplified coordinates.

[0245] Clause 39. The apparatus of Clause 38, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer operating on the input of the deep learning network and a second position encoding layer operating on the magnified coordinates.

[0246] Clause 40. The apparatus according to any one of Clauses 38 to 39, wherein the one or more layers of the deep learning network comprise three layers configured to progressively enlarge the reduced coordinates to generate a first enlarged coordinate, a second enlarged coordinate, and a third enlarged coordinate.

[0247] Clause 41. The apparatus of Clause 40, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer operating on the input of the deep learning network, a second position encoding layer operating on the first magnified coordinates, a third position encoding layer operating on the second magnified coordinates, and a fourth position encoding layer operating on the third magnified coordinates.

[0248] Clause 42. The apparatus of Clause 41, wherein at least one of the second position encoding layer, the third position encoding layer, or the fourth position encoding layer is positioned between the layer configured to amplify the coordinates and the initial residual block.

[0249] Clause 43. The apparatus according to any one of Clauses 37 to 42, wherein the at least one position coding layer comprises a learned position coding layer.

[0250] Clause 44. The apparatus of Clause 43, wherein the learned position encoding layer is configured as a convolutional layer, a fully connected layer, or a multilayer perceptron.

[0251] Clause 45. The apparatus of any one of Clauses 37 to 44, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use a different set of weights.

[0252] Clause 46. The apparatus according to any one of Clauses 37 to 44, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use the same set of weights.

[0253] Clause 47. The apparatus according to any one of Clauses 37 to 46, wherein the at least one position coding layer comprises one of a sine and cosine position coding layer, a linear position coding layer, a radial basis function position coding layer, a Fermi-Dirac position coding layer, a hybrid position coding layer, a relative position coding layer, or an image grid position coding layer.

[0254] Clause 48. An apparatus according to any one of Clauses 37 to 47, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein the plurality of positional encoding layers comprises at least two types of positional encoding.

[0255] Clause 49. The apparatus according to any one of Clauses 37 to 48, wherein the position encoding layer is configured to cascade the first feature and the second feature in order to combine the first feature and the second feature to generate a higher-dimensional feature.

[0256] Clause 50. The apparatus according to Clause 49, wherein the first feature has a feature size of 64, and wherein the higher-dimensional feature has a feature size of 192.

[0257] Clause 51. The apparatus according to any one of Clauses 37 to 50, wherein the processing circuitry is further configured to: decode the attributes of the frame of the encoded point cloud data using the deep learning network to generate decoded attributes, wherein the deep learning network includes one or more layers configured to generate a third feature of the attribute, wherein the deep learning network further includes at least one position encoding layer configured to generate a fourth feature from the attribute and combine the third feature with the fourth feature to generate a second higher-dimensional feature, and wherein the deep learning network further includes one or more layers configured to classify the higher-dimensional feature to generate the decoded attributes.

[0258] Clause 52. The apparatus according to Clauses 37 to 51, wherein the processing circuitry is further configured to: decode one or more syntax elements, the one or more syntax elements comprising: a first syntax element indicating whether positional encoding is enabled; a second syntax element indicating the positional encoding type of the at least one positional encoding layer; a third syntax element indicating the feature size of the at least one positional encoding layer; a fourth syntax element indicating the position of the at least one positional encoding layer; or a fifth syntax element indicating the input type of the at least one positional encoding layer.

[0259] Clause 53. The apparatus according to any one of Clauses 37 to 52, wherein the processing circuitry is further configured to: perform octree decoding on the encoded coordinates in the encoded bitstream to recover the encoded coordinates in the input tensor; and perform arithmetic decoding on the encoded features in the encoded bitstream to recover the corresponding features in the input tensor.

[0260] Clause 54. The apparatus according to any one of Clauses 37 to 53, the apparatus further comprising: a display configured to display the coded point cloud.

[0261] Clause 55. A method for decoding a point cloud, the method comprising: receiving a frame of encoded point cloud data, wherein the encoded point cloud data includes an input tensor, the input tensor including encoded coordinates and corresponding features; decoding coordinates of a geometric structure of the frame of the encoded point cloud data using a deep learning network and the input tensor to generate decoded coordinates, wherein the deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometry, and wherein the deep learning network further includes at least one position encoding layer configured to generate a second feature of the coordinates of the geometry and combine the first feature and the second feature to generate a higher-dimensional feature, and wherein the deep learning network further includes one or more classifier layers configured to classify the higher-dimensional feature to generate the decoded coordinates; and outputting the decoded coordinates in the decoded point cloud.

[0262] Clause 56. The method according to Clause 55, wherein the coordinates of the input tensor include reduced coordinates, and wherein the deep learning network includes one or more layers configured to amplify the reduced coordinates to generate amplified coordinates.

[0263] Clause 57. The method according to Clause 56, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer operating on the input of the deep learning network and a second position encoding layer operating on the magnified coordinates.

[0264] Clause 58. The method according to any one of Clauses 56 to 57, wherein the one or more layers of the deep learning network comprise three layers configured to progressively enlarge the reduced coordinates to generate a first enlarged coordinate, a second enlarged coordinate, and a third enlarged coordinate.

[0265] Clause 59. The method according to Clause 58, wherein the at least one position encoding layer of the deep learning network includes a first position encoding layer operating on the input of the deep learning network, a second position encoding layer operating on the first magnified coordinates, a third position encoding layer operating on the second magnified coordinates, and a fourth position encoding layer operating on the third magnified coordinates.

[0266] Clause 60. The method according to Clause 59, wherein at least one of the second position coding layer, the third position coding layer, or the fourth position coding layer is positioned between the layer configured to amplify the coordinates and the initial residual block.

[0267] Clause 61. The method according to any one of Clauses 55 to 60, wherein the at least one position coding layer comprises a learned position coding layer.

[0268] Clause 62. The method according to Clause 61, wherein the learned position encoding layer is configured as a convolutional layer, a fully connected layer, or a multilayer perceptron.

[0269] Clause 63. The method according to any one of Clauses 55 to 62, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use a different set of weights.

[0270] Clause 64. The method according to any one of Clauses 55 to 62, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use the same set of weights.

[0271] Clause 65. The method according to any one of Clauses 55 to 64, wherein the at least one position coding layer comprises one of a sine and cosine position coding layer, a linear position coding layer, a radial basis function position coding layer, a Fermi-Dirac position coding layer, a hybrid position coding layer, a relative position coding layer, or an image grid position coding layer.

[0272] Clause 66. The method according to any one of Clauses 55 to 65, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein the plurality of positional encoding layers comprises at least two types of positional encoding.

[0273] Clause 67. The method according to any one of Clauses 55 to 66, wherein, in order to combine the first feature and the second feature to generate a higher-dimensional feature, the positional encoding layer is configured to cascade the first feature and the second feature.

[0274] Clause 68. The method according to Clause 67, wherein the first feature has a feature size of 64, and wherein the higher-dimensional feature has a feature size of 192.

[0275] Clause 69. The method according to Clauses 55 to 68, the method further comprising: decoding the attributes of the frame of the encoded point cloud data using the deep learning network to generate decoded attributes, wherein the deep learning network includes one or more layers configured to generate a third feature of the attributes, wherein the deep learning network further includes at least one location encoding layer configured to generate a fourth feature from the attributes and combine the third feature with the fourth feature to generate a second higher-dimensional feature, and wherein the deep learning network further includes one or more layers configured to classify the higher-dimensional feature to generate the decoded attributes.

[0276] Clause 70. The method according to Clauses 55 to 69, further comprising: decoding one or more syntax elements, the one or more syntax elements comprising: a first syntax element indicating whether positional encoding is enabled; a second syntax element indicating the positional encoding type of the at least one positional encoding layer; a third syntax element indicating the feature size of the at least one positional encoding layer; a fourth syntax element indicating the position of the at least one positional encoding layer; or a fifth syntax element indicating the input type of the at least one positional encoding layer.

[0277] Clause 71. The method according to Clauses 55 to 70, the method further comprising: performing octree decoding on the encoded coordinates in the encoded bitstream to recover the encoded coordinates in the input tensor; and performing arithmetic decoding on the encoded features in the encoded bitstream to recover the corresponding features in the input tensor.

[0278] Clause 72. The method according to any one of Clauses 55 to 71, the method further comprising: displaying the coded point cloud.

[0279] Examples from the various aspects of this disclosure may be used individually or in any combination.

[0280] It should be recognized that, based on the examples, certain actions or events of any technique described herein may be performed in a different sequence, and may be added, combined, or omitted entirely (e.g., not all actions or events described are necessary for implementing the technique). Furthermore, in some examples, actions or events may be performed concurrently (e.g., through multithreading, interrupt handling, or multiple processors) rather than sequentially.

[0281] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0282] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs utilize lasers to optically reproduce data. The combinations described above should also be included within the scope of computer-readable media.

[0283] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, these techniques can be fully implemented in one or more circuit or logic elements.

[0284] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or IC sets (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Rather, as described above, various units can be combined in a codec hardware unit, or various units can be provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0285] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. An apparatus configured to encode a point cloud, the apparatus comprising: One or more memories configured to store point cloud data; and Processing circuitry, which communicates with the one or more memories, is configured to: Receive point cloud data frames; The coordinates of the geometric structure of the point cloud data frame are encoded using a deep learning network, wherein the deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometric structure, and wherein the deep learning network further includes at least one position encoding layer configured to generate a second feature of the coordinates of the geometric structure and combine the first feature and the second feature to generate a higher-dimensional feature; and The output includes the output tensor of the encoded coordinates and the higher-dimensional features.

2. The apparatus of claim 1, wherein the deep learning network includes one or more layers configured to reduce the coordinates of the geometry to generate reduced coordinates, and wherein the output tensor includes the reduced coordinates and the higher-dimensional features.

3. The apparatus of claim 2, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer that operates on the input of the deep learning network and a second position encoding layer that operates on the reduced coordinates.

4. The apparatus of claim 1, wherein the one or more layers of the deep learning network comprise three layers configured to progressively reduce the coordinates of the geometry to generate first reduced coordinates, second reduced coordinates, and third reduced coordinates, and wherein the output tensor comprises the third reduced coordinates after three reductions and the higher-dimensional features.

5. The apparatus of claim 4, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer operating on the input of the deep learning network, a second position encoding layer operating on the first reduced coordinates, a third position encoding layer operating on the second reduced coordinates, and a fourth position encoding layer operating on the third reduced coordinates.

6. The apparatus of claim 5, wherein at least one of the second position encoding layer, the third position encoding layer, or the fourth position encoding layer is positioned between the layer configured to reduce the coordinates of the geometry and the initial residual block.

7. The apparatus of claim 1, wherein the at least one position coding layer comprises a learned position coding layer.

8. The apparatus of claim 7, wherein the learned position encoding layer is configured as a convolutional layer, a fully connected layer, or a multilayer perceptron.

9. The apparatus of claim 1, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use a different set of weights.

10. The apparatus of claim 1, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use the same set of weights.

11. The apparatus of claim 1, wherein the at least one position coding layer comprises one of a sine and cosine position coding layer, a linear position coding layer, a radial basis function position coding layer, a Fermi-Dirac position coding layer, a hybrid position coding layer, a relative position coding layer, or an image grid position coding layer.

12. The apparatus of claim 1, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein the plurality of positional encoding layers comprises at least two types of positional encoding.

13. The apparatus of claim 1, wherein the position encoding layer is configured to cascade the first feature and the second feature in order to combine the first feature and the second feature to generate a higher-dimensional feature.

14. The apparatus of claim 13, wherein the first feature has a feature size of 64, and wherein the higher-dimensional feature has a feature size of 192.

15. The apparatus of claim 1, wherein the processing circuit is further configured to: The deep learning network is used to encode the attributes of the point cloud data frame, wherein the deep learning network includes one or more layers configured to generate a third feature of the attribute, and wherein the deep learning network further includes at least one location encoding layer configured to generate a fourth feature from the attribute and combine the third feature with the fourth feature to generate a second higher-dimensional feature.

16. The apparatus of claim 1, wherein the processing circuit is further configured to: Encoding one or more syntax elements, said one or more syntax elements including: The first syntax element indicates whether position encoding is enabled. The second syntax element indicates the position encoding type of the at least one position encoding layer. A third syntax element, which indicates the feature size of the at least one positional encoding layer. A fourth syntax element, which indicates the position of the at least one positional coding layer, or A fifth syntax element, which indicates the input type of the at least one positional coding layer.

17. The apparatus of claim 1, wherein the processing circuit is further configured to: Perform octree encoding on the coordinates of the output tensor to generate the encoded coordinates; The higher-dimensional features are quantized to generate quantized higher-dimensional features; The quantized higher-dimensional features of the output tensor are arithmetically encoded using an entropy model to generate encoded higher-dimensional features. as well as The encoded coordinates and the encoded higher-dimensional features are output in the encoded bitstream.

18. The apparatus according to claim 1, further comprising: A sensor configured to capture the point cloud data frame.

19. A method for encoding a point cloud, the method comprising: Receive point cloud data frames; The coordinates of the geometric structure of the point cloud data frame are encoded using a deep learning network, wherein the deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometric structure, and wherein the deep learning network further includes at least one position encoding layer configured to generate a second feature of the coordinates of the geometric structure and combine the first feature and the second feature to generate a higher-dimensional feature; and The output includes the output tensor of the encoded coordinates and the higher-dimensional features.

20. The method of claim 19, wherein the deep learning network includes one or more layers configured to reduce the coordinates of the geometry to generate reduced coordinates, and wherein the output tensor includes the reduced coordinates and the higher-dimensional features.

21. The method of claim 20, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer that operates on the input of the deep learning network and a second position encoding layer that operates on the reduced coordinates.

22. The method of claim 19, wherein the one or more layers of the deep learning network comprise three layers configured to progressively reduce the coordinates of the geometry to generate first reduced coordinates, second reduced coordinates, and third reduced coordinates, and wherein the output tensor comprises the third reduced coordinates after three reductions and the higher-dimensional features.

23. The method of claim 22, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer operating on the input of the deep learning network, a second position encoding layer operating on the first reduced coordinates, a third position encoding layer operating on the second reduced coordinates, and a fourth position encoding layer operating on the third reduced coordinates.

24. The method of claim 23, wherein at least one of the second position coding layer, the third position coding layer, or the fourth position coding layer is positioned between the layer configured to reduce the coordinates of the geometry and the initial residual block.

25. The method of claim 19, wherein the at least one position coding layer comprises a learned position coding layer.

26. The method of claim 25, wherein the learned position encoding layer is configured as a convolutional layer, a fully connected layer, or a multilayer perceptron.

27. The method of claim 19, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use a different set of weights.

28. The method of claim 19, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use the same set of weights.

29. The method of claim 19, wherein the at least one position coding layer comprises one of a sine and cosine position coding layer, a linear position coding layer, a radial basis function position coding layer, a Fermi-Dirac position coding layer, a hybrid position coding layer, a relative position coding layer, or an image grid position coding layer.

30. The method of claim 19, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein the plurality of positional encoding layers comprises at least two types of positional encoding.

31. The method of claim 19, wherein, in order to combine the first feature and the second feature to generate a higher-dimensional feature, the positional encoding layer is configured to cascade the first feature and the second feature.

32. The method of claim 31, wherein the first feature has a feature size of 64, and wherein the higher-dimensional feature has a feature size of 192.

33. The method according to claim 19, further comprising: The deep learning network is used to encode the attributes of the point cloud data frame, wherein the deep learning network includes one or more layers configured to generate a third feature of the attribute, and wherein the deep learning network further includes at least one location encoding layer configured to generate a fourth feature from the attribute and combine the third feature with the fourth feature to generate a second higher-dimensional feature.

34. The method according to claim 19, further comprising: Encoding one or more syntax elements, said one or more syntax elements including: The first syntax element indicates whether position encoding is enabled. The second syntax element indicates the position encoding type of the at least one position encoding layer. A third syntax element, which indicates the feature size of the at least one positional encoding layer. A fourth syntax element, which indicates the position of the at least one positional coding layer, or A fifth syntax element, which indicates the input type of the at least one positional coding layer.

35. The method according to claim 19, further comprising: Perform octree encoding on the coordinates of the output tensor to generate the encoded coordinates; The higher-dimensional features are quantized to generate quantized higher-dimensional features; The quantized higher-dimensional features of the output tensor are arithmetically encoded using an entropy model to generate encoded higher-dimensional features. as well as The encoded coordinates and the encoded higher-dimensional features are output in the encoded bitstream.

36. The method according to claim 19, further comprising: The point cloud data frames are captured using sensors.

37. An apparatus configured to decode coded point cloud data, the apparatus comprising: One or more memories configured to store the coded point cloud data; and Processing circuitry, which communicates with the one or more memories, is configured to: Receive a frame of the encoded point cloud data, wherein the encoded point cloud data includes an input tensor, and the input tensor includes encoded coordinates and corresponding features; The coordinates of the geometry of the frame of the encoded point cloud data are decoded using a deep learning network and the input tensor to generate decoded coordinates, wherein the deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometry, and wherein the deep learning network also includes at least one position encoding layer configured to generate a second feature of the coordinates of the geometry and combine the first feature and the second feature to generate a higher-dimensional feature, and wherein the deep learning network also includes one or more classifier layers configured to classify the higher-dimensional feature to generate the decoded coordinates; as well as Output the decoded coordinates from the decoded point cloud.

38. The apparatus of claim 37, wherein the coordinates of the input tensor include reduced coordinates, and wherein the deep learning network includes one or more layers configured to amplify the reduced coordinates to generate amplified coordinates.

39. The apparatus of claim 38, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer that operates on the input of the deep learning network and a second position encoding layer that operates on the magnified coordinates.

40. The apparatus of claim 38, wherein the one or more layers of the deep learning network comprise three layers configured to progressively amplify the reduced coordinates to generate a first amplified coordinate, a second amplified coordinate, and a third amplified coordinate.

41. The apparatus of claim 40, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer operating on the input of the deep learning network, a second position encoding layer operating on the first magnified coordinates, a third position encoding layer operating on the second magnified coordinates, and a fourth position encoding layer operating on the third magnified coordinates.

42. The apparatus of claim 41, wherein at least one of the second position encoding layer, the third position encoding layer, or the fourth position encoding layer is positioned between the layer configured to amplify the reduced coordinates and the initial residual block.

43. The apparatus of claim 37, wherein the at least one position coding layer comprises a learned position coding layer.

44. The apparatus of claim 43, wherein the learned position encoding layer is configured as a convolutional layer, a fully connected layer, or a multilayer perceptron.

45. The apparatus of claim 37, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use a different set of weights.

46. ​​The apparatus of claim 37, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use the same set of weights.

47. The apparatus of claim 37, wherein the at least one position coding layer comprises one of a sine and cosine position coding layer, a linear position coding layer, a radial basis function position coding layer, a Fermi-Dirac position coding layer, a hybrid position coding layer, a relative position coding layer, or an image grid position coding layer.

48. The apparatus of claim 37, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein the plurality of positional encoding layers comprises at least two types of positional encoding.

49. The apparatus of claim 37, wherein the position encoding layer is configured to cascade the first feature and the second feature in order to combine the first feature and the second feature to generate a higher-dimensional feature.

50. The apparatus of claim 49, wherein the first feature has a feature size of 64, and wherein the higher-dimensional feature has a feature size of 192.

51. The apparatus of claim 37, wherein the processing circuit is further configured to: The deep learning network is used to decode the attributes of the frame of the encoded point cloud data to generate decoded attributes, wherein the deep learning network includes one or more layers configured to generate a third feature of the attributes, wherein the deep learning network also includes at least one location encoding layer configured to generate a fourth feature from the attributes and combine the third feature with the fourth feature to generate a second higher-dimensional feature, and wherein the deep learning network also includes one or more layers configured to classify the higher-dimensional feature to generate the decoded attributes.

52. The apparatus of claim 37, wherein the processing circuit is further configured to: Decoding one or more syntax elements, said one or more syntax elements including: The first syntax element indicates whether position encoding is enabled. The second syntax element indicates the position encoding type of the at least one position encoding layer. A third syntax element, which indicates the feature size of the at least one positional encoding layer. A fourth syntax element, which indicates the position of the at least one positional coding layer, or A fifth syntax element, which indicates the input type of the at least one positional coding layer.

53. The apparatus of claim 37, wherein the processing circuit is further configured to: Perform octree decoding on the encoded coordinates in the encoded bitstream to recover the encoded coordinates in the input tensor; and Arithmetic decoding is performed on the encoded features in the encoded bitstream to recover the corresponding features in the input tensor.

54. The apparatus of claim 37, further comprising: A display configured to display the coded point cloud.

55. A method for decoding point clouds, the method comprising: Receive frames of encoded point cloud data, wherein the encoded point cloud data includes an input tensor, and the input tensor includes encoded coordinates and corresponding features; The coordinates of the geometry of the frame of the encoded point cloud data are decoded using a deep learning network and the input tensor to generate decoded coordinates, wherein the deep learning network includes one or more layers configured to generate a first feature of the coordinates of the geometry, and wherein the deep learning network also includes at least one position encoding layer configured to generate a second feature of the coordinates of the geometry and combine the first feature and the second feature to generate a higher-dimensional feature, and wherein the deep learning network also includes one or more classifier layers configured to classify the higher-dimensional feature to generate the decoded coordinates; as well as Output the decoded coordinates from the decoded point cloud.

56. The method of claim 55, wherein the coordinates of the input tensor include reduced coordinates, and wherein the deep learning network includes one or more layers configured to amplify the reduced coordinates to generate amplified coordinates.

57. The method of claim 56, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer that operates on the input of the deep learning network and a second position encoding layer that operates on the magnified coordinates.

58. The method of claim 56, wherein the one or more layers of the deep learning network comprise three layers configured to progressively enlarge the reduced coordinates to generate a first enlarged coordinate, a second enlarged coordinate, and a third enlarged coordinate.

59. The method of claim 58, wherein the at least one position encoding layer of the deep learning network comprises a first position encoding layer operating on the input of the deep learning network, a second position encoding layer operating on the first magnified coordinates, a third position encoding layer operating on the second magnified coordinates, and a fourth position encoding layer operating on the third magnified coordinates.

60. The method of claim 59, wherein at least one of the second position coding layer, the third position coding layer, or the fourth position coding layer is positioned between the layer configured to amplify the reduced coordinates and the initial residual block.

61. The method of claim 55, wherein the at least one position coding layer comprises a learned position coding layer.

62. The method of claim 61, wherein the learned position encoding layer is configured as a convolutional layer, a fully connected layer, or a multilayer perceptron.

63. The method of claim 55, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use a different set of weights.

64. The method of claim 55, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein each of the positional encoding layers is configured to use the same set of weights.

65. The method of claim 55, wherein the at least one position coding layer comprises one of a sine and cosine position coding layer, a linear position coding layer, a radial basis function position coding layer, a Fermi-Dirac position coding layer, a hybrid position coding layer, a relative position coding layer, or an image grid position coding layer.

66. The method of claim 55, wherein the deep learning network comprises a plurality of positional encoding layers, and wherein the plurality of positional encoding layers comprises at least two types of positional encoding.

67. The method of claim 55, wherein, in order to combine the first feature and the second feature to generate a higher-dimensional feature, the positional encoding layer is configured to cascade the first feature and the second feature.

68. The method of claim 67, wherein the first feature has a feature size of 64, and wherein the higher-dimensional feature has a feature size of 192.

69. The method according to claim 55, further comprising: The deep learning network is used to decode the attributes of the frame of the encoded point cloud data to generate decoded attributes, wherein the deep learning network includes one or more layers configured to generate a third feature of the attributes, wherein the deep learning network also includes at least one location encoding layer configured to generate a fourth feature from the attributes and combine the third feature with the fourth feature to generate a second higher-dimensional feature, and wherein the deep learning network also includes one or more layers configured to classify the higher-dimensional feature to generate the decoded attributes.

70. The method of claim 55, further comprising: Decoding one or more syntax elements, said one or more syntax elements including: The first syntax element indicates whether position encoding is enabled. The second syntax element indicates the position encoding type of the at least one position encoding layer. A third syntax element, which indicates the feature size of the at least one positional encoding layer. A fourth syntax element, which indicates the position of the at least one positional coding layer, or A fifth syntax element, which indicates the input type of the at least one positional coding layer.

71. The method according to claim 55, further comprising: Perform octree decoding on the encoded coordinates in the encoded bitstream to recover the encoded coordinates in the input tensor; as well as Arithmetic decoding is performed on the encoded features in the encoded bitstream to recover the corresponding features in the input tensor.

72. The method according to claim 55, further comprising: Display the coded point cloud 。

Images