Learning-based image coding for 3D point data

Abstract

This work involves the compression of point cloud data via alternative raster image representations, particularly for point clouds acquired from lidar sensors. In one embodiment, we present an image-based learning-based point cloud compression architecture that utilizes a 3D-to-2D image projection module called mesh alignment. Our proposal addresses the problem of existing distance image-based point cloud compression by a one-to-one assignment of the original Cartesian coordinates to a predefined mesh. The architecture we present can handle high-bit-depth lidar data and can also operate in a block-based manner.

Description

Technical Field

[0001] This embodiment generally relates to a method and apparatus for point cloud compression and processing. Background Technology

[0002] 3D point clouds are a universal data format spanning several business sectors, from autonomous driving, robotics, augmented reality / virtual reality (AR / VR), civil engineering, computer graphics to the animation / film industry. 3D LiDAR (light detection and ranging) sensors are already widely deployed in autonomous vehicles. Affordable LiDAR sensors have been released through Velodyne's Velabit, Apple's iPad Pro 2020, and Intel's RealSense LiDAR camera L515. With tremendous advancements in sensing technology, 3D point cloud data has become more practical than ever before and is expected to be the ultimate enabler in the aforementioned applications.

[0003] Point cloud data is also considered to consume a significant portion of network traffic, for example, in connected cars via 5G networks and in immersive communications (VR / AR). Efficient representation formats are essential for point cloud understanding and communication. In particular, for world modeling and sensing purposes, raw point cloud data needs to be properly organized and processed. Compression of raw point clouds is crucial when data needs to be stored and transmitted in relevant situations.

[0004] Furthermore, point clouds can represent sequential scans of the same scene containing multiple moving objects. This is called a dynamic point cloud, as opposed to a static point cloud captured from a static scene or static objects. Dynamic point clouds are typically organized into frames, with different frames captured at different times. Dynamic point clouds may require real-time or low-latency processing and compression. Summary of the Invention

[0005] According to one embodiment, a method for decoding point cloud data is proposed, comprising: decoding at least one 2D image, wherein each color component of the 2D image corresponds to the dimension of the 3D Cartesian coordinates of the point cloud data of the 3D point cloud; obtaining the 3D Cartesian coordinates of the point cloud data of the 3D point cloud from the color components of the at least one 2D image; and reconstructing the 3D point cloud from the 3D Cartesian coordinates of the 3D point cloud.

[0006] According to another embodiment, a method for encoding point cloud data is proposed, comprising: obtaining the point cloud data having 3D Cartesian coordinates; mapping the 3D Cartesian coordinates to Cartesian coordinates on a 2D grid to form a 2D image, wherein the corresponding color components of the 2D image correspond to the dimensions of the 3D Cartesian coordinates; and encoding the 2D image.

[0007] According to another embodiment, an apparatus for decoding point cloud data is proposed, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to: decode at least one 2D image, wherein each color component of the 2D image corresponds to a dimension of 3D Cartesian coordinates of point cloud data of a 3D point cloud; obtain the 3D Cartesian coordinates of the point cloud data of the 3D point cloud from the color components of the at least one 2D image; and reconstruct the 3D point cloud from the 3D Cartesian coordinates of the 3D point cloud.

[0008] According to another embodiment, an apparatus for encoding point cloud data is proposed, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to: obtain the point cloud data having 3D Cartesian coordinates; map the 3D Cartesian coordinates to Cartesian coordinates on a 2D grid to form a 2D image, wherein corresponding color components of the 2D image correspond to the dimensions of the 3D Cartesian coordinates; and encode the 2D image.

[0009] One or more embodiments also provide a computer program including instructions that, when executed by one or more processors, cause the one or more processors to perform an encoding or decoding method according to any of the above embodiments. One or more of these embodiments also provide a computer-readable storage medium storing instructions for encoding or decoding point cloud data according to the above methods.

[0010] One or more embodiments also provide a computer-readable storage medium storing video data generated according to the described method. One or more embodiments also provide a method and apparatus for transmitting or receiving video data generated according to the described method. Attached Figure Description

[0011] Figure 1 The figure shows a block diagram of a system in which aspects of this embodiment can be implemented.

[0012] Figure 2 The diagram illustrates an image-based point cloud compression system, where Proj2D is the projection onto spherical coordinates.

[0013] Figure 3 The diagram illustrates an image-based point cloud compression system, where SphQ is the projection onto spherical coordinates.

[0014] Figure 4 The diagram illustrates the projection from 3D Cartesian coordinates to spherical coordinates.

[0015] Figure 5 The figure shows an image-based point cloud compression system according to one embodiment.

[0016] Figure 6A The illustration shows a grid alignment module according to one embodiment, and Figure 6B The diagram illustrates the data flow of the grid alignment module.

[0017] Figure 7 The diagram illustrates an image-based point cloud compression system according to an embodiment, wherein the MSB and LSB are encoded separately.

[0018] Figure 8 The diagram illustrates an image-based point cloud compression system according to one embodiment, where the encoding of the LSB depends on the MSB.

[0019] Figure 9 The illustration shows an example of an image-based point cloud compression system according to one embodiment.

[0020] Figure 10 and Figure 11 The encoder and decoder are illustrated in the architecture of an image-based point cloud compression system according to one embodiment.

[0021] Figure 12 The diagram illustrates a point cloud compression system according to an embodiment, wherein blocks of a point cloud are compressed via image-based compression. Detailed Implementation

[0022] Figure 1 The diagram illustrates an example of a system in which various aspects and embodiments may be implemented. System 100 may be embodied as a device including the various components described below and configured to perform one or more of the aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100 may be embodied individually or in combination in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more of the aspects described in this application.

[0023] System 100 includes at least one processor 110 configured to execute instructions loaded thereon to implement various aspects, such as those described in this application. Processor 110 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 140 may include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0024] System 100 includes an encoder / decoder module 130 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents one or more modules that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Furthermore, as is known to those skilled in the art, the encoder / decoder module 130 may be implemented as a separate element of system 100 or may be incorporated within processor 110 as a combination of hardware and software.

[0025] Program code to be loaded onto processor 110 or encoder / decoder 130 to execute the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. According to various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of various items during execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0026] In several embodiments, the memory within processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be processor 110 or encoder / decoder module 130) is used for one or more of these functions. External memory may be memory 120 and / or storage device 140, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2, JPEG Pleno, MPEG-I, V-DMC, HEVC, or VVC.

[0027] Inputs to the components of system 100 can be provided by various input devices, as indicated in block 105. Such input devices include, but are not limited to: (i) an RF section that receives radio frequency (RF) signals, for example, transmitted over the air by a broadcasting device; (ii) a composite input terminal; (iii) a USB input terminal; and / or (iv) an HDMI input terminal.

[0028] In various embodiments, as known in the art, the input device of block 105 has associated respective input processing elements. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting the signal band to a band), (ii) down-converting the selected signal, (iii) band-limiting it again to a narrower band to select, for example, a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements to perform these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, down-converters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0029] Furthermore, the USB and / or HDMI terminals may include their respective interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed Solomon error correction) may be implemented as needed, for example, within a separate input processing IC or within processor 1010. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed, either within a separate interface IC or within processor 1010. Demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 110 and encoder / decoder 130, to operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0030] Various components of system 100 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data between them using a suitable connection arrangement 1140 (e.g., internal buses as known in the art, including I2C buses, wiring, and printed circuit boards).

[0031] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 190 may be implemented, for example, in a wired and / or wireless medium.

[0032] In various embodiments, data is streamed to system 100 using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signals in these embodiments are received via a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that delivers data via an HDMI connection to input block 105 to provide streaming data to system 100. Still other embodiments use an RF connection to input block 105 to provide streaming data to system 100.

[0033] System 100 can provide output signals to various output devices, including display 165, speaker 175, and other peripheral devices 185. In various examples of embodiments, other peripheral devices 185 include one or more of a standalone DVR, disc player, stereo system, lighting system, and other devices that provide functionality based on the output of system 100. In various embodiments, signaling such as AV link, CEC, or other communication protocols that enable inter-device control with or without user intervention is used to communicate control signals between system 100 and display 165, speaker 175, or other peripheral devices 185. Output devices can be communicatively coupled to system 100 via dedicated connections through their respective interfaces 160, 170, and 180. Alternatively, output devices can be connected to system 100 via communication interface 150 using communication channel 190. Display 165 and speaker 175 can be integrated into a single unit with other components of system 100 in electronic devices, such as a television. In various embodiments, display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0034] For example, if the RF portion of input 105 is part of a standalone set-top box, then display 165 and speaker 175 may alternatively be separate from one or more of the other components. In various embodiments where display 165 and speaker 175 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0035] Point cloud data use cases The automotive industry and autonomous vehicles are among the sectors where point clouds can be used. Autonomous vehicles should be able to "detect" their environment to make sound driving decisions based on the actual conditions of their immediate surroundings. Typical sensors, such as lidar, generate (dynamic) point clouds used by perception engines. These point clouds are not intended for human visual perception and are typically sparse, not necessarily colored, dynamic, and have a high acquisition frequency. They may possess other properties, such as reflectivity provided by lidar, as this property indicates the material of the sensed object and can aid in decision-making.

[0036] Virtual reality (VR) and immersive worlds are envisioned by many as the future of 2D video. In contrast to a standard television set where viewers can only see a virtual world in front of them, the basic idea is to immerse viewers in an environment surrounding them. Depending on the viewer's degree of freedom within the environment, there are several levels of immersion. Point clouds are a good candidate format for distributing VR content. They can be static or dynamic and are typically of moderate size, containing no more than a few million points at a time.

[0037] Point clouds can also be used for various purposes, such as cultural heritage / architecture, where objects like statues or buildings are scanned in 3D to share their spatial configuration without moving or accessing them. Furthermore, this is a way to preserve knowledge of an object even if it might be destroyed (e.g., a temple damaged by an earthquake). Such point clouds are typically static, colored, and large-scale.

[0038] Another use case is in topography and cartography, where 3D representations are used, and maps are not limited to a flat plane but can include topographic relief. Google Maps is now a good example of a 3D map, but it uses a grid instead of point clouds. However, point clouds may be a suitable data format for 3D maps, and such point clouds are typically static, colored, and large.

[0039] World modeling and sensing via point clouds may be a crucial technology that allows machines to gain knowledge about the 3D world around them (which is essential for the applications discussed in this article).

[0040] 3D point cloud data is essentially discrete samples of the surface of an object or scene. To fully represent the real world using these point samples, a massive number of points are actually needed. For example, a typical VR immersive scene contains millions of points, while a point cloud typically contains hundreds of millions. Therefore, processing such large-scale point clouds is computationally expensive, especially for consumer devices with limited computing power, such as smartphones, tablets, and car navigation systems.

[0041] The first step in any processing or inference of point clouds is to have an efficient storage method. To store and process the input point cloud at an affordable computational cost, one solution is to first downsample it, where the downsampled point cloud summarizes the geometry of the input point cloud while having far fewer points. The downsampled point cloud is then fed to subsequent machine tasks for further consumption. However, further reductions in storage space can be achieved by converting the raw point cloud data (raw or downsampled) into a bitstream via entropy coding techniques for lossless compression or feature map coding techniques for lossy compression.

[0042] There are three main methods for point cloud compression: octree-based compression, voxel-based compression, and image-based compression. Octtree-based compression first represents the point cloud as an octet tree and then losslessly compresses the tree structure based on the context of each node until the desired depth is reached. In contrast, both voxel-based and image-based compression methods extract features from the voxel and image representations of the point cloud (respectively), and then use entropy coding to encode quantized versions of these features into a bitstream.

[0043] Figure 2 The diagram illustrates a typical example of image-based point cloud compression. As can be seen from the diagram, the first step in image-based point cloud compression is to compress the 3D point cloud... Projection (210) to 2D image Next, any image encoder-decoder pair can be used to encode (220) and decode (230) the 2D image. Finally, the decoded 2D image is... Projecting (240) back into the 3D domain to obtain the reconstructed 3D point cloud. .

[0044] The traditional and well-known way to achieve projection is via 3D Cartesian coordinates. To spherical coordinates The projection (310) in the middle, such as Figure 3 As shown in the diagram. Then, an image encoder-decoder pair can be used to... Encoding (320) and decoding (330) are performed. Finally, the decoded... Projecting (340) back into the 3D domain to obtain the reconstructed 3D point cloud. .

[0045] from Figure 4 We can see the SphQ module (310), which consists of two steps. In the first step (410), the Cartesian coordinates are... Convert to its corresponding spherical coordinates The spherical coordinates are azimuth, elevation, and distance / depth values. Then, in the second step, only the elevation and azimuth are quantized to the desired grid resolution, followed by placing the corresponding distance / depth values ​​(420) on the quantized grid coordinates to form... This conversion is generally lossy in nature because (1) some precision of elevation and azimuth is lost during quantization, and (2) some depth values ​​are also lost depending on the specified resolution of the grid, i.e., the quantization of elevation and azimuth (e.g., Two points in the image are quantized to be the same point, and one depth of these two points is covered, thus resulting in a loss. It should be noted here that, in this case, the Proj3D module uses the decoded distance value and the quantized elevation (row) and azimuth (column) angles to transform from spherical coordinates back to 3D Cartesian coordinates. In this work, we focus on a better image-based representation of point clouds that avoids both problems and allows Cartesian coordinates to be directly projected onto the image without any information loss.

[0046] Figure 5 The figure shows an overall diagram of our proposed architecture for an image-based point cloud compression system according to one embodiment. Compared to the image-based compression system briefly described above, we can see that the contribution lies in the projection of the 3D point cloud onto a 2D image. The proposed mesh alignment module (510) uses Cartesian coordinates... Place directly into the image grid The proposed mesh alignment module aims to compress Cartesian coordinates, rather than some projected coordinates (e.g., depth values ​​in image compression), from the mesh alignment. A typical learning-based or non-learning-based image compression pipeline can be used to operate on the output of the proposed mesh alignment module to compress (520) and decompress (530) the raw point cloud. Proj3D (540) then extracts only the three color components (corresponding to the x, y, and z coordinates) from each pixel of the decoded image and places them in a list of 3D point locations.

[0047] Image generation module We now describe a novel image generation method, namely, Figure 6A The mesh alignment module is shown. As we can see, the first step (610) in the mesh alignment module is to obtain Cartesian coordinates. spherical projection Next, the spherical coordinates (elevation and azimuth coordinates) are matched one-to-one to the grid (630) using the grid matching module (620). Through the matching process, each spherical coordinate is associated with a grid point, and a matching index (640) representing the grid point index is obtained. The grid size is typically equal to or greater than the number of points in the point cloud to ensure no point loss during the transformation. Finally, the corresponding Cartesian coordinates are assigned (650) to the corresponding grid points using the matching index. In this way, all Cartesian coordinates are matched one-to-one to grid points, while some empty grid points (which default to zero values) may also be present. Therefore, we have matched the x, y, and z coordinates of the 3D Cartesian point cloud. Effectively convert to a 2D image with 3 channels (color components). ,like Figure 6A As shown, for example, the x, y, and z coordinates are transformed into the R, G, and B components of the 2D image. That is, each dimension of the Cartesian coordinates of the 3D point cloud is transformed into the color component of the 2D image.

[0048] exist Figure 6B The mathematical representation is also shown, where the input point cloud is... 3D points in Converted to spherical coordinates ( (middle), hence the elevation angle Converted into laser index (The number of angles equals the number of lasers, allowing the use of laser indices in the space of laser angles), and through factors Quantitative azimuth To generate q is selected based on the required resolution. Then, based on the results of grid matching (620), all Mapped one-to-one to predefined grid points (In the grid, 630), to generate point indices from the input cloud. Mapping to grid point indices in the grid (640), when Assigned to grid points At that time, this mapping is used to obtain a 2D image. .

[0049] Grid matching The mesh matching module (620) in our proposed architecture has the capability to map the projected spherical coordinates (elevation and azimuth values ​​only) to a predefined mesh. To achieve this, we can employ both conventional and learning-based methods. It should be noted that, for example, known knowledge about the lidar sensor can be used to define the predefined mesh. For instance, the Velodyne64 sensor has 64 lasers, each averaging approximately 2000 points. Therefore, a mesh of 64 rows and 2048 columns is sufficient for the point cloud from this sensor.

[0050] It should be emphasized here that the purpose of the mesh matching module is to achieve appropriate matching of 3D points to 2D images, so that the neighborhood relationships in 3D space are still approximately preserved in the 2D image. When neighborhood relationships are preserved, the resulting image retains the spatial correlation in the point cloud and can facilitate efficient compression of the resulting 2D image. Random mapping from 3D coordinates to 2D images is not suitable for compression.

[0051] In one embodiment, we use the well-known Moving Distance (EMD) (see "A metric for distributions with applications to image databases" by Rubner, Yossi et al., 6th International Conference on Computer Vision, IEEE, 1998) to find the corresponding mapping. EMD is a measure of the dissimilarity between two distributions (continuous or discrete) in a measurement space. It can be viewed as the minimum amount of work required to change one distribution to another. This distance implicitly preserves neighborhood information, as moving two neighboring points in one domain to distant points in another domain would require significantly more work.

[0052] In another embodiment, we can use methods such as RasterFairy or IsoMatch (see Fried, Ohad et al., “IsoMatch: Creating informative grid layouts,” Computer Graphics Forum, Vol. 34, No. 2, 2015), which can efficiently transform 2D point clouds into 2D meshes while attempting to preserve the neighborhood relationships present in the original 2D point cloud. These methods work by attempting to minimize an approximate objective function, which acts as a proxy for pairwise assignment objective functions.

[0053] In another embodiment, we can use either learning-based FoldingNet (see Yang, Yaoqing et al., “Foldingnet: Point cloud auto-encoder via deep grid deformation,” Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, 2018) or TearingNet (see Pang, Jiahao et al., “Tearingnet: Point cloud autoencoder to learn topology-friendly representations,” Proceedings of the IEEE / CVF International Conference on Computer Vision and Pattern Recognition, 2021) to transform the 2D mesh into projected spherical coordinates and thus find the associated mapping. Both methods use several learnable MLP (Multilayer Perceptron) layers to learn nonlinear mappings to fold (or tear and then fold) the fixed 2D mesh to a given point cloud. The learnable folding operation is not perfect, but using nearest neighbor search after folding can lead to the desired output of the mapping between the 2D mesh and the projected spherical coordinates.

[0054] High-bit depth compression using low-bit depth encoding In a typical LiDAR system, data is acquired as floating-point values ​​in meters and then quantized to the sensor's resolution, typically 1 mm. Given a typical LiDAR sensor range of 120 m, the quantized data ultimately reaches a bit depth of approximately 18 bits. Traditional image codecs typically process lower bit depth data, such as 8 or 10 bits. To compress this high bit depth data using a typical image compression pipeline, we propose dividing the bits into several smaller low bit depth blocks containing the highest and lowest (and possibly intermediate) significant bits. The choice of bit allocation between MSB and LSB can be user-defined and may depend on the type of LiDAR data being compressed.

[0055] Figure 7 The diagram illustrates this architecture according to one embodiment, where two separate image compressors are used to compress the most significant bit and the least significant bit, respectively. (As shown by...) Figure 7 We can see that when the high-level depth coordinates are... After projecting (710) onto the image / grid, the bits Split into two images MSB and LSB MSB images are typically very smooth and are transmitted via ENC because they contain more important information.LL (720) is compressed losslessly or with high fidelity. On the other hand, LSB images are compressed via ENC. L (730) is compressed in a lossy manner. At the decoder side, after both images pass through the corresponding decoder DEC... LL (740) and DEC L After decoding (750), combine (760) the highest effective image. and minimum effective image The information in the original 3D Cartesian coordinates is used to generate a lossy reconstruction. .

[0056] In such Figure 8 In another embodiment shown, the LSB encoder / decoder is not completely independent, but instead draws input from the MSB branch. The pipeline is similar to... Figure 7 In addition, the encoding (830) and decoding (850) of the LSB image are based on the encoded (820) and decoded MSB image (840), respectively.

[0057] It should be noted here that Figure 7 and Figure 8 The lossless encoder / decoder used for MSB can be any lossless (traditional or learning-based) image encoder / decoder pair. Furthermore, for the lossy encoder / decoder used for LSB, any existing lossy (traditional or learning-based) image encoder / decoder pair can be employed.

[0058] With traditional image compression codec JPEG2000 Figure 7 The example architecture shown in the settings is in Figure 9 In the above, the codec operates in lossless mode (920, 940) for MSB and in lossy mode (930, 950) for LSB.

[0059] This paper presents a traditional image compression codec, JPEG2000 (lossless), for MSB, and a learning-based scale-wise hyperprior model (lossy) for LSB (see Hu, Yueyu et al., “Learning end-to-end lossy image compression: A benchmark,” IEEE Transactions on Pattern Analysis and Machine Intelligence 44.8 (2021): 4194–4211). Figure 8 The example architectures shown in the examples are for the encoder and decoder, respectively. Figure 10 and Figure 11 middle.

[0060] Here, the learning-based model is conditioned on the input from the MSB branch during encoding / decoding. Within the learning-based model, This represents a 2D convolution with N output channels, a kernel size of kxk, and a downsampling factor of d. This represents a 2D convolution with N output channels, a kernel size of kxk, and an upsampling factor of d. Furthermore, ReLU (Revised Linear Unit) is a non-linear feature, Q represents quantization, AE / AD represents arithmetic encoding / decoding, and abs represents absolute value.

[0061] Specifically, on the encoder side, the LSB is encoded using a feature encoder (1010) and a super-prior encoder (1020). Both features and super-features are quantized (1030, 1050) and arithmetically encoded (1040, 1060). Furthermore, the super-features are arithmetically decoded (1070) and recovered (1080) to serve as input to the arithmetic encoder (1040) of that feature. On the decoder side, the MSB is reduced (1110) to be used for predicting LSB branches. The super-features are arithmetically decoded (1140) and recovered (1130) to serve as input to the arithmetic decoder (1150) of that feature. The output of the feature decoder (1120) is used to reconstruct the LSB, which is then combined with the MSB (1160) to reconstruct the point cloud.

[0062] Block-based coding So far, the description concerns the scenario of converting an entire point cloud (frame) into an image representation and then compressing it. However, in some cases, it is desirable to first divide the point cloud into several regions / blocks. And then compress each region / block, such as Figure 12 As shown in the figure. This can be achieved, for example, by first segmenting the (1210) point cloud using an octree, and then transforming and compressing each block via the proposed image-based compression. On the decoder side, the combination (1220) is decoded by the block. This is used to reconstruct the point cloud. It should be noted that, in order to encode the octree segmentation information, either a traditional or learning-based octree coding scheme can be used.

[0063] Various numerical values ​​are used in this application. Specific values ​​are used for illustrative purposes, and the aspects described are not limited to these specific values.

[0064] Various methods are described herein, and each of these methods includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined. Furthermore, in various embodiments, terms such as "first," "second," etc., may be used to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless explicitly stated otherwise, the use of such terms does not imply an ordering of the modified operations. Therefore, in this example, the first decoding need not be performed before the second decoding and may occur, for example, before, during, or within an overlapping time period of the second decoding.

[0065] The implementations and aspects described herein can be implemented, for example, as methods or processes, apparatus, software programs, data streams, or signals. Even if discussed only in the context of a single implementation form (e.g., discussed only as a method), implementations of the discussed features can be implemented in other forms (e.g., apparatus or program). Apparatus can be implemented, for example, in suitable hardware, software, and firmware. Methods can be implemented, for example, in apparatus, such as a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0066] The references to "an embodiment" or "an embodiment" or "an implementation" or "implementation", as well as other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the appearance of the phrases "in an embodiment" or "in an embodiment" or "in an implementation" or "in an implementation", as well as any other variations appearing throughout this application, do not necessarily refer to the same embodiment.

[0067] Furthermore, this application may refer to "determining" each piece of information. Determining information may include, for example, one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory.

[0068] Furthermore, this application may refer to "accessing" individual pieces of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0069] Furthermore, this application may refer to "receiving" individual pieces of information. Like "access," "receiving" is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) one or more of them. Moreover, "receiving" is generally referred to in one manner or another during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0070] To be understood, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many listed items as possible.

[0071] It will be apparent to those skilled in the art that implementations can generate various signals that are formatted to carry, for example, information that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of a spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

Claims

1. A method for decoding point cloud data, comprising: Decode at least one 2D image, wherein each color component of the 2D image corresponds to the dimension of the 3D Cartesian coordinates of the point cloud data of the 3D point cloud; The 3D Cartesian coordinates of the point cloud data obtained from the color components of the at least one 2D image; and The 3D point cloud is reconstructed from the 3D Cartesian coordinates of the 3D point cloud.

2. The method according to claim 1, wherein the decoding comprises: Decode at least the first 2D image and the second 2D image. The first 2D image corresponds to the first depth range of the point cloud data of the 3D point cloud, and the second 2D image corresponds to the second depth range of the point cloud data of the 3D point cloud. The 3D Cartesian coordinates of the point cloud data are obtained from both the first 2D image and the second 2D image.

3. The method of claim 2, wherein the first 2D image corresponds to a higher bit depth range and the second 2D image corresponds to a lower bit depth range, and wherein the first 2D image is decoded using a lossless decoder and the second 2D image is decoded using a lossy decoder.

4. The method according to claim 2 or 3, wherein the first 2D image is decoded using a neural network-based decoder, and the second 2D image is decoded using a non-neural network-based decoder.

5. The method according to any one of claims 2-4, wherein decoding of the second 2D image depends on the first 2D image.

6. The method according to any one of claims 1-5, wherein the 3D point cloud corresponds to a region of another 3D point cloud.

7. The method according to any one of claims 1-6, wherein the x-coordinates, y-coordinates, and z-coordinates of the point cloud data correspond to the R-components, G-components, and B-components of the 2D image.

8. A method for encoding point cloud data of 3D point clouds, comprising: Obtain the point cloud data with 3D Cartesian coordinates; The 3D Cartesian coordinates are mapped to Cartesian coordinates on a 2D grid to form a 2D image, wherein the corresponding color components of the 2D image correspond to the dimensions of the 3D Cartesian coordinates; as well as The 2D image is encoded.

9. The method of claim 8, wherein the mapping comprises: The 3D Cartesian coordinates are projected onto spherical coordinates to obtain the elevation and azimuth values; The elevation and azimuth values ​​are matched one-to-one to the grid points in the 2D grid; as well as Based on the matching result, the 3D Cartesian coordinates are placed on the matching grid coordinates in the 2D grid to output grid-aligned Cartesian coordinates for the 2D image.

10. The method of claim 9, wherein the matching is performed to preserve neighborhood relationships in the 3D point cloud.

11. The method according to any one of claims 8-10, wherein each spherical coordinate is associated with a grid point in the 2D grid, and a matching index representing the association is obtained.

12. The method according to any one of claims 8-11, wherein the 2D mesh is defined based on information about the lidar sensor.

13. The method according to any one of claims 8-12, further comprising: The 2D image is segmented into at least a first 2D image and a second 2D image, wherein the first 2D image corresponds to a first depth range and the second 2D image corresponds to a second depth range; and The first 2D image and the second 2D image are encoded.

14. The method of claim 13, wherein the first 2D image corresponds to a higher bit depth range and the second 2D image corresponds to a lower bit depth range, and wherein the first 2D image is encoded using a lossless encoder and the second 2D image is encoded using a lossy encoder.

15. The method according to any one of claims 8-14, wherein the encoding of the second 2D image depends on the first 2D image.

16. The method according to any one of claims 8-15, wherein the 3D point cloud corresponds to a region of another 3D point cloud.

17. The method according to any one of claims 8-16, wherein the x, y, and z coordinates of the 3D Cartesian coordinates of the point cloud data correspond to the R, G, and B components of the 2D image.

18. An apparatus comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform the method according to any one of claims 1-17.

19. A non-transitory computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1-17.

Images