Point cloud processing method, computer system and computer readable medium

Abstract

A point cloud data processing method, a computer system, and a computer-readable medium are provided. Quantized point cloud data is received, the quantized point cloud data including a plurality of stereo pixels. An occupancy map of the quantized point cloud corresponding to stereo pixels lost during quantization in the plurality of stereo pixels is generated. A point cloud is reconstructed based on the quantized point cloud data based on filling the lost stereo pixels.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 049,862, filed on July 9, 2020, and U.S. Provisional Application No. 17 / 345,063, filed on June 11, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to the field of data processing, and in particular to point cloud compression, and more specifically, to a point cloud processing method, a computer system, and a computer-readable medium. Background Art

[0004] Point clouds have been widely used in recent years. For example, point clouds can be used for object detection and object positioning in autonomous vehicles; point clouds can be used for mapping in Geographic Information Systems (GIS); point clouds can also be used in cultural heritage to visualize and archive cultural heritage objects and collections; point clouds can also be used to create and communicate virtual / augmented reality (VR / AR) content; and for remote presentation, etc. Point clouds contain a set of high-dimensional points, usually three-dimensional (3D) points, each of which includes 3D position information and additional attributes such as color, reflectivity, etc. Point clouds can be captured using multiple cameras and depth sensors or lidar in various settings, and can consist of thousands to billions of points to truly represent the original scene. Summary of the invention

[0005] Embodiments relate to a method, system, and computer-readable medium for point cloud processing. According to one aspect, a method for point cloud processing is provided. The method may include: receiving quantized point cloud data, the quantized point cloud data including a plurality of stereo pixels; generating an occupancy map of the quantized point cloud corresponding to stereo pixels lost during quantization in the plurality of stereo pixels; and reconstructing a point cloud based on the quantized point cloud data based on filling the lost stereo pixels.

[0006] According to another aspect, a computer system for point cloud processing is provided. The computer system may include one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices, the program instructions being executed by at least one of the one or more processors through at least one of the one or more memories, whereby the computer system is capable of performing a method. The method may include: receiving quantized point cloud data, the quantized point cloud data including a plurality of stereo pixels; generating an occupancy map of the quantized point cloud corresponding to stereo pixels lost during quantization in the plurality of stereo pixels; and reconstructing the point cloud based on the quantized point cloud data based on filling the lost stereo pixels.

[0007] According to another aspect, a computer-readable medium for point cloud processing is provided. The computer-readable medium may include one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices, and the program instructions may be executed by a processor. The program instructions may be executed by the processor to perform a method, which may accordingly include: receiving quantized point cloud data, the quantized point cloud data including a plurality of stereo pixels; generating an occupancy map of the quantized point cloud corresponding to stereo pixels lost during quantization in the plurality of stereo pixels; and reconstructing the point cloud based on the quantized point cloud data based on filling the lost stereo pixels. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] These and other objects, features and advantages will become apparent from the following detailed description of illustrative embodiments read in conjunction with the accompanying drawings. The various features of the drawings are not drawn to scale because the illustrations are intended to be clear and easy to understand by those skilled in the art in conjunction with the detailed description. In the drawings:

[0009] Figure 1 illustrates a networked computer environment according to at least one embodiment;

[0010] Figure 2A is a schematic diagram of an octree structure for point cloud data according to at least one embodiment;

[0011] Figure 2B is a schematic diagram of octree partitioning for point cloud data according to at least one embodiment;

[0012] Figure 3A is a diagram of a system for upsampling point cloud data according to at least one embodiment;

[0013] Figure 3B is a schematic diagram of a U-net structure according to at least one embodiment;

[0014] Figure 4An operational flow chart illustrating steps performed by a program for upsampling point cloud data according to at least one embodiment;

[0015] Figure 5 According to at least one embodiment Figure 1 block diagrams of internal and external components of computers and servers depicted in;

[0016] Figure 6 According to at least one embodiment, Figure 1 A block diagram of an illustrative cloud computing environment of a computer system depicted in; and

[0017] Figure 7 According to at least one embodiment Figure 6 A block diagram of the functional layers of an illustrative cloud computing environment. DETAILED DESCRIPTION

[0018] Detailed embodiments of the claimed structures and methods are disclosed herein; however, it is understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, which may be implemented in various forms. However, these structures and methods may be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Instead, these exemplary embodiments are provided so that the disclosure will be thorough and complete and fully convey the scope to those skilled in the art. In the description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0019] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0020] Embodiments generally relate to the field of data processing, and more particularly to point cloud compression. The exemplary embodiments described below provide a system, method, and computer program that include upsampling point cloud data. Thus, some embodiments have the ability to improve the computational field by allowing the level of detail of the point cloud data to be increased after quantization, which may enable higher levels of compression to be achieved.

[0021] As mentioned above, point clouds have been widely used in recent years. For example, point clouds can be used for object detection and object positioning in autonomous driving vehicles; point clouds can be used for mapping in Geographic Information Systems (GIS); point clouds can also be used in cultural heritage for visualization and archiving of cultural heritage objects and collections; point clouds can also be used for the creation and communication of virtual / augmented reality (VR / AR) content; and for remote presentation, etc. A point cloud contains a set of high-dimensional points, usually three-dimensional (3D) points, each of which includes 3D position information and additional attributes such as color, reflectivity, etc. Point clouds can be captured using multiple cameras and depth sensors or lidar in various settings, and can consist of thousands to billions of points to truly represent the original scene.

[0022] For faster transmission or reduced storage, compression techniques are needed to reduce the amount of data required to represent point clouds. ISO / IEC MPEG (JTC1 / SC 29 / WG 11) has created a self-organizing group (MPEG-PCC) to standardize compression techniques for static or dynamic clouds. However, current point cloud compression and processing techniques suffer from quantization loss, with the result that point clouds are represented with coarser subsampling, i.e., point clouds are lower level-of-detail (LoD) point clouds.

[0023] Quantization is a necessary step in most compression transmission pipelines. Quantization is achieved by compressing a series of values ​​into a single quantized value. When the possible values ​​of a symbol are reduced, the number of discrete symbols in a given stream will be reduced, making the stream more compressible. As a result of quantization, adjacent voxelized points are merged into a voxel (i.e., a single sample or data point, located on a regularly spaced three-dimensional grid similar to a pixel). Depending on the compression rate, the quantization step size (qs) determines the number of points lost and the LoD of the reconstructed point cloud. A point cloud subjected to voxel quantization loss can be modeled as:

[0024] Among them, qs is the quantization step size, X is the 3D position (coordinate) of the original point cloud, is the 3D position (coordinate) of the reconstructed point cloud. This quantization results in duplicate points, which are removed during the compression process.

[0025] Therefore, it is advantageous to upsample the point cloud data to minimize quantization loss and obtain a point cloud with a higher LoD. Upsampling point cloud data can also be useful for applications other than compression. For example, in some applications, such as in AR / VR or remote presentation, a point cloud with a lower LoD is transmitted to save bandwidth or reduce transmission latency. In such a scenario, when the point cloud is zoomed in, it may be beneficial to upsample the point cloud to a higher LoD point cloud. This application is called display adaptation, where upsampling the point cloud helps provide more display adaptation details when the point cloud is zoomed in.

[0026] Various aspects are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer-readable media according to various embodiments. It should be understood that each block of the flowchart illustrations and / or block diagrams and the combination of blocks in the flowchart illustrations and / or block diagrams can be implemented by computer-readable program instructions.

[0027] The exemplary embodiments described below provide systems, methods, and computer programs for point cloud processing. Figure 1 , a functional block diagram of a networked computer environment of a point cloud processing system 100 (hereinafter referred to as a “system”) for processing point cloud data after quantization is shown. It should be understood that Figure 1 This is merely an illustration of one implementation and does not imply any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0028] System 100 may include computer 102 and server computer 114. Computer 102 may communicate with server computer 114 via communication network 110 (hereinafter referred to as "network"). Computer 102 may include processor 104 and software program 108 stored on data storage device 106, and computer 102 may interface with a user and communicate with server computer 114. Figure 5 As discussed, the computer 102 may include internal components 800A and external components 900A, and the server computer 114 may include internal components 800B and external components 900B, respectively. The computer 102 may be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop, a tablet computer, a desktop computer, or any type of computing device capable of running programs, accessing a network, and accessing a database.

[0029] As referenced below Figure 6 and Figure 7As discussed, the server computer 114 may also operate in a cloud computing service model, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). The server computer 114 may also be located in a cloud computing deployment model, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud.

[0030] The server computer 114 can be used for point cloud processing and can run a point cloud processing program 116 (hereinafter referred to as a "program") that can interact with the database 112. Figure 4 The point cloud processing program method is described in more detail. In one embodiment, the computer 102 can operate as an input device including a user interface, and the program 116 can be primarily run on the server computer 114. In an alternative embodiment, the program 116 can be primarily run on one or more computers 102, and the server computer 114 can be used to process and store data used by the program 116. It should be noted that the program 116 can be a stand-alone program, or can be integrated into a larger point cloud processing program.

[0031] However, it should be noted that in some cases, the processing of program 116 may be shared in any proportion between computer 102 and server computer 114. In another embodiment, program 116 may be executed on more than one computer, server computer, or some combination of computers and server computers, such as, for example, on multiple computers 102 communicating with a single server computer 114 via network 110. In another embodiment, for example, program 116 may be executed on multiple server computers 114 communicating with multiple client computers via network 110. Alternatively, the program may be executed on a network server that communicates with the server and multiple client computers via the network.

[0032] The network 110 may include a wired connection, a wireless connection, a fiber optic connection, or some combination of these connections. In general, the network 110 may be any combination of connections and protocols that will support communication between the computer 102 and the server computer 114. The network 110 may include various types of networks, such as a local area network (LAN), a wide area network (WAN) such as the Internet, a telecommunication network such as a public switched telephone network (PSTN), a wireless network, a public switched network, a satellite network, a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a metropolitan area network (MAN), a private network, an ad hoc network, an intranet, a fiber-based network, etc., and / or a combination of these networks or other types of networks.

[0033] Figure 1 The number and arrangement of devices and networks shown are provided as examples. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different Figure 1 The devices and / or networks shown are arranged differently. Figure 1 Two or more of the devices shown may be implemented in a single device, or Figure 1 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one set of devices (eg, one or more devices) of system 100 may perform one or more functions described as being performed by another set of devices of system 100.

[0034] Reference now Figure 2A , depicting a schematic diagram of an octree structure 200A. In a point cloud compression solution such as reference software from the MPEG-PCC group (e.g., TMC13), if an octree geometry codec is used, the geometry encoding process proceeds as follows. First, a cube axis-aligned bounding box B is formed by two points (0, 0, 0) and (2 M-1 , 2 M-1 , 2 M-1 ) is limited, and 2 M-1The size of B is defined, and M is specified in the bitstream. The octree structure 200A is then constructed by recursively subdividing B. At each stage, the cube is subdivided into 8 sub-cubes. An 8-bit code, the occupancy code, is then generated by associating a 1-bit value with each sub-cube to indicate whether the cube contains a point (i.e., full and value 1) or does not contain a point (i.e., empty and value 0). Only complete sub-cubes (i.e., non-stereo pixels) of size greater than 1 are further subdivided.

[0035] Reference now Figure 2B , depicting a schematic diagram of an octree partition 200B. The octree partition 200B may include two levels of octree partitions 202 and corresponding occupancy codes 204, wherein dark cubes and nodes indicate that they have been occupied by points. The occupancy code 204 of each node is then compressed by an arithmetic encoder. The occupancy code 204 may be represented as an 8-bit integer S, and each bit in S represents the occupancy state of each child node. There are two encoding methods for the occupancy code 204 in TMC13, namely, a bit-by-bit encoding method and a byte-by-byte encoding method, and bit-by-bit encoding is enabled by default. Either method performs arithmetic coding with context modeling to encode the occupancy code 204, wherein the context state is initialized at the beginning of the entire encoding process and updated during the encoding process.

[0036] For bit-wise encoding, the 8 bins in S are encoded in a certain order, and each bin is encoded by referring to the occupancy status of neighboring nodes and the children of neighboring nodes, where the neighboring nodes are at the same level as the current node. For byte-wise encoding, S is encoded by referring to an adaptive lookup table (A-LUT), which identifies the N (e.g., 32) most frequently occupied codes and a cache that also identifies the M (e.g., 16) most recently observed occupied codes.

[0037] A binary flag indicating whether S is in the A-LUT is encoded. If S is in the A-LUT, the index into the A-LUT is encoded using a binary arithmetic encoder. If S is not in the A-LUT, a binary flag indicating whether S is in the cache is encoded. If S is in the cache, the binary representation of its index is encoded using a binary arithmetic encoder. Otherwise, if S is not in the cache, the binary representation of S is encoded using a binary arithmetic encoder. The decoding process first parses the dimensions of the bounding box B from the bitstream. The same octree structure is then constructed by subdividing B according to the decoded occupancy code.

[0038] Reference now Figure 3A, depicts a block diagram of a point cloud processing system 300. The point cloud processing system 300 may include, among other things, a U-net 302 and an occupancy map 304. The occupancy map 304 may correspond to a predicted probability of occupied stereo pixels, and thus, the top k stereo pixels may be selected, or a threshold defining occupied stereo pixels may be set and stereo pixels with predicted values ​​greater than the threshold may be selected. The point cloud processing system 300 may receive an input point cloud 306A as input. The input point cloud 306A may be divided into input blocks 308. The point cloud may be large, with millions of points. Current hardware computation and software computation and memory limitations make it impossible to transmit the entire point cloud to the network. In order to transmit the point cloud to the network and to make the system scalable, the input point cloud 306 may be subdivided into smaller cubic input blocks 308 and transmitted to the network. The input block 308 may be a stereo pixelated geometry of size Nx3, where N is the number of stereo pixels in the input cubic block, and 3 is the X, Y, Z coordinates. Upsampling is accomplished by performing geometric prediction on the input patches, which learns the underlying 3D structure and creates patches with more detail based on the occupancy map 304. The point cloud processing system 300 may generate output patches 310, which may be aggregated to form an upsampled point cloud 312.

[0039] Reference now Figure 3B , depicting a block diagram of a U-net architecture 302. The U-net architecture 302 may include one or more input nodes 314A-314X, a latent space 316, and one or more output nodes 318A-318X. The input nodes 314A-314X may form a contraction path for feature extraction, and the output nodes 318A-318X may form an expansion path for detail preservation. The input nodes 314A-314X and the output nodes 318A-318X may each include one or more convolution kernels (e.g., 3x3 convolution kernels) and adjusted linear units. Each input node 314A-314X may be connected continuously through a maximum pooling layer (e.g., a 2x2 maximum pooling layer). For example, each convolution may be followed by a batch norm and an adjusted linear unit. The latent space 316 may correspond to a representation of compressed data, in which similar data points are closer in space. The output nodes 318A-318X may be transmitted to each other in reverse order via upsampling connections (eg, upsampling factor 2). The final output node, output node 318A, may output point cloud data.

[0040] Since point clouds are sparse in nature, sparse convolutions are more memory efficient and allow very deep architectures to be built. A UNet architecture can be used with three Inception-Residual Network blocks per layer. UNet has been widely used for image segmentation. 3D libraries such as Minkowski Engine have created 3D versions of UNet. However, it uses residual blocks. The 3D UNet can be updated by using updated Inception-Residual Network blocks.

[0041] The input to the architecture is a point cloud of lower LoD, and the output is an occupancy prediction that is converted to a point cloud of higher LoD (explained in the next subsection). A binary cross entropy classification loss can be used to compare the occupancy map predictions from the network and the ground truth (original) point cloud.

[0042] The output of the network is an Nx8 occupancy map prediction of 2x2x2=8 voxels containing the input voxel. The output is the predicted probability that the voxel is occupied, so you can select the top k voxels, or set a threshold that defines occupied voxels and select voxels with predicted values ​​greater than the threshold. This occupancy map can be used to generate a denser point cloud. in The output blocks can be clustered together to form an upsampled point cloud.

[0043] It will be appreciated that improvements can be made to enhance or fine-tune performance based on the application and input point cloud. According to one or more embodiments, different deep learning architectures can be used to perform the same or better work (e.g., spatial pyramid pooling (i.e., adding new layers between the convolutional layers and the fully connected layers to map inputs of any size down to outputs of fixed size), or 3D convolutions, which involve standard convolutions, submanifold convolutions, or dilated / hole convolutions). Another embodiment will use an encoder-decoder architecture. Depending on the output, different loss functions can be used. For example, chamfer loss can be used to compare the output to the original (ground truth) point cloud.

[0044] The output of the architecture can also be changed. Currently, occupancy map predictions are output, which are 1 bit per voxel. However, it is also possible to directly learn the x, y, z coordinates of the output point cloud. The output of the network does not have to be the same size (N) as the input. Upsampling points can be learned directly. Without direct learning Coordinates or identification on a predefined coordinate map Occupancy prediction is used to learn an Nx8 occupancy map. Based on the prediction, a subset of points is selected to form coordinates. The output occupancy map may be changed depending on the input point cloud. For example, an 8-stereo pixel occupancy map may be applicable to qs=2. However, for qs=4, it is more appropriate to predict 4x4x4=64 stereo pixels around the input stereo pixels. It may also be applicable to training a multi-iteration system where the point cloud is upsampled in each iteration. Thus, a point cloud with qs=4 may first be upsampled to qs=2, and then, in the next iteration, upsampled to its original LoD. This method will be more applicable to point clouds with very low LoD, and may be upsampled in iterations on different trained deep learning architectures.

[0045] In addition to geometric upsampling, embodiments disclosed herein can be used for attribute (color, reflectivity, etc.) upsampling. In the case of attributes, these attributes can be input to the network along with the geometry (coordinates). For example, if the attribute is only color: RGB, the input to the network will be Nx6, and the output can be one of the following:

[0046] Nx(8+24), where 8 is the predicted occupancy map and 24 is the corresponding output attribute for each possible occupied voxel, and is converted to

[0047] where the network directly learns to output coordinates and attributes; or

[0048] Where 1 is the predicted occupancy on a predefined coordinate system, 3 is the attribute, and based on the occupancy prediction, a subset of coordinates will be selected to form Point cloud.

[0049] Reference now Figure 4 , shows an operational flow chart of the steps of a method 400 performed by a program for upsampling point cloud data.

[0050] At 402 , method 400 may include receiving quantized point cloud data, the quantized point cloud data including a plurality of voxels.

[0051] At 404 , method 400 may include generating an occupancy map of the quantized point cloud corresponding to voxels of the plurality of voxels that were lost during quantization.

[0052] At 406 , method 400 may include reconstructing the point cloud from the quantized point cloud data based on filling in the missing voxels.

[0053] Optionally, in one example, the occupancy map corresponds to a predicted probability of each of the plurality of lost voxels.

[0054] Optionally, in one example, based on the prediction probability being greater than a threshold, the missing stereo pixels are filled.

[0055] Optionally, in one example, the point cloud is reconstructed based on dividing the quantized point cloud data into one or more blocks and upsampling the blocks.

[0056] Optionally, in one example, the point cloud is reconstructed based on minimizing a binary cross entropy classification loss associated with the occupancy map.

[0057] Optionally, in one example, the point cloud is reconstructed by one or more of: U-net, spatial pyramid pooling, or three-dimensional 3D convolution.

[0058] Optionally, in one example, the 3D convolution includes one or more of the following: convolution, submanifold convolution, dilated convolution, and hole convolution.

[0059] Understandably, Figure 4 This is merely an illustration of one implementation and does not imply any limitations regarding how different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.

[0060] Figure 5 According to an illustrative embodiment Figure 1 A block diagram 500 of the internal and external components of a computer is depicted. It should be understood that Figure 5 This is merely an illustration of one implementation and does not imply any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0061] like Figure 5 As shown, the computer 102 ( Figure 1 ) and server computer 114 ( Figure 1 ) may include a set of internal components 800A, 800B and external components 900A, 900B, respectively. Each set of internal components 800 includes one or more processors 820, one or more computer-readable RAMs 822 and one or more computer-readable ROMs 824, one or more operating systems 828, and one or more computer-readable tangible storage devices 830 located on one or more buses 826.

[0062] The processor 820 is implemented in hardware, firmware, or a combination of hardware and software. The processor 820 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other types of processing components. In some implementations, the processor 820 includes one or more processors that can be programmed to perform functions. The bus 826 includes components that allow communication between internal components 800A and 800B.

[0063] Located on server computer 114 ( Figure 1 ) on one or more operating systems 828, software programs 108 ( Figure 1 ) and point cloud processing program 116 ( Figure 1 ) are stored on one or more of the corresponding computer-readable tangible storage devices 830 for execution by one or more of the corresponding processors 820 through one or more of the corresponding RAMs 822 (typically including cache). Figure 5 In the illustrated embodiment, each computer-readable tangible storage device 830 is a magnetic disk storage device of an internal hard drive. Alternatively, each computer-readable tangible storage device 830 is a semiconductor storage device such as ROM 824, EPROM, flash memory, optical disk, magneto-optical disk, solid state disk, compact disk (CD), digital versatile disk (DVD), floppy disk, cassette, magnetic tape, and / or other types of non-transitory computer-readable tangible storage devices that can store computer programs and digital information.

[0064] Each set of internal components 800A, 800B also includes an R / W drive or interface 832 to read from and write to one or more portable computer readable tangible storage devices 936 (e.g., CD-ROM, DVD, memory stick, tape, disk, optical disk, or semiconductor storage device). Figure 1 ) and point cloud processing program 116 ( Figure 1) can be stored on one or more of the corresponding portable computer-readable tangible storage devices 936, read via the corresponding R / W drive or interface 832 and loaded into the corresponding hard disk drive 830.

[0065] Each set of internal components 800A, 800B also includes a network adapter or interface 836, such as a TCP / IP adapter card; a wireless Wi-Fi interface card; or a 3G, 4G or 5G wireless interface card or other wired or wireless communication link. Server computer 114 ( Figure 1 ) on the software program 108 ( Figure 1 ) and point cloud processing program 116 ( Figure 1 ) can be downloaded from an external computer to the computer 102 via a network (eg, the Internet, a local area network, or other wide area network) and a corresponding network adapter or interface 836 ( Figure 1 ) and the server computer 114. The software program 108 and the point cloud processing program 116 on the server computer 114 are loaded into the corresponding hard disk drive 830 through the network adapter or interface 836. The network may include copper wire, optical fiber, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers.

[0066] Each set of external components 900A, 900B may include a computer display monitor 920, a keyboard 930, and a computer mouse 934. External components 900A, 900B may also include touch screens, virtual keyboards, touch pads, pointing devices, and other human-computer interface devices. Each set of internal components 800A, 800B also includes a device driver 840 to interface with the computer display monitor 920, keyboard 930, and computer mouse 934. Device driver 840, R / W driver or interface 832, and network adapter or interface 836 include hardware and software (stored in storage device 830 and / or ROM 824).

[0067] It should be understood in advance that although the present disclosure includes a detailed description of cloud computing, the implementation of the teachings described herein is not limited to a cloud computing environment. Instead, some embodiments can be implemented in conjunction with any other type of computing environment now known or later developed.

[0068] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or through interaction with a service provider. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0069] Features are as follows:

[0070] On-demand self-service: Cloud consumers can automatically and unilaterally provision computing capacity (such as server time and network storage) as needed without the need for manual interaction with the service provider.

[0071] Broad network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0072] Resource pooling: Pooling of a provider's computing resources to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. There is a sense of location independence because consumers generally do not control or know the exact location of the provided resources, but are able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0073] Rapid elasticity: Capacity can be provisioned quickly and elastically, in some cases automatically, to scale up quickly and release quickly to scale down quickly. To the consumer, the available capacity often appears to be unlimited and can be purchased at any time and in any quantity.

[0074] Measured services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be identified, controlled, and reported, providing transparency to both providers and consumers of the utilized services.

[0075] The service model is as follows:

[0076] Software as a Service (SaaS): The capability provided to the consumer is to use the provider's applications running on the cloud infrastructure. The applications are accessible from a variety of client devices through a thin client interface, such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0077] Platform as a Service (PaaS): The capability provided to consumers is to deploy applications created or acquired by consumers onto cloud infrastructure, where the applications are created using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure including networks, servers, operating systems or storage, but rather control the deployed applications and possible configuration of the application hosting environment.

[0078] Infrastructure as a Service (IaaS): The capabilities provided to consumers are the provision of processing, storage, networking, and other basic computing resources, where consumers can deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but control the operating system, storage, deployed applications, and may have limited control over selected networking components (e.g., host firewalls).

[0079] The deployment model is as follows:

[0080] Private cloud: Cloud infrastructure is run for the sole use of one organization. The cloud infrastructure can be managed by that organization or a third party and can exist on-premises or off-premises.

[0081] Community cloud: Cloud infrastructure is shared by several organizations and supports a specific community with common concerns (e.g., mission, security requirements, policy, and compliance considerations). Community clouds can be managed by these organizations or by a third party and can exist on-premises or off-premises.

[0082] Public cloud: Cloud infrastructure is available to the general public or large industrial groups and is owned by an organization that sells cloud services.

[0083] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain distinct entities but are tied together by standardized technologies or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0084] The cloud computing environment is service-oriented, with an emphasis on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is the infrastructure consisting of a network of interconnected nodes.

[0085] refer to Figure 6 , depicts an illustrative cloud computing environment 600. As shown, the cloud computing environment 600 includes one or more cloud computing nodes 10, and local computing devices used by cloud consumers can communicate with the cloud computing nodes 10, such as personal digital assistants (PDA) or cellular phones 54A, desktop computers 54B, laptop computers 54C and / or automobile computer systems 54N. The cloud computing nodes 10 are communicative with each other. The cloud computing nodes 10 can be physically or virtually grouped in one or more networks (not shown), such as a private cloud, community cloud, public cloud, or hybrid cloud as described above, or a combination thereof. This enables the cloud computing environment 600 to provide infrastructure, platforms, and / or software as services that do not require cloud consumers to maintain resources on local computing devices. It should be understood that Figure 6The types of computing devices 54A-N shown are for illustration only, and cloud computing node 10 and cloud computing environment 600 may communicate with any type of computerized device over any type of network and / or network addressable connection (eg, using a web browser).

[0086] Reference Figure 7 , showing a cloud computing environment 600 ( Figure 6 ) provides a set of functional abstraction layers 700. It should be understood in advance that Figure 7 The components, layers and functions shown in are for illustration only, and the embodiments are not limited thereto. As depicted in the figure, the following layers and corresponding functions are provided:

[0087] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: host 61; server 62 based on Reduced Instruction Set Computer (RISC) architecture; server 63; blade server 64; storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0088] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 71 ; virtual storage 72 ; virtual networks 73 , including virtual private networks; virtual applications and operating systems 74 ; and virtual clients 75 .

[0089] In one example, the management layer 80 may provide the functionality described below. Resource provisioning 81 provides dynamic acquisition of computing resources and other resources for performing tasks in a cloud computing environment. Metering and pricing 82 provides cost identification when resources are used in a cloud computing environment, and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, and protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides cloud computing resource allocation and management so that the required service levels are met. Service Level Agreement (SLA) planning and implementation 85 provides pre-scheduling and acquisition of cloud computing resources, for which future demand is anticipated based on the SLA.

[0090] Workload layer 90 provides examples of functionality that can use a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics processing 94; transaction processing 95; and point cloud processing 96. Point cloud processing 96 can upsample point cloud data to recover lost voxels after quantization.

[0091] Some embodiments may involve systems, methods, and / or computer-readable media incorporating any possible level of technical detail. A computer-readable medium may include a computer-readable non-transitory storage medium (or medium) having computer-readable program instructions thereon that cause a processor to perform operations.

[0092] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above storage devices. A non-exhaustive list of more specific examples of computer-readable storage media includes the following items: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device (e.g., a raised structure in a groove or a punched card with instructions recorded thereon), and any suitable combination of the above items. Computer-readable storage media as used herein should not be construed as being a transient signal in itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0093] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0094] The computer readable program code / instructions for performing operations may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., SmallTalk, C++, etc.) and procedural programming languages ​​(e.g., "C" programming language or similar programming languages). The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, which may be partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection may be established with an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuits to perform various aspects or operations.

[0095] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a way to implement the functions / actions specified in the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium, which may direct the computer, programmable data processing device, and / or other device to operate in a specific manner, so that the computer-readable storage medium storing the instructions includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in the flowchart and / or block diagram.

[0096] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device, so that a series of operating steps are executed on the computer, other programmable device, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable device, or other device implement the functions / actions specified in the flowchart and / or block diagram.

[0097] The flowchart and block diagram in the figure show the architecture, function and operation of the possible implementation of the system, method and computer-readable medium according to various embodiments.At this point, each frame in the flowchart or block diagram may represent a module, section or part of an instruction, which includes one or more executable instructions for implementing a specified logical function.Compared with those frames depicted in the figure, the method, computer system and computer-readable medium may include additional frames, fewer frames, different frames or frames of different arrangements.In some optional implementations, the function noted in the frame may not occur in the order noted in the figure.For example, the two frames shown in succession can actually be executed simultaneously or substantially simultaneously, or sometimes the frame can be executed in the opposite order, depending on the function involved.It should also be noted that each frame illustrated in the block diagram and / or the flowchart and the combination of the frames illustrated in the block diagram and / or the flowchart can be implemented by a system based on special-purpose hardware, which performs a specified function or action or realizes a combination of special-purpose hardware and computer instructions.

[0098] Obviously, the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0099] Unless otherwise clearly described, the elements, actions or instructions used herein should not be understood as critical or necessary. In addition, as used herein, the articles "one" and "an" are intended to include one or more projects, and can be used interchangeably with "one or more". In addition, as used herein, the term "set" is intended to include one or more projects (e.g., related projects, unrelated projects, combinations of related projects and unrelated projects, etc.), and can be used interchangeably with "one or more". In the case of only one project being meant, the term "one" or similar language is used. In addition, as used herein, the term "have", "have", "contain" or similar terms are intended to be open terms. In addition, the phrase "based on" means "based at least in part on", unless otherwise clearly stated.

[0100] Descriptions of various aspects and embodiments have been given for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Even if combinations of features are described in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways that are not specifically stated in the claims and / or disclosed in the specification. Although each dependent claim listed below may be directly subordinate to only one claim, the disclosure of possible implementations includes the combination of each dependent claim with every other claim in the claim set. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements relative to technologies found in the market, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A point cloud processing method, comprising: Receiving quantized point cloud data, wherein the quantized point cloud data includes a plurality of stereo pixels; generating an occupancy map of the quantized point cloud corresponding to the voxels lost during quantization, wherein the occupancy map corresponds to a predicted probability for each voxel of the plurality of voxels lost; as well as A point cloud is reconstructed from the quantized point cloud data based on filling the missing stereo pixels.

2. The method according to claim 1, wherein: The missing voxel is filled based on the prediction probability being greater than a threshold.

3. The method according to claim 1, wherein: The point cloud is reconstructed based on dividing the quantized point cloud data into one or more blocks and upsampling the blocks.

4. The method according to claim 1, wherein: The point cloud is reconstructed based on minimizing a binary cross entropy classification loss associated with the occupancy map.

5. The method according to claim 1, wherein: The point cloud is reconstructed by one or more of the following: U-net, spatial pyramid pooling, or 3D convolution.

6. The method according to claim 5, wherein: The 3D convolution includes one or more of the following: convolution, submanifold convolution, dilated convolution and hole convolution.

7. A computer system for point cloud processing, the computer system comprising: one or more computer readable non-transitory storage media configured to store computer program code; as well as One or more computer processors configured to access the computer program code and operate according to the instructions of the computer program code, wherein the computer program code includes: receiving code configured to cause the one or more computer processors to receive quantized point cloud data, the quantized point cloud data comprising a plurality of stereo pixels; generating code configured to cause the one or more computer processors to generate an occupancy map of a quantized point cloud corresponding to voxels of the plurality of voxels that were lost during quantization, wherein the occupancy map corresponds to a predicted probability for each voxel of the plurality of voxels that were lost; and Reconstruction code is configured to cause the one or more computer processors to reconstruct a point cloud from the quantized point cloud data based on filling in the missing voxels.

8. The computer system according to claim 7, wherein: The missing voxel is filled based on the prediction probability being greater than a threshold.

9. The computer system according to claim 7, wherein: The point cloud is reconstructed based on dividing the quantized point cloud data into one or more blocks and upsampling the blocks.

10. The computer system according to claim 7, wherein: The point cloud is reconstructed based on minimizing a binary cross entropy classification loss associated with the occupancy map.

11. The computer system according to claim 7, wherein: The point cloud is reconstructed by one or more of the following: U-net, spatial pyramid pooling, or 3D convolution.

12. The computer system according to claim 11, wherein: The 3D convolution includes one or more of the following: convolution, submanifold convolution, dilated convolution and hole convolution. 13 . A non-transitory computer-readable medium having stored thereon a computer program for point cloud processing, the computer program being configured to cause one or more computer processors to execute the method according to any one of claims 1 to 6.

Images