Methods and systems for generating synthetic point cloud data using generative models

Abstract

Methods and systems for generating synthetic point cloud data are described. A projected 2D data grid is generated by projecting a 3D point cloud into a 2D grid with rotational homogeneity. A generative model is learned using the projected 2D data grid, where the generative model is implemented using flexible convolution and transposed flexible convolution operations, for example, in a generative adversarial network. The learned generative model is used to generate synthetic point clouds.

Description

Technical Field

[0001] This application relates to a system and method for learning a generative model for generating synthetic point cloud data, and a system and method for generating synthetic point cloud data using the learned generative model based on data selected from a distribution. Background Technology

[0002] Understanding the environment plays a crucial role in many autonomous tasks. The success of autonomous devices (e.g., robots or vehicles) in performing autonomous tasks depends on robust sensory data input and the algorithms used to process that data. In many cases, sensory data is noisy or some sensory data is missing. To handle this, autonomous devices need to be able to “understand” this type of sensory data. Humans possess this ability. For example, if some pixels are missing from some video frames, a human can easily “predict / imagine” the missing data (e.g., generate those missing samples in their brain based on their temporal and spatial information about those pixels) and still have the same perception of the video.

[0003] Furthermore, many autonomous tasks require testing and even training in simulator environments because training and testing autonomous devices in real-world environments can be difficult, costly, and / or unsafe (e.g., in the case of autonomous driving). However, many traditional simulators cannot provide realistic sensory data for autonomous devices. As a result, autonomous devices trained and tested in simulators may not operate well in real-world environments. Therefore, there is a need to develop simulators that can generate more realistic data.

[0004] To address the aforementioned issues, an efficient method for generating synthetic data is needed. In recent years, researchers have successfully used generative models to generate image and video data, as described by Goodfellow et al. in their paper (Advances in Neural Information Processing Systems, pp. 2672-2680, 2014) and Zhu et al. in their paper (arXiv preprint, 2017). However, in most cases, autonomous devices require a three-dimensional (3D) understanding of the real environment for proper operation, which relies on accurate 3D sensory data (e.g., in the form of point clouds). Generating point clouds remains a challenging task.

[0005] Generative models are a class of machine learning methods that aim to generate samples from the same distribution as the training data. There are different types of generative models, such as variational autoencoders (VAEs) and generative adversarial networks (GANs). Many generative models that use convolutional neural networks (CNNs) in their architecture require deconvolution operations (e.g., in the decoder in a VAE or in the generator in a GAN environment). For a regular convolution, this operation is a transposed convolution. Many deep learning-based methods capture local features in an irreversible manner.

[0006] Therefore, a system and method for generating synthetic data is needed to address at least some of the aforementioned problems. Summary of the Invention

[0007] This application provides methods and systems for generating synthetic data in the form of point clouds. The disclosed methods and systems learn a generative model from the point cloud during the training phase and generate a synthetic point cloud from the learned generative model during the inference phase. In some examples, the disclosed methods and systems may utilize a deep neural network (DNN) architecture. This application also describes methods and systems for sorting and projecting data points from the point cloud onto a grid-based data structure, which helps improve the efficiency of learning the generative model during the training phase.

[0008] According to one aspect of this application, a method is provided, the method comprising: obtaining a first batch of point clouds representing a 3D environment; generating a projected 2D data grid for each point cloud in the first batch of point clouds by projecting each point cloud in the first batch of point clouds onto a projected two-dimensional (2D) data grid having rotational isomorphism; and, during a training phase, learning a generative model that generates one or more batches of synthetic point clouds by providing the generative model with the projected 2D data grids of each point cloud in the first batch of point clouds, wherein the generative model includes flexible convolution operations and transposed flexible convolution.

[0009] According to the foregoing aspects, the method further includes: using a learned generative model to generate one or more batches of synthetic point clouds from data sampled from a distribution.

[0010] According to any of the foregoing aspects, the generation of the 2D data grid for the projection further includes: filling the projected 2D data grid by wrapping the elements of the grid from one edge of the projected 2D data grid to the opposite edge; and during the training phase, learning the filled generative model using the projected 2D data grid.

[0011] According to any of the foregoing aspects, each point cloud in the first batch of point clouds is projected onto the projected 2D data grid using the following formula:

[0012]

[0013] Where x, y, and z represent the 3D coordinates of the data points in the point cloud.

[0014] According to the foregoing, the projected 2D data grid is filled by adding an additional leftmost column containing elements from the original rightmost column of the projected 2D data grid, and adding an additional rightmost column containing elements from the original leftmost column of the projected 2D data grid, wherein the number of columns filled is determined by the size of the convolution kernel.

[0015] According to any of the foregoing aspects, each row in the projected 2D data grid corresponds to a corresponding closed loop in the point cloud.

[0016] According to any of the foregoing aspects, the method further includes: using the one or more batches of synthetic point clouds to supplement any missing data points from the point cloud in the first batch.

[0017] According to the foregoing aspects, the method further includes: combining the generative model with a sequence model, wherein the sequence model generates synthetic temporal data for predicting any missing data points from the point cloud in the first batch.

[0018] Based on the foregoing, the learned generative model is a recurrent neural network (RNN).

[0019] According to any of the foregoing aspects, the method further includes: using the learned generative model to generate one or more batches of synthetic point clouds from distributed sampled data based on the actions of the autonomous device.

[0020] According to one aspect of this application, a method is provided, the method comprising: obtaining a first batch of point clouds representing a 3D environment; generating a projected 2D data grid for each point cloud in the first batch of point clouds by projecting each point cloud in the first batch of point clouds onto a projected 2D data grid having rotational isomorphism; and, during a training phase, learning a generative model that generates one or more batches of synthetic point clouds, the generative model being learned by providing the first batch of point clouds to the generative model, wherein the projected 2D data grid is used to identify nearest neighbors and to perform flexible convolution operations and transposed flexible convolution operations during the learning phase.

[0021] According to the foregoing aspects, the method further includes generating one or more batches of synthetic point clouds from distributed sampled data using the learned generative model.

[0022] According to the foregoing, the generation of the 2D data grid for the projection further includes: filling the 2D data grid for the projection by wrapping the elements of the grid from one edge of the 2D data grid to the opposite edge; and during the training phase, learning the filled generative model using the projected 2D data grid.

[0023] According to any of the foregoing aspects, the method further includes: combining the generative model with a sequence model, wherein the sequence model captures temporal data from the first batch of point clouds for predicting any missing data points.

[0024] According to any of the foregoing aspects, the learned generative model is a recurrent neural network (RNN).

[0025] According to another aspect of this application, a processing unit is provided, the processing unit comprising: a processor; and a memory storing computer-executable instructions, which, when executed by the processor, cause the processing unit to perform: obtaining a first batch of point clouds representing a 3D environment; generating a projected 2D data grid for each point cloud in the first batch of point clouds by projecting each point cloud in the first batch of point clouds onto a projected 2D data grid having rotational isomorphism; and, during a training phase, learning a generative model that generates one or more batches of synthetic point clouds by providing the generative model with the projected 2D data grids of each point cloud in the first batch of point clouds, wherein the generative model includes flexible convolution operations and transposed flexible convolution. Attached Figure Description

[0026] The accompanying drawings, by way of example, illustrate exemplary embodiments of this application, wherein:

[0027] Figure 1 This is a block diagram illustrating some components of an example autonomous vehicle;

[0028] Figure 2A block diagram of some components of the processing system for implementing a method for learning and generating a synthetic point cloud is shown.

[0029] Figure 3 It is a diagram that conceptually illustrates how 2D convolution can be applied to grid-based data;

[0030] Figure 4 This is a schematic diagram of 2D image coordinates relative to 3D light detection and ranging (LIDAR) coordinates;

[0031] Figure 5 This is a schematic diagram of the filling of a projected 2D mesh according to the example described in this article;

[0032] Figures 6A-6C This is a schematic diagram of an example projection of a point cloud onto a 2D mesh, based on the example described in this article;

[0033] Figure 7 This is a flowchart of an example method for learning a generative model to generate synthetic point clouds.

[0034] Similar reference numerals can be used in different accompanying drawings to denote similar parts. Detailed Implementation

[0035] Some examples of this application are described in the context of autonomous vehicles. However, the methods and systems disclosed herein are also applicable to implementations outside of autonomous devices, such as in the context of non-vehicle and / or semi-autonomous devices. For example, any system or device that needs to train and / or test point clouds and / or can benefit from the ability to synthesize point clouds during operation can benefit from this application. Furthermore, the examples of this application can be used to learn generative models for generating synthetic point clouds in other environments.

[0036] While the examples described herein may refer to automobiles as autonomous vehicles, the teachings of this application can be associated with other forms of autonomous or semi-autonomous devices, including, for example, trams, subways, trucks, buses, surface and underwater vessels, aircraft, unmanned aerial vehicles (also known as unmanned aerial vehicles, UAVs), warehouse equipment, construction equipment, or farm equipment, and may include both manned and unmanned vehicles. The methods and systems disclosed herein can also be associated with non-vehicle-mounted devices such as autonomous vacuum cleaners and lawnmowers.

[0037] Figure 1This is a block diagram of some components of an exemplary autonomous vehicle 100. Although described as autonomous, the vehicle 100 may operate in a fully autonomous, semi-autonomous, or fully user-controlled mode. In this application, the vehicle 100 is described in an embodiment of an automobile; however, as stated above, this application may be implemented in other in-vehicle or non-in-vehicle machines.

[0038] For example, the vehicle 100 includes a sensor system 110, a data analysis system 120, a path planning system 130, a vehicle control system 140, and an electromechanical system 150. Other systems and components may be suitably included in the vehicle 100. The various systems and components of the vehicle can communicate with each other, for example, via wired or wireless communication. For example, the sensor system 110 can communicate with the data analysis system 120, the path planning system 130, and the vehicle control system 140; the data analysis system 120 can communicate with the path planning system 130 and the vehicle control system 140; the path planning system 130 can communicate with the vehicle control system 140; and the vehicle control system 140 can communicate with the mechanical system 150.

[0039] The sensor system 110 includes various sensing units for collecting information about the interior and / or exterior environment of the vehicle 100. In the illustrated exemplary embodiment, the sensor system 110 includes a radar unit 112, a LiDAR unit 114, a camera 116, and a global positioning system (GPS) unit 118. The sensor system 110 may include other sensing units, such as a temperature sensor, a precipitation sensor, or a microphone.

[0040] In an exemplary embodiment, the LIDAR unit 114 may include one or more LIDAR sensors and may capture data in a wide view (e.g., a 360° view) of the vehicle 100. The LIDAR data (e.g., raw sensor data acquired by one or more LIDAR sensors) may include 3D information about the environment and may be processed to form a set of data points in 3D space. In this application, the term "3D point cloud" or "point cloud" will be used to refer to a set of data points having a 3D structure in space. Each data point in the 3D point cloud represents the 3D coordinates (e.g., x, y, and z values) of a sensed object in 3D space. The set of data points in the 3D point cloud may be irregularly spaced, depending on the sensed environment. In some examples, each data point in the point cloud may also contain other information besides 3D coordinates, such as the intensity of reflected light or the detection time.

[0041] Using various sensing units 112, 114, 116, and 118, the sensor system 110 can collect information about the local environment of the vehicle 100 (e.g., any directly surrounding obstacles) as well as information from a wider surrounding area (e.g., the radar unit 112 and the LIDAR unit 114 can collect information from an area with a radius of up to 100m or greater around the vehicle 100). The sensor system 110 can also collect information about the position and orientation of the vehicle 100 relative to a reference frame (e.g., using the GPS unit 118).

[0042] The sensor system 110 communicates with the data analysis system 120 to detect and identify objects in the environment of the vehicle 100, such as stationary obstacles, pedestrians, or other vehicles. The data analysis system 120 can be implemented using software, which may include any number of independent or interconnected modules or functions, such as machine learning algorithms and image processing capabilities. The data analysis system 120 can be implemented using one or more dedicated image processing units, or it can be implemented using one or more general-purpose processing units of the vehicle 100. The data analysis system 120 can repeatedly (e.g., at regular intervals) receive raw sensor data from the sensor system 110, process the raw sensor data, and perform image analysis in real-time or near real-time. The output of the data analysis system 120 may include, for example, identification of objects in 2D and / or 3D space, including object category, object location, and object boundaries.

[0043] Data acquired by the sensor system 110 and processed by the data analysis system 120 can be provided to the path planning system 130. The vehicle control system 140 is used to control the operation of the vehicle 100 based on a target set by the path planning system 130. The vehicle control system 140 can be used to provide complete, partial, or auxiliary control for the vehicle 100. The electromechanical system 150 receives control signals from the vehicle control system 140 to operate the mechanical and / or electromechanical components of the vehicle 100, such as the engine, transmission, steering system, and braking system.

[0044] The sensor system 110, the data analysis system 120, the path planning system 130, and the vehicle control system 140 may be implemented individually or in combination, at least partially, in one or more processing units of the vehicle 100.

[0045] Before operation in a real-world environment, the vehicle 100 may need to be trained and / or tested to perform the required operations. During training and / or testing, the data analysis system 120, the path planning system 130, and / or the vehicle control system 140 may be trained and / or tested outside the environment of the vehicle 100. For example, one or more modules of the data analysis system 120 may be machine learning-based modules that implement models learned using machine learning algorithms such as deep learning. The machine learning-based modules may be implemented using neural networks such as convolutional neural networks (CNNs), which may be trained and / or tested. Training and / or testing of the convolutional neural networks may be performed using real data (e.g., obtained by operating the vehicle 100 in a real-world environment) and / or synthetic data. Synthetic data is typically generated to simulate real data that will be received via the sensor system 110.

[0046] During actual operation, the vehicle 100 can also utilize synthetic data. For example, one or more sensors of the sensor system 110 may be blocked or otherwise prevented from acquiring sensing data at a point in actual operation (e.g., due to temporary occlusion of the LIDAR unit 114). The vehicle 100 can generate synthetic data to estimate at least some of the missing data. For example, the sensor system 110 may implement a learning generative model for generating such synthetic data in the event of missing data.

[0047] As an example, Figure 2 An example of a processing unit 200 is shown, which can be used to learn a generative model from multiple batches of real point clouds during the training phase, and to implement the learned generative model to generate synthetic data from data sampled from a distribution described further in detail below. In some embodiments, the processing unit 200 may, for example, be used in... Figure 1 The processing unit 200 is implemented in the vehicle 100 to implement a learned generative model for generating synthetic data during operation of the vehicle 100. The processing unit 200 may also be located outside the vehicle 100, for example, to implement the learned generative model, which generates synthetic data for training and / or testing the vehicle 100 outside a real environment (e.g., within a simulation).

[0048] In this example, the processing unit 200 includes one or more physical processors 210 (e.g., microprocessors, graphics processing units, digital signal processors, or other computing elements) coupled to electronic memory 220 and one or more input and output interfaces or devices 230. Electronic memory 220 may include tangible memory (e.g., flash memory) and transient memory (e.g., RAM). The tangible memory may store instructions, data, and / or software modules for the processor to execute to perform the examples described herein. Electronic memory 220 may include any suitable volatile and / or non-volatile storage and retrieval devices. Electronic memory 220 may include one or more of random access memory (RAM), read-only memory (ROM), hard disk, optical disk, subscriber identity module (SIM) card, memory stick, secure digital (SD) memory card, etc.

[0049] exist Figure 2 In the example, computer instructions and data are stored in the electronic memory 220 of the processing unit 200, which enables the processor 210 to generate synthetic data, as disclosed in this invention.

[0050] Synthetic data may need to be in the form of 3D point clouds. For example, raw data acquired by a LiDAR unit is often in the form of point clouds, and it may be desirable to generate synthetic data in a similar format.

[0051] The success of applying convolution to 2D images has led to the desire to use convolution to process 3D point clouds. However, unlike 2D images, 3D point clouds are generally invariant to permutations. Research has been conducted on using deep neural networks (DNNs) to process point cloud data. PointNet (e.g., as described by Qi et al. in the Proceedings of Computer Vision and Pattern Recognition (CVPR) (IEEE, 1(2), 4, 2017)) is an example of a DNN that uses permutation-invariant operations (maximum operations), which are effective at capturing global features. However, PointNet has been found to be unsuccessful in extracting local features. Other DNNs, such as Pointnet++ (e.g., as described by Qi et al. in Advances in Neural Information Processing Systems, pp. 5099-5108, 2017), Pointnet CNN (e.g., as described by Li et al. in the arXiv preprint (arXiv ID: 1801.07791, 2018), and other CNNs, such as ModelNet (e.g., as described by Simonovsky et al. in Proceedings of CVPR, July 2017), introduce variants of point cloud convolutions that are capable of extracting both local and global features.

[0052] For generative models, the expectation operator is differentiable relative to the position of the data point in the 3D point cloud (e.g., defined by x, y, z coordinates). This helps achieve better gradient flow, allowing for more efficient learning of the gradient model during training. Gradient flow is generally important for all machine learning algorithms.

[0053] Generative Adversarial Networks (GANs) are generative models that generate synthetic data. A GAN generative model includes a generator and a discriminator. The generator generates synthetic point clouds, and the discriminator evaluates the synthetic point clouds. The purpose of the generator is to generate synthetic point clouds that cannot be distinguished by the discriminator from real point clouds (e.g., point clouds generated from raw LiDAR sensor data). Typically, during the training phase, the GAN is trained using real point clouds until the synthetic point clouds generated by the generator (using the learned generative model) cannot be distinguished by the discriminator from the real point clouds. The discriminator can be implemented using neural networks such as CNNs, and the generator can be implemented using other types of neural networks (e.g., deconvolutional neural networks). Although this application describes examples of using neural networks to learn and deploy generative models, this is not limiting; other machine learning methods can also be used.

[0054] In the case of GANs, the gradient flow of the data is crucial because the only way the generator learns is by receiving gradients from the discriminator. In the proposed point cloud convolution, flexible convolution allows this propagation of gradients. Groh et al. described flexible convolution in an arXiv preprint (arXiv ID: 1803.07289, 2018). Flexible convolution extends common convolution operators on an image from a 2D grid to arbitrary 3D locations via spatial correlation through convolution weights. Therefore, flexible convolution provides a method for applying convolution to 3D point clouds where data points may have irregularly spaced neighbors.

[0055] When flexible convolution is used as a building block for the discriminator of a GAN or DNN encoder in a variational autoencoder (VAE), an inverse operator (similar to transposed convolution) can be defined in the decoder of the GAN generator or the VAE decoder. Other deep learning-based methods capture local features in an irreversible manner (e.g., grouping neighboring points and performing convolution on each group (called group operation) to extract local features, but it is unclear how to perform the inverse operation on the group operation). However, querying all neighboring points in 3D space is computationally expensive when using flexible convolution. To address this issue, this application describes a method for projecting 3D points in a point cloud onto a 2D mesh-based data structure.

[0056] To better understand this application, consider the data structure of a 2D image. 2D images are typically encoded as a 2D mesh-based data structure. 2D convolution can be applied to mesh-based data structures, such as images. In these structures, the order of each element is defined based on its position within the mesh. For example, as... Figure 3 As shown, the element in the i-th row and j-th column of the grid is indexed as element a. ij Similarly, the element in the i-th row and j+1-th column is indexed as element a. i(j+1) Similarly, the element in the (i+1)th row and the jth column is indexed as element a. (i+1)j This indexing order is used in 2D convolutions. For example, consider a 3×3 2D kernel containing 9 elements. To perform a 2D convolution on the element in the middle of the 3×3 grid, you could consider the 8 neighboring elements plus the center pixel, for example, using linear convolution.

[0057] For data in 3D point clouds, the points are not ordered in a 2D grid manner, and the neighbors of any given point in a 3D point cloud are not easily identified. Considering all possible neighbors of a given point in a 3D point cloud could also require a significant amount of computation. In this application, such problems are addressed by defining a grid-based structure for the points in a 3D point cloud.

[0058] For example, a 3D point cloud generated by a Velodyne 64 LIDAR can be decomposed into 64 scans over 360°. In some embodiments, the horizontal and vertical dimensions can be quantized (e.g., for efficiency).

[0059] When 2D convolution is applied to 2D data (i.e., 2D image data), the operation exhibits translational isomorphism. Similarly, the grid-based data structure onto which a point cloud is projected should also exhibit rotational isomorphism. To achieve this, a polar coordinate system can be used.

[0060] For example, consider 2D coordinate systems (e.g., for 2D image data) and 3D coordinate systems (e.g., for LiDAR data), such as Figure 4 As shown.

[0061] The projection of a 3D point cloud onto a 2D mesh-based data structure can be defined as follows:

[0062]

[0063] This projection can be conceptually understood as projecting closed loops in a point cloud onto single rows in a 2D grid. When projected in this way, the first and last elements in each row are actually adjacent to each other. To ensure that the projected 2D data accurately reflects this, the leftmost and rightmost columns of the 2D data grid are filled with values ​​from the other side (instead of zero-padding as is typically done when processing 2D images). Figure 5 An example illustrating this situation is provided. For a projected 2D data grid 510, a filled 2D data grid 520 is generated using an added leftmost column 522 and an added rightmost column 524, where the added leftmost column 522 contains the values ​​from the rightmost column 514 of the original projected data grid 510, and the added rightmost column 524 contains the values ​​from the leftmost column 512 of the projected data grid 510. The filled data grid 520 can be generated by adding the added columns 522 and 524 to the projected data grid 510. Alternatively, the filled data grid 520 can be a new data structure created in addition to the projected data grid 510.

[0064] Figures 6A-6C An example of how to project a point cloud onto a 2D data grid using the method described above is shown. Figure 6A This is a 2D representation of a 3D point cloud 610. The point cloud 610 contains information from a 360° view, such as a 360° view around a LiDAR sensor. In the 2D representation shown, the sensor would be located at the center of the image. Figure 6B The diagram illustrates how point cloud 610 is projected onto a 2D data grid format, as described above. Three closed loops 612, 614, and 616 are highlighted in point cloud 610. It should be noted that loops 612, 614, and 616 do not need to be circular; they can also be irregularly shaped and irregularly spaced. Each closed loop 612, 614, and 616 is projected onto corresponding rows 626, 624, and 626 in the 2D data grid 620. Although not explicitly shown, the 2D data grid 620 can be filled as described above. Figure 6C The diagram shows a 2D representation of the point cloud 610, and a visual representation of the 2D data grid 620b after projection.

[0065] When a point cloud is projected onto a 2D data grid, some information may be lost. This problem can be at least partially addressed by generating a filled data grid as described above, which helps ensure that adjacent elements in the leftmost and rightmost columns reflect the data in the point cloud. Other elements not located at the edges of the 2D data grid are likely to be less affected by the projection. Furthermore, the computational efficiency achieved through this method is sufficient to make any potential loss of information an acceptable trade-off.

[0066] In some examples, it may not be necessary to fill the 2D data grid as described above. For example (e.g., in the case of a projected 2D data grid with a large number of columns), the information lost by omitting the added leftmost and rightmost columns may be considered acceptable.

[0067] After projecting the point cloud onto a 2D data grid format, the projected 2D data grid is used as input to learn a generative model, such as a GAN generative model. Therefore, a generative model can be learned to generate synthetic 3D point clouds.

[0068] Figure 7 A flowchart of a method 700 for learning a generative model is shown, which generates one or more batches of synthetic point clouds from sampled data.

[0069] In section 702, the first batch of real point clouds was obtained. This first batch of point clouds can be obtained directly from a LiDAR sensor, for example... Figure 1 The LIDAR sensor of LIDAR unit 114 is shown. The first batch of point clouds can also be obtained, for example, from a database or memory storing point clouds previously obtained from LIDAR unit 114.

[0070] At 704, each point cloud in the first batch of point clouds is projected to generate a projected 2D data grid. The 2D data grid can have a format similar to a 2D image. Projection can be performed using the formula described above:

[0071]

[0072] This operation helps ensure the rotational isomorphism required for processing, for example, LiDAR point clouds. Other formulas can be used for projection, such as any other projection that achieves rotational isomorphism. For example, although this application describes a projection in which closed loops in a point cloud are projected onto rows in a 2D data grid, other methods can project closed loops in a point cloud onto columns in a 2D data grid.

[0073] Another example method for generating a projected 2D data grid, implementing rotational isomorphism, is now described. This example method generates a data grid with H rows and W columns. First, the data points in the point cloud are clustered together according to their elevation angles. This produces H clusters, corresponding to H LiDAR channels (each channel captures data at a specific elevation angle). For each cluster, the data points are sorted in ascending order of azimuth angle (this sorting may not be necessary in some cases where the raw data from the LiDAR cells has already been sorted). To ensure a fixed number of elements per row in the grid, 360° is divided into a fixed number of bins, resulting in W bins, each bin defining a range of azimuth angles. Data points within a cluster are placed into the corresponding bin according to the azimuth angle. For each bin, the average value of the data points belonging to that bin is calculated to obtain a single representative value. Thus, H × W elements are filled into the 2D data grid. In some implementations, the 2D grid can have a depth of 3, where each of the x, y, and z spatial coordinates is recorded at each depth.

[0074] Optionally, at 706, the projected data grid is filled such that elements at the edges of the projected data grid wrap around the opposite edges.

[0075] In the example where the projected data grid contains rows corresponding to closed loops in the 3D point cloud, the 2D data grid can be populated by adding an additional leftmost column containing elements from the rightmost column of the original projected data grid, and adding an additional rightmost column containing elements from the original leftmost column of the projected data grid. The number of columns that fill the edges of the resulting 2D data grid depends on the size of the kernel convolution.

[0076] When a ring in a 3D point cloud is projected onto a column in a 2D data grid, the fill can be modified. In such an example, the projected data grid can be filled by adding an additional top row containing the original bottom row of the projected data grid, and by adding an additional bottom row containing the original top row of the projected data grid. Other such modifications to the fill operation can be used for different projection methods.

[0077] The projected data grid can be populated by directly adding columns / rows to it, or by creating a new data structure for the populated grid. Populating the data grid in this way helps the generative model understand the relationships between points near the edges of the data grid.

[0078] At position 708, the generative model is learned using the projected data grid (with or without padding). The generative model can be learned using flexible convolution operations and transposed flexible convolution operations, and can be learned using a GAN as described above. Learning the generative model may involve repeatedly performing positions 702-708 using different batches of real point clouds.

[0079] A generative model can be learned by inputting a projected data grid into the generative model, so as to learn a generative model that generates synthetic point clouds from sampled 2D data.

[0080] In another possible approach, a generative model can be learned by feeding a 3D point cloud into the generative model, thereby learning to generate a synthetic point cloud from sampled 3D data. As described above, flexible convolution is used to learn the generative model. In this approach, the projected 2D data grid is not used directly as input to the generative model, but rather to aid in the execution of flexible convolution. The projected 2D data grid is used to identify the nearest neighbors for each data point in the point cloud, for computing the flexible convolution. For example, for a given data point in the point cloud, the corresponding data element in the 2D data grid, along with its neighboring elements (e.g., the k nearest neighbors), are identified. Those neighboring elements identified in the 2D data grid are then returned to the point cloud to identify neighboring data points in the point cloud. The flexible convolution can then be performed using those identified neighboring data points in the point cloud.

[0081] At point 710, after the generative model has been appropriately learned, it may optionally be used to generate a batch of synthetic 3D point clouds. For example, data sampled from a selected distribution (e.g., a normal distribution, a Gaussian distribution, or a multinomial distribution, and others, depending on the desired application) may be provided as input to the learned generative model for generating this batch of synthetic 3D point clouds.

[0082] In some examples, step 710 may be performed separately from other steps of method 700. For example, a generative model may be learned in one environment (e.g., a laboratory environment), and the learned generative model may be used to generate a batch of synthetic point clouds in different environments (e.g., in an autonomous vehicle). Therefore, this application also describes a method for generating a batch of synthetic point clouds using a generative model that has been learned in the manner described above.

[0083] In some examples, steps 702-708 can be performed repeatedly. For instance, learning the generative model may include generating a 2D data grid of projections using a second batch of point clouds, and further learning the generative model using the second batch of point clouds and the corresponding 2D data grids of projections. In some examples, steps 702-708 can be performed in parallel with 710. That is, the learning of the generative model can continue over time and can be performed in parallel with the generation of synthetic data through the generative model.

[0084] Method 700 can be used to generate batch synthetic point clouds for training and / or testing purposes, such as for training and / or testing autonomous or semi-autonomous devices.

[0085] The method 700 can be implemented at least partially in an autonomous device. For example, generating synthetic point cloud data using at least a trained generative model can be implemented in an autonomous vehicle (e.g., to supplement missing sensor data).

[0086] In some examples, method 700 may be executed together on the same processing system. In other examples, portions of method 700 may be executed separately or by different systems.

[0087] The method 700 can be at least partially derived from Figure 2 The processing unit 200 executes the above steps. For example, the processing unit 200 may execute instructions to perform steps 702-706 above, thereby generating a projected 2D data mesh from the point cloud. The processing unit 200 may execute instructions to perform step 708 above, thereby learning a generative model using the projected 2D data mesh as input. The processing unit 200 may execute step 710 above to realize the learned generative model for generating a synthetic point cloud.

[0088] In the example described herein, this application enables the generation of synthetic point clouds that more accurately reflect real point clouds (e.g., data from LiDAR units). These synthetic point clouds can be generated by a generative model learned in an unsupervised manner.

[0089] Synthetic point clouds generated in this way can be used to supplement missing point clouds and / or missing data points in a point cloud (e.g., real point clouds obtained using LIDAR cells).

[0090] In some examples, the learned generative model can be combined with a sequence model. A sequence model can be learned to generate sequences of synthetic point clouds. This can be used to generate synthetic temporal data, for example, to predict and / or supplement missing data in a real point cloud. The sequence model can be implemented using a recurrent neural network (RNN) with flexible convolutional operations. In such implementations, the learned generative model can be used within the neurons of the RNN. In some embodiments, the RNN itself can be the learned generative model.

[0091] In some examples, when a learned generative model is implemented in an autonomous device (e.g., an autonomous vehicle), a batch of synthetic point clouds can be generated based on (or in response to) actions of the autonomous device. These actions can be performed in reality or in a simulation. For example, an action of the autonomous device (e.g., a left turn) can be fed back to the generative model so that it can generate a synthetic point cloud reflecting that action.

[0092] In some examples, this application describes a method that includes using a generative model to generate a set of synthetic point cloud data, the generative model being implemented using flexible convolutions in a generative adversarial network. The generative model is trained using one or more projected 2D data grids to generate projected 2D data grids from a corresponding 3D point cloud data set by projecting each set of point cloud data onto the corresponding projected 2D data grid while maintaining rotational isomorphism.

[0093] Although this application describes methods and processes with steps in a certain order, one or more steps of the methods and processes may be omitted or changed as appropriate. One or more steps may be performed in an order different from the order in which they are described, as needed.

[0094] Although this application has been described at least in part according to method, those skilled in the art will understand that this application also relates to various components for performing at least some aspects and features of the described methods, whether by means of hardware components, software, or any combination of both. Accordingly, the technical solutions of this application may be embodied in the form of a software product. Suitable software products may be stored in pre-recorded storage devices or other similar non-volatile or non-transitory computer-readable media, including, for example, DVDs, CD-ROMs, USB flash drives, removable hard drives, or other storage media. The software product includes examples of instructions tangibly stored thereon that enable a processing device (e.g., a personal computer, server, or network device) to perform the methods disclosed herein.

[0095] This application may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are merely illustrative in all respects and not restrictive. Features selected from one or more of the foregoing embodiments may be combined to create alternative embodiments not explicitly described, and features suitable for such combinations should be understood to be included within the scope of this application.

[0096] All values ​​and subranges within the disclosed scope are also disclosed. Furthermore, although the systems, devices, and processes disclosed and illustrated herein may include a specific number of elements / components, systems, devices, and components may be modified to include additional or fewer such elements / components. For example, although any element / part disclosed may be referred to as the singular, the embodiments disclosed herein may be modified to include multiple such elements / parts. The subject matter described herein is intended to cover and encompass all suitable variations in the technology.

Claims

1. A method, characterized in that, include: Obtain the first batch of point clouds representing the 3D environment; A projected 2D data grid is generated for each point cloud in the first batch of point clouds by projecting each point cloud in the first batch of point clouds onto a projected 2D data grid with rotational isomorphism. as well as During the training phase, a generative model is learned to generate one or more batches of synthetic point clouds. The generative model is learned by providing the generative model with a projected 2D data grid of each point cloud from the first batch. The generative model includes flexible convolution operations and transposed flexible convolution operations. The 2D data grid used to generate the projections further includes: Elements of the projected 2D data grid are wrapped from one edge to the opposite edge to fill the projected 2D data grid; and During the training phase, a generative model is learned using a padded projected 2D data grid.

2. The method according to claim 1, characterized in that, Also includes: Using the learned generative model, one or more batches of synthetic point clouds are generated based on data sampled from the distribution.

3. The method according to claim 1, characterized in that, Each point cloud in the first batch of point clouds is projected onto the 2D data grid of the projection using the following formula: Where x, y, and z represent the 3D coordinates of the data points in the point cloud.

4. The method according to claim 3, characterized in that, The 2D data grid of the projection is filled by adding an additional leftmost column containing elements from the original rightmost column of the 2D data grid of the projection, and adding an additional rightmost column containing elements from the original leftmost column of the 2D data grid of the projection, wherein the number of columns filled is determined by the size of the convolution kernel.

5. The method according to claim 4, characterized in that, Each row in the 2D data grid of the projection corresponds to a corresponding closed loop in the point cloud.

6. The method according to any one of claims 1 to 5, characterized in that, Also includes: Use the one or more batches of synthetic point clouds to supplement any missing data points from the point cloud in the first batch.

7. The method according to any one of claims 1 to 5, characterized in that, Also includes: The generative model is combined with a sequence model, wherein the sequence model generates synthetic time data, which is used to predict any missing data points in the point cloud of the first batch.

8. The method according to any one of claims 1 to 5, characterized in that, The learned generative model is a recurrent neural network (RNN).

9. The method according to claim 1, characterized in that, Also includes: Using the learned generative model, one or more batches of synthetic point clouds are generated from data sampled from the distribution based on the actions of the autonomous device.

10. A processing unit, characterized in that, include: processor; A memory for storing computer-executable instructions, which, when executed by the processor, cause the processing unit to perform the method according to any one of claims 1 to 9.

11. A computer-readable storage device for storing instructions, characterized in that, When executed by the processor of the processing unit, the instructions cause the processing unit to perform the method according to any one of claims 1 to 9.

12. A method, characterized in that, include: Obtain the first batch of point clouds representing the 3D environment; A projected 2D data grid is generated for each point cloud in the first batch of point clouds by projecting each point cloud in the first batch of point clouds onto a projected 2D data grid with rotational isomorphism. as well as During the training phase, a generative model is learned, which generates one or more batches of synthetic point clouds. The generative model is learned by providing the first batch of point clouds to the generative model. A projected 2D data grid is used to identify nearest neighbors and to perform flexible convolution and transposed flexible convolution operations during learning. The 2D data grid used to generate the projected data further includes: Elements of the projected 2D data grid are wrapped from one edge to the opposite edge to fill the projected 2D data grid; and During the training phase, a generative model is learned using a padded projected 2D data grid.

13. The method according to claim 12, characterized in that, Also includes: Using the learned generative model, one or more batches of synthetic point clouds are generated based on data sampled from the distribution.

14. The method according to claim 12 or 13, characterized in that, Also includes: The generative model is combined with a sequence model, wherein the sequence model captures temporal data used to predict any missing data points from the first batch of point clouds.

15. The method according to claim 12, characterized in that, The learned generative model is a recurrent neural network.

16. A processing unit, characterized in that, include: processor; A memory for storing computer-executable instructions, which, when executed by the processor, cause the processing unit to perform the method according to any one of claims 12 to 15.

17. A computer-readable storage device for storing instructions, characterized in that, When executed by the processor of the processing unit, the instructions cause the processing unit to perform the method according to any one of claims 12 to 15.

Images