Systems, methods, and products for operating autonomous platforms using information from point cloud data

Abstract

The present disclosure relates to systems, methods, and products for operating an autonomous platform using information from point cloud data. A system for training a neural network for estimating surface normals for use in operating an autonomous platform is described. The system uses a parallelizable k-nearest neighbor sorting algorithm to provide a patch of points sampled from point cloud data as input to a neural network model. The points are transformed from Euclidean coordinates in Euclidean space to spherical coordinates. Polar angles of surface normals for the point cloud data are estimated in spherical coordinates. The trained neural network model is utilized on the autonomous platform, and the estimates of the polar angles of the surface normals are used to guide the operation of the autonomous platform within an environment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This is a non-provisional application of U.S. Provisional Patent Application No. 62 / 984,676, filed on March 3, 2020, entitled “A Neural Network Architecture for Small Lidar Processing Networks for Slope Estimation and Ground Plane Segmentation,” which is incorporated herein by reference in its entirety.

[0003] This is also a non-provisional application of U.S. Provisional Patent Application No. 62 / 984,693, filed on March 3, 2020, entitled “An Automatic Procedure for Either Proving or Identifying Counter-Examples to Correctness Properties for Neural Network Based Slope Estimation and Ground Plane Segmentation Systems that Process Point Cloud Data,” the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present invention relates to a system for training a neural network for estimating surface normals, and more particularly to a system for training a neural network for estimating surface normals for use in operating an autonomous platform. Background Art

[0005] As autonomous systems continue to expand into new and more complex environments, extracting salient information from sensor inputs in real time has become a driving force for developing new neural network models. Point clouds (e.g., those generated by LiDAR and stereo camera sensors) provide a dense stream of three-dimensional information about the world around an autonomous system (e.g., an autonomous vehicle or drone), from which one would like to efficiently infer information relevant to decision making (e.g., surface normals and object segmentation) for safe and successful operation of the autonomous system. However, processing these point clouds to extract this information using traditional analytical methods (see Reference No. 4 in the incorporated reference list) has proven to be neither robust to the noisy data provided by real-world sensors nor computationally complex enough to be applied in real time.

[0006] Neural network models have been shown to perform better when processing noisy point cloud data (see Reference No. 3) and have been successfully applied to the estimation of surface normals (see Reference No. 1); however, they remain computationally expensive because they contain approximately 1 million (M) to 15 million (M) parameters. Neural network models of this size require dedicated graphics processing units (GPUs) for real-time operation on autonomous systems, and the cost, power, and cooling requirements are very high. This makes existing neural network models unsuitable for applications that require both low power and thermal constraints, such as autonomous drones and autonomous underwater vehicles. In addition, for safety-critical applications that expect to provide some form of correctness verification, large networks cannot currently be easily verified using state-of-the-art formal verification techniques for neural networks.

[0007] The closest prior art for surface normal estimation includes a neural network model with 3.5M parameters that consumes local patches of points from a point cloud (see Reference No. 1). The prior art for point cloud segmentation involves a neural network model with 3.5M parameters that consumes the entire point cloud (see Reference No. 3); and a neural network model with 1.5M parameters that consumes the entire point cloud that is first transformed into a spherical image (see Reference No. 5). The work of Reference No. 5 specifically mentioned outperforms the performance of Reference No. 3 by noting the struggles in outdoor environments and solving the task with a smaller model to increase computational speed and reduce the resources required to deploy such a system on an autonomous platform.

[0008] Therefore, there is an ongoing need for a system and method for surface normal estimation for low-power applications that provides point cloud segmentation and performs well in outdoor environments. Summary of the Invention

[0009] The present invention relates to a system for training a neural network for estimating surface normals, and more particularly, to a system for training a neural network for estimating surface normals for use in operating an autonomous platform. The system includes a sensor that generates point cloud data, a non-transitory computer-readable medium having executable instructions encoded thereon, and one or more processors, the non-transitory computer-readable medium having executable instructions encoded thereon, such that when the executable instructions are executed, the one or more processors perform a plurality of operations. The system trains a neural network model, wherein training the neural network model includes providing a patch of points sampled from the point cloud data as input to the neural network model using a parallelizable k-nearest neighbors sorting algorithm, wherein the patch has a center point. The points are transformed from Euclidean coordinates in Euclidean space to spherical coordinates. The polar angle of the surface normal of the point cloud data is estimated in the spherical coordinates. The trained neural network model is utilized on the autonomous platform. The estimated polar angle of the surface normal is used to guide the autonomous platform within an environment.

[0010] In another aspect, the azimuth of the surface normal is estimated in spherical coordinates.

[0011] In another aspect, the system extracts angle and distance relationships between points in the patch and a center point; and provides the angle and distance relationships as input to a neural network model.

[0012] In another aspect, a neural network model is used to estimate the slope of point cloud data.

[0013] In another aspect, a neural network model is used to estimate whether point cloud data corresponds to ground data or non-ground data.

[0014] In another aspect, the autonomous platform is an autonomous ground vehicle, and wherein the autonomous ground vehicle within the operating environment includes a vehicle assembly that controls the autonomous ground vehicle.

[0015] Finally, the present invention also includes computer program products and computer-implemented methods. The computer program products include computer-readable instructions stored on a non-transitory computer-readable medium, which can be executed by a computer having one or more processors, such that when the instructions are executed, the one or more processors perform the operations listed herein. Alternatively, the computer-implemented methods include acts of causing a computer to execute these instructions and perform the resulting operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The objects, features and advantages of the present invention will become apparent from the following detailed description of various aspects of the present invention, taken in conjunction with the following drawings, in which:

[0017] Figure 1 is a block diagram depicting components of a system for training a neural network for estimating surface normals according to some embodiments of the present disclosure;

[0018] Figure 2 is an illustrative diagram of a computer program product according to some embodiments of the present disclosure;

[0019] Figure 3 is an illustration of ground plane segmentation of point cloud data generated by a LiDAR sensor according to some embodiments of the present disclosure;

[0020] Figure 4 is an example diagram of a neural network architecture according to some embodiments of the present disclosure;

[0021] Figure 5 is an illustration of spherical coordinates according to some embodiments of the present disclosure; and

[0022] Figure 6 is a flow chart illustrating a system for estimating surface normals for use in operating an autonomous platform, according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] The present invention relates to a system for training a neural network for estimating surface normals, and more particularly, to a system for training a neural network for estimating surface normals for use in operating an autonomous platform. The following description is presented to enable one of ordinary skill in the art to make and use the invention and incorporate it into the context of a particular application. Various modifications and multiple uses in different applications will be apparent to those skilled in the art, and the general principles defined herein can be applied to a wide range of aspects. Therefore, the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0024] In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without being limited to these specific details. In other cases, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the present invention.

[0025] The reader's attention is drawn to all papers and documents filed concurrently with this specification and open to public inspection together with this specification, and the contents of all such papers and documents are incorporated herein by reference. Unless expressly stated otherwise, all features disclosed in this specification (including any accompanying claims, abstracts, and drawings) may be replaced by alternative features serving the same, equivalent, or similar purposes, and therefore, unless expressly stated otherwise, each feature disclosed is merely one example of a series of equivalent or similar features.

[0026] Before describing the present invention in detail, a list of references is first provided. Next, a description of each major aspect of the present invention is provided. Finally, specific details of each embodiment of the present invention are provided to enable an understanding of the specific aspects.

[0027] (1) List of incorporated references

[0028] The following references are cited and incorporated throughout this application. For clarity and convenience, these references are listed here as a centralized resource for the reader. The following references are incorporated herein by reference as if fully set forth herein. These references are cited in this application by reference to the following corresponding reference numbers:

[0029] 1.Guerrero,P.,Kleiman,Y.,Ovsjanikov,M.,&Mitra,NJ(2018).PCPNet: Learning local shape properties from raw point clouds.Computer GraphicsForum,75-85.

[0030] 2.Iandola,FN(2016).SqueezeNet:AlexNet-level accuracy with 50x fewer parameters and<0.5MB model size.arXiv preprint arXiv:1602.07360.

[0031] 3.Qi, CR, Su, H., Mo, K., & Guibas, LJ (2017). PointNet: Deep learning onpoint sets for 3D classification and segmentation. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[0032] 4. Rusu, RB (2009). Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments. Muenchen, Germany: Computer Science department, Technische Universitaet.

[0033] 5.Wang,YS(2018).Pointseg:Real-time semantic segmentation based on3d LIDAR point cloud.arXiv preprint arXiv:1807.06288.

[0034] (2) Main aspects

[0035] Various embodiments of the present invention include three "main" aspects. The first main aspect is a system for training a neural network for estimating surface normals. The system is typically in the form of computer system operating software or in the form of a "hard-coded" instruction set. The system can be incorporated into a wide variety of devices that provide different functions. The second main aspect is a method, typically in the form of software, operating using a data processing system (computer). The third main aspect is a computer program product. A computer program product typically represents computer readable instructions stored on a non-transitory computer readable medium such as an optical storage device (e.g., a compact disk (CD) or digital versatile disk (DVD)) or a magnetic storage device (e.g., a floppy disk or tape). Other non-limiting examples of computer readable media include: a hard disk, a read-only memory (ROM), and a flash-type memory. These aspects will be described in more detail below.

[0036] Figure 1 A block diagram depicting an example of a system of the present invention (i.e., computer system 100) is provided in . The computer system 100 is configured to perform calculations, processes, operations, and / or functions associated with a program or algorithm. In one aspect, certain processes and steps discussed herein are implemented as a series of instructions (e.g., a software program) residing in one or more computer-readable memory units and executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform specific actions and exhibit specific behaviors, as described herein.

[0037] Computer system 100 may include an address / data bus 102 configured to transmit information. In addition, one or more data processing units (such as processor 104 (or multiple processors)) are coupled to address / data bus 102. Processor 104 is configured to process information and instructions. In one aspect, processor 104 is a microprocessor. Alternatively, processor 104 may be a different type of processor, such as a parallel processor, an application specific integrated circuit (ASIC), a programmable logic array (PLA), a complex programmable logic device (CPLD), or a field programmable gate array (FPGA).

[0038] Computer system 100 is configured to utilize one or more data storage units. Computer system 100 may include a volatile memory unit 106 (e.g., random access memory ("RAM"), static RAM, dynamic RAM, etc.) coupled to address / data bus 102, wherein volatile memory unit 106 is configured to store information and instructions for processor 104. Computer system 100 may also include a non-volatile memory unit 108 (e.g., read-only memory ("ROM"), programmable ROM ("PROM"), erasable programmable ROM ("EPROM"), electrically erasable programmable ROM ("EEPROM"), flash memory, etc.) coupled to address / data bus 102, wherein non-volatile memory unit 108 is configured to store static information and instructions for processor 104. Alternatively, computer system 100 may execute instructions retrieved from an online data storage unit, such as in "cloud" computing. In one aspect, computer system 100 may also include one or more interfaces, such as interface 110, coupled to address / data bus 102. The one or more interfaces are configured to enable the computer system 100 to connect to other electronic devices and computer systems. The communication interface implemented by the one or more interfaces can include wired (e.g., serial cable, modem, network adapter, etc.) and / or wireless (e.g., wireless modem, wireless network adapter, etc.) communication technology. In addition, one or more processors 104 (or devices, such as autonomous platforms) can be associated with one or more associated memories, wherein each associated memory is a non-transitory computer-readable medium. Each associated memory can be associated with a single processor 104 (or device) or a network of interactive processors 104 (or devices) (such as a network of autonomous platforms (e.g., autonomous vehicles, robots)).

[0039] In one aspect, computer system 100 may include an input device 112 coupled to address / data bus 102, wherein input device 112 (e.g., a lidar sensor) is configured to communicate information and command selections to processor 104. According to one aspect, input device 112 is an alphanumeric input device (such as a keyboard) that may include alphanumeric keys and / or function keys. Alternatively, input device 112 may be an input device other than an alphanumeric input device. In one aspect, computer system 100 may include a cursor control device 114 coupled to address / data bus 102, wherein cursor control device 114 is configured to communicate user input information and / or command selections to processor 104. In one aspect, cursor control device 114 is implemented using a device such as a mouse, trackball, trackpad, optical tracking device, or touch screen. Notwithstanding the foregoing, in one aspect, cursor control device 114 is directed and / or activated via input from input device 112, such as in response to the use of special keys and key sequence commands associated with input device 112. In an alternative aspect, cursor control device 114 is configured to be managed or directed by voice commands.

[0040] In one aspect, computer system 100 may also include one or more optional computer-usable data storage devices, such as a storage device 116 coupled to address / data bus 102. Storage device 116 is configured to store information and / or computer-executable instructions. In one aspect, storage device 116 is a storage device such as a magnetic or optical drive (e.g., a hard disk drive ("HDD"), a floppy disk, a compact disk read-only memory ("CD-ROM"), a digital versatile disk ("DVD")). In one aspect, a display device 118 is coupled to address / data bus 102, wherein display device 118 is configured to display video and / or graphics. In one aspect, display device 118 may include a cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a field emission display ("FED"), a plasma display, or any other display device suitable for displaying video and / or graphic images and user-readable alphanumeric characters.

[0041] The computer system 100 presented herein is an example computing environment according to one aspect. However, the non-limiting example of the computer system 100 is not strictly limited to being a computer system. For example, one aspect provides that the computer system 100 represents a class of data processing analysis that can be used according to the various aspects described herein. In addition, other computing systems can also be implemented. In fact, the spirit and scope of the present technology are not limited to any single data processing environment. Therefore, on the one hand, one or more operations of various aspects of the present technology are controlled or implemented using computer-executable instructions (such as program modules) executed by a computer. In one implementation, such program modules include routines, programs, objects, components and / or data structures configured to perform specific tasks or implement specific abstract data types. In addition, one aspect provides implementing one or more aspects of the present technology by utilizing one or more distributed computing environments, such as, in this computing environment, tasks are performed by remote processing devices linked by a communication network, or such as, in this computing environment, various program modules are located in local and remote computer storage media including memory-storage devices.

[0042] Figure 2 A diagram of a computer program product (i.e., storage device) embodying the present invention is depicted in . The computer program product is depicted as a floppy disk 200 or an optical disk 202 such as a CD or DVD. However, as previously mentioned, the computer program product generally represents computer-readable instructions stored on any compatible non-transitory computer-readable medium. The term "instructions" as used with respect to the present invention generally indicates a set of operations to be performed on a computer, and may represent a fragment of an entire program or a single, separate software module. Non-limiting examples of "instructions" include computer program code (source or object code) and "hard-coded" electronic devices (i.e., computer operations encoded into a computer chip). "Instructions" are stored on any non-transitory computer-readable medium, such as in the memory of a computer or on a floppy disk, CD-ROM, and flash drive. In any case, the instructions are encoded on a non-transitory computer-readable medium.

[0043] (3) Specific details of each implementation method

[0044] A system involving a preprocessing pipeline and a learned neural network model is described that: (1) estimates a surface normal vector in three-dimensional Euclidean space given three-dimensional point cloud data; (2) estimates whether the point cloud input points constitute a ground plane (approximately flat) or a non-ground plane; and implements (1) and (2) using a neural network model with approximately 10 thousand (k) parameters (i.e., learned weights of a neural network (e.g., a lidar processing neural network)) with six input points (i.e., points in a spatial sample from a lidar point cloud), compared to the current state of the art (see Reference No. 1) of 3.5 million (M) parameters and 500 input points, thereby allowing the invention to be applied to low-power applications, thermally constrained applications, low-cost applications, applications lacking a graphics processing unit (GPU), and applications requiring formal verification. The learned network model is then used to efficiently infer information relevant to decision making (e.g., surface normals and object segmentation) for safe and successful operation of an autonomous platform in an environment where the three-dimensional point cloud data is obtained. Non-limiting examples of autonomous platforms include autonomous land-based vehicles, autonomous drones, robots, and autonomous underwater vehicles.

[0045] The prior art involves building models that accept point sets in the form of point clouds, local patches from point clouds, or transformations on point sets such that the neural network model is permutation invariant with respect to the input. That is, the output of the neural network model will remain the same for a given point set, regardless of the order of the points in the set. This focus means that these methods necessarily rely on computationally expensive convolutional layers in their neural network architectures to build intermediate features for local patches of points.

[0046] In contrast, the present invention provides a significant improvement over the prior art by completely eliminating the need for convolutional layers using three strategies. First, a parallelizable k-nearest neighbor sorting algorithm provides local patches of points sorted by their distance from a central point as input to the neural network model described herein, thereby eliminating the need to develop a permutation-invariant neural network architecture. Second, the points are transformed from their x-, y-, and z-coordinate representation in Euclidean space into a The spherical coordinates are represented by is the azimuth angle and θ is the polar angle. Third, the task of estimating the surface normal from its (x, y, z) vector in Euclidean space is transformed into estimating the azimuth and polar angle of the surface normal in spherical space or simply estimating the polar angle θ.

[0047] By encoding more information into the input representation (i.e., the explicit relationship between all points and the center point and their sorted order, as described below) and reducing the information in the output representation to only what is necessary (i.e., assuming that surface normals have a magnitude of 1), the invention described herein is able to achieve the same task as the prior art using very small, dense neural networks on the order of 10k and 5k parameters. Furthermore, PointNet (see Reference No. 3) and PCPNet (see Reference No. 1) each demonstrated their approach on synthetic volumetric data (and 1 meter (m) voxels for 3D room scans) on the order of 3.5M parameters, where points can be arbitrarily sampled from surfels (surface elements) of a point cloud input set of 4096 and 500 points, respectively. ThetaNet (an embodiment of the present invention) has only 10k parameters, operates on an input of 6 points, and provides comparable performance to PCPNet on the constrained ground plane segmentation task, as described in detail below.

[0048] The present invention allows neural network models that perform tasks (1) and (2) above to be applied to low power, low cost, or thermally constrained applications and applications requiring formal verification. The invention described herein transforms the task of surface normal estimation into an estimate of the polar angle (θ in spherical coordinates) and azimuthal angle (φ in spherical coordinates) of the surface normal at the center point of the patch, or more simply an estimate of the polar angle θ alone. This transformation offloads the neural network from the burden of learning the magnitude of the surface normal, which is independent of the task and is assumed to be uniform by definition, thereby allowing the size of the neural network required to complete the task to be reduced. State-of-the-art methods require the processing power of a GPU to run the neural network models in real time, thus limiting their application to those systems that can support the high cost, power, and thermal load of a GPU.

[0049] Furthermore, state-of-the-art methods have been demonstrated on point cloud data generated from 3D models and room scans, which provide facets from which an arbitrary number of points can be sampled. In contrast, real-world tasks involve noisy LiDAR point cloud data that is biased by the number and orientation of "rings" on a given LiDAR sensor, and the current method of Reference No. 3 is known to have difficulties in this regard. Experimental studies of the disclosed invention demonstrate that the present invention performs well on the challenging noisy point cloud data of the prior art and can be transferred between varying vehicle platforms and LiDAR sensors without the need to retrain the neural network model.

[0050] One embodiment of the invention described herein (referred to as “CrossNet”) involves estimating the polar angles (θ in spherical coordinates) and azimuth angles (φ in spherical coordinates) of a patch of points sampled from a point cloud generated by a rotating lidar sensor or a stereo camera sensor. Figure 3 Depicted is the process by which CrossNet 300 consumes point cloud data 302 generated by a lidar sensor 304 to segment it into ground planes and non-ground planes (ie, segmented point cloud 306 ).

[0051] The second embodiment of the present invention (referred to as "ThetaNet") involves estimating only the polar angle θ of the patch of points. ThetaNet is a method for training a neural network to perform point-wise surface normal estimation and ground plane segmentation, which trains a neural network f to estimate the features of a local patch of points from a 3D point cloud. For a given point For n-many points within a radius r of x The sample is sampled, preprocessed using a preprocessing function η, and finally provided as input to a neural network model for local feature estimation. Figure 4 The neural network architecture of CrossNet 300 is illustrated, which has An input layer 400, three densely connected hidden layers 402, and two outputs consisting of polar angle θ 404 and azimuth angle φ 406. In order to perform applications for segmenting and estimating features of point cloud data (e.g., point cloud data of the type generated by lidar and stereo camera sensors), the present invention must provide a form of (i.e., the input must be an n-tuple of three floating-point values, where these values correspond to the x-, y-, and z-coordinates of a point in Euclidean space.) Additionally, during training, the invention must be provided with a ground-truth estimate of the surface normal for each point in the point cloud.

[0052] The invention described herein is constrained by the desire to operate in low power (e.g., ~0.6 Watt), low cost, and thermally constrained or safety critical (and therefore requiring formal verification) applications, which imposes strict constraints on the number of parameters of the models according to embodiments of the present disclosure. To provide such performance on a constrained computational budget, the present invention reformulates the task as an estimation of the polar angle θ and azimuthal angle φ of the surface normal of the center point of the sampled patch, or simply an estimation of the polar angle θ. Figure 5The spherical coordinate convention utilized is shown in Figure 1. Given an estimate of the surface normal f(η(X)), one can then simply set a threshold angle ∈ of deviation from the ground plane, where values below this threshold angle are classified as "ground" and values above this threshold angle are classified as "non-ground".

[0053] Since the invention described herein can be used to train a neural network to perform ground plane segmentation from lidar point cloud data via supervised learning, point cloud data with ground truth surface normal labels is required. In the experimental study, data was collected in simulation using Microsoft's AirSim autonomous vehicle simulation software. AirSim provides autonomous vehicle simulation using the NVIDIA PhysX vehicle dynamics model (developed by NVIDIA at 2788 San Tomas Expressway, Santa Clara, CA 95051), which has a complete sensor set including: camera, global positioning system (GPS), inertial measurement unit (IMU) and lidar, which are packaged as a plug-in for the 3D game engine Unreal Engine developed by Epic Games at 620 Crossroads Blvd., Cary, NC 27518.

[0054] During training, a batch of n-point patches with their associated ground truth labels is fed to the neural network and optimized using standard minibatch backpropagation. A patch consists of the n-1 nearest neighbors of a point in the point cloud for which the polar angle of the surface normal is to be estimated. In practice, this angle is estimated for each point in the point cloud by other strategies, such as assigning the angle estimate to all points in a given patch. Once each n-point patch is sampled from the original point cloud, it is then preprocessed using a preprocessing function η before being presented to the network. η is the Cartesian coordinate Convert a patch of n points to spherical coordinates The patch is first centered around its center point, although other methods such as centered the patch around the mean point are used. The centered points are then transformed from their Cartesian representation into spherical coordinates. The radii of the points are then normalized with respect to the patchwise sum of the radii. The points are presented to the neural network, which is sorted by their distance from the center point. Compared to existing techniques (such as PCPNet or PointNet), since all three methods in practice require a radius search around each point in the LiDAR point cloud, there is no additional computational cost in computing the k nearest neighbors of a point. The preprocessing function η extracts the angle and distance relationship between the points around the center point of the patch present in the Cartesian point representation, effectively offloading the burden of learning this intermediate representation from the neural network.

[0055] In practice, generalization from AirSim simulation to real-world lidar platforms was achieved by performing voxel-based downsampling of the lidar point cloud before sampling and preprocessing the patches. Voxel-based downsampling is used to bring the distribution of points closer to that of the training set, thereby reducing the number of points for which surface normals must be estimated, and to reduce the noise present in real-world sensor data by replacing points present in a voxel with a randomly selected single representative point or its mean. The final ThetaNet model, trained only in simulation, was utilized and demonstrated on both a remote control (RC) car platform using a Velodyne Puck LITE VLP-16 lidar sensor with 16 "rings" and an autonomous vehicle platform Polaris MRZR using a Velodyne HDL-32E lidar sensor with 32 "rings". Both vehicle demonstrations provided performance comparable to that observed in simulation. Velodyne Lidar is located at 5521 Hellyer Avenue, San Jose, CA 95138. Furthermore, experimental studies demonstrated successful formal verification of the properties of the resulting network.

[0056] In summary, the invention described herein utilizes a neural network model with 10k parameters (or less) that outperforms the prior art (e.g., Reference No. 5) in terms of cost, computation, and energy savings by not requiring any GPU whatsoever, which still requires a GPU, albeit as a lower cost embedded model. Furthermore, the work described in Reference No. 5 does not estimate surface normals and only provides point cloud segmentation. The invention according to an embodiment of the present disclosure provides surface normal estimates and utilizes these estimates to provide segmentation of a point cloud into ground plane points and non-ground plane points. While the work of Reference No. 1 can be similarly used to segment a point cloud into ground plane points and non-ground plane points given a surface normal, the invention described herein achieves the same goal using only 10k parameters, compared to the 3.5M parameters in the model disclosed in Reference No. 1. The work of Reference No. 3 does not provide surface normal estimation, although it does provide point cloud segmentation. However, as previously noted by Reference No. 5, the work of Reference No. 3 does not perform well in outdoor environments, which is precisely the type of environment in which many land, air, and underwater autonomous systems are intended to operate.

[0057] Figure 6 is a flow chart illustrating a system for operating an autonomous platform in an environment using information extracted from point cloud data in the environment. As described above, a neural network model is trained (element 600) using a parallelizable k-nearest neighbor sorting algorithm, wherein the neural network model is provided with patches of points sampled from the point cloud data as input (element 602). The points are transformed from Euclidean coordinates in Euclidean space represented by (x, y, z) to spherical space represented by (x, y, z). The spherical coordinates of the point cloud data are represented (element 604). The polar angles of the surface normals of the point cloud data are then estimated in spherical space (element 606). The trained neural network model is deployed on the autonomous platform (element 608). The estimation of the polar angles of the surface normals provides guidance for the operation of the autonomous platform within the environment (element 610) for safe and successful operation of the autonomous platform. Safe and successful operation requires preventing vehicle / platform collisions and rollovers. In addition, successful operation requires establishing a trajectory that enables the autonomous platform to operate and maneuver within the environment.

[0058] The system can be embedded and deployed in assisted-operation vehicles with human operators and autonomous platforms, such as unmanned aerial vehicles (UAVs), unmanned ground vehicles (UGVs), autonomous underwater platforms, and autonomous vehicles. For example, in an autonomous ground vehicle, the system can be used to determine trajectories for maneuvering and operating within an environment. Based on the estimated surface normals, the system can control vehicle components such as the throttle / brakes, acceleration / deceleration controls, steering, suspension, or safety devices (e.g., airbags, seatbelt tensioners, etc.), or any combination thereof, to safely and efficiently operate within the environment. Additionally, as the autonomous platform advances through the environment in real time, the estimated surface normals can be used to generate alerts to the vehicle's operating system (or human user). These alerts can be visual (e.g., a message displayed on a screen), audible (e.g., a beep), or any combination thereof. Furthermore, the alerts can be generated in conjunction with controls on vehicle components, causing the vehicle to brake, steer, or otherwise assert the alert. The system can also be configured to perform vehicle maneuvers by directly controlling the steering, brake, or throttle controls, rather than providing an alert to a human or another mechanical system.

[0059] The present invention is applicable to any program or product line that utilizes point cloud data, non-limiting examples of which include autonomous or semi-autonomous vehicle platforms that utilize lidar or stereo camera systems. For example, the invention described herein has been successfully deployed and demonstrated on a land-based autonomous vehicle, specifically the Polaris MRZR. For example, based on the capabilities and mission of the vehicle, slope estimates can be used to determine impassable terrain or estimate the "cost" of entering the terrain for utilization by the vehicle's path planning system. Slope estimates can also be fused with other sensor inputs (e.g., camera, radar) and utilized in a similar manner by the vehicle's path planning system (i.e., path planner).

[0060] Finally, while the present invention has been described in terms of several embodiments, one of ordinary skill in the art will readily recognize that the present invention may have other applications in other contexts. It should be noted that many embodiments and implementations are possible. Furthermore, the appended claims are in no way intended to limit the scope of the present invention to the specific embodiments described above. In addition, any recitation of "means for..." is intended to induce a means-plus-function reading of the element and claim, and any element not specifically recited using "means for..." should not be interpreted as a means-plus-function element, even if the claim otherwise includes the term "means." Furthermore, while particular method steps have been recited in a particular order, these method steps may be performed in any desired order and fall within the scope of the present invention.

Claims

1. A system for operating an autonomous platform in an environment using information extracted from point cloud data in the environment, the system comprising: a sensor that generates point cloud data; as well as A non-transitory computer-readable medium and one or more processors, the non-transitory computer-readable medium being encoded with executable instructions such that when the executable instructions are executed, the one or more processors perform the following operations: The neural network model is trained using image-free data, wherein the image-free data is entirely point cloud data, and the training includes: providing a patch of points sampled from the point cloud data as input to the neural network model using a parallelizable k-nearest neighbor sorting algorithm, wherein the patch of points has a center point; transforming the patch of points from Euclidean coordinates in Euclidean space to spherical coordinates; Generate normalized points by projecting onto the unit sphere; and estimating the polar angle of the surface normal of the normalized point in the spherical coordinates; utilizing the trained neural network model on the autonomous platform; and The estimate of the polar angle of the surface normal is used to guide operation of the autonomous platform within the environment.

2. The system according to claim 1, wherein: The one or more processors further perform the following operations: estimating an azimuth angle of the surface normal in the spherical coordinates.

3. The system according to claim 1, wherein: The one or more processors also perform the following operations: extracting the angle and distance relationship between points in the patch and the center point; and The angle and distance relationship is provided as input to the neural network model.

4. The system according to claim 1, wherein: The neural network model is used to estimate whether the point cloud data corresponds to ground data or non-ground data.

5. The system according to claim 1, wherein: The autonomous platform is an autonomous ground vehicle, and wherein operating the autonomous ground vehicle within the environment includes controlling a vehicle assembly of the autonomous ground vehicle.

6. A computer-implemented method for operating an autonomous platform in an environment using information extracted from point cloud data in the environment, the method comprising the following acts: causing one or more processors to execute instructions encoded on one or more associated memories, each of which is a non-transitory computer-readable medium, such that, when executed, the one or more processors perform the following operations: The neural network model is trained using image-free data, wherein the image-free data is entirely point cloud data, and the training includes: providing a patch of points sampled from the point cloud data as input to the neural network model using a parallelizable k-nearest neighbor sorting algorithm, wherein the patch of points has a center point; transforming the patch of points from Euclidean coordinates in Euclidean space to spherical coordinates; Generate normalized points by projecting onto the unit sphere; and estimating the polar angle of the surface normal of the normalized point in the spherical coordinates; utilizing the trained neural network model on the autonomous platform; and The estimate of the polar angle of the surface normal is used to guide operation of the autonomous platform within the environment.

7. The method according to claim 6, wherein: The one or more processors further perform the following operations: estimating an azimuth angle of the surface normal in the spherical coordinates.

8. The method according to claim 6, wherein: The one or more processors also perform the following operations: extracting the angle and distance relationship between points in the patch and the center point; and The angle and distance relationship is provided as input to the neural network model.

9. The method according to claim 6, wherein: The neural network model is used to estimate whether the point cloud data corresponds to ground data or non-ground data.

10. The method according to claim 6, wherein: The autonomous platform is an autonomous ground vehicle, and wherein operating the autonomous ground vehicle within the environment includes controlling a vehicle assembly of the autonomous ground vehicle.

11. A computer program product for operating an autonomous platform in an environment using information extracted from point cloud data in the environment, the computer program product comprising: Computer-readable instructions stored on a non-transitory computer-readable medium, the computer-readable instructions being executable by a computer having one or more processors to cause the processor to: The neural network model is trained using image-free data, wherein the image-free data is entirely point cloud data, and the training includes: providing a patch of points sampled from the point cloud data as input to the neural network model using a parallelizable k-nearest neighbor sorting algorithm, wherein the patch of points has a center point; transforming the patch of points from Euclidean coordinates in Euclidean space to spherical coordinates; Generate normalized points by projecting onto the unit sphere; and estimating the polar angle of the surface normal of the normalized point in the spherical coordinates; utilizing the trained neural network model on the autonomous platform; and The estimate of the polar angle of the surface normal is used to guide operation of the autonomous platform within the environment.

12. The computer program product of claim 11, the one or more processors further configured to estimate an azimuth angle of the surface normal in the spherical coordinates.

13. The computer program product of claim 11, wherein: The one or more processors also perform the following operations: extracting the angle and distance relationship between points in the patch and the center point; and The angle and distance relationship is provided as input to the neural network model.

14. The computer program product of claim 11, wherein: The neural network model is used to estimate whether the point cloud data corresponds to ground data or non-ground data.

15. The computer program product of claim 11, wherein: The autonomous platform is an autonomous ground vehicle, and wherein operating the autonomous ground vehicle within the environment includes controlling a vehicle assembly of the autonomous ground vehicle.

Images