Using deep learning to identify road geometries from point clouds

Abstract

The lidar generates a three-dimensional point cloud. Objects need to be detected and tracked from the point cloud. For example, embodiments detect from the point cloud what kind of the detected object is, what shape of the object is, and the current location and trajectory of the object. Each lidar sensor on the vehicle periodically generates a point cloud. The point cloud is input into a deep learning neural network that outputs road geometric features, such as road edges and lane separation lines. In some embodiments, other networks may also output information about other objects in the environment, such as other vehicles and pedestrians.

Description

Technical Field

[0001] This field involves interpreting point clouds, such as those generated from vehicle-mounted LiDAR. Background Art

[0002] Lidar (or light detection and ranging) is a remote sensing technology that uses laser pulses to measure distances and create high-resolution maps of the surface of the Earth or other objects. Lidar works by sending a beam of light from a transmitter and measuring the timing and intensity of the reflected signal from a receiver. By scanning the beam across a target area, Lidar generates a three-dimensional data point cloud that represents the shape, height, and features of terrain, vegetation, buildings, or other structures. Lidar can be mounted on vehicles to support driver assistance and autonomous vehicles.

[0003] ADAS (Advanced Driver Assistance Systems) are technologies that enhance the safety and convenience of drivers and passengers by providing assistance, warnings or interventions in various driving scenarios. ADAS can involve parking sensors, blind spot monitors, adaptive cruise control and lane keeping assist, as well as fully autonomous systems such as self-parking and self-driving. ADAS relies on a variety of sensors, cameras, radars, lidars and software to perceive the environment, detect potential hazards and communicate with the driver or other vehicles. ADAS can help reduce human errors, improve traffic flow, reduce fuel consumption and prevent collisions and injuries.

[0004] Autonomous vehicles (also known as self-driving cars) are vehicles that can operate without human intervention or supervision using sensors, cameras, software and artificial intelligence to perceive and navigate their environment. Autonomous vehicles have the potential to improve road safety, mobility, efficiency and environmental sustainability by reducing human error, traffic congestion, fuel consumption and greenhouse gas emissions.

[0005] As mentioned above, lidar sensors can generate point clouds. A point cloud is a collection of data points that represent the shape and surface of an object or scene in three-dimensional space. In addition to lidar, point clouds can also be generated from other sensors including radar and cameras. These point clouds can introduce challenges such as noise, incompleteness, redundancy, and complexity that require efficient and robust processing and representation methods.

[0006] Improved methods for interpreting point clouds are needed. Summary of the invention

[0007] According to one embodiment, a computer-implemented method for detecting road geometry from a point cloud is provided. In the method, at least one sensor on a vehicle is used to determine a point cloud representing the surroundings of the vehicle. The surroundings of the vehicle are partitioned into a plurality of voxels. Each voxel represents the volume of the surroundings of the vehicle. For each of the plurality of voxels, it is determined whether three-dimensional data exists for the corresponding voxel. When it is determined that three-dimensional data exists, data representing a point from a point cloud located within the corresponding voxel is input to a first neural network segment, which creates a fixed-size feature for each voxel. The first neural network segment uses full connection and pooling operations to accomplish this task. After the feature encoding segment, the voxel features enter the second neural network segment, which performs various types of sparse 3D convolutions on the input. The 3D sparse segment then leads to a 2D convolution segment before finally outputting road geometry and objects within the scene. Road geometry and object detection are implemented as two heads of the same network.

[0008] System, apparatus, and computer program product aspects are also disclosed.

[0009] Further features and advantages as well as the structure and operation of various aspects will be described in detail below with reference to the accompanying drawings. It should be noted that the specific aspects described herein are not intended to be limiting. The aspects presented herein are for illustrative purposes only. Based on the teachings contained herein, additional aspects will be apparent to those skilled in the relevant art. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The features and advantages of the example embodiments described herein will become apparent to those skilled in the art to which the present disclosure relates after reading the following description with reference to the accompanying drawings.

[0011] Figure 1 A flow chart for identifying road geometry from a point cloud is illustrated.

[0012] Figure 2 A flow chart for determining a point cloud from multiple lidar scans is illustrated.

[0013] Figure 3 is a diagram showing a vehicle having multiple lidar sensors.

[0014] Figure 4 is a diagram illustrating example point clouds generated from multiple lidar sensors.

[0015] Figure 5 is a diagram showing how the surroundings of a vehicle are partitioned into voxels and how these voxels are analyzed to determine whether data exists within these voxels.

[0016] Figure 6 is a diagram illustrating how features are generated from points within a voxel.

[0017] Figure 7 is a diagram showing how features of various point clouds are processed by a deep learning neural network to identify objects and road segments.

[0018] FIG. 8A to FIG. 8B is a diagram illustrating recognized objects and road geometry.

[0019] In the drawings, like reference numbers generally refer to the same or similar elements. Additionally, generally, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears.

[0020] Aspects of the present disclosure will be described with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] LiDAR generates a three-dimensional point cloud. Objects need to be detected and tracked from the point cloud. For example, an embodiment detects from the point cloud what kind of object is detected, what the shape of the object is, and the current position and trajectory of the object. Each LiDAR sensor on the vehicle periodically generates a point cloud. The three-dimensional point cloud is aggregated between LiDAR sensors and takes into account the ego-motion of the vehicle over time. Each point includes the reflectivity data detected for that voxel.

[0022] In the case where the point cloud is assembled, the cloud is partitioned into voxels. Each voxel may represent a voxel pattern of the surroundings occupying the vehicle. For those voxels that contain data, a multi-level 3D convolutional neural network is applied. The network outputs road geometry features, such as road edges and lane dividers. In some embodiments, the network may also output information about other objects in the environment (such as other vehicles and pedestrians) on a parallel network head.

[0023] The road geometry can be used to locate the vehicle by comparing it to a known map. In addition, the road geometry can be used to control the vehicle, for example to ensure that the vehicle remains in its proper lane.

[0024] Figure 1 A method 100 for identifying road geometry from a point cloud is illustrated. In one embodiment, the method 100 may be implemented in any combination of hardware, software, or firmware. In one example, the method 100 may be implemented by a computing device on a motor vehicle. A motor vehicle is a machine that uses an engine, motor, or other power source to transport people or objects over a road or other surface. A motor vehicle typically has wheels, brakes, steering, and a body or chassis that supports passengers and cargo. Some common types of motor vehicles are cars, trucks, buses, and motorcycles.

[0025] At 102, a point cloud representing the surroundings of the vehicle is determined using at least one sensor on the vehicle. As mentioned above, a point cloud is a collection of data points representing the shape and surface of an object or scene in three-dimensional space. In various embodiments, cameras, radars, and lidars may be used to generate the point cloud. In examples utilizing cameras, the point cloud is constructed using photogrammetry.

[0026] Radar is a technology that uses electromagnetic waves to detect and measure the distance, speed, and direction of objects. Radar can also be used to create a point cloud, which is a collection of points that represent the shape and surface of an object or scene in three dimensions. To create a point cloud using radar, a transmitter sends out a pulse of radio waves that travels in a specific direction and is reflected by any object in its path. The receiver then captures the reflected waves and records their time, frequency, and angle of arrival. By using multiple pulses and receivers, or by moving the transmitter and receiver, a radar system can scan a large area and collect multiple reflections from different points on an object or scene. The radar system then processes the data and converts it into a coordinate system that represents the location and intensity of each point.

[0027] As mentioned above, LiDAR is a remote sensing technology that uses laser pulses to measure the distance and reflectance of objects on the ground or in the air. A LiDAR system consists of a laser source, a scanner, a detector, and a computer. The laser source emits a beam of light that is directed by the scanner to scan a specific area or angle. The detector receives the reflected or scattered light from the target and measures the time it takes for the light to travel back. The computer then calculates the distance and position of the target based on the speed of light and the angle of the scanner. By rotating or scanning the device horizontally and vertically, LiDAR captures multiple reflection points and forms a scan, or a series of measurements covering a specific area or volume. About Figure 2 A more detailed description of how LiDAR can be used to generate point clouds

[0028] Figure 2 A flow chart illustrating an example of how a point cloud may be determined at step 102 is illustrated. Figure 2 In the example above, a point cloud is determined from aggregating multiple lidar scans.

[0029] At 202, a plurality of sensor scans are received. Each sensor scan includes a plurality of points detected in a surrounding environment at different times. The plurality of sensor scans may be collected from a plurality of lidar sensors on a vehicle.

[0030] Figure 33 is a diagram 300 showing a vehicle 302 and its surroundings. The surroundings include a road bounded by road edges 310A-310B. Road edges 310A-310B are the boundaries of the road surface that separate it from adjacent terrain such as shoulders, curbs, sidewalks, ditches, or vegetation. Road edges may be marked by various features (such as pavement markings, rumble strips, guardrails, or signs) to enhance visibility and safety of the road for drivers and pedestrians. Road edges may also be unmarked or indistinct, particularly on rural or unpaved roads, which may present challenges for navigation and lane keeping.

[0031] The road also has multiple lanes. A lane is a subdivision of a road that is used to separate traffic in the same or opposite directions, or for specific purposes such as turning, merging, or parking. Lanes help organize and regulate traffic, improve safety and efficiency, and reduce congestion and collisions. Lanes are typically marked by lane dividers (such as painted lines, signs, or symbols on the road surface) or by physical barriers (such as curbs, medians, or guardrails). Figure 3 The road depicted in FIG. 4 has two lanes separated by lane divider 312 .

[0032] exist Figure 3 , the surroundings of vehicle 302 further include another vehicle 306 and a pedestrian 308. In one example, vehicle 306 may be traveling in an opposite direction relative to vehicle 302 and pedestrian 308 may be stationary.

[0033] As mentioned above, the vehicle 302 may have multiple lidar sensors. The multiple lidar sensors include a long-range lidar sensor 304 and multiple near-field, short-range lidar sensors 306A-306C. The long-range lidar sensor 304 is positioned high on the vehicle, while the near-field lidar sensors 306A-C are positioned around the vehicle 302 to capture blind spots from the long-range lidar sensor 304. Example lidar sensors are available from Luminar Technologies Inc. of Palo Alto, California, and ZVISION of Beijing, China. In operation, as the vehicle 302 drives along a road, each of the sensors 304 and 306A-C may perform several scans. The multiple sensor scans received at step 202 may include multiple scans captured consecutively in time from each of the sensors 304 and 306A-C.

[0034] At 204, the points from each of the plurality of sensor scans are adjusted to correct for the ego-motion of the vehicle 302. Ego-motion is a term used to describe the motion of a vehicle relative to its own reference frame, or how it perceives its position, orientation, and velocity in the environment. After the adjustment in step 204, each scan received at step 202 should have a common reference frame. Ego-motion can be estimated from a variety of sensors such as inertial measurement units, global positioning systems (GPS), wheel odometry using techniques such as visual odometry, visual inertial odometry, or simultaneous localization and mapping.

[0035] At 206 , the points adjusted in 204 are aggregated to determine a point cloud describing the surroundings of the vehicle 302 . Figure 4 is a diagram 400 illustrating an example point cloud generated from multiple lidar sensors.

[0036] Each point in the point cloud includes a location in three-dimensional space, a timestamp when the location was detected, and the reflectivity detected at the location. Lidar reflectivity is a measure of how much light is scattered back to the lidar sensor by the target object. In further embodiments, each point may additionally include a Doppler value detected at the location. The Doppler value is a measure of the change in frequency or wavelength of an electromagnetic wave due to the relative motion of the source and the observer. In this way, the Doppler value may indicate the relative motion of the target object when the point was detected.

[0037] As mentioned above, once aggregated, the point cloud in step 206 may include points from multiple scans captured at different times. Since the points are captured at different times, objects that are moving relative to the earth may be blurred. Figure 3 In the example of , vehicle 306 may be in motion. Therefore, the point representing vehicle 306 is located at Figure 4 400 in FIG. As will be discussed below, the network can be trained so that this blur can be used to determine the speed of the object.

[0038] Steering Figure 1 At 104, the surroundings of the vehicle are partitioned into a plurality of voxels. Each voxel represents a volume of the surroundings of the vehicle. Partitioning the surroundings involves voxelization of the region. Voxelization converts the region into a discrete representation consisting of small cubic units called voxels. This is done in Figure 5 Shown in. Figure 5 is a diagram 500 showing how the surroundings of a vehicle may be partitioned into voxels, such as voxels 502A-502B.

[0039] At 106, for each of the plurality of voxels, it is determined whether three-dimensional data exists for the corresponding voxel. In one embodiment, if any point is in the point cloud, it may be determined that three-dimensional data exists. If the voxel lacks any point, it may be determined that three-dimensional data does not exist in the voxel. In another embodiment, it may be determined that three-dimensional data exists only when a threshold number of points exist.

[0040] At 108, when it is determined in step 106 that three-dimensional data exists, data representing points from the point cloud located within the corresponding voxel is input into the sparse convolutional neural network. By limiting the analysis to those voxels where data exists, embodiments can improve efficiency and avoid the need to analyze the entire 3D scene representing the surrounding environment of the vehicle.

[0041] Before inputting into the sparse convolutional neural network, it may be necessary to extract features from the points located within the corresponding voxels. As mentioned above, each point can be represented by coordinates in three-dimensional space, reflectivity values, timestamps, and possibly Doppler values. In one embodiment, an average value for each of these values ​​can be determined to calculate the features. In another embodiment, a neural network can be used to determine the features from the voxels. Figure 6 Features of the points shown.

[0042] Figure 6 600 is a diagram illustrating how features are generated from a point 504 within a voxel 502. The data represented by the point 504 can be formatted as a point-by-point input 602 in a fully connected neural network 604. The fully connected neural network 604 outputs a point-by-point feature 605. The point-by-point feature 605 is processed using an element-by-element max pool 606 to produce a local aggregate feature 607. The element-by-element max pool 606 is a pooling operation that calculates the maximum value for a patch of a feature map and uses it to create a downsampled (pooled) feature map. The point-by-point feature 605 is cascaded with the local aggregate feature 607 at a point-by-point concatenation 608 to produce a point-by-point concatenated feature 610. In this way, a feature representing a point cloud located within a corresponding voxel is determined.

[0043] Once features representing the point cloud located within the corresponding voxel have been determined, the features are input into a sparse convolutional neural network at 108. Figure 7 Shown in.

[0044] Figure 7 700 is a diagram showing how features of various point clouds are processed by a deep learning neural network to identify objects and road segments. Diagram 700 shows features 702 being input into a sparse convolutional neural network 704.

[0045] The sparse convolutional neural network 704 is a neural network. A neural network is a computational model that mimics the structure and function of biological neurons, which can learn from data and perform tasks such as classification, regression, or generation. A neural network consists of multiple layers of artificial neurons or units connected by weighted links or synapses, and processes information by applying activation functions and propagating signals forward and backward. A convolutional neural network (CNN) is a neural network specifically designed for processing spatial data (such as images, video, or audio). CNN uses convolutional layers (which apply filters or kernels to extract local features from inputs) and pooling layers (which reduce dimensionality and introduce invariance to translation, rotation, or scaling).

[0046] The sparse convolutional neural network 704 is a variant of the CNN that exploits the sparsity of the input or output, meaning that most values ​​are zero or close to zero. The sparse convolutional neural network 704 uses sparse convolutions, which only compute outputs for non-zero input values, and sparse pooling, which only retains non-zero output values. The sparse convolutional neural network 704 can reduce the computational cost and memory usage of the CNN, and can also capture more fine-grained and contextual information from sparse data.

[0047] Based on features 702, sparse convolutional neural network 704 assembles a two-dimensional grid 706 representing convolution values ​​determined by sparse convolutional neural network 704. Two-dimensional grid 706 may correspond to a top-down view of the vehicle's surroundings, and each element in two-dimensional grid 706 may include multiple channels of data.

[0048] Back to Figure 1 At 110, the convolution values ​​determined at 108 for the plurality of voxels are input into the second neural network segment. Figure 7 As shown, mesh 706 is input into convolutional neural network 708. Convolutional neural network 708 outputs road geometry 710 and objects 712.

[0049] The road geometry 710 includes road edges and lane dividers. In one example, the road geometry 710 may include a two-dimensional grid that represents the vehicle's surroundings from top to bottom. Each element of the grid may indicate whether the road edge or lane divider passes through the location of the grid element and its direction information. Based on this, the splines representing the corresponding road edges and lane dividers may be interpolated.

[0050] It is worth noting that lane dividers typically do not have a three-dimensional profile because they are simply painted on the street. However, according to one embodiment, lidar data can be used to detect lane dividers because reflectivity values ​​vary between painted and unpainted road surfaces.

[0051] Object 712 may identify the object's current location (e.g., its center point), what the object is (e.g., a sign, traffic light, vehicle, or pedestrian), what shape the object has (e.g., its size), and what its trajectory is (its speed, including direction of travel).

[0052] FIG. 8A to FIG. 8B is a diagram illustrating recognized objects and road geometry. Fig. 8A A top-down view 800 is shown and Figure 8B A perspective view 850 is shown. Splines 804 represent lane dividers, while ridges 802 represent road edges (shown in both views 800 and 850). View 800 includes an object at center point 806 representing a detected object, and view 850 shows a polygon 852 representing the detected object.

[0053] Return to Figure 7 In one embodiment, networks 704 and 708 may be trained together. The training set may be applied with examples of point clouds that have been labeled to identify objects and road geometry. The labeled examples may be applied to a back-propagation algorithm to update weights and biases in networks 704 and 708. Back-propagation may be used to train convolutional layers and fully connected layers. In one example, the training set may be human generated. Additionally or alternatively, the training set may be generated using a larger, more complex neural network, or using data generated by a lidar sensor with a higher resolution than that available on the vehicle.

[0054] Return to Figure 1 Once the road geometry and objects are determined at 110, the vehicle is controlled at 112 based on the identified road geometry and objects. In one embodiment, the identified road geometry and objects may be used to localize the vehicle. The vehicle may store a map with known locations of roads and objects in its vicinity. By comparing the identified road geometry objects to the known locations within the map, the precise location of the vehicle may be determined. Localization may be used to control the vehicle in driver assistance and autonomous driving because it allows the vehicle to:

[0055] Align its sensors and cameras with the environment and fuse their data to create a coherent representation of the scene

[0056] Compares its current location to its desired destination and generates a feasible and optimal path to follow

[0057] Detect and avoid obstacles, such as other vehicles, pedestrians or road signs

[0058] Adjust its speed, steering and braking according to traffic rules, road conditions and user preferences;

[0059] In a further example, the identified road geometry and objects can be used to control the vehicle without localization and mapping. The road geometry can specify the drivable space. It can be used to ensure that the vehicle remains in its lane until a lane change maneuver occurs. And, when a lane change maneuver occurs, the road geometry can be used to navigate between lanes. Object detection can be used to avoid obstacles and to detect relevant road features that may affect the driving operation. For example, object detection can be used to detect stop signs and traffic lights. Based on the detected objects, the vehicle can be safely driven in accordance with applicable road regulations.

[0060] As mentioned above, Figure 1 The method 100 in the embodiment of the present invention may be implemented on a computing device. The computing device may include one or more processors (also referred to as central processing units or CPUs). The processors may be connected to a communication infrastructure or bus. The computing device may also include user input / output devices, such as monitors, keyboards, pointing devices, etc., which may communicate with the communication infrastructure via a user input / output interface.

[0061] One or more of the processors may be a graphics processing unit (GPU). In one embodiment, a GPU may be a processor that is a dedicated electronic circuit designed to process mathematically intensive applications. A GPU may have a parallel structure that is effective for processing large blocks of data in parallel, such as mathematically intensive data common to computer graphics applications, images, videos, etc. Similarly, one or more of the processors may be a deep learning processor (DLP). A DLP is an electronic circuit designed for deep learning algorithms, typically with independent data memory and a dedicated instruction set architecture. Like a GPU, a DLP may utilize high data-level parallelism, relatively large on-chip buffers / memory to exploit data reuse patterns, and limited data width operators to achieve error resilience for deep learning.

[0062] The computing device may also include a main or primary memory, such as a random access memory (RAM). The main memory may include one or more levels of cache. The main memory may store control logic (ie, computer software) and / or data therein.

[0063] The computing device may also include one or more secondary storage devices or memories. Secondary storage may include, for example, a hard drive, flash memory storage, and / or a removable storage device or drive.

[0064] The computing device may further include a communication or network interface. The communication interface may allow the computing device to communicate and interact with any combination of external devices, external networks, external entities, etc. For example, the communication interface may allow the computer system to access external devices via a network, which may be wired and / or wireless (or a combination thereof), and which may include any combination of a LAN, a WAN, the Internet, etc.

[0065] To name a few non-limiting examples, the computing device may also be any of a rack computer, a server blade, a personal digital assistant (PDA), a desktop workstation, a laptop or notebook computer, a netbook, a tablet computer, a smart phone, a smart watch or other wearable device, an appliance, a part of the Internet of Things, and / or an embedded system, or any combination thereof.

[0066] Although several embodiments have been described, it will be appreciated by those skilled in the art that various modifications and changes may be made without departing from the scope of the embodiments described in detail herein. Therefore, the description and drawings should be regarded as illustrative rather than restrictive, and not all such modifications are intended to be included within the scope of the present teachings. Benefits, advantages, solutions to problems, and any elements that may cause any benefit, advantage, or solution to appear or become more obvious should not be construed as key, essential, or necessary features or elements of any or all claims. The present invention is limited only by the appended claims, which include any amendments made during the pendency of this application and all equivalents of those claims issued.

[0067] Identifiers such as "(a)", "(b)", "(i)", "(ii)", etc. are sometimes used for different elements or steps. These identifiers are used for clarity and do not necessarily specify the order of elements or steps.

[0068] In addition, in this document, relational terms such as first and second may be used only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises / comprising", "has / having", "includes / including", "contains / containing" or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes, has, includes or contains a series of elements includes not only those elements, but may also include other elements that are not explicitly listed or inherent to such processes, methods, articles or devices. An element preceded by "comprises...a", "has...a", "includes...a", "contains...a" does not exclude the presence of additional identical elements in a process, method, article and / or device that includes, has, includes and / or contains the element without additional constraints. Unless otherwise expressly stated herein, the terms "a" and "an" are defined as one or more. The terms "approximately," "about," or any other form thereof are defined as approximating what is understood by one of ordinary skill in the art. A device or structure that is "configured" in a certain manner is configured in at least that manner, but may also be configured in unlisted manners. For reference to an element, either the singular or plural form may be used, but this does not limit the scope of the present disclosure, and the same teachings may apply to multiple objects even if an object is referred to in its singular form in this application.

[0069] The embodiments detailed herein are provided to allow the reader to quickly determine the nature of the present technical disclosure. It should be understood that they will not be used to interpret or limit the scope or meaning of the claims when submitted. In addition, in the foregoing detailed description, it is demonstrated that multiple features are grouped together in various embodiments for the purpose of simplifying the present disclosure. The method of the present disclosure should not be interpreted as reflecting the intention that the claimed embodiments require more features than the features explicitly listed in each claim. On the contrary, as reflected in the attached claims, the subject matter of the invention lies in less than all the features of a single disclosed embodiment in at least some cases. Therefore, the following claims are hereby incorporated into the detailed description, with each claim independently serving as a subject matter claimed separately.

Claims

1. A computer-implemented method for detecting road geometry from a point cloud, the computer-implemented method comprising: (a) determining, using at least one sensor on a vehicle, a point cloud representing the surroundings of the vehicle; (b) partitioning the surrounding environment of the vehicle into a plurality of voxels, each voxel representing a volume of the surrounding environment of the vehicle; For each of the plurality of voxels: (c) determining whether three-dimensional data exists for a corresponding voxel based on the point cloud; (d) when it is determined in (c) that the three-dimensional data exists, inputting data representing a point from the point cloud located within the corresponding voxel into a first neural network segment to obtain a feature for the corresponding voxel, the first neural network segment being a fully connected (FC) feature encoding neural network; as well as (e) inputting the features determined for the corresponding voxels in (d) into a second neural network segment trained to recognize road geometry and detected objects.

2. The method of claim 1, wherein determining the point cloud (a) comprises detecting the point cloud using lidar data.

3. The method of claim 1 , wherein determining the point cloud (a) comprises: receiving a plurality of sensor scans, each sensor scan comprising a plurality of points detected at different times in the surrounding environment of the vehicle; adjusting points from each of the plurality of sensor scans to correct for ego-motion of the vehicle; as well as The plurality of sensor scans are aggregated to determine the point cloud. The method of claim 3 , wherein the point cloud represents objects moving relative to the earth as blurred.

5. The method of claim 3, wherein the plurality of sensor scans are collected from a plurality of lidar sensors on the vehicle.

6. The method of claim 5, wherein the plurality of lidar sensors include a long-range lidar positioned high above the vehicle and a plurality of near-field lidar sensors positioned around the vehicle to capture blind spots from the long-range lidar. 7 . The method of claim 1 , wherein each point in the point cloud comprises a location in three-dimensional space, a timestamp when the location was detected, and a reflectivity detected at the location. The method of claim 7 , wherein each point further comprises a Doppler value detected at the location.

9. The method of claim 1, wherein the road geometry features include road edges and lane dividers.

10. The method of claim 1, further comprising comparing the road geometry to a known map of the surroundings of the vehicle to locate the vehicle. The method of claim 1 , further comprising controlling the vehicle based on the road geometry.

12. The method according to claim 1, wherein, for a corresponding voxel among the plurality of voxels, the second neural network outputs whether a lane or a road edge is within the corresponding voxel, and at what angle the lane or the road edge passes through the voxel.

13. The method of claim 12, further comprising interpolating a spline representing the road geometry based on an output of the second neural network output.

14. The method according to claim 12, further comprising: (f) Assemble the two-dimensional grid presenting the convolution values ​​determined in (d), Wherein input (e) comprises inputting the two-dimensional grid.

15. The method of claim 1, wherein the second neural network detects objects in the point cloud based on the convolution values ​​determined in (d).

16. The method of claim 1, wherein the second neural network detects what the object is, what shape the object has, and what the current position and trajectory of the object is.

17. The method of claim 1, wherein the first neural network and the second neural network are trained together.

18. The method of claim 17, wherein the first and second neural networks are trained using examples labeled by a more computationally demanding neural network.

19. A non-transitory computer readable medium comprising instructions for determining road geometry from a point cloud, the instructions causing a computing system to perform operations comprising: (a) determining, using at least one sensor on a vehicle, a point cloud representing the surroundings of the vehicle; (b) partitioning the surrounding environment of the vehicle into a plurality of voxels, each voxel representing a volume of the surrounding environment of the vehicle; For each of the plurality of voxels: (c) determining whether three-dimensional data exists for a corresponding voxel based on the point cloud; (d) when it is determined in (c) that the three-dimensional data exists, inputting data representing a point from the point cloud located within the corresponding voxel into a first neural network segment to extract features for the corresponding voxel, and then inputting into a second neural network segment, the second neural network segment being a sparse convolutional neural network; and (e) Finally entering (d) the final neural network segment trained to recognize the road geometry features.

20. A processing device for determining road geometry from a point cloud, the processing device being configured to perform operations comprising: (a) determining, using at least one sensor on a vehicle, a point cloud representing the surroundings of the vehicle; (b) partitioning the surrounding environment of the vehicle into a plurality of voxels, each voxel representing a volume of the surrounding environment of the vehicle; For each of the plurality of voxels: (c) determining whether three-dimensional data exists for a corresponding voxel based on the point cloud; (d) when it is determined in (c) that the three-dimensional data exists, inputting data representing a point from the point cloud located within the corresponding voxel into a first neural network segment for feature encoding to extract features for the corresponding voxel; and (e) The features determined for the plurality of voxels in (d) are then input into a second neural network segment trained to recognize the road geometry features.

Images