Matching point lines from vehicle lidar to visually detected objects

Abstract

In one embodiment, a computer-implemented method combines data from a lidar sensor and a camera. A point cloud collected from the lidar sensor is received. The lidar sensor is mounted on a vehicle. A plurality of dotted lines from the point cloud are identified such that each of the identified plurality of dotted lines is at a substantially different angle from a plane of a road on which the vehicle is traveling. An image captured from the camera is received. The camera is mounted to the vehicle, the image having been captured substantially simultaneously with the point cloud. A plurality of objects in the image are identified using an image analysis algorithm. Each object from the plurality of objects is associated with the identified plurality of dotted lines.

Description

Technical Field

[0001] This area involves sensor fusion. Background Art

[0002] Lidar, or light detection and ranging, is a remote sensing technology that uses pulses of laser light to measure distances and create high-resolution maps of the surface of the ground or other objects. Lidar works by sending a beam of light from a transmitter and measuring the timing and intensity of the reflected signal from a receiver. By scanning the beam across a target area, Lidar can generate a three-dimensional point cloud of data representing the shape, height, and features of terrain, vegetation, buildings, or other structures. Lidar can be mounted on vehicles to support driver assistance and autonomous vehicles.

[0003] ADAS or Advanced Driver Assistance Systems are technologies that enhance the safety and convenience of drivers and passengers by providing assistance, warnings or interventions in various driving scenarios. ADAS can range from parking sensors, blind spot monitors, adaptive cruise control and lane keeping assist to fully autonomous systems such as automatic parking and autonomous driving. ADAS relies on a variety of sensors, cameras, radars, lidars and software to perceive the environment, detect potential hazards and communicate with the driver or other vehicles. ADAS helps reduce human error, improve traffic flow, reduce fuel consumption and prevent collisions and injuries.

[0004] Autonomous vehicles, also known as self-driving cars, are vehicles that can operate without human intervention or supervision, using sensors, cameras, software and artificial intelligence to perceive and navigate their environment. Autonomous vehicles have the potential to improve road safety, mobility, efficiency and environmental sustainability by reducing human error, traffic congestion, fuel consumption and greenhouse gas emissions.

[0005] As mentioned above, lidar sensors can generate point clouds. A point cloud is a collection of data points that represent the shape and surface of an object or scene in three-dimensional space. In addition to lidar, point clouds can also be generated from other sensors, including radar and cameras. These point clouds may pose several challenges that require efficient and robust processing and representation methods, such as noise, incompleteness, redundancy, and complexity.

[0006] Improved methods for interpreting point clouds are needed. Summary of the invention

[0007] In one embodiment, a computer-implemented method combines data from a lidar sensor and a camera. A point cloud collected from the lidar sensor is received. The lidar sensor is mounted to a vehicle. A plurality of point lines from the point cloud are identified such that each of the identified plurality of point lines is at a substantially different angle from a plane of a road on which the vehicle is traveling. An image captured from a camera is received. The camera is mounted to the vehicle, the image having been captured substantially simultaneously with the point cloud. Using an image analysis algorithm, a plurality of objects in the image are identified. Individual objects from the plurality of objects are associated with the identified plurality of point lines.

[0008] System, apparatus, and computer program product aspects are also disclosed.

[0009] Further features and advantages and the structure and operation of various aspects are described in detail below with reference to the accompanying drawings. It should be noted that the specific aspects described herein are not intended to be limiting. These aspects proposed herein are for illustrative purposes only. Based on the teachings contained herein, additional aspects will be apparent to those skilled in the relevant art. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The features and advantages of the example embodiments described herein will become apparent to those skilled in the art to which the present disclosure relates after reading the following description with reference to the accompanying drawings.

[0011] Figure 1 is a flow chart illustrating a method for associating visually detected objects with point lines detected using lidar.

[0012] Figure 2 is a flow chart illustrating a method for fusing data between vision and lidar data and controlling a vehicle based on the fused data.

[0013] Figure 3 An example of data collected from a lidar sensor on a vehicle is illustrated.

[0014] Figure 4A-4B An example of clustering points captured from a lidar sensor into lines along a common bearing is illustrated.

[0015] Figure 5A-5B An example of detecting objects using computer vision is illustrated.

[0016] Figure 6A-6B An example of associating objects using computer vision with lines along a common orientation is illustrated.

[0017] Figure 7A-7B An example of fusing associated objects and lines is illustrated.

[0018] In the drawings, like reference numbers generally refer to the same or similar elements. Additionally, generally, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears.

[0019] Aspects of the present disclosure will be described with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] Sensor fusion is the process of combining data from multiple sources (such as cameras, radar, lidar, or inertial measurement units) to obtain a more accurate and comprehensive representation of the environment. An embodiment combines lidar with computer vision into sensor fusion.

[0021] More specifically, embodiments relate to determining columns of lidar data, which are referred to as sticks. Lidar data is collected from a lidar sensor fixed to a vehicle. As the term is used herein, a stick is a line of lidar data captured at a common azimuth angle, with the line direction representing an angle that is substantially different from the plane of the road. Typically, a lidar sensor scans by measuring all points detected in a single azimuth column. A single azimuth column can have multiple sticks. For example, at the same azimuth, a lidar sensor can detect a curb and a fire hydrant. Both the curb and the fire hydrant have substantially different angles from the plane of the road.

[0022] In addition to lidar sensing, embodiments also relate to determining objects from computer vision data. A camera is mounted to the vehicle and collects image data substantially simultaneously with the lidar sensor. Computer vision algorithms are applied to the collected image data to identify objects, such as other vehicles and pedestrians. The object detection algorithm can determine the classification of the object as well as the size and shape of the object.

[0023] According to one embodiment, an object identified as a rod from the lidar data is associated with an object identified using computer vision. In one example, the association can enrich the lidar data to include a classification of the object. In another example, the association can be used to identify false positives. For example, if the computer vision identifies an object that was not identified from the lidar data, the object identified from the computer vision can be determined to be a false positive and ignored.

[0024] In one possible implementation, this sensor fusion can be used only in operations related to driving the vehicle for optimal comfort. In operations related to safety, vision and lidar can be processed separately.

[0025] Figure 1 is a flow chart illustrating a method for associating visually detected objects with point lines detected using lidar.

[0026] At 102, a point cloud collected from a lidar sensor is received. As described above, a point cloud is a collection of data points representing the shape and surface of an object or scene in three-dimensional space. Each point in the point cloud includes a location in three-dimensional space (e.g., altitude, orientation, and distance), a timestamp when the location was detected, and the reflectivity detected at the location. The lidar reflectivity is a measure of the amount of light scattered back to the lidar sensor by the target object. In another embodiment, each point may also include a Doppler value detected at the location. The Doppler value is a measure of the change in the frequency or wavelength of an electromagnetic wave due to the relative motion of the source and the observer. In this way, the Doppler value can indicate the relative motion of the target object when the point is detected.

[0027] As mentioned above, LiDAR is a remote sensing technology that uses pulses of laser light to measure the distance and reflectivity of objects on the ground or in the air. A LiDAR system consists of a laser source, a scanner, a detector, and a computer. The laser source emits a beam of light that is guided by a scanner to scan a certain area or angle. The detector receives the reflected or scattered light from the target and measures the time required for the light to propagate back. The computer then calculates the distance and position of the target based on the speed of light and the angle of the scanner. By rotating or scanning the device horizontally and vertically, LiDAR can capture multiple reflection points and form a series of measurements that scan or cover a specific area or volume. Examples of different types of LiDAR sensors available include direct detection based on time-of-flight LiDAR sensors, frequency modulated continuous wave (FMCW) LiDAR sensors, and amplitude modulated continuous (AMCW) LiDAR sensors.

[0028] In this way, by emitting short pulses of laser light and detecting the echoes, LiDAR can create an accurate image of the terrain or environment. LiDAR scanning involves scanning a specified range or direction using a laser beam. For example, a LiDAR system can guide a laser along multiple altitude angles along a common azimuth angle, sampling each return signal at each altitude angle to determine the range (and possibly the Doppler value). When all altitude angles are sampled at a common azimuth angle, the azimuth angle can be incremented and the altitude angle can be sampled again. In this way, LiDAR sensors can sample points in a wide range of three-dimensional space. In order to steer the laser beam to a specific altitude and azimuth, different scanning can be used, including mechanical scanning, solid-state scanning, MEMs mirror steering, and optical phased array (OPA) steering.

[0029] In other examples, beam steering can be avoided entirely by using flash lidar systems. Instead of a point-by-point scan of the field of view, these systems collect time-of-flight data points for the entire horizontal plane simultaneously with each flash.

[0030] Figure 3is a diagram 300 illustrating an example of points collected from a lidar sensor on a vehicle. The sampled points may be from a single scan over the field of view of the lidar sensor. In another embodiment, the points may be sampled from multiple consecutive or nearly consecutive scans of the lidar sensor. In yet another embodiment, the points may be captured over multiple scans of different lidar sensors mounted at different points around the vehicle that are captured nearly simultaneously.

[0031] At 104, a plurality of point lines from the point cloud are identified such that each of the identified plurality of point lines is at a substantially different angle to the plane of the road on which the vehicle is traveling. As described above, the plurality of point lines may be referred to as "sticks." Each of the plurality of sticks may include points captured from the lidar sensor at a vertical common azimuth angle. For example, points may be captured continuously from a single vertical scan.

[0032] As described above, each point in the bar is at a substantially different angle to the plane of the road on which the vehicle is traveling, and the plane of the road is substantially horizontal. The substantially different angles are set to a minimum value based on the driving performance on the road. In some embodiments, the substantially different angles can be in the range of 45-90 degrees, in other embodiments in the range of 75-90 degrees, and in other embodiments substantially close to 90 degrees.

[0033] refer to Figure 3 The diagram 300 in FIG. 300 illustrates an example operation. As described above, the diagram 300 illustrates a point cloud. The point cloud includes a plurality of points captured at azimuth angles 302. The plurality of points captured at azimuth angles 302 include a plurality of points 304 at nearly horizontal angles. Figure 3 In the example of FIG. 3 , the object detected at an almost horizontal angle may be a street. The plurality of points captured at azimuth angle 302 include a plurality of points 306 at an almost vertical angle. Figure 3 In the example in , the object detected at a nearly vertical angle may be a bus. The plurality of points 306 comprises a rod. The process is repeated at a plurality of azimuth angles across the field of view to determine a plurality of rods.

[0034] In order to determine whether a sequence of points along an azimuth angle is at a horizontal or vertical angle, each point may be converted, for example, from spherical coordinates (altitude, azimuth, distance) to rectangular coordinates (x, y, z). If the variation in the sequence of points on the z plane is below a threshold, the sequence of points may have different x, y coordinates on substantially the same z plane. If the sequence of points is on substantially the same z plane, the sequence of points may be at a horizontal angle. Similarly, if the variation in the sequence of points on the x and y planes is below a threshold, the sequence of points may have substantially the same horizontal xy position with varying z heights. If the sequence of points has substantially the same horizontal xy position with varying z heights, the sequence of points may be at a vertical angle.

[0035] Figure 4A-4B An example of clustering points captured from a lidar sensor into lines along a common bearing is illustrated. Figure 4A Shows that from Figure 3 4. Diagram 400 illustrates a plurality of bars 402 determined from a point cloud illustrated in FIG. Diagram 400 illustrates a plurality of bars 402 from the perspective of a lidar sensor on a vehicle. Figure 4B A diagram 450 is shown from a nadir perspective with a plurality of bars 402. The diagram 450 shows an origin 452 at which a vehicle is located.

[0036] Continue from Figure 3 , a plurality of points 306 at substantially perpendicular angles are grouped into a rod 404. In rectangular coordinates, the rod 404 can be specified by an xy coordinate and two z coordinates, one specifying the bottom of the rod 404 and the other specifying the top of the rod 404. Figure 4A FIG. 400 and Figure 4B Rod 404 is illustrated in both diagrams 450 in FIG.

[0037] At 106, an image captured from a camera is received. The camera is mounted to the vehicle and the image may be a photographic image captured substantially simultaneously with the point cloud. In one embodiment, multiple cameras are mounted around the exterior of the vehicle, such as the front mirror, rear mirror, side mirror, or roof. These cameras may capture images of the vehicle's surroundings from different angles and perspectives and transmit them to a display screen or computer system inside the vehicle. The camera may help the driver or the vehicle itself sense and monitor traffic conditions, road hazards, blind spots, parking spaces, or other objects or events around the vehicle.

[0038] In embodiments where multiple cameras are used, multiple images captured from the various cameras may be stitched together to produce the image received at step 106. Stitching together images captured from multiple cameras is the process of combining overlapping or adjacent photos into a larger and more complete scene. In order to stitch images together, the images may need to have some common features or reference points that can be matched and aligned. The images may then be warped, blended, and cropped to create a seamless and natural-looking result.

[0039] At 108, a plurality of objects are identified in the image received at step 106. The identification is done using an image analysis algorithm. The image analysis algorithm may output a set of bounding boxes indicating the spatial location and size of each detected object, as well as a class label indicating the class of the object, such as a person, car, bus, dog, etc. The image analysis algorithm may be three-dimensional in that its purpose is to estimate not only the two-dimensional location and size of objects, but also their three-dimensional shape, pose, and orientation in the image.

[0040] The 3D object detection model typically outputs a set of cuboids or other geometric primitives representing the 3D bounding volume of each detected object, as well as a class label and a pose vector indicating the class and orientation of the object, respectively. Example image analysis algorithms that can be used to detect objects at step 108 include YOLO (You Only Look Once), Faster R-CNN (Faster Region-based Convolutional Neural Network), PointRCNN, and CenterNet.

[0041] Figure 5A-5B An example of detecting objects using computer vision is illustrated. Figure 5A A diagram 500 is shown having a photo image 502 that may be received, for example, in step 106. Diagram 500 also includes a plurality of bounding boxes, including bounding box 504 overlaid on photo image 502. The plurality of bounding boxes are shown in diagram 500 from the perspective of a camera that captured photo image 502. Figure 5B Diagram 550 is shown from a nadir perspective with a plurality of bounding boxes, including bounding box 504. Diagram 550 also shows origin 452 at which the vehicle is located.

[0042] At 110, each object from the plurality of objects determined at step 104 is associated with the rod identified at step 108. The association may involve determining whether each object from the plurality of objects is located in substantially the same position as each rod. For example, when a rod is within a bounding box of an object or within a threshold distance from a bounding box of an object, the rod may be associated with the object. Alternatively or additionally, the association may involve determining whether each object from the plurality of objects has substantially the same shape as each rod. For example, when a rod has a similar height to a bounding box of an object, the rod may be more likely to be associated with the object. Furthermore, when a group of adjacent rods together have a shape similar to a surface of a bounding box of an object, the group of rods may be more likely to be associated with the object.

[0043] Figure 6A-6B An example of associating objects using computer vision with lines along a common orientation is illustrated. Fig. 6A A diagram 600 is shown having a plurality of bars 602 associated with an object having a bounding box 604. Diagram 600 illustrates the plurality of bars 602 and bounding box 604 from the perspective of a lidar sensor and camera on a vehicle. Figure 6B Diagram 650 with multiple bars 602 and bounding box 604 is shown from a nadir perspective.

[0044] Figure 2 is a flow chart illustrating a method 200 for fusing data between vision and lidar data and controlling a vehicle based on the fused data.

[0045] At 202, it is determined whether a rod from the rods determined at step 104 is associated with the object determined at step 108. Figure 1 This step is described in step 110 of FIG. This analysis may be performed for each line identified in step 104 and / or for each object determined at step 108. When the line and the object are associated, the operation proceeds to step 204. When it is determined at 206 that the object is not associated with any of the bars, the operation proceeds to step 216. When it is determined at 216 that an object from the object is not associated with any of the bars, the operation proceeds to step 218.

[0046] At 204, the data representing the stick is fused with the data representing the object. As described above, sensor fusion is the process of combining data from multiple sensors to obtain a more accurate and reliable representation of the environment or phenomenon of interest. Sensor fusion can enhance the performance, robustness, and functionality of adaptive driver assistance and autonomous driving. Sensor fusion can also bring benefits such as reducing uncertainty, resolving conflicts, improving resolution, and utilizing complementary information from different sensor modalities.

[0047] As described above, the image analysis performed at step 108 may involve determining a classification of an object. A classification is a category of an object, such as a person, car, bus, dog, etc. Step 104 may include enriching the stick data with the classification determined for the corresponding object. Data enrichment may include associating a stick with a classification. Thus, in situations where an autonomous driving or ADAS system may control a vehicle differently based on whether an object at a particular location has a different classification, the autonomous driving or ADAS system may behave as if an object with an associated classification is located at the location of the stick.

[0048] As described above, data fusion may also involve adjusting the data from the various sensors based on the relative accuracy of the sensors. A lidar sensor may have a finer range resolution than a camera. Thus, fusing data from various sources may involve adjusting the position of the bounding box determined at step 108 to the position of the corresponding line of bars. For example, in Fig. 7A This situation is exemplified in -B.

[0049] Figure 7A-7B An example of fusing associated objects and lines is illustrated. Fig. 7A A diagram 700 is shown having a plurality of bars 602 associated with an object having a bounding box 604. Diagram 700 illustrates the plurality of bars 602 and bounding box 604 from the perspective of a lidar sensor and camera on a vehicle. Figure 7B A diagram 750 is shown from a nadir perspective with the plurality of bars 602 and the bounding box 604. As shown in these figures, the position of the bounding box 604 has been adjusted so that one side of the bounding box 604 is along the plurality of bars 602.

[0050] In an embodiment, the fusion described with respect to step 204 may be used to control the vehicle via the autonomous driving or ADAS system in some situations, but not in other situations. For example, in some situations, the autonomous driving or ADAS system may control the vehicle for safety, while in other situations, the autonomous driving or ADAS system may control the vehicle for comfort.

[0051] When taking control of the vehicle for safety, an automated driving or ADAS system may determine a safe following distance from the vehicle ahead of it; a safe lateral distance to ensure the vehicle can recognize when lateral safety may be compromised due to the driver unsafely cutting into the lane; right-of-way decisions based on lane markings, traffic signs, traffic lights, and the actions of other drivers; areas of limited visibility where hazards may exist but are contained; and methods used to avoid collisions.

[0052] However, keeping the vehicle at the safe minimum following distance or safe lateral distance may cause the vehicle to drive so aggressively that it affects the comfort of the passengers in the vehicle. Therefore, in addition to determining the safe minimum following distance or safe lateral distance, the autonomous driving or ADAS system may also determine the minimum distance for controlling the vehicle to obtain maximum comfort for the vehicle. In an embodiment, the fusion in step 204 may occur when the vehicle is controlled for comfort, but not when the vehicle is controlled for safety. In this way, the camera and lidar data remain independent, redundant data sources to ensure safe operation of the vehicle.

[0053] When the object determined using image analysis (decision box 206) is determined not to be associated with any rods in the scene, the operation may again vary depending on whether the vehicle is being controlled for comfort or safety. At 210, a determination is made as to whether the vehicle is being controlled for the comfort of the occupants or for safety. When the vehicle is being controlled only for comfort, the object may be ignored as a false positive at 212. At 214, when the vehicle is being controlled only for safety, the autonomous driving or ADAS system may control the vehicle based on the object to ensure that the vehicle is being driven safely according to the object.

[0054] When the rod using LiDAR is determined not to be associated with any object in the scene (at 216), it is determined at 218 whether the rod is an airborne particle. Airborne particles are tiny solid or liquid particles suspended in the air, such as dust, smoke, fog, precipitation, pollen, or pollutants. LiDAR can detect airborne particles by sending a beam of light into the air and measuring the amount and direction of light scattered or reflected back by the particles. One possible method for detecting airborne particles in an image is to use remote sensing technology that measures the amount and characteristics of light reflected or scattered by particles at different wavelengths.

[0055] Another possible approach for detecting airborne particles in images is to use machine learning algorithms that can classify and quantify aerosols based on their features and patterns. For example, convolutional neural networks (CNNs) can learn to extract and identify features of aerosols from a large number of labeled images and then apply them to new images to estimate the optical depth, size distribution, and composition of aerosols.

[0056] When the stick is determined to be an airborne particle, the stick is ignored in controlling the vehicle at 220. For example, the stick is not used in the minimum following distance or safe lateral distance for the vehicle, whether the vehicle is being controlled for comfort or safety.

[0057] At 222 , when the stick is determined not to be airborne particulate matter, the vehicle is controlled based on the stick. For example, the stick is used to determine a minimum following distance or a safe lateral distance to control the vehicle for both comfort and safety.

[0058] As mentioned above, Figure 1 Method 100 and Figure 2 The method 200 in can be implemented on a computing device. The computing device may include one or more processors (also referred to as central processing units, or CPUs). The processor may be connected to a communication infrastructure or bus. The computer device may also include user input / output devices, such as monitors, keyboards, pointing devices, etc., which can communicate with the communication infrastructure through a user input / output interface.

[0059] One or more of the processors may be a graphics processing unit (GPU). In one embodiment, a GPU may be a processor that is a dedicated electronic circuit designed to process mathematically intensive applications. The GPU may have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data commonly found in computer graphics applications, images, videos, etc. Similarly, one or more of the processors may be a deep learning processor (DLP). A DLP is an electronic circuit designed for deep learning algorithms, typically with independent data memory and a dedicated instruction set architecture. Like a GPU, a DLP may utilize high data-level parallelism, relatively large on-chip buffers / memories for exploiting data reuse patterns, and limited data width operators for implementing error resilience capabilities for deep learning.

[0060] The computing device may also include a main memory, such as a random access memory (RAM). The main memory may include one or more levels of cache. The main memory may store control logic (ie, computer software) and / or data therein.

[0061] The computing device may also include one or more secondary storage devices or memories. The secondary storage may include, for example, a hard drive, flash memory, and / or a removable storage device or drive.

[0062] The computing device may further include a communication or network interface. The communication interface may allow the computing device to communicate and interact with any combination of external devices, external networks, external entities, etc. For example, the communication interface may allow the computer system to access external devices via a network, which may be wired and / or wireless (or a combination thereof) and may include any combination of LAN, WAN, the Internet, etc.

[0063] The computing device may also be a rack computer, a server blade, a personal digital assistant (PDA), a desktop workstation, a laptop or notebook computer, a netbook, a tablet computer, a smartphone, a smart watch or other wearable device, part of the Internet of Things, and / or an embedded system, to name a few non-limiting examples, or any combination thereof.

[0064] Although several embodiments have been described, it will be appreciated by those skilled in the art that various modifications and changes may be made without departing from the scope of the embodiments described in detail herein. Therefore, the description and drawings should be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the present teachings. Benefits, advantages, solutions to problems, and any elements that may cause any benefit, advantage, or solution to appear or become more apparent should not be construed as key, essential, or necessary features or elements of any or all claims. The present invention is limited only by the appended claims, including any amendments made during the pendency of this application and all equivalents of those claims issued.

[0065] Identifiers, such as "(a)", "(b)", "(i)", "(ii)", are sometimes used for different elements or steps. These identifiers are used for clarity and do not necessarily specify an order for the elements or steps.

[0066] However, in this document, relational terms such as first and second can be used only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. As used herein, the term "comprises", "comprising", "has", "having", "includes", "including", "contains", "containing" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including, having, containing or containing a series of elements not only includes those elements, but also can include other elements that are not explicitly listed or inherent to such processes, methods, articles or devices. In the absence of additional restrictions, the elements starting with "including ... one", "having ... one", "including ... one", "containing ... one" do not exclude the presence of other identical elements in the process, method, article and / or device including, having, containing and / or containing the element. Unless otherwise expressly stated herein, the terms "one" and "one" are defined as one or more. The terms "approximately," "about," or any other form thereof are defined as approximating what is understood by one of ordinary skill in the art. A device or structure that is "configured" in a certain manner is configured in at least that manner, but may also be configured in unlisted manners. For reference to an element, either the singular or plural form may be used, but this does not limit the scope of the present disclosure, and the same teachings may apply to multiple objects even if an object is referred to in its singular form in this application.

[0067] The embodiments detailed herein are provided to allow the reader to quickly determine the nature of the present technical disclosure. It should be understood that they will not be used to interpret or limit the scope or meaning of the claims when submitted. In addition, in the foregoing detailed description, it is demonstrated that multiple features are grouped together in various embodiments for the purpose of simplifying the present disclosure. The method of the present disclosure should not be interpreted as reflecting the intention that the claimed embodiments require more features than the features explicitly listed in each claim. On the contrary, as reflected in the attached claims, the subject matter of the invention lies in less than all the features of a single disclosed embodiment in at least some cases. Therefore, the following claims are hereby incorporated into the detailed description, with each claim independently serving as a separately claimed subject matter.

Claims

1. A computer-implemented method for combining data from a lidar sensor and a camera, the computer-implemented method comprising: receiving a point cloud collected from the lidar sensor, the lidar sensor being mounted on a vehicle; identifying a plurality of point lines from the point cloud such that each of the identified plurality of point lines is at a substantially different angle to a plane of a road on which the vehicle is traveling; receiving an image captured from the camera, the camera being mounted to the vehicle and the image having been captured substantially simultaneously with the point cloud; identifying a plurality of objects in the image using an image analysis algorithm; and Each object from the plurality of objects is associated with the identified plurality of point lines. 2 . The method of claim 1 , further comprising, when a point line from the plurality of point lines is associated with an object from the plurality of objects, enriching the line with a classification of the object determined using the image analysis algorithm. 3 . The method of claim 1 , further comprising identifying an object from the plurality of objects as a false positive when the object is determined not to be associated with any of the plurality of dotted lines.

4. The method according to claim 3, further comprising: ignoring the object identified as the false positive when controlling the vehicle for the comfort of an occupant of the vehicle; and When the vehicle is controlled for safety, the vehicle is controlled based on the object identified as the false alarm.

5. The method of claim 1 , further comprising, when a point line from the plurality of point lines is determined to be not associated with any of the plurality of objects: The vehicle is controlled based on the line.

6. The method of claim 5, further comprising, when the one point line from the plurality of point lines is determined to be not associated with any of the plurality of objects: determining whether the line is an airborne particle; When the line is determined to be the airborne particulate matter, ignoring the line when controlling the vehicle; and When the line is determined not to be the airborne particulate matter, the vehicle is controlled based on the line.

7. The method of claim 1, wherein each of the plurality of point lines is captured from the lidar sensor at a vertical common azimuth angle.

8. The method of claim 7, wherein each point in the plurality of point lines is captured sequentially by the lidar sensor.

9. The method of claim 1, wherein the plane of the road is substantially horizontal.

10. The method of claim 9, wherein the substantially different angle is set to a minimum value based on drivability on the road.

11. The method of claim 1, wherein the associating comprises determining whether the respective objects from the plurality of objects are located in substantially the same position as respective lines from the plurality of point lines. 12 . The method of claim 12 , wherein the associating comprises determining whether the respective objects from the plurality of objects have substantially the same shape as the respective lines from the plurality of point lines.

13. A non-transitory computer readable medium comprising instructions for combining data from a lidar sensor and a camera, the instructions causing a computing system to perform operations comprising: receiving a point cloud collected from the lidar sensor, the lidar sensor being mounted on a vehicle; identifying a plurality of point lines from the point cloud such that each of the identified plurality of point lines is at a substantially different angle to a plane of a road on which the vehicle is traveling; receiving an image captured from the camera, the camera being mounted to the vehicle and the image having been captured substantially simultaneously with the point cloud; identifying a plurality of objects in the image using an image analysis algorithm; and Each object from the plurality of objects is associated with the identified plurality of point lines.

14. The non-transitory computer readable medium of claim 13, the operations further comprising, when a point line from the plurality of point lines is associated with an object from the plurality of objects, enriching the line with a classification of the object determined using the image analysis algorithm. 15 . The non-transitory computer readable medium of claim 13 , the operations further comprising identifying an object from the plurality of objects as a false positive when the object is determined not to be associated with any of the plurality of dotted lines.

16. The non-transitory computer readable medium of claim 15, the operations further comprising: ignoring the object identified as the false positive when controlling the vehicle for the comfort of an occupant of the vehicle; and When the vehicle is controlled for safety, the vehicle is controlled based on the object identified as the false alarm.

17. The non-transitory computer readable medium of claim 13, the operations further comprising, when a point line from the plurality of point lines is determined to be not associated with any of the plurality of objects: The vehicle is controlled based on the line.

18. The non-transitory computer readable medium of claim 17, the operations further comprising, when the one point line from the plurality of point lines is determined to be not associated with any of the plurality of objects: determining whether the line is an airborne particle; When the line is determined to be the airborne particulate matter, ignoring the line when controlling the vehicle; and When the line is determined not to be the airborne particulate matter, the line is controlled based on the line.

19. The non-transitory computer readable medium of claim 13, wherein each of the plurality of point lines is captured from the lidar sensor at a perpendicular common azimuth angle.

20. The non-transitory computer readable medium of claim 13, wherein the associating comprises determining whether the respective objects from the plurality of objects have substantially the same shape as the respective lines from the plurality of point lines.

Images