Comparison of point lines from vehicle LiDAR with vision-detected objects

Abstract

In one embodiment, a computer-implemented method combines data from a lidar sensor and a camera. A point cloud collected by the lidar sensor is received. The lidar sensor is mounted on a vehicle. A plurality of point lines from the point cloud are identified such that each of the identified plurality of point lines lies at a substantially different angle to a plane of a road on which the vehicle is traveling. An image captured by the camera is received. The camera is mounted on the vehicle, and the image was captured substantially simultaneously with the point cloud. Using an image analysis algorithm, a plurality of objects in the image are identified. Each object from the plurality of objects is correlated with the identified plurality of point lines.

Description

REGIONThe field relates to sensor fusion.BACKGROUNDLight Detection and Ranging is a remote sensing technique in which pulses of laser light are used to measure distances and to generate high resolution maps of the surface of the earth or other objects. Lidar functions by emitting a beam of light from a transmitter and measuring the time and intensity of the signal reflected by a receiver. By scanning the beam across a target area, lidar can generate a three-dimensional point cloud of data representing the shape, height, and characteristics of the terrain, vegetation, buildings, or other structures. Lidar may be mounted on vehicles to support driver assistance and autonomous vehicles.ADAS or advanced driver assistance systems are technologies that increase the safety and comfort of drivers and passengers by providing support, warnings or interventions in various driving situations. ADAS may include parking sensors, blind spot indicators, adaptive cruise control, and lane keeping assistants up to fully autonomous systems such as self-parking and self-driving. ADAS relies on various sensors, cameras, radars, lidars, and software to sense the environment, detect potential hazards, and communicate with the driver or other vehicles. ADAS may help reduce human failure, improve traffic flow, reduce fuel consumption, and avoid collisions and injuries.Autonomous vehicles, also known as self-driving cars, are vehicles that can travel without human intervention or supervision, using sensors, cameras, software, and artificial intelligence to perceive and navigate their environment. Autonomous vehicles have the potential to improve safety, mobility, efficiency, and environmental compliance in road traffic by reducing human failure, traffic jams, fuel consumption, and greenhouse gas emissions.As mentioned above, lidar sensors may produce point clouds. Point clouds are collections of data points representing the shape and surface of an object or scene in three-dimensional space. In addition to lidar, point clouds may also be generated by other sensors including radars and cameras. These point clouds may involve some challenges, such as noise, incompleteness, redundancy, and complexity, that require efficient and robust processing and rendering methods.Improved methods for interpreting point clouds are needed.SUMMARYIn one embodiment, a computer-implemented method combines data from a lidar sensor and a camera. A point cloud collected by the lidar sensor is received. The lidar sensor is mounted on a vehicle. A plurality of score lines from the point cloud is identified such that each of the identified plurality of score lines is at a substantially different angle to a plane of a road on which the vehicle is traveling. An image captured by the camera is received. The camera is mounted on the vehicle, wherein the image has been captured substantially simultaneously with the point cloud. Using an image analysis algorithm, a plurality of objects in the image are identified. Respective objects of the plurality of objects are correlated with the identified plurality of score lines.System, apparatus, and computer program product aspects are also disclosed.Further features and advantages and the structure and mode of operation of various aspects are described in detail below with reference to the attached drawings. It should be noted that the specific aspects described herein are not to be understood as limiting. These aspects are presented herein for illustrative purposes only. Additional aspects will be apparent to those skilled in the relevant area(s) based on the teachings herein.DESCRIPTION OF THE DRAWINGSThe features and advantages of the exemplary embodiments described herein will become apparent to those skilled in the art to which this disclosure pertains upon reading the following description with reference to the accompanying drawings. FIG. 1 is a flow diagram illustrating a method for correlating vision detected objects to score lines detected with lidar. FIG. 2 is a flow diagram illustrating a method for merging data between vision and lidar data and for controlling the vehicle based on the merged data. FIG. 3 illustrates an example of data collected from a lidar sensor on a vehicle.FIGS. 4A-4B illustrate an example of clustering points detected by a lidar sensor into lines along a common azimuth.FIGS. 5A-5B illustrate an example of detecting objects using computer vision.FIGS. 6A-6B illustrate an example of correlating the objects using computer vision with the lines along a common azimuth.FIGS. 7A-7B illustrate an example of fusing the correlated objects and lines.In the drawings, like reference numerals generally designate identical or similar elements. In addition, the left(s) digit(s) of a reference number generally identifies (identify) the drawing in which the reference number first appears.Aspects of the present disclosure will be described with reference to the accompanying drawings.DETAILED DESCRIPTIONSensor fusion is a process of combining data from multiple sources, such as cameras, radars, lidars, or inertial measurement units, to obtain a more accurate and comprehensive representation of the environment. Embodiments combine lidar with computer vision for sensor fusion.In particular, embodiments include determining columns of lidar data referred to as rods. The lidar data is collected by a lidar sensor mounted on a vehicle. Rods, as the term is used herein, are lines of lidar data captured at a common azimuth angle, where the line orientation represents an angle that is substantially different from the plane of the road. Typically, lidar sensors make passes by measuring all points detected in a single azimuth column. A single azimuth column may comprise a plurality of rods. For example, the lidar sensor can detect a curb and a fire hydrant in the same azimuth. Both the curb and the fire hydrant have an angle that is substantially different from the road plane.In addition to lidar sensing, embodiments include determining objects from computer vision data. Cameras are mounted on the vehicle and collect image data substantially simultaneously with the lidar sensor. Computer vision algorithms are applied to the collected image data to identify objects such as other vehicles and pedestrians. The object detection algorithm may determine a classification of the object as well as a size and shape of the object.According to one embodiment, the objects identified as rods from the lidar data are correlated with the objects identified using computer vision. In one example, this correlation may enrich the lidar data to include classification of the object. In another example, this correlation may be used to identify false positives. For example, if the computer vision identifies an object not identified by the lidar data, the object identified by the computer vision may be determined to be false positive and ignored.In one possible implementation, this sensor fusion may be used only in vehicle driving operations for optimal comfort. In security operations, vision and lidar may be handled independently of each other.FIG. 1 is a flow diagram illustrating a method for correlating vision detected objects to score lines detected with lidar.At 102, a point cloud collected by a lidar sensor is received. As mentioned above, a point cloud is a collection of data points representing the shape and surface of an object or scene in three-dimensional space. Each point in the point cloud includes a location in three-dimensional space (e.g., height, azimuth, and distance), a time stamp when the location was detected, and a reflectivity detected at the location. Lidar reflectivity is a measure of how much light is backscattered from a target object to a lidar sensor. In another embodiment, each point may additionally include a Doppler value detected at that location. A Doppler value is a measure of the change in frequency or wavelength of an electromagnetic wave due to the relative movement of the source and the observer. In this way, the Doppler value may indicate relative motion of the target object when the point has been detected.As described above, lidar is a remote sensing technique that uses pulses of laser light to measure the distance and reflectivity of objects on the ground or in the air. A lidar system is comprised of a laser source, a scanner, a detector, and a computer. The laser source emits a beam of light that is directed by the scanner to scan a particular range or angle. The detector receives the light reflected or scattered from the target and measures the time required for the light to return. The computer then calculates the range and position of the target based on the speed of light and the angle of the scanner. By rotating or scanning the device in the horizontal and vertical directions, the lidar can capture multiple points of reflection and make a run or series of measurements covering a particular area or volume. Examples of different types of available lidar sensors include direct detection based on a time-of-flight lidar sensor, a frequency modulated continuous wave (FMCW) lidar sensor, and an amplitude modulated continuous wave (AMCW) lidar sensor.In this way, by emitting short bursts of laser light and detecting the echoes, lidar can generate an accurate image of the terrain or the environment. Lidar traversal includes traversal of a laser beam over a specified range or direction. For example, a lidar system may align a laser along a plurality of elevation angles along a common azimuth angle, with each return signal sampled at each elevation angle to determine a distance (and possibly a Doppler value). If all elevation angles have been sampled with the common azimuth angle, the azimuth angle may be incremented and the elevation angles may be sampled again. In this way, a lidar sensor can scan points over a wide range of three-dimensional space. To direct the laser beams to a particular height and azimuth, a different scan may be used, including mechanical scanning, solid state scanning, MEMS mirror steering, and optical phased array (OPA) steering.In other examples, beam steering may be completely avoided by using a flash lidar system. Instead of point-by-point scanning of the field of view, these systems gather time-of-flight data points of entire horizontal planes simultaneously on each flash.FIG. 3 is a diagram 300 illustrating an example of points collected by a lidar sensor on a vehicle. The points scanned may originate from a single pass over a field of view of the lidar sensor. In another embodiment, the points may be scanned from multiple successive or nearly successive passes of the lidar sensor. In another embodiment, the points may be detected over multiple passes of different lidar sensors that are mounted at different points around the vehicle and are detected nearly simultaneously.At 104, a plurality of score lines are identified from the point cloud such that each of the identified plurality of score lines is at a substantially different angle to a plane of a road on which the vehicle is traveling. As mentioned above, the plurality of dotted lines may be referred to as "bars.". Each of the plurality of rods may include points detected at a common vertical azimuth angle by the lidar sensor. For example, the points can be detected successively from a single vertical pass.As mentioned above, the points in each bar are at a substantially different angle than the plane of a road on which the vehicle is travelling and the plane of the road is substantially horizontal. The substantially different angle is set to a minimum according to the driveability across the road. In some embodiments, the substantially different angle may be in the range of 45-90 degrees, in other embodiments in the range of 75-90 degrees, and in other embodiments substantially close to 90 degrees.An example operation is described with reference to the diagram 300 in FIG. 3. As mentioned above, the diagram 300 illustrates a point cloud. The point cloud includes a plurality of points captured at an azimuth angle 302. The plurality of points detected at azimuth angle 302 include a plurality of points 304 at a near horizontal angle. In the example in FIG. 3, the object detected at the nearly horizontal angle may be a road. The plurality of points detected at azimuth angle 302 include a plurality of points 306 at a near vertical angle. In the example in FIG. 3, the object detected at a nearly vertical angle may be a bus. The plurality of points 306 comprise a rod. The process is repeated for a plurality of azimuth angles across the field of view to determine a plurality of rods.To determine whether a point sequence along an azimuth forms a horizontal or vertical angle, each point may be converted from spherical (height, azimuth, range) to rectangular coordinates (x, y, z), for example. If the deviation of a point sequence on the z-plane is below a threshold, then the point sequence may have different x, y coordinates on substantially the same z-plane. If the point sequence lies substantially on the same z-plane, the point sequence can form a horizontal angle. Similarly, if the deviation of a point sequence on the x and y planes is below a threshold, the point sequence may have substantially the same horizontal x-y position with varying z elevations. If the point sequence has substantially the same horizontal x-y position with varying z-elevations, the point sequence can form a vertical angle.FIGS. 4A-4B illustrate an example of clustering points detected by a lidar sensor into lines along a common azimuth. FIG. 4A shows a diagram 400 having a plurality of rods 402 determined from the point cloud illustrated in FIG. 3. The diagram 400 illustrates the plurality of rods 402 from the perspective of a lidar sensor on a vehicle. FIG. 4B shows a diagram 450 with a plurality of rods 402 from a nadir perspective. The diagram 450 shows a starting point 452 at which the vehicle is located.Continuing with the example of FIG. 3, a plurality of points 306 are clustered at a substantially vertical angle to a rod 404. In rectangular coordinates, the rod 404 may be specified by an x-y coordinate and two z coordinates, one of which specifies the bottom of the rod 404 and the other of which specifies the top of the rod 404. The rod 404 is illustrated in both the diagram 400 in FIG. 4A and the diagram 450 in FIG. 4B.At 106, an image captured by a camera is received. The camera is mounted to the vehicle and the image may be a photographic image captured substantially simultaneously with the point cloud. In one embodiment, multiple cameras are mounted on the exterior of the vehicle, such as the front, rear, side mirrors, or roof. These cameras may capture images from the surroundings of the vehicle from different viewpoints and perspectives and forward them to a display screen or a computer system in the vehicle. The cameras may help the driver or vehicle itself recognize and monitor traffic conditions, road hazards, blind spots, parking spots, or other objects or events around the vehicle.In the embodiment where multiple cameras are used, multiple images captured by the respective cameras may be composited to produce the image received at step 106. Composing images captured by multiple cameras is a process in which overlapping or adjacent photos are combined into a larger and more complete scene. To assemble images, the images must have some common features or reference points that can be matched and matched. Then, the images can be distorted, overlaid and cropped to produce a seamless and naturally-acting result.At step 108, a plurality of objects are identified in the image received at step 106. Identification is made using an image analysis algorithm. The image analysis algorithm may output a series of bounding boxes indicating the spatial location and size of each detected object, as well as a class label indicating the category of the object, such as person, car, bus, dog, etc. The image analysis algorithm may be three-dimensional in that it attempts to estimate not only the two-dimensional location and size of objects but also their three-dimensional shape, posture and orientation in the image.Three-dimensional object detection models typically output a series of cuboids or other geometric primitives representing the three-dimensional bounding volume of each detected object, as well as a class designation and a posture vector that indicate the category and orientation of the object, respectively. Example image analysis algorithms that may be used to detect objects at step 108 include You Only Look Once (YOLO), Fast Region-based Convolutional Neural Network (R-CNN), PointRCNN, and CenterNet.FIGS. 5A-5B illustrate an example of detecting objects using computer vision. FIG. 5A shows a diagram 500 with a photographic image 502, as can be received, for example, in step 106. The diagram 500 also includes a plurality of bounding boxes, including a bounding box 504 overlaid on the photographic image 502. The plurality of bounding boxes are shown in diagram 500 from the perspective of a camera that captured the photographic image 502. FIG. 5B shows a diagram 550 with a plurality of bounding boxes, including bounding box 504, from a nadir perspective. The diagram 550 also shows the origin 452 where the vehicle is located.At step 110, the respective objects from the plurality of objects determined at step 104 are correlated with the identified rods at step 108. The correlation may include determining whether the respective objects of the plurality of objects are at substantially the same position as the respective rod. For example, if a stick is within the bounding box of an object or within a threshold distance from the bounding box of the object, the stick may be correlated with the object. Alternatively or additionally, the correlation may include determining whether the respective objects of the plurality of objects have substantially the same shape as the respective rods. For example, if a rod has a similar height to the bounding box of an object, the rod may be more likely to be correlated with the object. In addition, when a group of adjacent rods has a shape that resembles the surface of a bounding box of an object, the group of rods may be more likely to be correlated with the object.FIGS. 6A-6B illustrate an example of correlating the objects using computer vision with the lines along a common azimuth. FIG. 6A shows a diagram 600 having a plurality of rods 602 correlated with an object having the bounding box 604. The diagram 600 illustrates the plurality of rods 602 and the bounding box 604 from the perspective of a lidar sensor and a camera on a vehicle. FIG. 6B shows a diagram 650 with a plurality of rods 602 and the delimiting frame 604 from a nadir perspective.FIG. 2 is a flow diagram illustrating a method 200 for merging data between vision and lidar data and for controlling the vehicle based on the merged data.At 202, it is determined whether a rod from the rods determined at step 104 is correlated with an object determined at step 108. This step has been described above with reference to step 110 in FIG. 1. This analysis may be performed for each row identified at step 104 and / or for each object determined at step 108. If a line and an object are correlated, the process continues to step 204. If it is determined at 206 that an object is not correlated with any of the rods, the process continues with step 216. If it is determined at 216 that an object of the object is not correlated with any of the posts, the process continues to step 218.At 204, the data representing the rod is fused to the data representing the object. As mentioned above, sensor fusion is a process in which data from multiple sensors is combined to obtain a more accurate and reliable representation of the environment or phenomenon of interest. The sensor fusion may improve the performance, robustness, and functionality of the adaptive driver assistance and autonomous driving. Sensor fusion may also provide advantages such as reducing uncertainty, resolving conflicts, increasing resolution, and exploiting supplemental information from different sensor modalities.As mentioned above, the image analysis performed at step 108 may include determining a classification of the object. The classification is the category of the object, such as person, car, bus, dog, etc. Step 104 may include accumulating the rod data with the classification determined for the corresponding object. Data enrichment may include associating the stick with the classification. Thus, to the extent that an autonomous driving or ADAS system can differently control the vehicle based on the classification of an object at a particular location, the autonomous driving or ADAS system may behave as if an object with the associated classification is at the location of the stick.As mentioned above, data fusion may also include adjusting the data from various sensors based on the relative accuracy of the sensors. A lidar sensor may have a finer range resolution than cameras. Thus, merging data from different sources may include matching a boundary frame location determined at step 108 to a location of a corresponding line of rods. This is illustrated, for example, in Figures 7A-B.FIGS. 7A-7B illustrate an example of fusing the correlated objects and lines. FIG. 7A shows a diagram 700 having a plurality of rods 602 correlated with an object having the bounding box 604. The diagram 700 illustrates the plurality of rods 602 and the bounding box 604 from the perspective of a lidar sensor and a camera on a vehicle. FIG. 7B shows a diagram 750 with the plurality of rods 602 and the delimiting frame 604 from a nadir perspective. As shown in these figures, the location of the bounding box 604 has been adjusted such that one side of the bounding box 604 runs along the plurality of rods 602.In one embodiment, the fusion described in step 204 may be used to control a vehicle by an autonomous driving or ADAS system in some contexts, but not in others. For example, in some cases, an autonomous driving or ADAS system may be to control a vehicle for safety, while in other contexts, an autonomous driving or ADAS system may be to control a vehicle for comfort.When a vehicle is controlled for safety, an autonomous driving or ADAS system may determine a safe distance to the preceding vehicle, a safe lateral distance to ensure that a vehicle may identify when lateral safety is compromised by a driver entering its lane in an unsafe manner, a driving forward decision based on lanes, traffic signs, traffic lights and other driver behavior, limited vision areas where hazards may exist but are included, and collision avoidance possibilities.However, continuously bringing the vehicle to a safe minimum distance or a safe lateral distance may cause the vehicle to drive very aggressively to the point where it may interfere with the comfort of passengers in the car. Thus, in addition to determining a safe minimum distance or a safe lateral distance, an autonomous driving or ADAS system may determine minimum distances for controlling the vehicle for maximum comfort of the vehicle. In one embodiment, the fusion may occur in step 204 when the vehicle is controlled for comfort, but not when the vehicle is controlled for safety. In this way, the camera and lidar data remain independent, redundant data sources to ensure safe operation of the vehicle.If a particular object using image analysis (decision block 206) is determined not to correlate with one of the rods in the scene, operation may in turn depend on whether the vehicle is being controlled for comfort or safety. At 210, a determination is made as to whether the vehicle is being controlled for occupant comfort or safety. If the vehicle is being controlled for comfort only, the object may be ignored as false positive at 212. At 214, if the vehicle is controlled for safety only, the autonomous driving or ADAS system may control the vehicle based on the object to ensure that the vehicle is driven safely in accordance with the object.If a wand using lidar is determined not to correlate with any of the objects in the scene (at 216), it is determined at 218 whether the wand is an air particle. An air particle is a minute solid or liquid particle floating in the air, such as dust, smoke, mist, precipitate, pollen, or dirt. Lidar can detect air particles by transmitting a beam of light into the air and measuring the amount and direction of light scattered or reflected back by the particles. One possible method of detecting air particles in an image is the use of remote sensing techniques which measure the amount and characteristics of light reflected or scattered by the particles at different wavelengths.Another possible method to detect air particles in an image is the use of machine learning algorithms that can classify and quantify the aerosols based on their features and patterns. For example, convolutional neural networks (CNNs) may learn to extract and recognize the features of aerosols from large sets of marked images and then apply them to new images to estimate the optical depth, size distribution, and composition of the aerosols.If it is determined that the wand is the air particle, the wand is ignored at 220 in controlling the vehicle. For example, the rod is not used at a minimum distance or a safe lateral distance for the vehicle, neither in controlling the vehicle for comfort nor for safety.If it is determined at 222 that the wand is not the air particles, the vehicle is controlled based on the wand. For example, the rod is used to determine a minimum distance or a safe lateral distance to steer the vehicle for both comfort and safety.As mentioned above, the method 100 in FIG. 1 and the method 200 in FIG. 2 may be implemented on a computing device. A computing device may include one or more processors (also called central processing units or CPUs). The processor may be connected to a communication infrastructure or a bus. The computing device may also include user input / output device(s) such as monitors, keyboards, pointing devices, etc., that may communicate with the communication infrastructure via user input / output interface(s).One or more of the processors may be a graphics processing unit (GPU). In one embodiment, a GPU may be a processor that is specialized electronic circuitry designed to process mathematically intensive applications. The GPU may have a parallel structure suitable for parallel processing of large blocks of data, such as mathematically intensive data common to computer graphics applications, images, videos, etc. Similarly, one or more of the processors may be a deep learning processor (DLP). A DLP is an electronic circuit designed for deep learning algorithms and typically has a separate data store and dedicated instruction set architecture. Like a GPU, DLP may utilize high data-level parallelism, relatively large on-chip buffer / memory to utilize data reuse patterns and operators with limited data width for deep learning fault resistance.The computing device may also include main or primary memory, such as random access memory (RAM). Main memory may include one or more levels of cache. The main memory may have control logic (i.e., computer software) and / or data stored therein.The computing device may also include one or more secondary storage devices or memories. The secondary storage may be, for example, a hard disk drive, flash memory, and / or removable storage device or drive.The computing device may further include a communication or network interface. The communication interface may allow the computing device to communicate and interact with any combination of external devices, external networks, external entities, etc. For example, the communication interface may allow the computer system to access external devices over a network, which may be wired and / or wireless (or a combination thereof), and which may include any combination of LANs, WANs, the Internet, etc.The computing device may also be a rack computer, a server blade, a personal digital assistant (PDA), a desktop workstation, a laptop or notebook, a netbook, a tablet, a smart phone, a smart watch or other wearable device, a device, a portion of the Internet of Things, and / or an embedded system, just to name a few non-limiting examples, or any combination thereof.Although several embodiments have been described, it will be apparent to one of ordinary skill in the art that various modifications and changes may be made without departing from the scope of the embodiments described herein. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present teachings. The benefit, advantages, solution issues, and (any) element(s) that may result in a benefit, advantage, or solution occurring or being more pronounced are not to be understood as critical, required, or essential features or elements of any or all claims. The invention(s) is(are) defined solely by the appended claims, including all changes made during the pendency of this application and all equivalents of these claims as appended hereto.Identifiers such as "(a)", "(b)", "(i)", "(ii)", etc. are sometimes used for different elements or steps. These identifiers are used for clarity and do not necessarily dictate an order of the elements or steps.Moreover, in this document, relational terms such as first and second and the like are used exclusively to distinguish one entity or action from another entity or action without necessarily requiring or implying an actual relationship or order between these entities or actions. The terms "comprises," "comprising," "has," "having," "includes," "including," "contains," "containing," or any other modification thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or device comprising a list of elements includes, includes, or contains not only those elements but may also include other elements not expressly listed or belonging to that process, method, article, or device. An element preceded by "comprises... a", "has... a", "includes... a", "includes... a" does not exclude, without further limitations, the existence of additional identical elements in the process, method, article and / or apparatus comprising the element. The terms "a" and "an" are defined as one or more unless expressly stated otherwise herein. The terms "about," "about," or any other versions thereof are defined as being close to what would be understood by one of ordinary skill in the art. A device or structure that is "configured" in a particular manner is configured at least in this manner, but may be configured in other ways that are not listed. For the specification of elements, a singular or plural form may be used, but this does not limit the scope of the disclosure, and the same teaching may apply to plural objects even if an object is mentioned in its singular form in the present application.The embodiments described in detail herein are provided to enable the reader to quickly take an image of the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing detailed description, it is shown that several features are summarized in various embodiments for the purpose of streamlining the disclosure. This type of disclosure is not to be interpreted as requiring more features than are expressly recited in the individual claims. Rather, as the following claims reflect, inventive subject matter lies in at least some instances in less than all features of a single disclosed embodiment. Therefore, the following claims are hereby incorporated into the Detailed Description, each claim standing on its own on a separate claimed subject matter.

Claims

A computer-implemented method for combining data from a lidar sensor and a camera, comprising: receiving a point cloud collected by the lidar sensor, the lidar sensor mounted on a vehicle; identifying a plurality of score lines from the point cloud such that each of the identified plurality of score lines is at a substantially different angle to a plane of a road on which the vehicle is traveling; receiving an image captured by the camera, wherein the camera is mounted on the vehicle and the image has been captured substantially simultaneously with the point cloud; identifying, using an image analysis algorithm, a plurality of objects in the image; and correlating the respective objects of the plurality of objects with the identified plurality of score lines.The method of claim 1, further comprising, when correlating a score line from the plurality of score lines to an object from the plurality of objects, enriching the score with a classification of the object determined using the image analysis algorithm.The method of claim 1, further comprising, if an object of the plurality of objects is determined to be not correlated with any of the plurality of score lines, identifying the object as false positive.The method of claim 3, further comprising: when controlling the vehicle for driver comfort of the vehicle, ignoring the object identified as false positive; and when controlling the vehicle for safety, controlling the vehicle based on the object identified as false positive.The method of claim 1, further comprising, when a score line of the plurality of score lines does not correlate with any of the plurality of objects: controlling the vehicle based on the score line.The method of claim 5, further comprising, when a line of points out of the plurality of lines of points is determined to be non-correlated with any of the plurality of objects: determining whether the line is an air particle; when the line is determined to be an air particle, ignoring the line in controlling the vehicle; and when the line is not determined to be an air particle, controlling the vehicle based on the line.The method of claim 1, wherein each of the plurality of score lines is detected at a common vertical azimuth angle by the lidar sensor.The method of claim 7, wherein each point in the plurality of score lines is sequentially detected by the lidar sensor.The method of claim 1, wherein the plane of the road is substantially horizontal.The method according to claim 9, wherein the substantially different angle is set to a minimum in accordance with the drivability of the road.The method of claim 1, wherein correlating comprises determining whether the respective objects of the plurality of objects are in substantially the same position as respective lines of the plurality of score lines.The method of claim 12, wherein correlating comprises determining whether the respective objects of the plurality of objects have substantially the same shape as the respective lines of the plurality of score lines.A non-transitory computer readable medium including instructions for combining data from a lidar sensor and a camera, causing a computer system to perform operations comprising: receiving a point cloud collected by the lidar sensor, the lidar sensor mounted on a vehicle; identifying a plurality of score lines from the point cloud such that each of the identified plurality of score lines is at a substantially different angle to a plane of a road on which the vehicle travels; receiving an image captured by the camera, wherein the camera is mounted on the vehicle and the image has been captured substantially simultaneously with the point cloud; identifying, using an image analysis algorithm, a plurality of objects in the image; and correlating the respective objects of the plurality of objects with the identified plurality of score lines.The non-transitory computer readable medium of claim 13, wherein the operations further comprise, when correlating a score line from the plurality of score lines to an object from the plurality of objects, enriching the line with a classification of the object determined using the image analysis algorithm.The non-transitory computer-readable medium of claim 13, wherein the operations further comprise, if an object of the plurality of objects is determined to be not correlated with any of the plurality of score lines, identifying the object as false positive.The non-transitory computer readable medium of claim 15, wherein the operations further comprise: in controlling the vehicle for comfort of the driver of the vehicle, ignoring the object identified as false positive; and in controlling the vehicle for safety, controlling the vehicle based on the object identified as false positive.The non-transitory computer readable medium of claim 13, wherein the operations further comprise, when a score line of the plurality of score lines is determined to be non-correlated with any of the plurality of objects: controlling the vehicle based on the score line.The non-transitory computer readable medium of claim 17, wherein the operations further comprise, when the line of points from the plurality of lines of points is determined to be non-correlated with any of the plurality of objects: determining whether the line is an air particle; when the line is determined to be an air particle, ignoring the line in controlling the vehicle; and when the line is not determined to be an air particle, controlling the vehicle based on the line.The non-transitory computer readable medium of claim 13, wherein each of the plurality of score lines is detected at a vertical common azimuth angle by the lidar sensor.The non-transitory computer readable medium of claim 13, wherein the correlating comprises determining whether the respective objects of the plurality of objects have substantially the same shape as respective lines of the plurality of dotted lines.

Images