A multimodal technique for generating 3D road marking labels

Abstract

receiving images captured during movement of the vehicle along a road using a camera mounted on the vehicle; a planar surface is defined relative to the camera; receiving a first annotation for the images identifying features of the road and defined by two-dimensional coordinates in a camera image plane; receiving LiDAR data captured during the movement of the vehicle; fitting a surface to the LiDAR data to generate a fitted surface representing a ground plane for the vehicle; performing a first transformation in which the first annotation on the image plane is projected onto the planar surface to generate a second annotation; performing a second transformation in which the second annotation on the planar surface is projected onto the fitted surface to generate a third annotation; and training a 3D lane detection model using the images and the third annotation to perform image-based predictions without LiDAR input.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Patent Application No. 63 / 265,867, filed December 22, 2021, and entitled “MULTIMODAL TECHNIQUES FOR 3D ROAD MARKING LABEL GENERATION,” the disclosure of which is incorporated by reference in its entirety herein.

[0002] This paper concerns a multimodal technique for three-dimensional (3D) road marking label generation. [Background technology]

[0003] Some vehicles currently manufactured are equipped with one or more types of systems that can at least partially handle the operations related to driving the vehicle. Some such assistance involves automatically surveying the vehicle's surroundings and being able to take action in light of detected roadways, vehicles, pedestrians, and / or other objects. The development of such systems typically involves significant efforts in refining the system's ability to accurately detect and interpret its surroundings. Summary of the Invention

[0004] In an aspect, a method comprises receiving, at a computer system separate from a vehicle, images captured during movement of the vehicle along a road using a camera mounted on the vehicle, where a planar surface is defined relative to the camera; receiving, at the computer system, a first annotation of the images, where the first annotation identifies a feature of the road and is defined by two-dimensional coordinates in an image plane of the camera; receiving, at the computer system, LiDAR data captured during the movement of the vehicle using a LiDAR mounted on the vehicle; fitting a surface to the LiDAR data using the computer system to generate a fitted surface representing a ground plane for the vehicle; performing a first transformation in which the first annotation of the image plane is projected onto the planar surface to generate a second annotation; performing a second transformation in which the second annotation of the planar surface is projected onto the fitted surface to generate a third annotation; and training a 3D lane detection model using the images and the third annotation, the 3D lane detection model being trained to make image-based predictions without LiDAR input.

[0005] Implementations may include any or all of the following features: At least one of the first and second transformations is based on a roll and pitch of the camera relative to a ground surface; The LiDAR data comprises LiDAR point cloud data; The road features include road lanes; Fitting a surface to the LiDAR data comprises performing a convex optimization; The first transformation is a static transformation and the second transformation is a dynamic transformation per frame of the image; Training the 3D lane detection model comprises supervising a machine learning algorithm using a loss function. [Brief description of the drawings]

[0006] [Figure 1] 4 shows examples of sensor outputs generated during vehicle motion.

[0007] [Diagram 2] Examples of annotations and transformations are given below.

[0008] [Diagram 3] Here is an example of training a 3D lane detection model.

[0009] [Figure 4] An example of the method is given below.

[0010] [Diagram 5] An example of a vehicle is shown.

[0011] [Figure 6] 1 illustrates an exemplary architecture of a computing device that may be used to implement aspects of the present disclosure.

[0012] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Described herein are examples of systems and techniques that use inputs of multiple modalities to generate labels for three-dimensional (3D) road markings. Such 3D road marking labels may be used to train 3D lane detection models to perform image-based predictions without LiDAR input. For example, the present disclosure may eliminate restrictive and limiting assumptions of prior approaches while allowing 3D labels to be generated with better performance than in conventional approaches. As another example, 3D road marking labels may be generated with reliable accuracy beyond the typical reach of LiDAR technology.

[0014] In some prior art methods, two-dimensional (2D) lane positions were predicted based on 2D labels. Then, assumptions were made to obtain lane positions on the ground (i.e., in 3D). First, prior art methods may have been based on or otherwise considered the assumption that the camera was at a fixed height above the ground. This assumption may have adversely affected prior art methods in that the height of the camera above the ground may vary depending on whether the vehicle was traveling uphill or downhill. Second, prior art methods may have assumed that the vehicle did not bounce up and down. This assumption may have adversely affected prior art methods in that an uneven road surface may change the vertical position of the vehicle due to the suspension of its wheels. Either or both of the above assumptions may make the mathematical calculations of 3D road marking label generation somewhat inaccurate, thus affecting the quality of the 3D model.

[0015] In contrast, the present disclosure may use 3D labels of road markings to enable the model to directly detect in 3D space from 2D image inputs, without requiring LiDAR data as input when the 3D model is run. In particular, 3D road marking label generation according to the present subject matter may be performed without considering the above assumptions. Hard-coded transformations that may otherwise be used to obtain detections in 3D space may be omitted.

[0016] The examples herein refer to vehicles. A vehicle is a machine that transports passengers or cargo, or both. A vehicle may have one or more motors that use at least one type of fuel, or other energy source (e.g., electricity). Examples of vehicles include, but are not limited to, cars, trucks, and buses. The number of wheels may vary between vehicle types, and one or more (e.g., all) of the wheels may be used to propel the vehicle. A vehicle may include a passenger compartment that accommodates one or more people. At least one vehicle occupant may be considered to be the driver; in this case, various tools, implements, or other devices may be provided to the driver. In the examples herein, any person carried by a vehicle may be referred to as the "driver" or "passenger" of the vehicle, regardless of whether that person is driving the vehicle, whether that person has access to the controls to drive the vehicle, or whether that person lacks the controls to drive the vehicle. The vehicles in this example are shown as similar or identical to each other for illustrative purposes only.

[0017] Examples herein refer to driving assistance. In some implementations, driving assistance may be performed by a driving assistance (AD) system, including but not limited to an autonomous driving system. For example, an AD system may include an advanced driver assistance system (ADAS). Driving assistance involves at least partially automating one or more dynamic driving tasks. An ADAS may perform driving assistance and is an example of a driving assistance system. Driving assistance is typically performed in part based on the output of one or more sensors located on, under, or within the vehicle. The AD system may plan one or more trajectories for the vehicle before and / or while controlling the motion of the vehicle. The planned trajectory may define a path for the vehicle's travel. Thus, propelling the vehicle according to the planned trajectory may correspond to controlling one or more aspects of the vehicle's motion behavior, such as but not limited to the vehicle's steering angle, gear (e.g., forward or reverse), speed, acceleration, and / or braking.

[0018] Although an autonomous vehicle is an example of a system that performs driver assistance, not all driver assistance systems are designed to provide a fully autonomous vehicle. Several levels of driving automation have been defined by SAE International, typically referred to as levels 0, 1, 2, 3, 4, and 5, respectively. For example, a level 0 system or driving mode may not involve persistent vehicle control by the system. For example, a level 1 system or driving mode may include adaptive cruise control, emergency braking assist, automatic emergency braking assist, lane keeping, and / or lane centering. For example, a level 2 system or driving mode may include highway assist, autonomous obstacle avoidance, and / or autonomous parking. For example, a level 3 or 4 system or driving mode may include incremental control of the vehicle by the driver assistance system. For example, a level 5 system or driving mode may not require human intervention in the driver assistance system.

[0019] Examples herein refer to sensors. A sensor is configured to detect one or more aspects of its environment and output a signal reflective of the detection. The detected aspect may be static or dynamic at the time of detection. By way of illustrative example only, a sensor may indicate one or more of the following: distance between the sensor and an object, speed of a vehicle carrying the sensor, trajectory of the vehicle, or acceleration of the vehicle. A sensor may generate an output without probing the surroundings with anything (e.g., passive sensing such as an image sensor capturing electromagnetic radiation), or the sensor may probe the surroundings (e.g., active sensing by sending out electromagnetic radiation and / or sound waves) and detect a response to the probe. Examples of sensors that may be used with one or more embodiments include, but are not limited to, optical sensors (e.g., cameras); light-based sensing systems (e.g., LiDAR); radio-based sensors (e.g., radar); acoustic sensors (e.g., ultrasonic devices and / or microphones); inertial measurement units (e.g., gyroscopes and / or accelerometers); speed sensors (e.g., for a vehicle or a component thereof); position sensors (e.g., for a vehicle or a component thereof); orientation sensors (e.g., for a vehicle or a component thereof); torque sensors; temperature sensors (e.g., primary or secondary thermometers); pressure sensors (e.g., for the ambient air or a component of the vehicle); humidity sensors (e.g., rain detectors); or occupancy sensors.

[0020] Examples herein refer to LiDAR. As used herein, LiDAR includes any object detection system based at least in part on light, which emits light in one or more directions. The light may be generated by a laser and / or by a light emitting diode (LED), to name just two examples. The LiDAR may emit light pulses in different directions (e.g., characterized by different polar angles and / or different azimuthal angles) to survey the surroundings. For example, one or more laser beams may be incident on an orientable reflector for aiming the laser pulses. In some implementations, the LiDAR may include a frequency modulated continuous wave (FMCW) LiDAR. For example, the FMCW LiDAR may use a non-pulsed scanning beam with a modulated (e.g., swept or "chirped") frequency, and a beat between the emitted and detected signals is determined. The LiDAR may detect a return signal by a suitable sensor to generate an output. As used herein, a region of higher resolution in a LiDAR's field of view includes any region in which higher resolution occurs relative to another area of ​​the field of view. To name just a few examples, the LiDAR can be a scanning LiDAR or a non-scanning LiDAR (e.g., flash LiDAR). A scanning LiDAR can operate based on mechanical scanning or non-mechanical scanning. To name just a few examples, a non-mechanical scanning LiDAR can operate using an optical phased array or a tunable metasurface (e.g., including liquid crystals) with structures smaller than the wavelength of light.

[0021] Examples herein refer to machine learning algorithms. As used herein, machine learning algorithms may include implementations of artificial intelligence that allow a machine, such as a driver assistance system, to perceive its environment and take action to achieve one or more goals. Machine learning algorithms may apply one or more principles of data mining to derive a driving envelope from data collected about a vehicle and its associated circumstances. Machine learning algorithms may be trained in one or more perspectives. For example, supervised, semi-supervised, and / or unsupervised training may be performed. In some implementations, machine learning algorithms may utilize one or more classifiers. For example, a classifier may assign one or more labels to instances recognized in the processed data. In some implementations, machine learning algorithms may utilize one or more forms of regression analysis. For example, a machine learning algorithm may apply regression to determine one or more numerical values.

[0022] 1 shows examples of sensor outputs 100 generated during the movement of a vehicle 102. Any or all of the sensor outputs 100 may be used in conjunction with one or more other examples described elsewhere herein. As shown generally, the vehicle 102 is currently moving along a road 104. The vehicle 102 is oriented in at least a forward direction and has an image sensor (e.g., a camera) that captures frames of image data showing part or all of the road 104, which data is represented here as an image 106 that is part of the sensor output 100.

[0023] An annotator (e.g., a person viewing images and assigning labels) annotates the image 106 (and others) based on recognizing one or more features in the image 106. One or more annotations may be generated for the features. Here, annotations 108 and 110 (e.g., lane boundaries) are added to the image 106. For example, annotation 108 or annotation 110 may include one or more point-like structures and / or spatial structures defined for the features of the image 106. The road features may include road lanes. Annotation 108 or annotation 110 is created using 2D coordinates defined with respect to the image plane of the image 106. For example, the 2D coordinates may be defined using respective values ​​of image plane variables u and v. Annotation 108 or annotation 110 may be combined with the image 106 as a modified image, or the annotations may be stored separately in association with the image 106.

[0024] The vehicle 102 has a LiDAR oriented at least in a forward direction and capturing LiDAR data about the vehicle's 102 surroundings, represented here as a LiDAR point cloud 112 that is part of the sensor output 100. The LiDAR point cloud 112 includes points 114 that are defined using 3D coordinates with respect to a LiDAR coordinate system. The LiDAR coordinate system may be centered on the LiDAR or may be transformed to any other origin. Various features of the road 104 may be reflected by the LiDAR point cloud 112, where set 116A (e.g., an essentially linear series of points 114) is a portion of the LiDAR point cloud 112 that reflects features of the road 104 (e.g., lane boundaries). Similarly, sets 116B-116C (e.g., each essentially linear series of points 114) reflect other features of the road 104. The features represented by sets 116A-C may correspond, for example, to lane boundaries relative to a position 118 of the vehicle 102. The LiDAR point cloud 112 may effectively extend only a finite distance in one or more directions from the vehicle 102. In some implementations, the individual points 114 of the LiDAR point cloud 112 may begin to become sparse at some approximate distance. For example, the LiDAR point cloud 112 may extend substantially only about 60 meters to about 80 meters ahead of the vehicle 102. 3D road marking labels may be generated using the images 106 and the LiDAR point cloud 112, for example, as described below.

[0025] Turning now to FIG. 2, this figure illustrates an example of annotations and transformations. Any or all of the annotations and transformations may be used with one or more other examples described elsewhere herein. The annotations and transformations are now represented diagrammatically using a diagram 200 representing a 3D Cartesian space. A camera 202 is shown diagrammatically in the diagram 200. Referring briefly back to FIG. 1, the camera 202 may be mounted in a forward direction of the vehicle 102 and may generate an image 106. A close-up diagram illustrates diagrammatically that the camera 202 includes an image sensor 204 (e.g., a charge-coupled device or any other light-sensitive component) and that the image plane 206 may be considered to extend parallel to the image sensor 204. For example, the plane of the image 106 may be referred to as extending along the image plane 206. A camera axis 208 illustrates diagrammatically the direction from which light reaches the image sensor 204.

[0026] Coordinate system 210 may be defined relative to, for example, based on, the coordinate system of camera 202. Coordinate system 210 may include axes 210A-210B that are perpendicular to each other. For example, axis 210A may lie in a plane that extends such that axis 210B is perpendicular to that plane. This plane of axis 210A is sometimes referred to as a planar plane, and may be imagined here as extending horizontally into diagram 200.

[0027] Here, the surface 212 is similarly extended to the diagram 200. Here, the surface 212 is represented using the lines of the surface 212 that lie in the plane of the coordinate system 210. The surface 212 may be obtained by applying a plane-fitting technique to some or all of the LiDAR point cloud 112. In some implementations, applying the plane-fitting technique may involve determining a maximum density of points 114 associated with a maximum pre-defined height. For example, this may ensure that the surface 212 fits as well as possible to the ground of the road 104 and does not fit, for example, to vehicles or other non-ground structures. The surface 212 is sometimes referred to as a fitting surface, or a ground surface.

[0028] In some implementations, iterative surface fitting techniques may be used in defining the surface 212. For example, the surface fitting techniques may include a random sample consensus process. Any of several types of optimization may be applied in fitting the surface 212. In some implementations, a convex optimization may be performed. For example, one may define a convex function that represents the mismatch or error of the fit between each candidate for the surface 212 and the one of the points 114 with the greatest density, and then attempt to minimize the convex function over a convex set.

[0029] To generate the 3D road marking label, transformations may be applied to the annotations 108 or 110. At least one of the transformations may be based on the roll and pitch of the camera 202 relative to the fitted surface 212 (e.g., the ground plane). Here, angle θ is shown generally as defined by the camera axis 208 and the fitted surface 212. Angle θ' is shown generally as defined by the camera axis 208 and the planar plane of axis 210A. Angle θ'' is shown generally as defined by the fitted surface 212 and the planar plane of axis 210A. Thus, the relationship is θ''=θ'−θ.

[0030] In some implementations, the first transformation involves projecting the annotation 108 or annotation 110 onto the planar surface of the axis 210A. This may be a static transformation. This first transformation may generate new annotations for the planar surface of the axis 210A, represented here diagrammatically as annotation 108' and annotation 110' in the planar surface of the axis 210A. For example, this transformation may be performed using a transformation matrix defined at least in part based on the angle θ'.

[0031] In some implementations, the second transformation involves projecting the annotation 108' or annotation 110' onto the fitted surface 212. This may be a dynamic transformation that is performed for each frame of the camera image (e.g., image 106). The second transformation may generate new annotations for the fitted surface 212, represented here generally as annotation 108'' and annotation 110'' within the fitted surface 212. For example, this transformation may involve using a transformation matrix that is defined at least in part based on the angle θ''.

[0032] That is, one can transform from the image plane 206 to the flat plane of axis 210A, and then transform from the flat plane of axis 210A to the fitted plane 212.

[0033] Referring again to FIG. 1 , representation 112′ here shows both a portion of LiDAR point cloud 112 (e.g., point 114) and road markings 116A′-116C′ based on annotations 108″ and 110″ to fitting surface 212. Road markings 116A′ added to representation 112′ may essentially correspond to set 116A; similarly, road markings 116B′-116C′ added to representation 112′ may essentially correspond to sets 116B-116C, respectively. Road markings 116A′-116C′ represent 3D coordinates for road features. The combination of road markings 116A′-116C′ with LiDAR point cloud 112 to generate representation 112′ is shown here for illustrative purposes only. Road markings 116A'-116C' may exist separately (e.g., as a set of 3D coordinates) from LiDAR point cloud 112 or other LiDAR data. Camera images (including, for example, image 106) and annotations 108'' and 110'' may be used, for example, to train a model, as will now be described.

[0034] 3 shows an example 300 of training a 3D lane detection model 302. Some or all of the example 300 may be used with one or more other examples described elsewhere herein. The 3D lane detection model 302 may be generated, trained, or otherwise calibrated using a machine learning algorithm. For example, an iterative process may be applied.

[0035] Example 300 involves an image 304. In some implementations, image 304 may be captured using a vehicle-mounted camera (e.g., camera 202 in FIG. 2) used for data collection purposes. For example, image 304 may be received by a system separate from the vehicle that is being used to develop and refine 3D lane detection model 302. Image 304 may be annotated. For example, annotations 108 or annotations 110 (FIG. 1) may be defined in or for image 304.

[0036] The 3D lane detection model 302 generates an output 306. In some implementations, the output 306 represents estimates or predictions for the 3D coordinates of features that are visible in at least a portion of the image 304. In some implementations, the output 306 includes 3D coordinates in a Cartesian coordinate system (e.g., (x, y, z)-coordinates) that indicate that the 3D lane detection model 302 has determined where lane features defined in image space will be located in the 3D coordinate system.

[0037] Input 308 illustrates generally that 3D road marking labels may be received. In some implementations, the 3D road marking labels have been determined from image 304 based on transforming, or otherwise performing a projection of, one or more annotations of image 304. For example, the transformation described using angles θ, θ′, and θ″ in FIG. 2 may be used. Ground truth 310 may be used to train (e.g., develop or otherwise improve, optimize, or complete) 3D lane detection model 302. Ground truth 310 includes 3D coordinates in a Cartesian coordinate system (e.g., (x*, y*, z*)-coordinates).

[0038] The output 306 and the ground truth 310 may be compared in one or more ways. In some implementations, a loss function 312 may be applied to the (x, y, z)-coordinates and the (x*, y*, z*)-coordinates. For example, if the output 306 partially predicts the coordinates (1, 1, 1) and the ground truth 310 shows the coordinates (10, 10, 10) instead, the result 314 of applying the loss function 312 may represent some function of the value (9, 9, 9), which is the difference between the respective sets of coordinates. The result 314 may be applied to train or otherwise tune the 3D lane detection model 302. The training may be supervised using iterative feedback with the output 306 and the ground truth 310 applied to the loss function 312 to improve the accuracy and reliability of the 3D lane detection model 302. In some implementations, this may enable the 3D lane detection model 302 to eventually make image-based predictions without LiDAR input. For example, this may enable a vehicle's ADAS to omit the LiDAR device and instead rely on camera output to perform lane detection and other functions in an autonomous vehicle.

[0039] 4 illustrates an example of a method 400. The method 400 may be used with one or more other examples described elsewhere herein. More or fewer operations may be performed than shown. Two or more operations may be performed in a different order unless otherwise indicated.

[0040] At operation 402, the method 400 includes receiving, at a computer system separate from the vehicle, images captured during the movement of the vehicle along a road using a camera mounted on the vehicle. For example, image 304 (FIG. 3) may be received (e.g., including image 106) by a system training a 3D lane detection model 302. A planar plane is defined relative to the camera. For example, a planar plane of axis 210A may be defined relative to camera 202 (FIG. 2).

[0041] At operation 404, method 400 includes receiving, at a computer system, a first annotation for the image. For example, annotation 108 or annotation 110 (FIG. 1) may be received. The annotation identifies a feature of a road (e.g., road 104) and is defined by 2D coordinates (e.g., variables u and v) in an image plane of a camera (e.g., image plane 206 in FIG. 2).

[0042] At operation 406, the method 400 includes receiving, at a computer system, LiDAR data captured during motion of the vehicle using a LiDAR mounted on the vehicle. For example, the LiDAR point cloud 112 (e.g., including points 114) may be received.

[0043] At operation 408, the method 400 includes using a computer system to fit a surface to the LiDAR data to generate a fitted surface that represents the contact surface for the vehicle. In some implementations, the fitted surface 212 (FIG. 2) may be defined based on point density in the LiDAR data. For example, a convex optimization may be performed.

[0044] At operation 410, the method 400 includes performing a first transformation in which a first annotation in the image plane is projected onto a planar plane to generate a second annotation. For example, annotation 108 or annotation 110 in the image plane 206 may be transformed to annotation 108' or annotation 110' in the planar plane of axis 210A.

[0045] At operation 412, the method 400 includes performing a second transformation in which the second annotation on the planar surface is projected onto the fitted surface to generate a third annotation. For example, annotation 108' or annotation 110' on the planar surface of axis 210A may be transformed to annotation 108'' or annotation 110'' on the fitted surface 212.

[0046] At operation 414, the method 400 includes training a 3D lane detection model using the images and the third annotations. For example, the 3D lane detection model 302 may be trained. The 3D lane detection model may be trained to perform image-based predictions without LiDAR input. For example, in this case, only the images 304 may be used to create the output 306, without any LiDAR point cloud.

[0047] FIG. 5 illustrates an example of a vehicle 500. The vehicle 500 may be used with one or more other examples described elsewhere herein. The vehicle 500 includes an ADAS / AD system 502 and a vehicle control 504. The ADAS / AD system 502 may be implemented using some or all of the components described with reference to FIG. 6 below. The ADAS / AD system 502 includes a sensor 506 and a planning algorithm 508. The planning algorithm 508 may include or otherwise utilize a 3D lane detection model trained using one or more examples described herein. Other aspects of the vehicle 500, including but not limited to other components of the vehicle 500 in which the ADAS / AD system 502 may be implemented, are omitted here for the sake of brevity.

[0048] The sensor 506 is also described herein as including appropriate circuitry and / or executable programming for processing the sensor output and performing detection based on that processing. The sensor 506 may include a LiDAR 510. In some implementations, the LiDAR 510 may include any object detection system based at least in part on laser light. For example, the LiDAR 510 may be oriented in any direction relative to the vehicle and may be used to detect at least the distance to one or more other objects (e.g., another vehicle). The LiDAR 510 may detect the surroundings of the vehicle 500 by sensing the presence of an object in relation to the vehicle 500. In some implementations, the LiDAR 510 is a scanning LiDAR or a non-scanning LiDAR (e.g., a flash LiDAR). The LiDAR 510 is shown here in dashed outline: the LiDAR 510 may be used during collection of data to be processed to provide ground truth for training a 3D lane detection model, and may be omitted (from the vehicle 500 or another vehicle) when the 3D lane detection model is applied to make image-based predictions.

[0049] Sensors 506 may include a camera 512. In some implementations, camera 512 may include any image sensor whose signals the vehicle 500 considers. For example, camera 512 may be oriented in any direction relative to the vehicle and may be used to detect vehicles, lanes, lane markings, curbs, and / or road signs. Camera 512 may detect the surroundings of vehicle 500 by visually registering the situation in relation to vehicle 500. In some implementations, one or more other types of sensors may additionally be included in sensor 506.

[0050] The planning algorithm 508 may plan the ADAS / AD system 502 to perform one or more actions, or no action, in response to monitoring the vehicle's 500 surroundings and / or driver input. One or more outputs of the sensors 506 may be taken into account. In some implementations, the planning algorithm 508 may perform motion planning and / or plan a trajectory for the vehicle 500. For example, a 3D lane detection model may perform image-based predictions without LiDAR input.

[0051] Vehicle control 504 may include steering control 514. In some implementations, the ADAS / AD system 502 and / or another driver of the vehicle 500 controls the trajectory of the vehicle 500 by adjusting the steering angle of at least one wheel by manipulating the steering control 514. The steering control 514 may be configured to control the steering angle through a mechanical connection between the steering control 514 and the adjustable wheels, or may be part of a steer-by-wire system.

[0052] Vehicle control 504 may include a gear control 516. In some implementations, the ADAS / AD system 502 and / or another operator of the vehicle 500 uses the gear control 516 to select among multiple operating modes of the vehicle (e.g., a drive mode, a neutral mode, or a park mode). For example, the gear control 516 may be used to control an automatic transmission in the vehicle 500.

[0053] Vehicle control 504 may include a signal control 518. In some implementations, signal control 518 may control one or more signals that may be generated by vehicle 500. For example, signal control 518 may control the turn signals and / or horn of vehicle 500.

[0054] Vehicle controls 504 may include brake controls 520. In some implementations, brake controls 520 may control one or more types of braking systems designed to slow the vehicle, stop the vehicle, and / or keep the vehicle stationary when stopped. For example, brake controls 520 may be actuated by the ADAS / AD system 502. As another example, brake controls 520 may be actuated by the driver using a brake pedal.

[0055] Vehicle control 504 may include a vehicle dynamics system 522. In some implementations, vehicle dynamics system 522 may control one or more functions of vehicle 500 in addition to, in the absence of, or in place of driver control. For example, when the vehicle is stopped on a hill, vehicle dynamics system 522 may hold the vehicle stationary if the driver does not activate brake control 520 (e.g., by pressing the brake pedal).

[0056] Vehicle controls 504 may include an acceleration control 524. In some implementations, acceleration control 524 may control one or more types of traction motors of the vehicle. For example, acceleration control 524 may control an electric motor and / or an internal combustion motor of the vehicle 500.

[0057] Vehicle 500 may include a user interface 526. User interface 526 may include an audio interface 528. In some implementations, audio interface 528 may include one or more speakers positioned within the passenger compartment. For example, audio interface 528 may operate, at least in part, with an infotainment system within the vehicle.

[0058] User interface 526 may include a visual interface 530. In some implementations, visual interface 530 may include at least one display device within a passenger compartment of vehicle 500. For example, visual interface 530 may include a touch screen device and / or an instrument cluster display.

[0059] FIG. 6 illustrates an example architecture of a computing device 600 that may be used to implement aspects of the disclosure, including any of the systems, devices, and / or techniques described herein or any other systems, devices, and / or techniques that may be utilized in various possible embodiments.

[0060] The computing device illustrated in FIG. 6 may be used to execute the operating system, application programs, and / or software modules (including software engines) described herein.

[0061] In some embodiments, the computing device 600 includes at least one processing device 602 (e.g., processor), such as a central processing unit (CPU). A variety of processing devices are available from a variety of manufacturers, such as Intel or Advanced Micro Devices. In this example, the computing device 600 also includes a system memory 604, and a system bus 606 that couples various system components including the system memory 604 to the processing device 602. The system bus 606 is one of a number of types of bus structures that can be used, including, but not limited to, a memory bus or memory controller; a peripheral bus; and a local bus, using any of a variety of bus architectures.

[0062] Examples of computing devices that may be implemented using computing device 600 include desktop computers, laptop computers, tablet computers, mobile computing devices (such as smartphones, touchpad mobile digital devices, or other mobile devices), or other devices configured to process digital instructions.

[0063] The system memory 604 includes a read-only memory 608 and a random access memory 610. A basic input / output system 612, containing the basic routines that function to transfer information within the computing device 600, such as during start-up, may be stored in the read-only memory 608.

[0064] In some embodiments, computing device 600 also includes a secondary storage device 614, such as a hard disk drive for storing digital data. The secondary storage device 614 is connected to system bus 606 by a secondary storage interface 616. The secondary storage device 614 and its associated computer-readable media provide non-volatile, non-transitory storage of computer-readable instructions (including application programs and program modules), data structures, and other data for computing device 600.

[0065] Although the exemplary environment described herein employs a hard disk drive as a secondary storage device, other embodiments use other types of computer-readable storage media. Examples of these other types of computer-readable storage media include magnetic cassettes, flash memory cards, solid-state drives (SSDs), digital video disks, Bernoulli cartridges, compact disk read-only memories, digital versatile disk read-only memories, random access memories, or read-only memories. Some embodiments include non-transitory media. For example, a computer program product may be tangibly embodied in a non-transitory storage medium. Additionally, such computer-readable storage media may include local storage or cloud-based storage.

[0066] A number of program modules may be stored in the secondary storage device 614 and / or system memory 604, including an operating system 618, one or more application programs 620, other program modules 622 (such as a software engine described herein), and program data 624. The computing device 600 may utilize any suitable operating system.

[0067] In some embodiments, a user provides input to the computing device 600 through one or more input devices 626. Examples of input devices 626 include a keyboard 628, a mouse 630, a microphone 632 (e.g., for voice and / or other audio input), a touch sensor 634 (such as a touchpad or touch-sensitive display), and a gesture sensor 635 (e.g., for gesture input). In some implementations, the input devices 626 provide presence, proximity, and / or motion-based detection. Other embodiments include other input devices 626. The input devices may be connected to the processing device 602 through an input / output interface 636 that is coupled to the system bus 606. These input devices 626 may be connected by any number of input / output interfaces (such as a parallel port, a serial port, a game port, or a universal serial bus). In some contemplated embodiments, wireless communication between the input device 626 and the input / output interface 636 is also possible, including infrared, BLUETOOTH® wireless technology, 802.11a / b / g / n, cellular, ultra-wideband (UWB), ZIGBEE®, or other radio frequency communication systems, to name just a few.

[0068] In this exemplary embodiment, a display device 638, such as a monitor, liquid crystal display device, light emitting diode display device, projector, or touch-sensitive display device, is also connected to the system bus 606 via an interface, such as a video adapter 640. In addition to the display device 638, computing device 600 may also include various other peripheral devices (not shown), such as speakers or printers.

[0069] Computing device 600 may be connected to one or more networks via network interface 642. Network interface 642 may provide wired and / or wireless communication. In some implementations, network interface 642 may include one or more antennas for transmitting and / or receiving wireless signals. When used in a local area networking environment or a wide area networking environment (such as the Internet), network interface 642 may include an Ethernet interface. Other possible embodiments use other communication devices. For example, some embodiments of computing device 600 include a modem for communicating across a network.

[0070] Computing device 600 may include at least some form of computer readable media. Computer readable media includes any available media that can be accessed by computing device 600. By way of example, computer readable media include computer readable storage media and computer readable communication media.

[0071] Computer readable storage media includes volatile and nonvolatile, removable and non-removable media implemented in any device configured to store information, such as computer readable instructions, data structures, program modules, or other data. Computer readable storage media includes, but is not limited to, random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory, or other memory technology, compact disc read-only memory, digital versatile disk, or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 600.

[0072] Computer-readable communication media typically embodied computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, computer-readable communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency, infrared and other wireless media. Combinations of any of the above are also included within the scope of computer-readable media.

[0073] The computing device illustrated in FIG. 6 is also an example of a programmable electronic apparatus that may include one or more such computing devices, and when multiple computing devices are included, such computing devices may be coupled together to a suitable data communications network to collectively perform various functions, methods, or operations disclosed herein.

[0074] In some implementations, the computing device 600 may be characterized as an ADAS computer. For example, the computing device 600 may include one or more components that are possibly used to process tasks that arise in the field of artificial intelligence (AI). In turn, the computing device 600 generally includes sufficient forward power and necessary support architecture for the demands of ADAS or AI. For example, the processing device 602 may include a multi-core architecture. As another example, the computing device 600 may include one or more co-processors in addition to or as part of the processing device 602. In some implementations, at least one hardware accelerator may be coupled to the system bus 606. For example, a graphics processing unit may be used. In some implementations, the computing device 600 may implement neural network specific hardware to handle one or more ADAS tasks.

[0075] The terms "substantially" and "about" used throughout this specification are used to describe and report small variations, such as those due to processing variations. For example, they can refer to less than or equal to ±5%, such as less than or equal to ±2%, such as less than or equal to ±1%, such as less than or equal to ±0.5%, such as less than or equal to ±0.2%, such as less than or equal to ±0.1%, such as less than or equal to ±0.05%. Also, as used herein, indefinite articles such as "a" or "an" mean "at least one".

[0076] It should be understood that all combinations of the above concepts, and additional concepts discussed in more detail below (provided that such concepts are not mutually inconsistent), are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein.

[0077] Several implementations have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of the specification.

[0078] Additionally, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Additionally, other processes may be provided or processes may be eliminated from the described flows, and other components may be added to or removed from the described systems. Accordingly, other implementations are within the scope of the following claims.

[0079] While certain features of the described implementations have been shown as described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It should therefore be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of these implementations. It should be understood that they have been presented by way of example only and not limitation, and that various changes in form and details may be made. Any part of the apparatus and / or method described herein may be combined in any combination, except in mutually exclusive combinations. The implementations described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different implementations described.

Claims

1. receiving, in a computer system separate from the vehicle, images captured during movement of the vehicle along a roadway using a camera mounted on the vehicle, where a planar surface is defined relative to the camera; receiving, at the computer system, a first annotation for the image, where the first annotation identifies a feature of the road and is defined by two-dimensional coordinates within an image plane of the camera; receiving, in the computer system, LiDAR data captured during the motion of the vehicle using a LiDAR mounted on the vehicle; using the computer system to fit a surface to the LiDAR data to generate a fitted surface representative of a contact surface for the vehicle; performing a first transformation in which the first annotation on the image plane is projected onto the planar plane to generate a second annotation; performing a second transformation in which the second annotation of the planar surface is projected onto the fitted surface to generate a third annotation; and training a 3D lane detection model using the images and the third annotations, the 3D lane detection model being trained to perform image-based predictions without LiDAR input. A method comprising:

2. The method of claim 1 , wherein at least one of the first transformation and the second transformation is based on a roll and pitch of the camera relative to the ground plane.

3. The method of claim 1 , wherein the LiDAR data comprises LiDAR point cloud data.

4. The method of claim 1 , wherein the features of the road include lanes of the road.

5. The method of claim 1 , wherein fitting the surface to the LiDAR data comprises performing a convex optimization.

6. The method of claim 1 , wherein the first transformation is a static transformation and the second transformation is a frame-by-frame dynamic transformation of the image.

7. The method of any one of claims 1 to 6, wherein training the 3D lane detection model comprises supervising a machine learning algorithm using a loss function.

Images