Predicting three-dimensional features for autonomous driving

Abstract

A processor coupled to a memory is configured to receive image data based on an image captured by a camera of a vehicle. The image data is used as a basis for input to a trained machine learning model trained to predict a three-dimensional trajectory of a machine learning feature. The three-dimensional trajectory of the machine learning feature is provided for autonomous control of the vehicle.

Description

[0001] Cross Reference to Related Applications

[0002] This application is a continuation of, and claims priority to, U.S. Patent Application No. 16 / 265,720, filed February 1, 2019, entitled “PREDICTING THREE-DIMENSIONAL FEATURES FOR AUTONOMOUS DRIVING,” the disclosure of which is incorporated by reference herein in its entirety. BACKGROUND

[0003] Deep learning systems for applications such as autonomous driving are developed by training machine learning models. Generally, the performance of a deep learning system is limited, at least in part, by the quality of the training set used to train the model. In many cases, significant resources are expended in collecting, organizing, and annotating training data. Traditionally, much of the work of organizing a training data set is done manually by reviewing potential training data and correctly labeling features associated with the data. The work required to create a training set with accurate labels can be significant, but is often tedious. Moreover, it is often difficult to collect and accurately label data that a machine learning model needs to improve. Accordingly, there is a need to improve the process for generating training data with accurately labeled features. BRIEF DESCRIPTION OF DRAWINGS

[0004] Various embodiments of the application are disclosed in the following detailed description and the accompanying drawings.

[0005] Figure 1 is a block diagram illustrating an embodiment of a deep learning system for autonomous driving.

[0006] Figure 2 is a flow diagram illustrating an embodiment of a process for training and applying a machine learning model for autonomous driving.

[0007] Figure 3 is a flow diagram illustrating an embodiment of a process for creating training data using a time series of elements.

[0008] Figure 4 is a flow diagram illustrating an embodiment of a process for training and applying a machine learning model for autonomous driving.

[0009] Figure 5 is a diagram illustrating an example of an image captured from a vehicle sensor.

[0010] Figure 6 is a diagram illustrating an example of an image captured from a vehicle sensor with a predicted three-dimensional trajectory of a lane line. DETAILED DESCRIPTION

[0011] The application can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product of a computer readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the application can take, can be referred to as techniques. In general, the order of the steps of disclosed processes can be altered, except

[0012] The accompanying drawings, together with the specification, illustrate various embodiments of the application. Figure 1 A detailed description of one or more embodiments of the application is provided below along with accompanying figures that illustrate the principles of the application. The application is described in connection with such embodiments, but the application is not limited to any embodiment. The scope of the application is limited only by the claims and the application encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the application. These details are provided for the purpose of example and the application can be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the application has not been described in detail so that the application is not unnecessarily obscured.

[0013] A machine learning training technique for generating highly accurate machine learning results is disclosed. A training dataset is created using data captured by sensors on a vehicle that capture the vehicle's environment and vehicle operating parameters. For example, sensors fixed on the vehicle capture data such as image data of the road and surrounding environment that the vehicle is driving on. The sensor data can capture vehicle lane lines, vehicle lanes, other vehicle traffic, obstacles, traffic control signs, etc. Odometer and other similar sensors capture vehicle operating parameters such as vehicle speed, steering, orientation, directional changes, location changes, elevation changes, speed changes, etc. The captured dataset is transmitted to a training server to create a training dataset. The training dataset is used to train a machine learning model to generate highly accurate machine learning results. In some embodiments, a time series of the captured data is used to generate the training data. For example, ground truth is determined based on a time series group of elements and is associated with a single element from the group. As one example, a series of images over a time period, such as 30 seconds, is used to determine the actual path of a vehicle lane line over the time period that the vehicle is driving. The vehicle lane line is determined by using the most accurate vehicle lane image over the time period. Different portions (or locations) of the lane line can be identified from different image data of the time series. More accurate data is captured for different portions of the lane line as the vehicle drives on the lane next to the lane line. In some examples, occluded portions of the lane line are revealed as the vehicle drives, for example, along a hidden curve or over a hill. The most accurate portion of the lane line from each image of the time series can be used to identify the lane line over the entire group of image data. Image data of the lane line far away is typically not as detailed as image data of the lane line near the vehicle. By capturing a time series of image data as the vehicle drives along the lane, accurate image data of all portions of the corresponding lane line and corresponding odometer data is collected.

[0014] In some embodiments, a three-dimensional representation of a feature, such as a lane line, is created from a time series group of elements that correspond to ground truth. The ground truth is then associated with a subset of the time series of elements, such as a single image frame of the group of captured image data. For example, a first image in the group of images is associated with ground truth of a lane line represented in three-dimensional space. Although the ground truth is determined based on the group of images, the selected first frame and ground truth are used to create training data. As an example, training data is created to predict a three-dimensional representation of a vehicle lane using only a single image. In some embodiments, any element or group of elements of the time series group of elements is associated with ground truth and used to create training data. For example, ground truth can be applied to an entire video sequence to create training data. As another example, a middle element or last element of the time series group of elements is associated with ground truth and used to create training data.

[0015] In various embodiments, the selected images and real data can be applied to different features, such as lane lines, path prediction of vehicles including proximate vehicles, depth distance of objects, traffic control signs, etc. For example, a series of images of a vehicle in an adjacent lane are used to predict a path of the vehicle. Using a time series of images and the actual path taken by the adjacent vehicle, a single image in the set and the actual path taken can be used as training data to predict the path of the vehicle. The information can also be used to predict whether the adjacent vehicle will cut into the path of the autonomous vehicle. For example, the path prediction can predict whether the adjacent vehicle will merge in front of the autonomous vehicle. The autonomous vehicle can be controlled to minimize the likelihood of a collision. For example, the autonomous vehicle can slow down to prevent a collision, adjust the speed and / or steering of the vehicle to prevent a collision, initiate a warning to the adjacent vehicle and / or occupants of the autonomous vehicle, and / or change lanes, etc. In various embodiments, the ability to accurately infer path predictions, including vehicle path predictions, significantly improves the safety of the autonomous vehicle.

[0016] In some embodiments, the trained machine learning model is used to predict a three-dimensional representation of one or more features for autonomous driving, including lane lines. For example, rather than identifying two-dimensional lane lines from image data by segmenting images of lane lines, a three-dimensional representation is generated using a time series of elements and odometer data corresponding to the time series. The three-dimensional representation includes elevation changes, which greatly improves the accuracy of lane line detection and the detection of corresponding lanes and identified drivable paths. In some embodiments, one or more splines or another parametric representation is used to represent the lane lines. Using piecewise polynomials to represent the lane lines greatly reduces the computational resources required to evaluate the three-dimensional objects. This reduction in computational resources corresponds to an improvement in processing speed and efficiency without significantly sacrificing the accuracy of the representation. In various embodiments, a piecewise polynomial, a set of three-dimensional points, or another suitable representation can be used to represent the lane lines, particularly lanes including curves in the lane lines. For example, a piecewise polynomial inserts an actual lane line using highly accurate segments of the lane line that are identified from a set of elements captured over time using sensor data.

[0017] In some embodiments, sensor data is received. The sensor data can include images (such as video and / or still images), radar, audio, lidar, inertial, odometry, location, and / or other forms of sensor data. The sensor data includes a time series group of elements. For example, the time series group of elements can include a group of images captured from a camera sensor of a vehicle over a time period. In some embodiments, a training dataset is determined, including determining corresponding ground truth data for at least a selected time series of elements of the time series group of elements based on the time series of multiple elements of the time series group of elements. For example, the ground truth data is determined by examining the most relevant portion of each element of the time series group of elements, including time series of preceding and / or subsequent elements of the group. In some scenarios, only the preceding and / or subsequent time series of elements include data that is not present in the earlier time series of elements, such as a lane line that first disappears around a curve and only emerges in the later time series of elements. The determined ground truth data can be a three-dimensional representation of a vehicle lane, a predicted path of the vehicle, or another similar prediction. The elements of the time series group of elements are selected and associated with the ground truth data. The selected elements and the ground truth data are part of the training dataset. In some embodiments, a processor is used to train a machine learning model using the training dataset. For example, the training dataset is used to train a machine learning model to infer features for self-driving or driver-assist operations of a vehicle. Using the trained machine learning model, a neural network can infer features associated with autonomous driving, such as vehicle lanes, drivable space, objects (e.g., pedestrians, stationary vehicles, moving vehicles, etc.), weather (e.g., rain, hail, fog, etc.), traffic control objects (e.g., traffic lights, traffic signs, road signs, etc.), traffic patterns, etc.

[0018] In some embodiments, a system includes a processor and a memory coupled to the processor. The processor is configured to receive image data based on images captured by a camera of a vehicle. For example, a camera sensor fixed to the vehicle captures images of the environment of the vehicle. The camera can be a front-facing camera, a door pillar camera, or another appropriately positioned camera. The image data captured from the camera is processed using a processor on the vehicle, such as a GPU or an AI processor. In some embodiments, the image data is used as a basis for input to a trained machine learning model trained to predict a three-dimensional trajectory of a vehicle lane. For example, the image data is used as input to a neural network trained to predict a vehicle lane. The machine learning model infers a three-dimensional trajectory of the detected lane. Rather than segmenting the image into two-dimensional lane and non-lane segments, a three-dimensional representation is inferred. In some embodiments, the three-dimensional representation is a spline, a parametric curve, or another representation capable of describing a curve in three dimensions. In some embodiments, the three-dimensional trajectory of the vehicle lane is provided in autonomous control of the vehicle. For example, the three-dimensional trajectory is used to determine lane lines and corresponding drivable space.

[0019] Figure 1 is a block diagram illustrating embodiments of a deep learning system for autonomous driving. The deep learning system includes different components that can be used together for self-driving and / or driver-assisted operations of a vehicle, as well as for collecting and processing data to train machine learning models for autonomous driving. In various embodiments, the deep learning system is installed on a vehicle. Data from the vehicle can be used to train and improve autonomous driving features of the vehicle or other similar vehicles.

[0020] In the illustrated example, the deep learning system 100 is a deep learning network that includes a sensor 101, an image pre-processor 103, a deep learning network 105, an artificial intelligence (AI) processor 107, a vehicle control module 109, and a network interface 111. In various embodiments, the different components are communicatively connected. For example, sensor data from the sensor 101 is fed to the image pre-processor 103. Processed sensor data from the image pre-processor 103 is fed to the deep learning network 105 running on the AI processor 107. The output of the deep learning network 105 running on the AI processor 107 is fed to the vehicle control module 109. In various embodiments, the vehicle control module 109 is connected to the vehicle and controls the operation of the vehicle, such as the speed, braking, and / or steering of the vehicle. In various embodiments, sensor data and / or machine learning results can be sent to a remote server via the network interface 111. For example, sensor data can be transmitted to a remote server via the network interface 111 to collect training data for improving the performance, comfort, and / or safety of the vehicle. In various embodiments, the network interface 111 is used to communicate with a remote server, make phone calls, send and / or receive text messages, and transmit sensor data based on the operation of the vehicle and for other reasons. In some embodiments, the deep learning system 100 can include additional or fewer components as desired. For example, in some embodiments, the image pre-processor 103 is an optional component. As another example, in some embodiments, a post-processing component (not shown) is used to perform post-processing on the output of the deep learning network 105 before providing the output to the vehicle control module 109.

[0021] In some embodiments, sensor 101 includes one or more sensors. In various embodiments, sensor 101 can be affixed to the vehicle at various locations and / or oriented in one or more different directions. For example, sensor 101 can be affixed to the front, side, rear, and / or roof of the vehicle in various orientations, such as front, rear, or side. In some embodiments, sensor 101 can be an image sensor, such as a high dynamic range camera. In some embodiments, sensor 101 includes non-visual sensors. In some embodiments, sensor 101 includes radar, audio, LiDAR, inertial, odometer, location, and / or ultrasonic sensors. In some embodiments, sensor 101 is not mounted on the vehicle with vehicle control module 109. For example, sensor 101 can be mounted on a neighboring vehicle and / or affixed to the road or environment and included as part of a deep learning system to capture sensor data. In some embodiments, sensor 101 includes one or more cameras that capture the road surface on which the vehicle is traveling. For example, one or more front-facing cameras and / or door post cameras capture lane markings for the lane in which the vehicle is traveling. As another example, the cameras capture neighboring vehicles, including those attempting to cut into the vehicle's lane. Additional sensors capture odometer, location, and / or vehicle control information, including information related to the vehicle's trajectory. Sensor 101 may include two image sensors capable of capturing still images and / or video. Data may be captured over a period of time, such as a sequence of captured data over a period of time. For example, images of lane markings may be captured along with vehicle odometer data over a 15 second period or another suitable period of time. In some embodiments, sensor 101 includes a location sensor, such as a global positioning system (GPS) sensor for determining the vehicle's location and / or changes in location.

[0022] In some embodiments, the image preprocessor 103 is used to pre-process sensor data of the sensor 101. For example, the image preprocessor 103 can be used to pre-process the sensor data, split the sensor data into one or more components, and / or post-process one or more components. In some embodiments, the image preprocessor 103 is a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the image preprocessor 103 is a tone mapping processor that processes high dynamic range data. In some embodiments, the image preprocessor 103 is implemented as part of the artificial intelligence (AI) processor 107. For example, the image preprocessor 103 can be a component of the AI processor 107. In some embodiments, the image preprocessor 103 can be used to normalize an image or transform an image. For example, an image captured with a fisheye lens can be distorted, and the image preprocessor 103 can be used to transform the image to remove or modify the distortion. In some embodiments, noise, distortion, and / or blur intensity is removed or reduced during the pre-processing step. In various embodiments, the image is adjusted or normalized to improve the results of machine learning analysis. For example, the white balance of the image is adjusted to account for different lighting operating conditions, such as daylight, sunny, overcast, dusk, sunrise, sunset, and nighttime conditions, among others.

[0023] In some embodiments, the deep learning network 105 is a deep learning network used to determine vehicle control parameters, including analyzing a driving environment to determine lane markings, lanes, drivable space, obstacles, and / or potential vehicle paths, among others. For example, the deep learning network 105 can be an artificial neural network, such as a convolutional neural network (CNN), trained based on inputs, such as sensor data, and whose output is provided to the vehicle control module 109. As one example, the output can include at least a three-dimensional representation of lane markings. As another example, the output can include at least potential vehicles that can merge into the vehicle’s lane. In some embodiments, the deep learning network 105 receives at least sensor data as input. Additional inputs can include scene data describing the environment around the vehicle and / or vehicle specifications, such as the vehicle’s operating characteristics. The scene data can include scene labels describing the environment around the vehicle, such as rain, wet road, snow, mud, high-density traffic, highway, city, school zone, among others. In some embodiments, the output 105 of the deep learning network is a three-dimensional trajectory of the vehicle’s lane. In some embodiments, the output of the deep learning network 105 is a potential vehicle cut-in. For example, the deep learning network 105 identifies a neighboring vehicle that can enter the vehicle’s forward lane.

[0024] In some embodiments, the artificial intelligence (AI) processor 107 is a hardware processor for running a deep learning network 105. In some embodiments, the AI processor 107 is a specialized AI processor for performing inferences on sensor data using a convolutional neural network (CNN). The AI processor 107 can be optimized for the bit depth of the sensor data. In some embodiments, the AI processor 107 is optimized for deep learning operations, such as neural network operations, including convolutions, dot products, vector and / or matrix operations, etc. In some embodiments, the AI processor 107 is implemented using a graphics processing unit (GPU). In various embodiments, the AI processor 107 is coupled to a memory configured to provide the AI processor with instructions that, when executed, cause the AI processor to perform deep learning analysis on received input sensor data and determine machine learning results for autonomous driving. In some embodiments, the AI processor 107 is used to process sensor data to prepare the data for use as training data.

[0025] In some embodiments, the vehicle control module 109 is used to process the output of the artificial intelligence (AI) processor 107 and translate the output into vehicle control operations. In some embodiments, the vehicle control module 109 is used to control a vehicle for autonomous driving. In various embodiments, the vehicle control module 109 can adjust the speed, acceleration, steering, braking, etc. of the vehicle. For example, in some embodiments, the vehicle control module 109 is used to control a vehicle to maintain the position of the vehicle within a lane, merge the vehicle into another lane, adjust the speed and lane positioning of the vehicle to account for a merging vehicle, etc.

[0026] In some embodiments, the vehicle control module 109 is used to control vehicle lighting, such as brake lights, turn signals, headlights, etc. In some embodiments, the vehicle control module 109 is used to control vehicle audio conditions, such as the vehicle’s sound system, playing audio alerts, enabling microphones, enabling speakers, etc. In some embodiments, the vehicle control module 109 is used to control a notification system including a warning system to notify the driver and / or passengers of driving events, such as a potential collision or approaching a predetermined destination. In some embodiments, the vehicle control module 109 is used to adjust sensors, such as the vehicle’s sensors 101. For example, the vehicle control module 109 can be used to change parameters of one or more sensors, such as modifying orientation, changing output resolution and / or format type, increasing or decreasing capture rate, adjusting captured dynamic range, adjusting camera focus, enabling and / or disabling sensors, etc. In some embodiments, the vehicle control module 109 can be used to change parameters of the image preprocessor 103, such as modifying frequency range of filters, adjusting feature and / or edge detection parameters, adjusting channels and bit depth, etc. In various embodiments, the vehicle control module 109 is used to implement self-driving and / or driver-assisted control of the vehicle. In some embodiments, the vehicle control module 109 is implemented using a processor coupled with memory. In some embodiments, the vehicle control module 109 is implemented using an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or other suitable processing hardware.

[0027] In some embodiments, the network interface 111 is a communication interface used to send and / or receive data, including voice data. In various embodiments, the network interface 111 includes a cellular or wireless interface used to interface with a remote server to connect and make voice calls, send and / or receive text messages, transmit sensor data, receive updates to the deep learning network including updated machine learning models, obtain environmental conditions including weather conditions and forecasts, traffic conditions, etc. For example, the network interface 111 can be used to receive updates to instructions and / or operating parameters of the sensors 101, the image preprocessor 103, the deep learning network 105, the AI processor 107, and / or the vehicle control module 109. The machine learning models of the deep learning network 105 can be updated using the network interface 111. As another example, the network interface 111 can be used to update firmware of the sensors 101 and / or operating parameters of the image preprocessor 103, such as image processing parameters. As yet another example, the network interface 111 can be used to transmit potential training data to a remote server for training machine learning models.

[0028] Figure 2is a flowchart illustrating an embodiment of a process for training and applying machine learning models for autonomous driving. For example, input data including sensor and odometry data is received and processed to create training data for training machine learning models. In some embodiments, the sensor data corresponds to image data captured via an autonomous driving system. In some embodiments, the sensor data corresponds to sensor data captured based on a particular use case, such as a user manually exiting autonomous driving. In some embodiments, the process is used to create and deploy machine learning models for a deep learning system 100 of Figure 1

[0029] In 201, training data is prepared. In some embodiments, sensor data including image data and odometry data is received to create a training data set. The sensor data can include still images and / or video from one or more cameras. Additional sensors, such as radar, lidar, ultrasound, etc., can be used to provide relevant sensor data. In various embodiments, the sensor data is paired with corresponding odometry data to help identify features of the sensor data. For example, a location and changes in location data can be used to identify a location of relevant features in the sensor data, such as lane lines, traffic control signals, objects, etc. In some embodiments, the sensor data is a time series of elements and is used to determine ground truth data. The ground truth data is then associated with a subset of the time series, such as a first frame of image data. The selected elements of the time series and the ground truth data are used to prepare the training data. In some embodiments, the training data is prepared to train a machine learning model to identify only features from the sensor data, such as lane lines, vehicle paths, traffic patterns, etc. The prepared training data can include data for training, validation, and testing. In various embodiments, the sensor data can be in different formats. For example, the sensor data can be still images, video, audio, etc. The odometry data can include vehicle operational parameters, such as applied acceleration, applied braking, applied steering, vehicle location, vehicle orientation, changes in vehicle location, changes in vehicle orientation, etc. In various embodiments, the training data is organized and annotated for creating a training data set. In some embodiments, a portion of the training data preparation can be performed by a human curator. In various embodiments, a portion of the training data is automatically generated from data captured from a vehicle, significantly reducing the effort and time required to build a robust training data set. In some embodiments, the format of the data is compatible with the machine learning models used on the deployed deep learning application. In various embodiments, the training data includes validation data for testing the accuracy of the trained models.

[0030] ​In 203, a machine learning model is trained. For example, the machine learning model is trained using the data prepared in 201. In some embodiments, the model is a neural network, such as a convolutional neural network (CNN). In various embodiments, the model includes multiple intermediate layers. In some embodiments, the neural network can include multiple layers, including multiple convolution and pooling layers. In some embodiments, the trained model is validated using a validation data set created from received sensor data. In some embodiments, the machine learning model is trained to predict a three-dimensional representation of features from a single input image. For example, a three-dimensional representation of a lane line can be inferred from an image captured from a camera. As another example, a predicted path of a neighboring vehicle including whether the neighboring vehicle will attempt to merge is predicted from an image captured from a camera.

[0031] In 205, the trained machine learning model is deployed. For example, the trained machine learning model is installed on a vehicle as an update to a deep learning network, such as Figure 1 the deep learning network 105 of FIG. 1. In some embodiments, over-the-air updates are used to install newly trained machine learning models. In some embodiments, the update is a firmware update transmitted using a wireless network, such as WiFi or a cellular network. In some embodiments, the new machine learning model can be installed when the vehicle is serviced.

[0032] In 207, sensor data is received. For example, the sensor data is captured from one or more sensors of a vehicle. In some embodiments, the sensor is Figure 1 the sensor 101 of FIG. 1. The sensor can include an image sensor, such as a fisheye lens mounted behind a windshield, a front-facing camera or side-facing camera mounted in a door pillar, a rear-facing camera, etc. In various embodiments, the sensor data is in a format used as input by the machine learning model trained in 203 or converted to that format. For example, the sensor data can be raw or processed image data. In some embodiments, the data is captured from an ultrasonic sensor, radar, LiDAR sensor, microphone, or other suitable technology. In some embodiments, the sensor data is pre-processed during a pre-processing step using an image pre-processor, such as the image pre-processor 103 of FIG. 1. For example, the image can be normalized to remove distortion, noise, etc. Figure 1

[0033] In 209, the trained machine learning model is applied. For example, the machine learning model trained in 203 is applied to the sensor data received in 207. In some embodiments, the application of the model is performed by an AI processor, such as the AI processor 107 of FIG. 1, using a deep learning network, such as the deep learning network 105 of FIG. 1. Figure 1 Figure 1 ​​by the deep learning network 105) is performed. In various embodiments, a three- dimensional representation of a feature, such as a lane line, is identified and / or predicted by applying the trained machine learning model. For example, two splines of a lane line representing a lane in which the vehicle is traveling are inferred. As another example, a predicted path of a neighboring vehicle is inferred, including whether the neighboring vehicle is likely to cut into the current lane. In various embodiments, a vehicle, obstacle, lane, traffic control signal, map feature, object distance, speed limit, drivable space, etc. is identified by applying the machine learning model. In some embodiments, the feature is identified in three dimensions.

[0034] In 211, the autonomous vehicle is controlled. For example, one or more autonomous driving features are implemented by controlling various aspects of the vehicle. Examples can include controlling the steering, speed, acceleration, and / or braking of the vehicle, maintaining the position of the vehicle in a lane, maintaining the position of the vehicle relative to other vehicles and / or obstacles, providing notifications or warnings to occupants, etc. Based on the analysis performed in 209, the steering and speed of the vehicle are controlled to maintain the vehicle between two lane lines. For example, the left and right lane lines are predicted, and the corresponding vehicle lane and drivable space are identified. In various embodiments, a vehicle control module, such as the vehicle control module 109 of FIG. 1, controls the vehicle. Figure 1

[0035] Figure 3 is a flowchart illustrating an embodiment of a process for creating training data using a time series of elements. For example, a time series of elements consisting of sensor and odometry data is collected from a vehicle and used to automatically create training data. In various embodiments, the process of FIG. 2 is used to automatically label training data with corresponding ground truth data. The results corresponding to the time series are associated with the elements of the time series. The results and selected elements are packaged as training data to predict future results. In various embodiments, the process of FIG. 2 is used to capture sensor data and related data. For example, in various embodiments, sensor data is captured from the sensor(s) 101 of FIG. 1. Figure 3 Figure 1 Figure 1 Figure 2 Figure 3 Figure 3 Figure 3 ​​​​​​​The process of applying prediction-related data to create an organized collection of examples to improve machine learning models.

[0036] At 301, a time series of elements is received. In various embodiments, the elements are sensor data, such as image data captured at a vehicle and transmitted to a training server. Sensor data is captured over a period of time to create the time series of elements. In various embodiments, the elements are timestamped to maintain ordering of the elements. As the elements progress through the time series, further events in the time series are used to help predict outcomes for earlier elements in the time series. For example, the time series may capture a vehicle in an adjacent lane that signals to merge, accelerates, and positions itself closer to a nearby lane line. Using the entire time series, the results can be used to determine if the vehicle is merging into a shared lane. This result can be used to predict that the vehicle will merge based on a selected element of the time series (such as an image from an earlier image in the time series). As another example, the time series captures a curve in a lane line. The time series captures various potholes, turns, crests, etc. in the lane that are not apparent based on a single element of the time series. In various embodiments, the elements are sensor data in a format that can be used as input by a machine learning model. For example, the sensor data can be raw or processed image data. In some embodiments, the data is captured from ultrasonic sensors, radar, LiDAR sensors, or other suitable technology.

[0037] In various embodiments, a time series is organized by associating a timestamp with each element of the time series. For example, a timestamp is associated with at least the first element in the time series. The timestamp can be used to calibrate the time series of elements using relevant data (such as odometer data). In various embodiments, the length of the time series can be a fixed time length, such as 10 seconds, 30 seconds, or another suitable length. The time length can be configurable. In various embodiments, the time series can be based on the speed of the vehicle, such as the vehicle's average speed. For example, at slower speeds, the time length of the time series can be increased to capture data for a longer distance traveled than would be possible using a shorter time length at the same speed. In some embodiments, the number of elements in the time series is configurable. For example, the number of elements can be based on the distance traveled. For example, for a fixed time period, a faster-moving vehicle will include more elements in the time series than a slower-moving vehicle. The additional elements increase the fidelity of the captured environment and can improve the accuracy of predicted machine learning results. In various embodiments, the number of elements is adjusted by adjusting the frames per second at which the sensor captures data and / or by discarding unnecessary intermediate frames.

[0038] In 303, data related to the elements of the time series is received. In various embodiments, the related data is received at the training server along with the elements received in 301. In some embodiments, the related data is odometry data of the vehicle. Using the location, orientation, location changes, orientation changes, and / or other related vehicle data, the location data of features identified in the elements of the time series can be labeled. For example, lane lines can be labeled with very accurate locations by examining the time series of elements of the lane lines. Generally, the lane lines closest to the vehicle camera are accurate and closely related to the location of the vehicle. In contrast, the XYZ location of the lines farthest from the vehicle are difficult to determine. The far sections of the lane lines can be occluded (e.g., behind a turn or a hill) and / or difficult to accurately capture (e.g., due to distance or lighting, etc.). The data related to the elements is used to label the portions of the features identified in the time series that are identified with high accuracy. In various embodiments, a threshold is used to determine whether to associate an identified portion of a feature, such as a portion of a lane line, with the related data. For example, portions of a lane line identified with a high degree of certainty, such as portions close to the vehicle, are associated with the related data, while portions of the lane line identified with a degree of certainty below the threshold, such as portions far from the vehicle, are not associated with the related data of the element. Instead, another element of the time series, such as a subsequent element with a higher degree of certainty and its related data, is used. In some embodiments, the related data is an output of a neural network, such as the output of the deep learning network 105 of Figure 1 In some embodiments, the related data is an output of a vehicle control module, such as the vehicle control module 109 of Figure 1 The related data can include vehicle operating parameters, such as speed, speed changes, acceleration, acceleration changes, steering, steering changes, braking, braking changes, etc. In some embodiments, the related data is radar data used to estimate the distance of objects, such as obstacles.

[0039] In some embodiments, the data related to the elements of the time series includes map data. For example, offline data, such as road and / or satellite level map data, is received in 303. The map data can be used to identify features, such as roads, vehicle lanes, intersections, speed limits, school zones. For example, the map data can describe the path of a vehicle lane. As another example, the map data can describe speed limits associated with different roads of the map.

[0040] In various embodiments, data related to elements of a time series is organized by associating timestamps with the related data. The time series from the elements and corresponding timestamps of the related data can be used to synchronize the two data sets. In some embodiments, the data is synchronized at the time of capture. For example, as each element of the time series is captured, a corresponding set of related data is captured and saved with the time series of elements. In various embodiments, the time period of the related data is configurable and / or matches the time period of the time series of elements. In some embodiments, the related data is sampled at the same rate as the time series of elements.

[0041] In 305, ground truth data is determined for the time series. In various embodiments, the time series is analyzed to determine ground truth data associated with machine learning features. For example, lane lines are identified from the time series of ground truth data corresponding to the lane lines. As another example, ground truth data for a path of a moving object, such as a vehicle, pedestrian, bicyclist, animal, etc., is identified from the time series for the detected moving object. In some embodiments, in the case of a moving vehicle entering the lane of the autonomous vehicle within the time series, the moving vehicle is annotated as a cut-in vehicle. In some embodiments, the ground truth data is represented as a three-dimensional representation, such as a three-dimensional trajectory. For example, ground truth data associated with a lane line can be represented as a three-dimensional parametric spline or curve. As another example, a predicted path of a detected vehicle is determined and represented as a three-dimensional trajectory. The predicted path can be used to determine whether the vehicle merges into the space occupied. In various embodiments, ground truth data can only be determined by examining the time series of elements. For example, analysis of only a subset of the time series can cause portions of a lane line to be occluded. By extending the analysis over the time series of elements, the occluded portions of the lane line are revealed. Also, captured data at the end of the time series (e.g., with higher fidelity) more accurately captures details of portions of the lane line that are further away in distance. Additionally, the related data is also more accurate as the related data is based on data captured closer in proximity (both distance and time). In various embodiments, simultaneous localization and mapping techniques are applied to different portions of a detected object, such as a lane line, identified in different elements of the time series of elements to map the different portions of the object to precise three-dimensional locations including elevation. The set of mapped three-dimensional locations represent the ground truth data for the object, such as a segment of a lane line captured over the time series. In some embodiments, the localization and mapping techniques produce a set of precise points, e.g., a set of points corresponding to different points along a vehicle lane line. The set of points can be converted to a more efficient format, such as a spline or parametric curve. In some embodiments, ground truth data is determined to detect objects in three dimensions, such as lane lines, drivable space, traffic controls, vehicles, etc.

[0042] In some embodiments, real data is determined to predict semantic labels. For example, a detected vehicle can be labeled as being in the left lane or the right lane. In some embodiments, a detected vehicle can be labeled as being in a blind spot, a vehicle that should yield, or having another suitable semantic label. In some embodiments, the vehicle is assigned to a road or lane in a map based on the determined real data. As an additional example, the determined real data can be used to label traffic lights, lanes, drivable spaces, or other features that assist in autonomous driving.

[0043] In some embodiments, the relevant data is depth (or distance) data of detected objects. By associating distance data with objects identified in the time series of elements, a machine learning model can be trained to estimate object distances by using the relevant distance data as ground truth data for detected objects. In some embodiments, the distances are for detected objects such as obstacles, barriers, moving vehicles, stationary vehicles, traffic control signals, pedestrians, etc.

[0044] In 307, the training data is packaged. For example, elements of a time series are selected and associated with the real data determined in 305. In various embodiments, the selected elements are early elements in the time series. The selected elements represent sensor data input to the machine learning model, and the real data represents the predicted results. In various embodiments, the training data is packaged and prepared as training data. In some embodiments, the training data is packaged into training, validation, and test data. Based on the determined real data and the selected elements of the time series, the training data can be packaged to train the machine learning model to identify lane lines, predicted paths of vehicles, speed limits, vehicle cut-ins, object distances and / or drivable spaces, and other useful features for autonomous driving. The packaged training data can now be used to train the machine learning model.

[0045] Figure 4 is a flow chart illustrating an embodiment of a process for training and applying a machine learning model for autonomous driving. In some embodiments, Figure 4 The process is used to collect and retain sensor and odometry data to train machine learning models for autonomous driving. In some embodiments, whether autonomous driving control is enabled or not, Figure 4 The processes are all implemented on a vehicle that is enabled for autonomous driving. For example, sensor and odometer data may be collected immediately after exiting autonomous driving, while the vehicle is being driven by a human driver, and / or while the vehicle is being driven autonomously. In some embodiments, using Figure 1 Deep learning system to implement Figure 4 In some embodiments, Figure 4 Part of the processFigure 2 of 207, 209, and / or 211 as part of a process of applying a machine learning model to autonomous driving.

[0046] In 401, sensor data is received. For example, a vehicle equipped with sensors captures sensor data and provides the sensor data to a neural network running on the vehicle. In some embodiments, the sensor data can be visual data, ultrasonic data, LiDAR data, or other suitable sensor data. For example, images are captured from a high dynamic range front-facing camera. As another example, ultrasonic data is captured from a side-facing ultrasonic sensor. In some embodiments, a vehicle is fixed with multiple sensors for capturing data. For example, in some embodiments, eight surround cameras are fixed to a vehicle and provide 360 degrees of visibility around the vehicle, with a range of up to 250 meters. In some embodiments, the camera sensors include a wide forward-facing camera, a narrow forward-facing camera, a rearview camera, a forward-facing side camera, and / or a rear-facing side camera. In some embodiments, ultrasonic and / or radar sensors are used to capture surrounding details. For example, twelve ultrasonic sensors can be fixed to a vehicle to detect hard and soft objects. In some embodiments, a front-facing radar is used to capture data of the surrounding environment. In various embodiments, the radar sensors are able to capture surrounding details despite heavy rain, heavy fog, dust, and other vehicles. The various sensors are used to capture the environment around the vehicle and provide the captured data for deep learning analysis.

[0047] In some embodiments, the sensor data includes odometry data, including a location, an orientation, a change in location, and / or a change in orientation of the vehicle, among others. For example, location data is captured and associated with other sensor data captured during the same time frame. As one example, location data captured at the same time as image data is used to associate location information with the image data.

[0048] In 403, the sensor data is pre-processed. In some embodiments, one or more pre-processing passes can be performed on the sensor data. For example, the data can be pre-processed to remove noise, correct for alignment issues, and / or blur, among others. In some embodiments, one or more different filtering passes are performed on the data. For example, a high-pass filter can be performed on the data and a low-pass filter can be performed on the data to isolate different components of the sensor data. In various embodiments, the pre-processing steps performed in 403 are optional and / or can be incorporated into the neural network.

[0049] In 405, a deep learning analysis of the sensor data is initiated. In some embodiments, the deep learning analysis is performed on the sensor data optionally pre-processed in 403. In various embodiments, the deep learning analysis is performed using a neural network, such as a convolutional neural network (CNN). In various embodiments, a machine learning model is trained offline using the process of Figure 2 and deployed onto a vehicle to perform inference on sensor data. For example, the model can be trained to identify, as needed, road lane lines, obstacles, pedestrians, moving vehicles, parked vehicles, drivable space, etc. In some embodiments, multiple trajectories of lane lines are identified. For example, several potential trajectories of lane lines are detected, and each trajectory has a corresponding probability of occurrence. In some embodiments, a predicted lane line is the lane line with the highest probability of occurrence and / or the highest associated confidence value. In some embodiments, a predicted lane line from the deep learning analysis needs to exceed a minimum confidence threshold. In various embodiments, the neural network includes multiple layers, including one or more intermediate layers. In various embodiments, the sensor data and / or results of the deep learning analysis are retained and transmitted in 411 for use in the automatic generation of training data.

[0050] In various embodiments, the deep learning analysis is used to predict additional features. The predicted features can be used to assist in autonomous driving. For example, a detected vehicle can be assigned to a lane or road. As another example, a detected vehicle can be determined to be in a blind spot, a vehicle that should yield, a vehicle in a left adjacent lane, a vehicle in a right adjacent lane, or have another suitable attribute. Similarly, the deep learning analysis can identify traffic lights, drivable space, pedestrians, obstacles, or other suitable driving features.

[0051] In 407, the results of the deep learning analysis are provided to vehicle control. For example, the results are provided to a vehicle control module to control the vehicle for autonomous driving and / or to implement autonomous driving functionality. In some embodiments, the results of the deep learning analysis in 405 are passed through one or more additional deep learning pass using one or more different machine learning models. For example, a predicted path of a lane line can be used to determine a vehicle lane, and the determined vehicle lane is used to determine drivable space. The drivable space is then used to determine a path for the vehicle. Similarly, in some embodiments, a predicted vehicle cut-in is detected. The determined path for the vehicle takes into account the predicted cut-in to avoid a potential collision. In some embodiments, various outputs of the deep learning are used to construct a three-dimensional representation of the environment of the vehicle for autonomous driving, including a predicted path of the vehicle, identified obstacles, identified traffic control signals including speed limits, etc. In some embodiments, the vehicle control module utilizes the determined results to control the vehicle along the determined path. In some embodiments, the vehicle control module isFigure 1 The vehicle control module 109.

[0052] In 409, the vehicle is controlled. In some embodiments, a vehicle control module (such as Figure 1 The vehicle control module 109 of the present invention is used to control the vehicle in which autonomous driving is activated. The vehicle control can adjust the speed and / or steering of the vehicle taking into account the environment surrounding the vehicle, for example, to maintain the vehicle in the lane at an appropriate speed. In some embodiments, the results are used to adjust the vehicle when it is expected that adjacent vehicles will merge into the same lane. In various embodiments, using the results of the deep learning analysis, the vehicle control module determines an appropriate way to operate the vehicle, for example, along the determined path at an appropriate speed. In various embodiments, the results of the vehicle control (such as speed changes, brake applications, steering adjustments, etc.) are retained and used for automatic generation of training data. In various embodiments, the vehicle control parameters are retained and transmitted in 411 for automatic generation of training data.

[0053] At 411, sensor and related data are transmitted. For example, the sensor data received at 401, along with the results of the deep learning analysis at 405 and / or the vehicle control parameters used at 409, are transmitted to a computer server for automatic generation of training data. In some embodiments, the data is a time series of data, and the various collected data are correlated by the computer server. For example, odometer data is correlated with captured image data to generate real-world data. In various embodiments, the collected data is wirelessly transmitted from the vehicle to a training data center, for example, via WiFi or a cellular connection. In some embodiments, metadata is transmitted along with the sensor data. For example, metadata may include time of day, timestamp, location, vehicle type, vehicle control and / or operating parameters such as speed, acceleration, braking, whether autonomous driving is enabled, steering angle, odometer data, etc. Additional metadata may include the time since the last sensor data was transmitted, vehicle type, weather conditions, road conditions, etc. In some embodiments, the transmitted data is anonymized, for example, by removing the vehicle's unique identifier. As another example, data from similar vehicle models is merged to prevent individual users and their vehicle usage from being identified.

[0054] In some embodiments, data is transmitted only in response to a trigger. For example, in some embodiments, an erroneous prediction triggers transmission of the sensor and related data to automatically collect data in order to create an example set of organizations for improving predictions of a deep learning network. For example, a prediction made in 405 related to whether a vehicle is attempting to merge is determined to be erroneous by comparing the prediction to the observed actual outcome. The data associated with the erroneous prediction (including the sensor and related data) is then transmitted and used to automatically generate training data. In some embodiments, a trigger can be used to identify particular scenarios, such as a sharp turn, a road fork, a lane merge, a sudden stop, or other suitable scenario where additional training data is helpful and can be difficult to collect. For example, a trigger can be based on a sudden deactivation or exit of an autonomous driving feature. As another example, vehicle operating characteristics, such as speed changes and acceleration changes, can form the basis of a trigger. In some embodiments, a prediction having less than a particular threshold of accuracy triggers transmission of the sensor and related data. For example, in certain scenarios, a prediction can not have a Boolean type correct or erroneous outcome, but rather is evaluated by determining an accuracy value of the prediction.

[0055] In various embodiments, sensor and related data is captured over a period of time and the entire time series of data is transmitted together. The time period can be configured and / or can be based on one or more factors, such as the speed of the vehicle, the distance traveled, speed changes, etc. In some embodiments, the sampling rate of the captured sensor and / or related data is configurable. For example, the sampling rate can increase at higher speeds, during sudden braking, during sudden acceleration, during sharp turns, or another suitable scenario where additional fidelity is needed.

[0056] Figure 5 is a diagram illustrating an example of an image captured from a vehicle sensor. In the illustrated example, Figure 5The image includes image data 500 captured from a vehicle driving in a lane between two lane lines. The vehicle and the location of the sensor used to capture the image data 500 are represented by label A. The image data 500 is sensor data and can be captured from a camera sensor, such as a front-facing camera of the vehicle, while driving. The image data 500 captures portions of lane lines 501 and 511. The lane lines 501 and 511 turn to the right as the lane lines 501 and 511 approach the horizon. In the example shown, the lane lines 501 and 511 are visible, but become increasingly difficult to detect as the lane lines straighten out in the distance as they turn away from the location of the camera sensor. Without any additional input, a white line drawn at the top of the lane lines 501 and 511 approximates the detectable portions of the lane lines 501 and 511 from the image data 500. In some embodiments, the detected portions of the lane lines 501 and 511 can be detected by segmenting the image data 500.

[0057] In some embodiments, the labels A, B, and C correspond to different locations on the road and different times in the time series. Label A corresponds to the time and location of the vehicle when the image data 500 is captured. Label B corresponds to a location on the road ahead of the location of label A and at a time after the time of label A. Similarly, label C corresponds to a location on the road ahead of the location of label B and at a time after the time of label B. As the vehicle drives, the vehicle passes through the locations of labels A, B, and C (from label A to label C), and a time series of sensors and related data is captured while driving. The time series includes elements captured at the locations (and times) of labels A, B, and C. Label A corresponds to the first element of the time series, label B corresponds to a middle element of the time series, and label C corresponds to a middle (or potentially last) element of the time series. At each label, additional data is captured, such as odometer data of the vehicle in the label location. Depending on the length of the time series, additional or less data is captured. In some embodiments, a timestamp is associated with each element of the time series.

[0058] In some embodiments, ground truth data (not shown) is determined for lane lines 501 and 511. For example, using the processes disclosed herein, the locations of lane lines 501 and 511 are identified by identifying different portions of lane lines 501 and 511 from different elements of the time series of elements. In the example shown, portions 503 and 513 are identified using image data 500 and related data (such as odometry data) obtained at the location and time of label A. Portions 505 and 515 are identified using image data (not shown) and related data (such as odometry data) obtained at the location and time of label B. Portions 507 and 517 are identified using image data (not shown) and related data (such as odometry data) obtained at the location and time of label C. By analyzing the time series of elements, the locations of the different portions of lane lines 501 and 511 are identified, and the ground truth data can be determined by combining the different identified portions. In some embodiments, the portions are identified as points along each portion of the lane line. In the example shown, only three portions of each lane line (portions 503, 505, and 507 of lane line 501 and portions 513, 515, and 517 of lane line 511) are highlighted to explain the process, but additional portions can be captured within the time series to determine the location of the lane lines with higher resolution and / or higher accuracy.

[0059] In various embodiments, the locations of the portions of the image data capturing lane lines 501 and 511 closest to the sensor are determined with high accuracy. For example, the locations of portions 503 and 513 are identified with high accuracy using image data 500 and related data (such as odometry data) of label A. The locations of portions 505 and 515 are identified with high accuracy using image and related data of label B. The locations of portions 507 and 517 are identified with high accuracy using image and related data of label C. By utilizing the time series of elements, the locations of the various portions of lane lines 501 and 511 captured by the time series can be identified with high accuracy in three dimensions, and used as a basis for ground truth data for lane lines 501 and 511. In various embodiments, the determined ground truth data is associated with selected elements of the time series, such as image data 500. The ground truth data and selected elements can be used to create training data for predicting lane lines. In some embodiments, the training data is created automatically and without human labeling. The training data can be used to train a machine learning model to predict a three-dimensional trajectory of a lane line from captured image data, such as image data 500.

[0060] Figure 6 is a diagram illustrating an example of utilizing a predicted three-dimensional trajectory of a lane line from image captured by a vehicle sensor. In the example shown, Figure 6the image data 600 includes image data 600 captured from a vehicle driving in a lane between two lane lines. The vehicle and the location of the sensor used to capture the image data 600 are represented by label A. In some embodiments, label A corresponds to the same location as label A of Figure 5 the image data 600 is sensor data and can be captured from a camera sensor, such as a front-facing camera of a vehicle, while driving. The image data 600 captures portions of lane lines 601 and 611. The lane lines 601 and 611 turn to the right as the lane lines 601 and 611 approach the horizon. In the illustrated example, the lane lines 601 and 611 are visible, but become increasingly difficult to detect as the lane lines turn away from the location of the camera sensor and straighten out in the distance. The red lines drawn at the top of the lane lines 601 and 611 are the predicted three-dimensional trajectories of the lane lines 601 and 611. Using the processes disclosed herein, the image data 600 is used as input to a trained machine learning model to predict the three-dimensional trajectories. In some embodiments, the predicted three-dimensional trajectories are represented as a three-dimensional parametric spline or another parametric form of representation.

[0061] In the illustrated example, the portions 621 of the lane lines 601 and 611 are portions of the lane lines 601 and 611 that diverge in the distance. The three-dimensional locations (i.e., longitude, latitude, and altitude) of the portions 621 of the lane lines 601 and 611 are determined with high accuracy using the processes disclosed herein and are included in the predicted three-dimensional trajectories of the lane lines 601 and 611. Using the trained machine learning model, the three-dimensional trajectories of the lane lines 601 and 611 can be predicted using the image data 600 and without needing location data at the locations of the portions 621 of the lane lines 601 and 611. In the illustrated example, the image data 600 is captured at the location and time of label A.

[0062] In some embodiments, Figure 6 label A corresponds to Figure 5 label A, and only the image data 600 is used as input to the trained machine learning model to determine the predicted three-dimensional trajectories of the lane lines 601 and 611. By training the machine learning model using real data (determined using images and related data of elements obtained at the locations of labels A, B, and C of Figure 5 the lane lines 601 and 611 with high accuracy, even at portions of the lane lines in the distance, such as the portions 621. Although the image data 600 and Figure 5The image data 500 is relevant, but the prediction of the trajectory does not require the image data 600 to be included in the training data. By training on sufficient training data, lane lines can be predicted even for newly encountered scenarios. In various embodiments, the predicted three-dimensional trajectories of the lane lines 601 and 611 are used to maintain the position of the vehicle within the detected lane lines and / or to autonomously navigate the vehicle along the detected lane of the predicted lane lines. By predicting the lane lines in three dimensions, the performance, safety, and accuracy of the navigation is significantly improved.

[0063] While the forgoing embodiments have been described in some detail for purposes of clarity of understanding, the application is not limited to the details provided. There are many alternative ways of implementing the application. The disclosed embodiments are illustrative in nature and not restrictive.

Claims

1. A system for autonomous driving, comprising: The processor is configured to: receiving image data based on an image captured by a camera of the vehicle; Determining a three-dimensional trajectory of machine-learned features for autonomous driving based on inputting the image data into a machine-learning model, The machine learning model is trained based on training data, and the training data includes: real data associated with a plurality of time series elements, the real data comprising three-dimensional trajectories associated with machine-learned features, the machine-learned features generated based on the time series elements, the time series elements comprising respective images captured at respective times within a time period, wherein the real data is generated based on portions of a single time series element, and a selected time series element of the plurality of time series elements, wherein the machine learning model is trained to output the real data associated with the plurality of time series elements based on an input of the selected time series elements; and providing the three-dimensional trajectory of the machine-learned features while automatically controlling the vehicle; and A memory is coupled to the processor and configured to provide instructions to the processor.

2. The system of claim 1, wherein the machine learning features are associated with vehicle lane markings. 3 . The system of claim 2 , wherein portions of the three-dimensional trajectory of the vehicle lane markings are occluded in the image data. 4 . The system of claim 2 , wherein the processor is further configured to use the vehicle lane marking to identify a vehicle lane occupied by the vehicle.

5. The system of claim 2 or 3, wherein the vehicle lane is used to identify a drivable space. The system of claim 5 , wherein the identified drivable space is used to maintain the vehicle in the vehicle lane. 7 . The system according to claim 1 , wherein the camera of the vehicle comprises at least one of the following: a front camera, a side camera, or a rear camera.

8. The system of any one of claims 1 to 3, wherein the predicted three-dimensional trajectory is one identified potential trajectory among a plurality of identified potential trajectories. 9 . The system of claim 8 , wherein the predicted three-dimensional trajectory has an occurrence probability value that is a highest occurrence probability value of the plurality of identified potential trajectories.

10. The system of claim 1, wherein the training data further comprises a correlated set of odometry data associated with the time series elements. The system of claim 10 , wherein at least a portion of the training data is automatically labeled as the real data.

12. The system according to any one of claims 1 to 3, wherein controlling the vehicle comprises: Adjust the speed and steering of the vehicle.

13. The system of any one of claims 1 to 3, wherein the three-dimensional trajectory is represented by a spline.

14. The system of any one of claims 1 to 3, wherein the three-dimensional trajectory is represented by one or more piecewise polynomials.

15. The system of any one of claims 1 to 3, wherein the machine learning feature is associated with a predicted path of a second vehicle.

16. The system of claim 15, wherein the predicted path of the second vehicle is used to determine whether the second vehicle is likely to enter a lane occupied by the automatically controlled vehicle.

17. The system of claim 16, wherein controlling the vehicle comprises: Adjust speed or steering to avoid collision with the second vehicle.

18. A computer program product embodied in a non-transitory computer-readable storage medium and comprising computer instructions for: receiving image data based on an image captured by a camera of the vehicle; Determine the three-dimensional trajectory of the lane line based on inputting the image data into a machine learning model, The machine learning model is trained based on training data, and the training data includes: Real data associated with a plurality of time series elements, the plurality of time series elements comprising a respective image, the real data comprising three-dimensional trajectories associated with lane lines represented in the image, and a selected time series element of the plurality of time series elements, wherein the machine learning model is trained to output the real data associated with the plurality of time series elements based on an input of the selected time series elements; as well as The three-dimensional trajectory of the vehicle lane is provided while automatically controlling the vehicle.

19. A method for autonomous driving, comprising: receiving image data based on an image captured by a camera of the vehicle; Determine the three-dimensional trajectory of the lane line based on inputting the image data into a machine learning model, The machine learning model is trained based on training data, and the training data includes: Real data associated with a plurality of time series elements, the plurality of time series elements comprising a respective image, the real data comprising three-dimensional trajectories associated with lane lines represented in the image, and a selected time series element of the plurality of time series elements, wherein the machine learning model is trained to output the real data associated with the plurality of time series elements based on an input of the selected time series elements; and The three-dimensional trajectory of the vehicle lane is provided while automatically controlling the vehicle.

Images