Generating ground truth for machine learning from time series elements

Abstract

Sensor data including a group of time series elements is received. A training data set is determined by determining ground truths corresponding to at least one selected time series element within the group of time series elements. The corresponding ground truths are based on multiple time series elements within the group of time series elements. A processor is used to train a machine learning model using the training data.

Description

Technology Field

[0001] This application is a continuation of U.S. Patent Application No. 16 / 265,729, filed on February 1, 2019, with the title of the invention “Generating Ground Truth for Machine Learning from Time Series Elements,” and the disclosure of said U.S. Patent Application is incorporated by reference in its entirety into this application. Background Technology

[0002] Deep learning systems used in applications such as autonomous driving are developed by training machine learning models. Generally, the performance of a deep learning system is limited, at least in part, by the quality of the training set used to train the model. In many cases, significant resources are invested in collecting, curating, and annotating training data. Traditionally, much effort is performed manually to curate training data sets by reviewing potential training data and appropriately labeling features related to the data. The effort required to create a training set with accurate labels can be substantial and often tedious. Furthermore, collecting and accurately labeling the data necessary to improve machine learning models can often be difficult. Therefore, there is a need to improve the process for generating training data with accurately labeled features.

[0003] A machine learning training technique for generating highly accurate machine learning results is disclosed. A training data set is created using data captured by vehicle sensors to capture the vehicle's environment and vehicle operating parameters. For example, sensors attached to the vehicle capture data such as image data of the road and surrounding environment on which the vehicle is traveling. Sensor data can capture vehicle lane lines, vehicle lanes, other vehicle traffic, obstacles, traffic control signs, etc. Odometry and other similar sensors capture vehicle operating parameters such as vehicle speed, steering, orientation, changes in direction, changes in location, changes in elevation, and changes in speed. The captured dataset is transmitted to a training server to generate a training dataset. The training dataset is used to train a machine learning model for generating highly accurate machine learning results. In some embodiments, the time series of the captured data is used to generate training data. For example, ground truth is determined based on a group of time series elements and associated with a single element of the group.For example, a series of images over a specific time period, such as 30 seconds, is used to determine the actual path of the vehicle lane line over the period during which the vehicle travels. The vehicle lane line is determined using the most accurate vehicle lane image from that time period. Different portions (or locations) of the lane line can be identified from different image data in the time series. As the vehicle moves along the lane line, more accurate data regarding different portions of the lane line is captured. In some instances, occluded portions of the lane line are revealed, for example, when the vehicle moves along a hidden curve or the ridge of a hill. The most accurate portion of the lane line from each image in the time series can be used to identify the lane line for the entire group of image data. Image data of the lane line at a distance is generally less detailed than image data of the lane line near the vehicle. By capturing the time series of image data as the vehicle moves along the lane, accurate image data for all parts of the lane and corresponding odometry data are collected. Brief explanation of the drawing

[0004] Various embodiments of the present invention are disclosed in the following detailed description and accompanying drawings. FIG. 1 is a block diagram illustrating an example of a deep learning system for autonomous driving. FIG. 2 is a flowchart illustrating an example of a process for training and applying a machine learning model for autonomous driving. FIG. 3 is a flowchart illustrating an example of a process for generating training data using time series of elements. Figure 4 is a flowchart illustrating an example of a process for training and applying a machine learning model for autonomous driving. Figure 5 is a diagram showing an example of an image captured from a vehicle sensor. Figure 6 is a diagram showing an example of an image captured from a vehicle sensor by predicting the three-dimensional trajectory of a lane. Specific details for implementing the invention

[0005] The present invention may be implemented in various ways, including a processor, such as a processor configured to execute instructions stored and / or provided by memory connected to the process, a process, an apparatus, a system, a composition of matter, and / or a computer program product embodied in a computer-readable storage medium. In this specification, such implementations, or any other forms that the present invention may take, may be referred to as the technology. Generally, the order of steps of the disclosed process may be altered within the scope of the present invention. Unless otherwise noted, components such as a processor or memory described as configured to perform a task may be implemented as general components temporarily configured to perform a task at a given time or as specific components manufactured to perform a task. As used herein. The term 'processor' refers to a processing core configured to process data such as one or more devices, circuits, and / or computer program instructions.

[0006] A detailed description of one or more embodiments of the present invention is provided below, together with the accompanying drawings illustrating the principles of the present invention. While the present invention is described in relation to these embodiments, the present invention is not limited to any of the embodiments. The scope of the present invention is limited only by the claims and includes numerous alternatives, modifications, and equivalents. To provide a complete understanding of the present invention, many specific details are described in the description. These details are provided for illustrative purposes only, and the present invention may be practiced according to the claims without some or all of these specific details. For clarity, technical information known in the art related to the present invention has not been described in detail so as not to unnecessarily obscure the present invention.

[0007] A machine learning training technique for generating highly accurate machine learning results is disclosed. A training data set is created using data captured by vehicle sensors to capture the vehicle's environment and vehicle operating parameters. For example, sensors attached to the vehicle capture data such as image data of the road and surrounding environment on which the vehicle is traveling. Sensor data can capture vehicle lane lines, vehicle lanes, other vehicle traffic, obstacles, traffic control signs, etc. Odometry and other similar sensors capture vehicle operating parameters such as vehicle speed, steering, orientation, changes in direction, changes in location, changes in elevation, and changes in speed. The captured dataset is transmitted to a training server to generate a training dataset. The training dataset is used to train a machine learning model for generating highly accurate machine learning results. In some embodiments, the time series of the captured data is used to generate training data. For example, ground truth is determined based on a group of time series elements and associated with a single element of the group.For example, a series of images over a specific period, such as 30 seconds, is used to determine the actual path of the vehicle lane line over the period during which the vehicle travels. The vehicle lane line is determined using the most accurate vehicle lane image from that period. Different portions (or locations) of the lane line can be identified from different image data in the time series. As the vehicle moves along the lane line, more accurate data regarding different portions of the lane line is captured. In some instances, occluded portions of the lane line are revealed, for example, when the vehicle moves along a hidden curve or the ridge of a hill. The most accurate portion of the lane line from each image in the time series can be used to identify the lane line for the entire group of image data. Image data of the lane line at a distance is generally less detailed than image data of the lane line near the vehicle. By capturing the time series of image data as the vehicle moves along the lane, accurate image data for all parts of the lane and corresponding odometry data are collected.

[0008] In some embodiments, a three-dimensional representation of a feature, such as a lane boundary, is generated from a group of time series elements corresponding to ground truth. This ground truth is then associated with a subset of time series elements, such as a single image frame of a group of captured image data. For example, the first image of the image group is associated with the ground truth for the lane boundary represented in three-dimensional space. Although the ground truth is determined based on the image group, the selected first frame and the ground truth are used to generate training data. For example, training data is generated to predict the three-dimensional representation of a vehicle lane using only a single image. In some embodiments, a group of elements or any element belonging to a group of time series elements is associated with the ground truth and used to generate training data. For example, the ground truth may be applied to the entire video sequence to generate training data. In another embodiment, the middle or last element of a group of time series elements is associated with ground truth and used to generate training data.

[0009] In various embodiments, selected images and ground truths may be applied to other features such as lane lines, path prediction for a vehicle including neighboring vehicles, depth distances of objects, traffic control signs, etc. For example, a series of images of a vehicle in an adjacent lane is used to predict the path of that vehicle. A single image from a group and the actual path taken by the adjacent vehicle can be used as training data to predict the vehicle's path using the time series of images and the actual path taken by the adjacent vehicle. This information can be used to predict whether the adjacent vehicle will cut into the path of the autonomous vehicle. For example, path prediction can predict whether the adjacent vehicle will merge in front of the autonomous vehicle. The autonomous vehicle may be controlled to minimize the possibility of a collision. For example, an autonomous vehicle may reduce speed to prevent a collision, adjust the vehicle's speed and / or steering to prevent a collision, initiate warnings to adjacent vehicles and / or occupants of the autonomous vehicle, and / or change lanes, etc. In various embodiments, the ability to accurately infer path prediction, including vehicle path prediction, significantly improves the safety of the autonomous vehicle.

[0010] In some embodiments, a trained machine learning model may be used to predict a three-dimensional representation of one or more features for autonomous driving, including lane boundaries. For example, instead of identifying two-dimensional lane boundaries from image data by image segmentation of lane boundaries, a three-dimensional representation is generated using time series of elements and mileage measurement data corresponding to the time series. The three-dimensional representation includes changes in elevation, which significantly improves the accuracy of lane boundary detection and the accuracy of the lane detection and identified drivable path. In some embodiments, lane boundaries are represented using one or more splines or other parameterized representations. The use of a piecewise polynomial to represent lane boundaries significantly reduces the computational resources required to evaluate the three-dimensional object. This reduction in computational resources corresponds to improvements in processing speed and efficiency without significantly sacrificing the accuracy of the representation. In various embodiments, lane boundaries, particularly including curves of lane boundaries, may be represented using discriminative polynomials, a set of three-dimensional points, or other suitable representations. For example, discriminative polynomials interpolate actual lane boundaries using highly accurate sections of lane boundaries identified from a group of elements captured over time using sensor data.

[0011] In some embodiments, sensor data is received. The sensor data may include images (such as video and / or still images), radar, audio, lidar, inertia, odometry, location, and / or other forms of sensor data. The sensor data includes a group of time series elements. For example, a group of time series elements may include a group of images captured from a sensor camera of a vehicle over a certain period. In some embodiments, a training dataset is determined by determining a ground truth corresponding to at least one selected time series element within a group of time series elements based on a plurality of time series elements within the group of time series elements. For example, the ground truth is determined by examining the most relevant part of each element of the group of time series elements, including the previous and / or subsequent time series elements of the group. In some scenarios, only earlier and / or subsequent time series elements contain data not present in earlier time series elements, such as vehicle lane boundaries that initially disappear around the curve and appear only in later elements of the time series. The determined ground truth may be a 3D representation of vehicle lane boundaries, a predicted path for a vehicle, or other similar prediction. Elements of a group of time series elements are selected and associated with the ground truth. The selected elements and the ground truth are part of the training dataset. In some embodiments, a processor is used to train a machine learning model using the training dataset.For example, training datasets are used to train machine learning models for inferring features used in self-driving or driver-assisted operations of vehicles. Using the trained machine learning models, neural networks can infer features related to autonomous driving, such as vehicle lanes, drivable space, objects (e.g., pedestrians, stationary vehicles, moving vehicles, etc.), weather (e.g., rain, hail, fog, etc.), traffic control objects (e.g., traffic lights, traffic signs, street signs, etc.), and traffic patterns.

[0012] In some embodiments, the system includes a processor and memory coupled to the processor. The processor is configured to receive image data based on images captured by the vehicle's camera. For example, a camera sensor attached to the vehicle captures an image of the vehicle's environment. The camera may be a forward-facing camera, a pillar camera, or other appropriately positioned camera. The image data captured from the camera is processed using a processor, such as the vehicle's GPU or AI processor. In some embodiments, the image data is used as the basis for input to a trained machine learning model trained to predict the three-dimensional trajectory of the vehicle lane. For example, the image data is used as input to a neural network trained to predict the vehicle lane. The machine learning model infers the three-dimensional trajectory for the detected lane. Instead of segmenting the image into lane and non-lane segments of a two-dimensional image, a three-dimensional representation is inferred. In some embodiments, the three-dimensional representation is a spline, a parametric curve, or another representation capable of describing a curve in three dimensions. In some embodiments, a three-dimensional trajectory of the vehicle lane is provided when the vehicle is automatically controlled. For example, the three-dimensional trajectory is used to determine lane boundaries and the corresponding drivable space.

[0013] FIG. 1 is a block diagram illustrating an embodiment of a deep learning system for autonomous driving. The deep learning system includes data collection and processing for training machine learning models for autonomous driving, as well as other components that can be used together for the self-driving and / or driver-assisted operation of a vehicle. In various embodiments, the deep learning system is installed in a vehicle. Data from the vehicle can be used to train and improve the autonomous driving features of the vehicle or other similar vehicles.

[0014] In the illustrated example, the deep learning system (100) is a deep learning network comprising sensors (101), an image pre-processor (103), a deep learning network (105), an artificial intelligence (AI) processor (107), a vehicle control module (109), and a network interface (111). In various embodiments, different components are communicatively connected. For example, sensor data from the sensor (101) is fed to the image pre-processor (103). The processed sensor data from the image pre-processor (103) is fed to the deep learning network (105) running on the AI ​​processor (107). The output of the deep learning network (105) running on the AI ​​processor (107) is fed to the vehicle control module (109). In various embodiments, the vehicle control module (109) is connected to the vehicle and controls the operation of the vehicle, such as the vehicle's speed, braking, and / or steering. In various embodiments, sensor data and / or machine learning results may be sent to a remote server via a network interface (111). For example, sensor data may be transmitted to a remote server via the network interface (111) to collect training data to improve performance, comfort, and / or safety of the vehicle.In various embodiments, the network interface (111) is used to communicate with a remote server, make phone calls, send and / or receive text messages, and transmit sensor data based on the operation of the vehicle for various reasons. In some embodiments, the deep learning system (100) may include appropriately added or fewer components. For example, in some embodiments, an image preprocessing processor (103) is an optional component. As another example, in some embodiments, a post-processing component (not shown) is used to perform post-processing on the output of the deep learning network (105) before the output is provided to the vehicle control module (109).

[0015] In some embodiments, the sensor (101) comprises one or more sensors. In various embodiments, the sensor (101) may be affixed to the vehicle at different locations on the vehicle and / or oriented in one or more different directions. For example, the sensor (101) may be affixed to the front, side, rear, and / or roof, etc., in forward-facing, rear-facing, side-facing, etc. In some embodiments, the sensor (101) may be an image sensor, such as a high dynamic range camera. In some embodiments, the sensor (101) comprises non-visual sensors. In some embodiments, the sensor (101) comprises, among others, radar, audio, LiDAR, inertia, odometry, location, and / or ultrasonic sensor data. In some embodiments, the sensor (101) is not mounted on the vehicle along with the vehicle control module (109). For example, the sensor (101) may be mounted on a neighboring vehicle and / or attached to the road or environment, and may be included as part of a deep learning system to capture sensor data. In some embodiments, the sensor (101) includes one or more cameras that capture the road surface on which the vehicle is traveling. For example, one or more front-facing and / or pillar cameras capture the lane markings of the lane on which the vehicle is traveling. As another example, the cameras capture neighboring vehicles, including a vehicle attempting to cut into the lane on which the vehicle is traveling.Additional sensors capture mileage measurement, location, and / or vehicle control information, including information related to the vehicle trajectory. Sensors (101) may include sensors capable of capturing both still images and / or video. Data may be captured over a period of time, such as a sequence of data captured during a period of time. For example, an image of a lane marking may be captured along with vehicle mileage measurement data for 15 seconds or another appropriate period. In some embodiments, sensors (101) include a location sensor, such as a global position system (GPS) sensor, for determining the vehicle's position and / or changes in position.

[0016] In some embodiments, an image preprocessing processor (103) is used to preprocess sensor data of a sensor (101). For example, the image preprocessing processor (103) may be used to preprocess sensor data, split sensor data into one or more components, and / or post-process one or more components. In some embodiments, the image preprocessing processor (103) is a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the image preprocessing processor (103) is a tone-mapper processor for processing high dynamic range data. In some embodiments, the image preprocessing processor (103) is implemented as part of an artificial intelligence (AI) processor (107). For example, the image preprocessing processor (103) may be a component of the AI ​​processor (107). In some embodiments, the image preprocessing processor (103) may be used to normalize or transform the image. For example, an image captured with a fisheye lens may be warped, and the image preprocessing processor (103) may be used to remove or modify the warping. In some embodiments, noise, distortion, and / or blurriness are removed or reduced through the preprocessing step. In various embodiments, the image is adjusted or normalized to improve the results of machine learning analysis.For example, the white balance of an image is adjusted by taking into account different lighting operating conditions, such as daylight, sunny, cloudy, dusk, sunrise, sunset, and night conditions, among others.

[0017] In some embodiments, the deep learning network (105) is a deep learning network used to determine vehicle control parameters, including analyzing the driving environment to determine lane markers, lanes, drivable space, obstacles, and / or potential vehicle paths. For example, the deep learning network (105) may be an artificial neural network, such as a convolutional neural network (CNN), which is trained on inputs such as sensor data and whose output is provided to the vehicle control module (109). For example, the output may include at least a three-dimensional representation of lane markers. As another example, the output may include at least a potential vehicle that is likely to merge into the vehicle's lane. In some embodiments, the deep learning network (105) receives at least sensor data as input. Additional inputs may include scene data describing the environment surrounding the vehicle and / or vehicle specifications such as the operating characteristics of the vehicle. Scene data may include scene tags describing the environment surrounding the vehicle, such as raining, wet roads, snowing, muddy, high-density traffic, highway, urban, school zone, etc. In some embodiments, the output of the deep learning network (105) is a three-dimensional trajectory of the vehicle's lane. In some embodiments, the output of the deep learning network (105) is a potential vehicle entering. For example, the deep learning network (105) identifies neighboring vehicles that are likely to enter the lane ahead of the vehicle.

[0018] In some embodiments, the artificial intelligence (AI) processor (107) is a hardware processor for running a deep learning network (105). In some embodiments, the AI ​​processor (107) is a specialized AI processor for performing inference on sensor data using a convolutional neural network (CNN). The AI ​​processor (107) may be optimized for the bit depth of the sensor data. In some embodiments, the AI ​​processor (107) is optimized for deep learning operations, such as neural network operations including convolution, dot-product, vector, and / or matrix operations, among others. In some embodiments, the AI ​​processor (107) is implemented using a graphics processing unit (GPU). In various embodiments, the AI ​​processor (107) is coupled to memory. The memory is configured to provide instructions to the AI ​​processor that, at execution, allow the AI ​​processor to perform deep learning analysis on the received input sensor data and determine a machine learning result to be used for autonomous driving. In some embodiments, the AI ​​processor is used to process the sensor data in preparation to make the data available as training data.

[0019] In some embodiments, the vehicle control module (109) is utilized to process the output of the artificial intelligence (AI) processor (107) and translate it into vehicle control operations. In some embodiments, the vehicle control module (109) is used for vehicle control for autonomous driving. In some embodiments, the vehicle control module (109) can adjust the vehicle's speed, acceleration, steering, braking, etc. For example, in some embodiments, the vehicle control module (109) is used to control the vehicle, such as maintaining the vehicle's position within a lane, merging the vehicle into another lane, and lane positioning considering the vehicle's speed and merging.

[0020] In some embodiments, the vehicle control module (109) is used to control vehicle lighting, such as brake lights, turn signals, and headlights. In some embodiments, the vehicle control module (109) is used to control vehicle audio conditions, such as a vehicle sound system, playback of audio alerts, activation of a microphone, and activation of a horn. In some embodiments, the vehicle control module (109) is used to control a notification system, including a warning system, that notifies the driver and / or passenger of driving events, such as a potential collision or approach to an intended destination. In some embodiments, the vehicle control module (109) is used to coordinate sensors, such as the vehicle's sensors (101). For example, the vehicle control module (109) may be used to change parameters of one or more sensors, such as correcting orientation, changing output resolution and / or format type, increasing or decreasing capture rate, adjusting captured dynamic range, adjusting camera focus, enabling and / or disabling sensors. In some embodiments, the vehicle control module (109) may be used to change parameters of an image preprocessing processor (103), such as correcting the frequency range of a filter, adjusting feature and / or edge detection parameters, and adjusting channel and bit depth. In various embodiments, the vehicle control module (109) is used to implement self-driving and / or driver-assisted control of the vehicle.In some embodiments, the vehicle control module (109) is implemented using a processor combined with memory. In some embodiments, the vehicle control module (109) is implemented using an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or other suitable processing hardware.

[0021] In some embodiments, the network interface (111) is a communication interface for transmitting and / or receiving data including voice data. In various embodiments, the network interface (111) includes a cellular or wireless interface for interfacing with a remote server to enable connecting and making voice calls, to send and / or receive text messages, to transmit sensor data, to receive updates to a deep learning network including an updated machine learning model, and to retrieve environmental conditions including weather conditions and forecasts, traffic conditions, etc. For example, the network interface (111) may be used to receive updates to commands and / or operating parameters for the sensor (101), image preprocessing processor (103), deep learning network (105), AI processor (107), and / or vehicle control module (109). The machine learning model of the deep learning network (105) may be updated using the network interface (111). As another example, the network interface (111) may be used to update the firmware of the sensor (101) and / or operating parameters of the image preprocessing processor (103), such as image processing parameters. As yet another embodiment, the network interface (111) may be used to transmit potential training data to a remote server to train a machine learning model.

[0022] FIG. 2 is a flowchart illustrating an example of a process for training and applying a machine learning model for autonomous driving. For example, input data including sensor and driving distance measurement data is received and processed to generate training data for training the machine learning model. In some embodiments, the sensor data corresponds to image data captured through the autonomous driving system. In some embodiments, the sensor data corresponds to sensor data captured based on a specific use case, such as a user manually disengaging autonomous driving. In some embodiments, the process is used to create and deploy a machine learning model for the deep learning system (100) of FIG. 1.

[0023] In (201), training data is prepared. In some embodiments, sensor data including image data and mileage measurement data is received to create a training data set. The sensor data may include still images and / or videos from one or more cameras. Additional sensors, such as radar, lidar, ultrasound, etc., may be used to provide relevant sensor data. In various embodiments, the sensor data is paired with corresponding mileage measurement data to help identify features of the sensor data. For example, position and position change data may be used to identify the location of relevant features in the sensor data, such as lane boundaries, traffic control signals, objects, etc. In some embodiments, the sensor data is a time series of elements and is used to determine ground truth. The ground truth of a group is associated with a subset of the time series, such as the first frame of the image data. Selected elements of the time series and ground truth are used to prepare the training data. In some embodiments, training data is prepared to train a machine learning model to identify only features from sensor data, such as lane lines, vehicle paths, and traffic patterns. The prepared training data may include data for training, validation, and testing. In various embodiments, the sensor data may be in different formats. For example, the sensor data may be still images, videos, audio, etc. The mileage measurement data may include vehicle operation parameters such as applied acceleration, applied braking, applied steering, vehicle location, vehicle orientation, changes in vehicle location, and changes in vehicle orientation.In various embodiments, training data is curated and annotated to generate a training data set. In some embodiments, part of the preparation of the training data may be performed by a human curator. In various embodiments, part of the training data is automatically generated from data captured from a vehicle, significantly reducing the effort and time required to build a robust training data set. In some embodiments, the format of the data is compatible with machine learning models used in deployed deep learning applications. In various embodiments, the training data includes validation data to test the accuracy of the trained model.

[0024] In (203), a machine learning model is trained. For example, the machine learning model is trained using the data prepared in (201). In some embodiments, the model is a neural network such as a convolutional neural network (CNN). In various embodiments, the model includes a number of intermediate layers. In various embodiments, the neural network may include a number of layers including a number of convolution and pooling layers. In some embodiments, the training model is validated using a validation dataset generated from received sensor data. In various embodiments, the machine learning model is trained to predict a three-dimensional representation of a feature from a single input image. For example, a three-dimensional representation of a lane boundary can be inferred from an image captured from a camera. As another example, the predicted path of neighboring vehicles, including whether the vehicle will attempt to merge, is predicted from an image captured from a camera.

[0025] In (205), a trained machine learning model is deployed. For example, the trained machine learning model is installed in the vehicle as an update to a deep learning network, such as the deep learning network (105) of FIG. 1. In some embodiments, an over-the-air update is used to install the newly trained machine learning model. In some embodiments, the update is a firmware update transmitted using a wireless network, such as WiFi or a cellular network. In some embodiments, the new machine learning model may be installed when the vehicle is serviced.

[0026] In (207), sensor data is received. For example, sensor data is captured from one or more sensors of the vehicle. In some embodiments, the sensors are the sensors (101) of FIG. 1. The sensors may include image sensors such as a fisheye camera mounted behind the windshield, a front or side camera mounted on a pole, a rear camera, etc. In various embodiments, the sensor data is converted into the above format or into a format used as input by the machine learning model trained in (203). For example, the sensor data is raw or processed image data. In some embodiments, the data is data captured from an ultrasonic sensor, radar, lidar sensor, microphone, or other suitable technology. In some embodiments, the sensor data is preprocessed during the preprocessing step using an image preprocessing processor such as the image preprocessing processor (103) of FIG. 1. For example, the image may be normalized to remove distortion, noise, etc.

[0027] In (209), a trained machine learning model is applied. For example, the machine learning model trained in (203) is applied to the sensor data received in (207). In some embodiments, the application of the model is performed by an AI processor, such as the AI ​​processor (107) of FIG. 1, using a deep learning network, such as the deep learning network (105) of FIG. 1. In various embodiments, by applying the trained machine learning model, a three-dimensional representation of features, such as lane boundaries, is identified and / or predicted. For example, two splines representing the lane boundaries of the lane in which the vehicle is moving are inferred. In another example, the predicted path of a neighboring vehicle is inferred, including whether there is a possibility that the neighboring vehicle will enter the current lane. In various embodiments, vehicles, obstacles, lanes, traffic control signals, map features, object distance, speed limit, drivable space, etc. are identified by applying a machine learning model. In some embodiments, features are identified in three dimensions.

[0028] In (211), the autonomous vehicle is controlled. For example, one or more autonomous driving features are implemented by various aspects of the vehicle. Examples may include steering, speed, acceleration, and / or braking control of the vehicle, maintaining the vehicle's position in a lane, maintaining the vehicle's position relative to other vehicles and / or obstacles, and providing notifications or warnings to the occupants. Based on the analysis performed in (209), the steering or speed of the vehicle is controlled to maintain the vehicle between two lane boundaries. For example, the left and right lane boundaries are predicted, and the corresponding vehicle lane and drivable space are identified. In various embodiments, a vehicle control module, such as the vehicle control module (109) of FIG. 1, controls the vehicle.

[0029] FIG. 3 is a flowchart illustrating an example of a process for generating training data using a time series of elements. For example, a time series of elements consisting of sensor and mileage measurement data is collected from a vehicle and used to automatically generate training data. In various embodiments, the process of FIG. 3 is used to automatically label the training data along with the corresponding ground truth. An outcome corresponding to the time series is associated with an element of the time series. The outcome and the selected element are packaged into training data used to predict future outcomes. In various embodiments, the sensor and associated data are captured using the deep learning system of FIG. 1. For example, in various embodiments, sensor data is captured from the sensor(s) (101) of FIG. 1. In some embodiments, the process of FIG. 3 is performed at (201) of FIG. 2. In some embodiments, the process of FIG. 3 is performed to automatically collect data when existing predictions are inaccurate or can be improved. For example, a prediction is made by the autonomous vehicle to determine whether the vehicle will enter the path of the autonomous vehicle. After waiting for a certain period and analyzing the captured sensor data, a determination can be made as to whether the prediction is correct or incorrect. In some embodiments, a determination is made that the prediction can be improved. If the prediction is incorrect or can be improved, the process of FIG. 3 can apply the data related to the prediction to generate a curated set of examples to improve the machine learning model.

[0030] In (301), elements of a time series are received. In various embodiments, the elements are sensor data, such as image data captured from a vehicle and transmitted to a training server. Sensor data is captured over a period of time to generate a time series of elements. In various embodiments, the elements are timestamps to maintain the ordering of the elements. As the elements progress through the time series, additional events in the time series are used to help predict results at earlier elements of the time series. For example, the time series can capture vehicles in adjacent lanes sending merging signals, accelerating, and positioning themselves close to the nearby lane boundary. Using the entire time series, the results can be used to determine whether a vehicle has merged into a shared lane. This result can be used to predict whether a vehicle will merge based on selected elements of the time series, such as one of the initial images of the time series. As another example, the time series captures the curve of the lane boundary. The time series captures various dips, bends, crests, etc. of the lanes that are not evident in only a single element of the time series. In various embodiments, the elements are sensor data in a format used as input by a machine learning model. For example, the sensor data is raw or processed image data. In some embodiments, the data is data captured from an ultrasonic sensor, radar, lidar sensor, or other suitable technology.

[0031] In various embodiments, the time series is organized by associating a timestamp with each element of the time series. For example, the timestamp is associated with at least the first element of the time series. Timestamps may be used to calibrate the time series elements with relevant data, such as mileage measurement data. In various embodiments, the length of the time series may be a fixed length of time, such as 10 seconds, 30 seconds, or other appropriate lengths. The length of time is configurable. In various embodiments, the time series may be based on the speed of the vehicle, such as the average speed of the vehicle. For example, at slower speeds, the length of time for the time series may be increased compared to using a shorter length of time at the same speed to capture data during a longer distance of travel. In some embodiments, the number of elements in the time series is configurable. For example, the number of elements may be based on the distance traveled. For example, a vehicle moving fast over a fixed period contains more elements in the time series than a vehicle moving slowly. Additional elements can increase the fidelity of the captured environment and improve the accuracy of the predicted machine learning results. In various embodiments, the number of elements is adjusted by adjusting the frames per second captured by the sensor and / or by discarding unnecessary intermediate frames.

[0032] In (303), data related to the elements of the time series is received. In various embodiments, the related data is received from a training server along with the elements received in (301). In some embodiments, the related data is the vehicle's mileage measurement data. Positional data of features identified in the elements of the time series are labeled using position, direction, change in position, change in direction, and / or other related vehicle data. For example, lane boundaries can be labeled with very accurate positions by examining the time series of elements of the lane boundaries. Generally, the lane boundary closest to the vehicle camera is accurate and closely related to the vehicle's position. In contrast, the XYZ position of the lane furthest from the vehicle is difficult to determine. The distant part of the lane boundary may be occluded (e.g., behind a bend or hill) and / or difficult to capture accurately (e.g., due to distance or lighting, etc.). The data related to the elements is used to label parts of the features identified in the time series identified with high accuracy. In various embodiments, a threshold value may be used to determine whether to associate relevant data with a portion of the identified feature (such as a portion of a lane boundary). For example, a portion of the lane boundary identified with high certainty (such as a portion near the vehicle) is associated with relevant data, whereas a portion of the lane boundary identified with high certainty below the threshold (such as a portion far from the vehicle) is not associated with the relevant data of the element. Instead, another element of the time series with higher certainty, such as a subsequent element, and the associated data are used. In some embodiments, the relevant data is the output of a neural network, such as the output of the deep learning network (105) of FIG. 1.In some embodiments, the relevant data is the output of a vehicle control module, such as the vehicle control module (109) of FIG. 1. The relevant data may include vehicle operating parameters such as speed, acceleration, change in acceleration, steering, change in steering, braking, change in braking, etc. In some embodiments, the relevant data is radar data for estimating the distance of an object, such as an obstacle.

[0033] In some embodiments, data associated with elements of a time series includes map data. For example, offline data, such as road and / or satellite-level map data, is received at (303). Map data is used to identify features such as roads, vehicle lanes, intersections, speed limits, school zones, etc. For example, map data can describe the path of a vehicle lane. As another example, map data can describe speed limits associated with other roads on the map.

[0034] In various embodiments, data related to the elements of a time series is organized by associating timestamps with the related data. Corresponding timestamps from the time series elements and the related data may be used to synchronize the two data sets. In some embodiments, the data is synchronized at the capture time. For example, when each element of the time series is captured, the corresponding set of related data is captured and saved along with the time series element. In various embodiments, a specific period of the related data may be set and / or match a specific period of the time series of the elements. In some embodiments, the related data is sampled at the same rate as the time series elements.

[0035] In (305), ground truth for a time series is determined. In various embodiments, the time series is analyzed to determine ground truth associated with machine learning features. For example, lane boundaries are identified in the time series corresponding to ground truth for those lane boundaries. As another example, ground truth for the path of a moving object (such as a vehicle, pedestrian, cyclist, animal, etc.) is the path identified for the moving object detected in the time series. In some embodiments, if a moving vehicle enters the lane of an autonomous vehicle over the time series, the moving vehicle is annotated as a cut-in vehicle. In some embodiments, ground truth is represented as a three-dimensional representation, such as a three-dimensional trajectory. For example, ground truth associated with a lane can be represented as a three-dimensional parameterized spline or curve. As another example, the predicted path for a detected vehicle is determined and represented as a three-dimensional trajectory. The predicted path can be used to determine whether the vehicle will merge into the occupied space. In various embodiments, ground truth can be determined simply by examining the time series of elements. For example, analyzing only a subset of the time series may leave some occluded lane boundaries hidden. Expanding the analysis across the time series of elements reveals the hidden parts of the vehicle lane. Furthermore, captured data at the end of the time series captures details of lane boundaries further away more accurately (e.g., with higher fidelity).Additionally, the relevant data is also more accurate because it is based on data captured at close range (both distance and time). In various embodiments, simultaneous localization and mapping techniques are applied to different portions of a detected object, such as lane boundaries, identified in different elements of a time series of elements, to map different parts of the object to accurate 3D locations including elevation. The set of mapped 3D locations represents the ground truth of the object, such as segments of lane boundaries captured over the time series. In some embodiments, the localization and mapping techniques result in a set of precise points, for example, a set of points corresponding to different points along a vehicle lane boundary. The set of points can be converted into a more efficient format, such as a spline or a parametric curve. In some embodiments, ground truth is determined to detect objects such as lane boundaries, drivable space, traffic control, vehicles, etc., in three dimensions.

[0036] In some embodiments, ground truth is determined to predict semantic labels. For example, a detected vehicle may be labeled as being in the left lane or the right lane. In some embodiments, a detected vehicle may be labeled as being in a blind spot, as a vehicle that must yield, or with other appropriate semantic labels. In some embodiments, the vehicle is assigned to a road or lane on a map based on the determined ground truth. In additional examples, the determined ground truth may be used to label traffic lights, lanes, drivable spaces, or other features that assist autonomous driving.

[0037] In some embodiments, the relevant data is depth (or distance) data of the detected object. By associating the object identified in the time series of elements with the distance data, a machine learning model can be trained to estimate the object distance using the relevant distance data as the ground truth of the detected object. In some embodiments, the distance is for the detected object, such as an obstacle, barrier, moving vehicle, stationary vehicle, traffic control sign, pedestrian, etc.

[0038] In (307), training data is packaged. For example, elements of a time series are selected and associated with ground truth determined in (305). In various embodiments, the selected elements are initial elements of the time series. The selected elements represent sensor data input to the machine learning model, and the ground truth represents the predicted results. In various embodiments, the training data is packaged and prepared as training data. In some embodiments, the training data is packaged as training, validation, and testing data. Based on the determined ground truth and the selected elements of the time series, the training data may be packaged to train the machine learning model to identify lane boundaries, predicted vehicle paths, speed limits, vehicle cut-ins, object distances, and / or drivable space, and other useful features for autonomous driving. The packaged training data is now available to train the machine learning model.

[0039] FIG. 4 is a flowchart illustrating an example of a process for training and applying a machine learning model for autonomous driving. In some embodiments, the process of FIG. 4 is used to collect and retain sensor and mileage measurement data for training a machine learning model for autonomous driving. In some embodiments, the process of FIG. 4 is implemented in a vehicle capable of autonomous driving regardless of whether autonomous driving control is enabled. For example, sensor and mileage measurement data may be collected immediately after autonomous driving is disengaged, while the vehicle is being driven by a human driver, and / or while the vehicle is autonomous driving. In some embodiments, the technique described in FIG. 4 may be implemented using the deep learning system of FIG. 1. In some embodiments, parts of the process of FIG. 4 are performed in (207), (209), and / or (211) of FIG. 2 as part of the process for applying a machine learning model for autonomous driving.

[0040] In (401), sensor data is received. For example, a vehicle equipped with sensors captures sensor data and provides the sensor data to a neural network running in the vehicle. In some embodiments, the sensor data may be vision data, ultrasonic data, LiDaR data, or other suitable sensor data. For example, images are captured from a high dynamic range forward-facing camera. As another example, ultrasonic data is captured from a side ultrasonic sensor. In some embodiments, the vehicle is affixed with a number of sensors for capturing data. For example, in some embodiments, eight surround cameras are affixed to the vehicle and provide 360-degree visibility around the vehicle with a range of up to 250 meters. In some embodiments, the camera sensors include a wide forward camera, a narrow forward camera, a rear view camera, a forward-looking side camera, and / or a rearward-looking side camera. In some embodiments, ultrasonic and / or radar sensors are used to capture surrounding details. For example, 12 ultrasonic sensors may be attached to the vehicle to detect both hard and soft objects. In some embodiments, a forward-facing radar is utilized to capture data of the surrounding environment. In various embodiments, the radar sensors can capture surrounding details despite heavy rain, fog, dust, and other vehicles.Various sensors are used to capture the environment around the vehicle, and the captured data is provided for deep learning analysis.

[0041] In some embodiments, sensor data includes mileage measurement data including the vehicle's position, orientation, change of position, and / or change of orientation. For example, position data is captured and associated with other sensor data captured during the same time frame. For example, when image data is captured, the captured position data is used to associate the image data with the position information.

[0042] In (403), sensor data is pre-processed. In some embodiments, one or more pre-processing passes are performed on the sensor data. For example, the data may be pre-processed to remove noise, correct alignment issues and / or blurring. In some embodiments, one or more other filtering passes are performed on the data. For example, a high-pass filter may be performed on the data and a low-pass filter may be performed on the data to separate different components of the sensor data. In various embodiments, the pre-processing steps performed in (403) are optional and / or may be incorporated into a neural network.

[0043] In (405), deep learning analysis of the sensor data is initiated. In some embodiments, deep learning analysis is performed on the sensor data optionally preprocessed in (403). In various embodiments, deep learning analysis is performed using a neural network such as a convolutional neural network (CNN). In various embodiments, the machine learning model is trained offline using the process of FIG. 2 and deployed to a vehicle to perform inference on the sensor data. For example, the model can be trained to appropriately identify road lane lines, obstacles, pedestrians, moving vehicles, parked vehicles, drivable spaces, etc. In some embodiments, multiple trajectories for the lane lines are identified. For example, multiple potential trajectories for the lane lines are detected, and each trajectory has a corresponding probability of occurring. In some embodiments, the predicted lane boundary is the lane boundary having the highest probability of occurrence and / or the highest associated confidence value. In some embodiments, the lane boundary predicted from the deep learning analysis is required to exceed a minimum confidence threshold value. In various embodiments, the neural network includes a plurality of layers, including one or more intermediate layers. In various embodiments, sensor data and / or the results of the deep learning analysis are retained for the automatic generation of training data and transmitted at (411).

[0044] In various embodiments, deep learning analysis is used to predict additional features. The predicted features can be used to assist autonomous driving. For example, a detected vehicle can be assigned to a lane or road. As another example, it can be determined whether the detected vehicle is in a blind spot, is a vehicle that must yield, is a vehicle in the adjacent left lane, is a vehicle in the adjacent right lane, or possesses other appropriate attributes. Similarly, deep learning analysis can identify traffic lights, drivable space, pedestrians, obstacles, or other appropriate features for driving.

[0045] In (407), the results of the deep learning analysis are provided to the vehicle control. For example, the results are provided to the vehicle control module to control the vehicle for autonomous driving and / or to implement autonomous driving functionality. In some embodiments, the results of the deep learning analysis in (405) are passed through one or more additional deep learning passes using one or more different machine learning models. For example, the predicted path for the lane boundary can be used to determine the vehicle lane, and the determined vehicle lane can be used to determine the drivable space. The drivable space is used to determine the path for the vehicle. Similarly, in some embodiments, a predicted vehicle cut-in is detected. The determined path of the vehicle takes into account the predicted cut-in to avoid potential collisions. In some embodiments, various outputs of deep learning are used to construct a three-dimensional representation of the vehicle's environment for autonomous driving, including the predicted path of the vehicle, identified obstacles, identified traffic control signs including speed limits, etc. In some embodiments, the vehicle control module utilizes the determined results to control the vehicle along the determined path. In some embodiments, the vehicle control module is the vehicle control module (109) of FIG. 1.

[0046] In (409), the vehicle is controlled. In some embodiments, the vehicle with autonomous driving enabled is controlled using a vehicle control module such as the vehicle control module (109) of FIG. 1. Vehicle control may modulate the speed and / or steering of the vehicle to maintain the vehicle in the lane at an appropriate speed, for example, by considering the vehicle's surrounding environment. In some embodiments, the results are used to adjust the vehicle in anticipation of neighboring vehicles merging into the same lane. In various embodiments, using the results of deep learning analysis, the vehicle control module determines the appropriate manner to operate the vehicle at an appropriate speed along a determined path, for example. In various embodiments, the results of vehicle control, such as changes in speed, application of braking, and adjustment of steering, are retained and used for the automatic generation of training data. In various embodiments, vehicle control parameters are retained for the automatic generation of training data and transmitted in (411).

[0047] In (411), the sensor and related data are transmitted. For example, the sensor data received in (401), along with the results of the deep learning analysis in (405) and / or the vehicle control parameters used in (409), is transmitted to a computer server for the automatic generation of training data. In some embodiments, the data is a time series of data, and various collected data are associated together by the computer server. For example, mileage measurement data is associated with captured image data to generate ground truth. In various embodiments, the collected data is transmitted wirelessly from the vehicle to a training data center, for example, via Wi-Fi or a cellular connection. In various embodiments, metadata is transmitted along with the sensor data. For example, metadata may include operational parameters such as time of day, timestamp, location, type of vehicle, vehicle control and / or speed, acceleration, braking, whether autonomous driving is enabled, steering angle, and mileage measurement data. Additional metadata includes the time since the last previous sensor data was transmitted, vehicle type, weather conditions, road conditions, etc. In some embodiments, the transmitted data is anonymized, for example, by removing unique identifiers of the vehicle. As another example, data of similar vehicle models is merged to prevent individual users and their use of their vehicles from being identified.

[0048] In some embodiments, data is transmitted only in response to a trigger. For example, in some embodiments, an inaccurate prediction triggers the transmission of sensors and related data to automatically collect data in order to generate a set of curated examples to improve the prediction of a deep learning network. For example, a prediction made in (405) regarding whether a vehicle attempts to merge is determined to be inaccurate by comparing the prediction with the actually observed result. Data including sensors and sensor data related to the incorrect prediction is transmitted and used to automatically generate training data. In some embodiments, the trigger may be used to identify specific scenarios, such as sharp curves, forks in the roads, lane merges, sudden stops, or other appropriate scenarios where additional training data would be helpful and might be difficult to gather. For example, the trigger may be based on the sudden deactivation or disengagement of autonomous driving features. As another example, vehicle operating properties, such as changes in speed or acceleration, may form the basis of the trigger. In some embodiments, a prediction with an accuracy below a certain threshold triggers the transmission of sensors and related data. For example, in some scenarios, the prediction may not have a Boolean correct or Boolean incorrect result, but is instead evaluated by determining the accuracy value of the prediction.

[0049] In various embodiments, sensors and associated data are captured over a set period of time, and the entire time series of data is transmitted together. The set period may be composed of and / or based on one or more factors, such as vehicle speed, distance traveled, and changes in speed. In some embodiments, the sampling rate of the captured sensors and / or associated data is configurable. For example, the sampling rate is increased at higher speeds, during sudden braking, during sudden acceleration, during hard steering, or in other appropriate scenarios requiring additional fidelity.

[0050] FIG. 5 is a diagram illustrating an example of an image captured from a vehicle sensor. In the illustrated example, the image of FIG. 5 contains image data (500) captured from a vehicle moving in a lane between two lane boundaries. The location of the vehicle and sensor used to capture the image data (500) is represented by label (A). The image data (500) is sensor data and may be captured from a camera sensor, such as a front camera of the vehicle, while driving. The image data (500) captures parts (501 and 511) of the lane boundaries. The lane boundaries (501 and 511) curve to the right as the lane boundaries (501 and 511) approach the horizon. In the illustrated example, the lane boundaries (501 and 511) are visible, but become increasingly difficult to detect as they curve away from the location of the camera sensor. The white line drawn on top of the lane boundary lines (501 and 511) approximates a detectable portion of the lane boundary lines (501 and 511) from the image data (500) without any additional input. In some embodiments, the detected portion of the lane boundary lines (501 and 511) may be detected by segmenting the image data (500).

[0051] In some embodiments, labels (A), (B), and (C) correspond to different locations on the road and different times in a time series. Label (A) corresponds to the location of the vehicle at the time and image data (500) is captured. Label (B) corresponds to a location on the road ahead of the location of label (A) and a time after the time of label (A). Similarly, label (C) corresponds to a location on the road ahead of the location of label (B) and a time after the time of label (B). As the vehicle travels, it passes through the locations of labels (A), (B), and (C) (from label (A) to label (C)) and captures the sensor's time series and related data while moving. The time series includes elements captured at the locations (and times) of labels (A), (B), and (C). Label (A) corresponds to the first element of the time series, Label (B) corresponds to the middle element of the time series, and Label (C) corresponds to the middle (or potentially last) element of the time series. At each label, additional data, such as vehicle odometry data, is captured at the location of the label. Depending on the length of the time series, additional or less data is captured. In some embodiments, a timestamp is associated with each element of the time series.

[0052] In some embodiments, ground truth (not shown) for lane boundaries (501 and 511) is determined. For example, using the process disclosed herein, the location of lane boundaries (501 and 511) is identified by identifying different portions of lane boundaries (501 and 511) from different elements of a time series of elements. In the illustrated example, portions (503 and 513) are identified using image data (500) and related data (such as mileage measurement data) taken at the time and location of label (A). Portions (505 and 515) are identified using image data (not shown) and related data (such as mileage measurement data) taken at the time and location of label (B). Parts (507 and 517) are identified using image data (not shown) and related data (such as mileage measurement data) taken at the time and location of label (C). By analyzing the time series of elements, the locations of different parts of the lane boundaries (501 and 511) are identified, and ground truth can be determined by combining the different identified parts. In some embodiments, parts are identified as points along each part of the vehicle lane. In the illustrated example, only three parts of each vehicle lane are highlighted to explain the process (parts (503), (505), and (507) for lane boundary (501) and parts (513), (515), and (517) for lane boundary (511), but additional parts can be captured over the time series to determine the locations of the lane boundaries with high resolution and / or higher accuracy.

[0053] In various embodiments, locations of portions in image data capturing the lane boundaries (501 and 511) closest to the sensor location are determined with a high degree of accuracy. For example, the locations of portions (503 and 513) are identified with a high degree of accuracy using the image data (500) of label (A) and related data (such as mileage measurement data). The locations of portions (505 and 515) are identified with a high degree of accuracy using the image and related data of label (B). The locations of portions (507 and 517) are identified with a high degree of accuracy using the image data and related data of label (C). By utilizing a time series of elements, the locations of various portions of the lane boundaries (501 and 511) captured by the time series can be identified with a high degree of accuracy in three dimensions and can be used as a basis for ground truth of the lane boundaries (501 and 511). In various embodiments, the determined ground truth is associated with selected elements of a time series, such as image data (500). The ground truth and selected elements can be used to generate training data for predicting lane boundaries. In some embodiments, the training data is generated automatically without human labeling. The training data can be used to train a machine learning model to predict the three-dimensional trajectory of lane boundaries from captured data, such as image data (500).

[0054] FIG. 6 illustrates an example of an image captured from a vehicle sensor by predicting the three-dimensional trajectory of a lane boundary. In the illustrated example, the image of FIG. 6 includes image data (600) captured from a vehicle moving in a lane between two lane boundaries. The location of the vehicle and sensor used to capture the image data (600) is represented by a label (A). In some embodiments, the label (A) corresponds to the same location as the label (A) in FIG. 5. The image data (600) is sensor data and may be captured from a camera sensor, such as a front camera of a moving vehicle. The image data (600) captures a portion of the lane boundaries (601 and 611). The lane boundaries (601 and 611) curve to the right as the lane boundaries (601 and 611) approach the horizontal line. In the illustrated example, the lane boundaries (601 and 611) are visible, but curve in a direction away from the position of the camera sensor and become increasingly difficult to detect as they get further away from the distance. The red line drawn on the lane boundaries (601 and 611) is the predicted three-dimensional trajectory of the lane boundaries (601 and 611). Using the process disclosed herein, the three-dimensional trajectory is predicted using image data (600) as input to a trained machine learning model. In some embodiments, the predicted three-dimensional trajectory is represented as a three-dimensional parameterized spline or a parameterized form of other representation.

[0055] In the illustrated example, portions (621) of lane boundaries (601 and 611) are parts of lane boundaries (601 and 611) that are far apart. The three-dimensional location (i.e., longitude, latitude, and altitude) of portions (621) of lane boundaries (601 and 611) is determined with a high degree of accuracy using the process disclosed herein and is included in the predicted three-dimensional trajectory of lane boundaries (601 and 611). Using a trained machine learning model, the three-dimensional trajectory of lane boundaries (601 and 611) can be predicted using image data (600) without requiring location data at the location of portions (621) of lane boundaries (601 and 611). In the illustrated example, image data (600) is captured at the location and time of the label (A).

[0056] In some embodiments, the label (A) of FIG. 6 corresponds to the label (A) of FIG. 5, and the predicted three-dimensional trajectory of the lane boundary lines (601 and 611) is determined using only image data (600) as input to a trained machine learning model. By training the machine learning model using ground truth determined using relevant data and images of a time series including elements taken at the locations of labels (A), (B), and (C) of FIG. 5, the three-dimensional trajectory of the lane boundary lines (601 and 611) is predicted with a high degree of accuracy, even for distant portions of the lane boundary lines such as portion (621). Even though the image data (600) and the image data (500) of FIG. 5 are associated, the prediction of trajectories does not require the image data (600) to be included in the training data. Through training with sufficient training data, lane boundaries can be predicted even in counter scenarios. In various embodiments, the predicted three-dimensional trajectories of lane boundaries (601 and 611) are used to maintain the vehicle's position within the detected lane boundaries and / or for the vehicle to autonomously navigate along the detected lanes of the predicted lane boundaries. By predicting lane boundaries in three dimensions, the performance, safety, and accuracy of navigation are greatly improved.

[0057] Although the foregoing embodiments have been described in some detail for the purpose of clarity of understanding, the invention is not limited by the provided details. There are numerous alternative methods for implementing the invention. The embodiments disclosed herein are exemplary and not limiting.

Claims

Claim 1 A method comprising: acquiring sensor data including a plurality of images captured at points in time within a period of time by one or more processors; determining ground truths related to a three-dimensional representation of a feature by one or more processors based on the sensor data; and training a machine learning model by one or more processors using a training dataset including a portion of the sensor data captured at points in time within the period of time related to the three-dimensional representation of the feature, wherein the machine learning model is trained to output an identification of a path related to a vehicle as the three-dimensional representation of the feature based on an individual image of another vehicle separate from the plurality of images as input to sensor data related at points in a different period of time. Claim 2 In claim 1, the three-dimensional representation of the above feature is a method related to lane boundaries. Claim 3 A method according to paragraph 2, wherein the sensor data comprises a plurality of images captured at the points in time within the time period, and portions of the lane boundary are represented by a set of images from among the plurality of images. Claim 4 A method in which, in paragraph 3, images representing the lane boundary are selected from the set of images to train the machine learning model based on a measure of the relevancy of the images to the remaining images of the set of images representing the portion of the lane boundary. Claim 5 In paragraph 2, the three-dimensional representation of the above feature is a method that reflects the trajectory of the lane boundary line. Claim 6 In claim 1, the three-dimensional representation of the above features is a method that reflects a path related to a vehicle. Claim 7 In claim 6, a method in which the vehicle is in the first lane adjacent to the second lane at the time when the sensor data is captured by the sensor of another vehicle located in the second lane. Claim 8 A method according to claim 1, wherein the training dataset further includes scene data describing the real-world environment surrounding the sensor of the vehicle that captured the sensor data. Claim 9 A system comprising one or more processors, wherein the one or more processors: acquire sensor data comprising a plurality of images captured at points in time within a time period; determine ground truths related to a three-dimensional representation of a feature based on the sensor data; and train a machine learning model using a training dataset comprising a portion of the sensor data captured at points in time within the time period related to the three-dimensional representation of the feature, wherein the machine learning model is trained to output the identification of a path related to a vehicle as the three-dimensional representation of the feature based on an individual image of another vehicle separate from the plurality of images as input to sensor data related at points in a different time period. Claim 10 In paragraph 9, the three-dimensional representation of the above feature is a system related to lane boundaries. Claim 11 In claim 10, the sensor data comprises a plurality of images captured at the points in time within the time period, and portions of the lane boundary are represented by a set of images from among the plurality of images, in a system. Claim 12 In claim 11, the images representing the lane boundary are selected from the image set to train the machine learning model based on a measure related to the relevance of the images to the remaining images of the image set representing the portion of the lane boundary. Claim 13 In item 10, the three-dimensional representation of the above feature is a system that reflects the trajectory of the lane boundary line. Claim 14 In claim 9, the three-dimensional representation of the above features is a system that reflects a path related to a vehicle. Claim 15 In claim 14, a system in which the vehicle is in the first lane adjacent to the second lane at the time when the sensor data is captured by the sensor of another vehicle located in the second lane. Claim 16 In claim 9, the system further comprises scene data describing the real-world environment surrounding the sensor of the vehicle that captured the sensor data. Claim 17 A non-transient computer-readable storage medium for storing computer instructions, wherein the computer instructions, when executed by one or more processors, cause the one or more processors to: acquire sensor data including a plurality of images captured at points in time within a time period; determine ground truths related to a three-dimensional representation of a feature based on the sensor data; and train a machine learning model using a training dataset including a portion of the sensor data captured at points in time within the time period related to the three-dimensional representation of the feature, and the machine learning model is trained to output an identification of a path related to a vehicle as the three-dimensional representation of the feature based on an individual image of another vehicle separate from the plurality of images as input to sensor data related at points in another time period. Claim 18 In paragraph 17, the three-dimensional representation of the above feature is a computer-readable storage medium related to lane boundaries.

Images