Generating ground truth for machine learning from time series elements

Abstract

Embodiments of the present application relate to generating ground truth for machine learning from time series elements. Sensor data is received including a set of time series elements. A training data set is determined including determining a corresponding ground truth for at least selected ones of the set of time series elements. The corresponding ground truth is based on a plurality of time series elements of the set of time series elements. A processor is used to train a machine learning model using the training data set.

Description

Divisional Application Instructions

[0001] This application is a divisional application of Chinese invention patent application No. 202080023715.2 entitled "Generating Ground Truth Values ​​for Machine Learning from Time Series Elements", which was filed internationally on January 28, 2020, entered the Chinese national phase on September 23, 2021. Cross-reference to related applications

[0002] This application is a continuation-to-priority application of U.S. Patent Application No. 16 / 265729, entitled “GENERATING GROUND TRUTH FOR MACHINE LEARNING FROM TIMESERIES ELEMENTS”, filed on February 1, 2019, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] Various embodiments of this application relate to generating ground truth values ​​for machine learning from time series elements. Background Technology

[0004] Deep learning systems, used for applications such as autonomous driving, are developed by training machine learning models. Typically, the performance of a deep learning system is at least in part limited by the quality of the training set used to train the model. In many cases, significant resources are devoted to collecting, curating, and labeling training data. Traditionally, much of the work involved in curating training datasets is done manually by reviewing potential training data and appropriately labeling the features associated with the data. The work required to create a training set with accurate labels can be substantial and often tedious. Furthermore, it is often difficult to collect and accurately label the data that machine learning models need to improve upon. Therefore, there is a need for improved processes for generating training data with accurately labeled features. Summary of the Invention

[0005] Various embodiments of this application relate to generating ground truth values ​​for machine learning from time series elements.

[0006] According to a first aspect of this application, a method is provided comprising: obtaining sensor data associated with time points within a time period by one or more processors; determining ground truth values ​​based on the sensor data by one or more processors, the ground truth values ​​being associated with a three-dimensional representation of features; and training a machine learning model by one or more processors using a training dataset comprising a portion of the sensor data captured at time points within the time period associated with the three-dimensional representation of features, wherein the machine learning model is trained to output an identifier of the three-dimensional representation of features based on inputs of sensor data associated with time points within different time periods.

[0007] According to a second aspect of this application, a system is provided, comprising one or more processors, the processors being configured to: acquire sensor data associated with time points within a time period; determine ground truth based on the sensor data, the ground truth being associated with a three-dimensional representation of features; and train a machine learning model using a training dataset comprising a portion of the sensor data captured at time points within the time period associated with the three-dimensional representation of features, wherein the machine learning model is trained to output an identifier of the three-dimensional representation of features based on inputs of sensor data associated with time points within different time periods.

[0008] According to a third aspect of this application, a non-transient computer-readable storage medium is provided, comprising computer instructions that, when executed by one or more processors, cause the one or more processors to: acquire sensor data associated with time points within a time period; determine ground truth based on the sensor data, the ground truth being associated with a three-dimensional representation of features; and train a machine learning model using a training dataset comprising a portion of the sensor data captured at time points within the time period associated with the three-dimensional representation of features, wherein the machine learning model is trained to output an identifier of the three-dimensional representation of features based on inputs of sensor data associated with time points within different time periods. Attached Figure Description

[0009] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0010] Figure 1 This is a block diagram illustrating an embodiment of a deep learning system for autonomous driving.

[0011] Figure 2 This is a flowchart illustrating an embodiment of the process for training and applying a machine learning model for autonomous driving.

[0012] Figure 3 This is a flowchart illustrating an embodiment of the process for creating training data using time series of elements.

[0013] Figure 4 This is a flowchart illustrating an embodiment of the process for training and applying a machine learning model for autonomous driving.

[0014] Figure 5 This is an illustration of an example of an image captured from a vehicle's sensors.

[0015] Figure 6 This is an illustration of an example of an image captured by a vehicle sensor showing a predicted 3D trajectory with lane lines. Detailed Implementation

[0016] This invention can be implemented in various ways, including as: a process; an apparatus; a system; a material composition; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, these implementations or any other forms of the invention may be referred to as techniques. Generally, the order of steps of the disclosed process can be varied within the scope of this invention. Unless otherwise stated, components such as processors or memories described as being configured to perform tasks can be implemented as general-purpose components temporarily configured to perform tasks at a given time or manufactured as special-purpose components for performing tasks. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0017] The following is an appendix explaining the principles of the invention. Figure 1 This description provides a detailed account of one or more embodiments of the invention. The invention is described in conjunction with such embodiments, but is not limited to any particular embodiment. The scope of the invention is limited only by the claims, and the invention covers numerous alternatives, modifications, and equivalents. Numerous specific details are set forth in the following description to provide a complete understanding of the invention. These details are provided for illustrative purposes, and the invention may be practiced without some or all of these specific details. For clarity, technical materials known in the art related to the invention have not been described in detail so as not to unnecessarily obscure the invention.

[0018] A machine learning training technique for generating highly accurate machine learning results is disclosed. A training dataset is created using data captured by sensors on a vehicle to capture the vehicle's environment and vehicle operating parameters. For example, sensors fixed to the vehicle capture data such as image data of the road and surrounding environment on which the vehicle is driving. Sensor data can capture lane lines, lanes, other vehicle traffic, obstacles, traffic control signs, etc. Mileage and other similar sensors capture vehicle operating parameters such as vehicle speed, steering, orientation, changes in direction, changes in position, changes in altitude, changes in speed, etc. The captured dataset is transmitted to a training server to create a training dataset. The training dataset is used to train a machine learning model to generate highly accurate machine learning results. In some embodiments, time series of the captured data are used to generate training data. For example, ground truth is determined based on a set of time series elements and associated with a single element from that set. As an example, a sequence of images for a time period such as 30 seconds is used to determine the actual path of the lane lines during the time period in which the vehicle travels. Lane lines are determined using the most accurate image of the lanes within the time period. Different portions (or locations) of the lane lines can be identified from different image data in the time series. As a vehicle travels along the lane lines, more accurate data is captured for different sections of the lane lines. In some examples, occluded portions of the lane lines are revealed as the vehicle travels, for example, along hidden curves or over ridges. The most accurate portion of the lane lines from each image in the time series can be used to identify the lane lines over the entire set of image data. Image data of lane lines further away from the vehicle is typically less detailed than image data of lane lines near the vehicle. Through time-series capture of image data as the vehicle travels along the lane, accurate image data and corresponding mileage data for all sections of the corresponding lane lines are collected.

[0019] In some embodiments, a three-dimensional representation of a feature, such as lane lines, is created from a set of time-series elements corresponding to ground truth. This ground truth is then associated with a subset of the time-series elements, such as a single image frame from the set of captured image data. For example, the first image in a set of images is associated with a ground truth representing lane lines in three-dimensional space. Although the ground truth is determined based on this set of images, the selected first frame and the ground truth are used to create training data. As an example, training data is created to predict a three-dimensional representation of a lane using only a single image. In some embodiments, any element or set of elements in a set of time-series elements is associated with a ground truth and used to create training data. For example, the ground truth may be applied to the entire video sequence to create training data. As another example, a middle or last element in a set of time-series elements is associated with a ground truth and used to create training data.

[0020] In various embodiments, the selected images and ground truth can be applied to different features, such as lane lines, vehicle path prediction including adjacent vehicles, object depth distance, traffic control signs, etc. For example, a sequence of images of vehicles in adjacent lanes is used to predict the path of that vehicle. Using the time series of images and the actual paths taken by adjacent vehicles, the individual images of that set and the actual paths taken can be used as training data to predict vehicle paths. Information can also be used to predict whether adjacent vehicles will merge into the path of the autonomous vehicle. For example, path prediction can predict whether adjacent vehicles will merge in front of the autonomous vehicle. The autonomous vehicle can be controlled to minimize the probability of a collision. For example, the autonomous vehicle can decelerate to prevent a collision, adjust the vehicle's speed and / or steering to prevent a collision, issue warnings to adjacent vehicles and / or occupants of the autonomous vehicle, and / or change lanes, etc. In various embodiments, the ability to accurately infer path predictions including vehicle path prediction significantly improves the safety of autonomous vehicles.

[0021] In some embodiments, the trained machine learning model is used to predict a three-dimensional representation of one or more features, including lane lines, for autonomous driving. For example, instead of identifying two-dimensional lane lines from image data by segmenting images of lane lines, a three-dimensional representation is generated using time series of elements and mileage data corresponding to the time series. The three-dimensional representation includes elevation variations, which significantly improves the accuracy of lane line detection and the detection of corresponding lanes and identified drivable paths. In some embodiments, lane lines are represented using one or more splines or another parametric form of representation. Using piecewise polynomials to represent lane lines significantly reduces the computational resources required to evaluate three-dimensional objects. This reduction in computational resources corresponds to improvements in processing speed and efficiency without significantly sacrificing the accuracy of the representation. In various embodiments, lane lines (especially curves including lane lines) can be represented using piecewise polynomials, sets of three-dimensional points, or another suitable representation. For example, a piecewise polynomial interpolates the actual lane lines using a height-accurate portion of the lane lines identified from a set of elements captured over time using sensor data.

[0022] In some embodiments, sensor data is received. Sensor data may include images (such as video and / or still images), radar, audio, light, inertia, odometer, location, and / or other forms of sensor data. The sensor data includes a set of time-series elements. For example, a set of time-series elements may include a set of images captured from a vehicle's camera sensors within a time period. In some embodiments, a training dataset is determined by determining corresponding ground truth values ​​for at least a selected time-series element within the set of time-series elements, based on multiple time-series elements in the set. For example, ground truth values ​​are determined by examining the most relevant portion of each element in the set of time-series elements, including previous and / or subsequent time-series elements in the set. In some scenarios, only previous and / or subsequent time-series elements include data not present in earlier time-series elements, such as lane lines that initially disappear around a curve and only appear in later elements of the time series. The determined ground truth values ​​may be a three-dimensional representation of the vehicle's lane lines, the vehicle's predicted path, or another similar prediction. Elements in the set of time-series elements are selected and associated with ground truth values. The selected elements and ground truth values ​​are part of the training dataset. In some embodiments, the processor is used to train a machine learning model using a training dataset. For example, the training dataset is used to train a machine learning model for inferring features for autonomous driving or driver-assisted operation of a vehicle. Using the trained machine learning model, the neural network can infer features associated with autonomous driving, such as lanes, drivable space, objects (e.g., pedestrians, stationary vehicles, moving vehicles, etc.), weather (rain, hail, fog, etc.), traffic control objects (e.g., traffic lights, traffic signs, street signs, etc.), traffic patterns, etc.

[0023] In some embodiments, the system includes a processor and memory coupled to the processor. The processor is configured to receive image data based on images captured by a camera of the vehicle. For example, a camera sensor fixed to the vehicle captures images of the vehicle's environment. The camera may be a forward-facing camera, a pillar camera, or another suitably positioned camera. A processor on the vehicle, such as a GPU or AI processor, is used to process the image data captured from the camera. In some embodiments, the image data is used as the basis for input to a trained machine learning model that is trained to predict the three-dimensional trajectory of a lane. For example, the image data is used as input to a neural network trained to predict lanes. The machine learning model infers the three-dimensional trajectory of the detected lane. Instead of segmenting the image into lane and non-lane segments of a two-dimensional image, a three-dimensional representation is inferred. In some embodiments, the three-dimensional representation is a spline, a parametric curve, or another representation capable of describing a curve in three-dimensional space. In some embodiments, the three-dimensional trajectory of the lane is provided in an autonomous vehicle. For example, the three-dimensional trajectory is used to determine lane lines and corresponding drivable spaces.

[0024] Figure 1 This is a block diagram illustrating an embodiment of a deep learning system for autonomous driving. The deep learning system includes various components that can work together for autonomous driving and / or driver assistance operations of a vehicle, as well as for collecting and processing data for training machine learning models for autonomous driving. In various embodiments, the deep learning system is mounted on a vehicle. Data from the vehicle can be used to train and improve the autonomous driving characteristics of that vehicle or other similar vehicles.

[0025] In the illustrated example, the deep learning system 100 is a deep learning network including a sensor 101, an image preprocessor 103, a deep learning network 105, an artificial intelligence (AI) processor 107, a vehicle control module 109, and a network interface 111. In various embodiments, the different components are communicatively connected. For example, sensor data from the sensor 101 is fed to the image preprocessor 103. The processed sensor data from the image preprocessor 103 is fed to the deep learning network 105 running on the AI ​​processor 107. The output of the deep learning network 105 running on the AI ​​processor 107 is fed to the vehicle control module 109. In various embodiments, the vehicle control module 109 is connected to and controls the operation of the vehicle, such as vehicle speed, braking, and / or steering. In various embodiments, sensor data and / or machine learning results may be sent to a remote server via the network interface 111. For example, sensor data may be transmitted to a remote server via the network interface 111 to collect training data for improving the vehicle's performance, comfort, and / or safety. In various embodiments, network interface 111 is used to communicate with a remote server for making phone calls, sending and / or receiving short messages, transmitting sensor data based on vehicle operation, and other reasons. In some embodiments, deep learning system 100 may include additional or fewer components as appropriate. For example, in some embodiments, image preprocessor 103 is an optional component. As another example, in some embodiments, post-processing components (not shown) are used to perform post-processing on the output of deep learning network 105 before providing it to vehicle control module 109.

[0026] In some embodiments, sensor 101 includes one or more sensors. In various embodiments, sensor 101 may be fixed to the vehicle at different locations and / or oriented in one or more different directions. For example, sensor 101 may be fixed to the front, side, rear, and / or top of the vehicle in forward, rearward, lateral, and other directions. In some embodiments, sensor 101 may be an image sensor, such as a high dynamic range camera. In some embodiments, sensor 101 includes a non-visual sensor. In some embodiments, sensor 101 includes radar, audio, LiDAR, inertia, odometer, position, and / or ultrasonic sensors, as well as other sensors. In some embodiments, sensor 101 is not mounted to a vehicle having vehicle control module 109. For example, sensor 101 may be mounted on an adjacent vehicle and / or fixed to a road or environment and is included as part of a deep learning system for capturing sensor data. In some embodiments, sensor 101 includes one or more cameras that capture the road surface on which the vehicle is traveling. For example, one or more forward and / or pillar cameras capture lane markings of the lane in which the vehicle is traveling. As another example, cameras capture adjacent vehicles, including vehicles attempting to merge into the lane in which the vehicle is traveling. Additional sensors capture mileage, location, and / or vehicle control information, including information related to the vehicle's trajectory. Sensor 101 may include two image sensors capable of capturing additional images and / or video. Data may be captured within a time period, such as a sequence of captured data within a time period. For example, an image of lane markings may be captured along with vehicle mileage data within a 15-second time period or another suitable time period. In some embodiments, sensor 101 includes a location sensor, such as a Global Positioning System (GPS) sensor, for determining the vehicle's position and / or changes in position.

[0027] In some embodiments, image preprocessor 103 is used to preprocess sensor data from sensor 101. For example, image preprocessor 103 may be used to preprocess sensor data, split sensor data into one or more components, and / or postprocess the one or more components. In some embodiments, image preprocessor 103 is a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a dedicated image processor. In various embodiments, image preprocessor 103 is a tone mapper processor for processing high dynamic range data. In some embodiments, image preprocessor 103 is implemented as part of artificial intelligence (AI) processor 107. For example, image preprocessor 103 may be a component of AI processor 107. In some embodiments, image preprocessor 103 may be used to normalize or transform an image. For example, an image captured using a fisheye lens may be distorted, and image preprocessor 103 may be used to transform the image to remove or modify the distortion. In some embodiments, noise, distortion, and / or blur are removed or reduced during the preprocessing step. In various embodiments, the image is adjusted or normalized to improve the results of machine learning analysis. For example, the white balance of the image is adjusted to take into account different lighting conditions, such as daylight, sunny, cloudy, dusk, sunrise, sunset, and night conditions, as well as others.

[0028] In some embodiments, the deep learning network 105 is a deep learning network used to determine vehicle control parameters, including analyzing the driving environment to determine lane markings, lanes, drivable space, obstacles, and / or potential vehicle paths. For example, the deep learning network 105 may be an artificial neural network, such as a convolutional neural network (CNN) trained on inputs such as sensor data, and its output is provided to the vehicle control module 109. As an example, the output may include at least a three-dimensional representation of lane markings. As another example, the output may include at least potential vehicles in lanes that may merge into the vehicle. In some embodiments, the deep learning network 105 receives at least sensor data as input. Additional inputs may include scenario data describing the environment surrounding the vehicle and / or vehicle specifications, such as the vehicle's operating characteristics. Scenario data may include scenario labels describing the environment surrounding the vehicle, such as rain, slippery road surface, snow, mud, high-density traffic, highway, urban area, school zone, etc. In some embodiments, the output of the deep learning network 105 is a three-dimensional trajectory of the vehicle's lane. In some embodiments, the output of the deep learning network 105 is a potential vehicle insertion. For example, a deep learning network 105 can identify adjacent vehicles that may enter the lane ahead of the vehicle.

[0029] In some embodiments, the artificial intelligence (AI) processor 107 is a hardware processor for running a deep learning network 105. In some embodiments, the AI ​​processor 107 is a dedicated AI processor for performing inference on sensor data using a convolutional neural network (CNN). The AI ​​processor 107 may be optimized for the bit depth of the sensor data. In some embodiments, the AI ​​processor 107 is optimized for deep learning operations, such as neural network operations including convolution, dot product, vector and / or matrix operations, and other operations. In some embodiments, the AI ​​processor 107 is implemented using a graphics processing unit (GPU). In various embodiments, the AI ​​processor 107 is coupled to a memory configured to provide instructions to the AI ​​processor that, when executed, cause the AI ​​processor to perform deep learning analysis on received input sensor data and determine machine learning results for autonomous driving. In some embodiments, the AI ​​processor 107 is used to process sensor data to prepare the data for use as training data.

[0030] In some embodiments, the vehicle control module 109 is used to process the output of the artificial intelligence (AI) processor 107 and translate the output into vehicle control operations. In some embodiments, the vehicle control module 109 is used to control a vehicle for autonomous driving. In various embodiments, the vehicle control module 109 can adjust the vehicle's speed, acceleration, steering, braking, etc. For example, in some embodiments, the vehicle control module 109 is used to control the vehicle to maintain its position within the lane, merge the vehicle into another lane, or adjust the vehicle's speed and lane position to accommodate merging vehicles, etc.

[0031] In some embodiments, the vehicle control module 109 is used to control vehicle lighting, such as actuator lights, turn signals, headlights, etc. In some embodiments, the vehicle control module 109 is used to control vehicle audio conditions, such as the vehicle's sound system, playing audio warnings, enabling the microphone, enabling the horn, etc. In some embodiments, the vehicle control module 109 is used to control a notification system, including an alert system, to inform the driver and / or passengers of driving events such as a potential collision or the imminent arrival at the intended destination. In some embodiments, the vehicle control module 109 is used to adjust sensors, such as the vehicle's sensors 101. For example, the vehicle control module 109 may be used to change parameters of one or more sensors, such as modifying orientation, changing output resolution and / or format type, increasing or decreasing capture rate, adjusting the captured dynamic range, adjusting camera focus, enabling and / or disabling sensors, etc. In some embodiments, the vehicle control module 109 may be used to change parameters of the image preprocessor 103, such as modifying the frequency range of filters, adjusting feature and / or edge detection parameters, adjusting channel and bit depth, etc. In various embodiments, the vehicle control module 109 is used to implement autonomous driving and / or driver-assisted control of the vehicle. In some embodiments, the vehicle control module 109 is implemented using a processor coupled to memory. In some embodiments, the vehicle control module 109 is implemented using an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or other suitable processing hardware.

[0032] In some embodiments, network interface 111 is a communication interface for sending and / or receiving data (including voice data). In various embodiments, network interface 111 includes a cellular or wireless interface for interfacing with a remote server to: connect and make voice calls, send and / or receive short messages, transmit sensor data, receive updates to a deep learning network including an updated machine learning model, retrieve environmental conditions including weather conditions and weather forecasts, traffic conditions, etc. For example, network interface 111 can be used to receive updates to instructions and / or operating parameters for sensor 101, image preprocessor 103, deep learning network 105, AI processor 107, and / or vehicle control module 109. The machine learning model of deep learning network 105 can be updated using network interface 111. As another example, network interface 111 can be used to update the firmware of sensor 101 and / or operating parameters of image preprocessor 103, such as image processing parameters. As yet another example, network interface 111 can be used to transmit potential training data to a remote server for training a machine learning model.

[0033] Figure 2This is a flowchart illustrating an embodiment of a process for training and applying a machine learning model for autonomous driving. For example, input data, including sensor and mileage data, is received and processed to create training data for training the machine learning model. In some embodiments, the sensor data corresponds to image data captured via the autonomous driving system. In some embodiments, the sensor data corresponds to sensor data captured based on specific usage scenarios (such as a user manually disengaging from autonomous driving). In some embodiments, the process is used to create and deploy training data for autonomous driving. Figure 1 The deep learning system 100 machine learning model.

[0034] At point 201, training data is prepared. In some embodiments, sensor data, including image data and odometer data, is received to create a training dataset. Sensor data may include still images and / or video from one or more cameras. Additional sensors, such as radar, LiDAR, and ultrasound, may be used to provide relevant sensor data. In various embodiments, sensor data is paired with corresponding odometer data to help identify features of the sensor data. For example, changes in location and location data may be used to identify the location of relevant features (such as lane lines, traffic control signals, objects, etc.) in the sensor data. In some embodiments, the sensor data is a time series of elements and is used to determine ground truth. The ground truth of this set is then associated with a subset of the time series (such as the first frame of image data). Selected elements of the time series and ground truth are used to prepare training data. In some embodiments, training data is prepared to train a machine learning model to identify only features from the sensor data, such as lane lines, vehicle paths, traffic patterns, etc. The prepared training data may include data for training, validation, and testing. In various embodiments, sensor data may be in different formats. For example, sensor data may be still images, video, audio, etc. Mileage data may include vehicle operating parameters such as applied acceleration, applied braking, applied steering, vehicle position, vehicle orientation, changes in vehicle position, and changes in vehicle orientation. In various embodiments, training data is programmed and labeled to create a training dataset. In some embodiments, the preparation of the training data may be performed by a human programmer. In various embodiments, portions of the training data are automatically generated from data captured from the vehicle, significantly reducing the workload and time required to build a robust training dataset. In some embodiments, the data format is compatible with the machine learning model used in the deployed deep learning application. In various embodiments, the training data includes validation data for testing the accuracy of the trained model.

[0035] At point 203, a machine learning model is trained. For example, the machine learning model is trained using the data prepared at point 201. In some embodiments, the model is a neural network, such as a convolutional neural network (CNN). In various embodiments, the model includes multiple intermediate layers. In some embodiments, the neural network may include multiple layers, including multiple convolutional and pooling layers. In some embodiments, the trained model is validated using a validation dataset created from the received sensor data. In some embodiments, the machine learning model is trained to predict a 3D representation of features from a single input image. For example, a 3D representation of lane lines can be inferred from an image captured by a self-portrait camera. As another example, a predicted path, including whether adjacent vehicles will attempt to merge, can be predicted from an image captured by a self-portrait camera.

[0036] At point 205, the trained machine learning model is deployed. For example, the trained machine learning model is installed in a vehicle as a target for deep learning networks (such as...). Figure 1 Updates to the deep learning network 105. In some embodiments, over-the-air updates are used to install newly trained machine learning models. In some embodiments, the update is a firmware update transmitted using a wireless network such as WiFi or a cellular network. In some embodiments, the new machine learning model can be installed during vehicle maintenance.

[0037] At point 207, sensor data is received. For example, sensor data is captured from one or more sensors of the vehicle. In some embodiments, the sensor is... Figure 1 Sensor 101. The sensor may include an image sensor, such as a fisheye camera mounted behind the windshield, a forward or side camera mounted in a pillar, a rear-facing camera, etc. In various embodiments, sensor data is, or is converted into, a format used as input to a machine learning model trained at 203. For example, the sensor data may be raw or processed image data. In some embodiments, the data is data captured from an ultrasonic sensor, radar, LiDAR sensor, microphone, or other suitable technology. In some embodiments, an image preprocessor (such as...) is used during the preprocessing step. Figure 1 An image preprocessor 103 is used to preprocess sensor data. For example, the image can be normalized to remove distortion, noise, etc.

[0038] At point 209, the trained machine learning model is applied. For example, the machine learning model trained at point 203 is applied to the sensor data received at point 207. In some embodiments, this is accomplished by an AI processor (such as...) Figure 1 The AI ​​processor 107 uses deep learning networks (such as...) Figure 1A deep learning network 105 is used to perform the application of the model. In various embodiments, by applying the trained machine learning model, a three-dimensional representation of features (such as lane lines) is identified and / or predicted. For example, two splines representing the lane in which a vehicle is traveling are inferred. As another example, predicted paths of adjacent vehicles are inferred, including whether an adjacent vehicle is likely to merge into the current lane. In various embodiments, by applying the machine learning model, vehicles, obstacles, lanes, traffic control signals, map features, object distances, speed limits, drivable space, etc., are identified. In some embodiments, features are identified in three-dimensional space.

[0039] At 211, the autonomous vehicle is controlled. For example, one or more autonomous driving features are implemented by controlling various aspects of the vehicle. Examples may include controlling the vehicle's steering, speed, acceleration, and / or braking; maintaining the vehicle's position within its lane; maintaining the vehicle's position relative to other vehicles and / or obstacles; providing notifications or warnings to the user, etc. Based on the analysis performed at 209, the vehicle's steering and speed are controlled to keep the vehicle between two lane lines. For example, the left and right lane lines are predicted and correspond to lanes and drivable space. In various embodiments, the vehicle control module (such as...) Figure 1 The vehicle control module 109 controls the vehicle.

[0040] Figure 3 This is a flowchart illustrating an embodiment of a process for creating training data using time series of elements. For example, time series of elements, consisting of sensor and mileage data, are collected from a vehicle and used to automatically create training data. In various embodiments, Figure 3 The process is used to automatically label training data with corresponding ground truth values. Results corresponding to time series are associated with elements of the time series. Results and selected elements are packaged into training data to predict future results. In various embodiments, the process uses... Figure 1 Deep learning systems are used to capture sensor data and related data. For example, in various embodiments, sensor data is obtained from... Figure 1 Multiple sensors 101 are captured. In some embodiments, Figure 3 The process in Figure 2 At point 201, the operation is performed. In some embodiments, when the existing prediction is incorrect or can be improved, Figure 3 The process is performed to automatically collect data. For example, the autonomous vehicle makes a prediction to determine whether a vehicle is cutting into its path. After waiting for a period of time and analyzing the captured sensor data, a determination can be made as to whether the prediction is correct or incorrect. In some embodiments, the determination of the prediction can be improved. If the prediction is incorrect or can be improved, Figure 3The process can be applied to data related to prediction to create a sample set of warp-knitted strategies for improving machine learning models.

[0041] At 301, elements of the time series are received. In various embodiments, elements are sensor data, such as image data captured at the vehicle and transmitted to a training server. Sensor data is captured over a period of time to create a time series of elements. In various embodiments, elements are timestamps used to maintain the order of elements. As elements progress in the time series, further events in the time series are used to help predict outcomes from earlier elements in the time series. For example, the time series may capture vehicles in adjacent lanes that indicate merging, acceleration, and positioning themselves closer to nearby lane lines. Using the entire time series, the results can be used to determine if a vehicle will merge into a shared lane. This result can be used to predict if a vehicle will be merged based on selected elements of the time series (such as an image from an earlier image in the time series). As another example, the time series captures the curves of lane lines. The time series captures various dips, bends, ridges, etc., of a lane that are not apparent from a single element of the time series. In various embodiments, elements are sensor data in a format used as input by a machine learning model. For example, sensor data can be raw or processed image data. In some embodiments, the data is data captured from ultrasonic sensors, radar, LiDAR sensors, or other suitable technologies.

[0042] In various embodiments, the time series is organized by associating a timestamp with each element of the time series. For example, a timestamp is associated with at least a first element in the time series. The timestamp can be used to calibrate time series elements with related data such as mileage data. In various embodiments, the length of the time series can be a fixed length such as 10 seconds, 30 seconds, or another suitable length. The length of the time can be configurable. In various embodiments, the time series can be based on the vehicle's speed, such as the vehicle's average speed. For example, at slower speeds, the length of the time for the time series can be increased to capture data over a longer distance than might be traveled if a shorter time length were used at the same speed. In some embodiments, the number of elements in the time series is configurable. For example, the number of elements can be based on the distance traveled. For example, a faster-moving vehicle may include more elements in the time series than a slower-moving vehicle within a fixed time period. Additional elements increase the fidelity of the captured environment and can improve the accuracy of predicted machine learning results. In various embodiments, the number of elements is adjusted by adjusting the number of frames per second of sensor-captured data and / or by discarding unwanted intermediate frames.

[0043] At point 303, data associated with the elements of the time series is received. In various embodiments, the associated data, along with the elements received at point 301, is received at a training server. In some embodiments, the associated data is vehicle mileage data. Location data of features identified in the elements of the time series can be labeled using position, orientation, changes in position, changes in orientation, and / or other associated vehicle data. For example, lane lines can be labeled with very accurate location by examining the time series of lane line elements. Typically, the lane lines closest to the vehicle camera are accurate and closely associated with the vehicle's location. In contrast, the XYZ location of the lines furthest from the vehicle is difficult to determine. Distant portions of lane lines may be occluded (e.g., behind curves or slopes) and / or difficult to capture accurately (e.g., due to distance or lighting conditions). The data associated with the elements is used to label portions of the identified features in the time series with high accuracy. In various embodiments, a threshold is used to determine whether to associate the identified portions of features (such as portions of lane lines) with the associated data. For example, portions of lane lines identified with high certainty (such as those near vehicles) are associated with relevant data, while portions of lane lines identified with less than a threshold certainty (such as those farther from vehicles) are not associated with relevant data for that element. Instead, another element of the time series with higher certainty (such as subsequent elements) and its relevant data are used. In some embodiments, the relevant data is the output of a neural network, such as... Figure 1 The output of the deep learning network 105. In some embodiments, the relevant data is from a vehicle control module (such as...) Figure 1 The output of the vehicle control module 109. Related data may include vehicle operating parameters such as speed, speed changes, acceleration, acceleration changes, steering, steering changes, braking, braking changes, etc. In some embodiments, the related data is radar data used to estimate the distance to objects such as obstacles.

[0044] In some embodiments, the data associated with the elements of the time series includes map data. For example, at 303, offline data such as road and / or satellite horizontal map data is received. Map data can be used to identify features such as roads, lanes, intersections, speed limits, school zones, etc. For example, map data can describe the path of a lane. As another example, map data can describe speed limits associated with different roads on the map.

[0045] In various embodiments, data related to elements of a time series is organized by associating timestamps with related data. Corresponding timestamps from the time series elements and the related data can be used to synchronize the two datasets. In some embodiments, the data is synchronized at the capture time. For example, when each element of the time series is captured, the corresponding set of related data is captured and saved along with the time series element. In various embodiments, the time period of the related data is configurable and / or matches the time period of the element's time series. In some embodiments, the related data is sampled at the same rate as the time series element.

[0046] At 305, ground truth is determined for the time series. In various embodiments, the time series is analyzed to determine ground truth associated with machine learning features. For example, lane lines are identified from the time series corresponding to the ground truth of that lane line. As another example, the ground truth of the path of a moving object (such as a vehicle, pedestrian, cyclist, animal, etc.) is the path identified from the time series for the detected moving object. In some embodiments, a moving vehicle is labeled as an intervening vehicle if it enters the lane of an autonomous vehicle within the time series. In some embodiments, ground truth is represented as a three-dimensional representation, such as a three-dimensional trajectory. For example, the ground truth associated with a lane line can be represented as a three-dimensional parametric spline or curve. As another example, the predicted path of a detected vehicle is determined and represented as a three-dimensional trajectory. The predicted path can be used to determine whether a vehicle has merged into an occupied space. In various embodiments, ground truth can only be determined by examining the time series of elements. For example, analyzing only a subset of the time series may leave partially occluded lane lines. By expanding the analysis across the time series of elements, the occluded portions of the lane lines are revealed. Furthermore, captured data towards the end of the time series more accurately captures (e.g., with higher fidelity) details of further sections of lane lines at a distance. Additionally, the correlated data is more accurate because it is based on data captured closer (both in distance and time). In various embodiments, simultaneous localization and mapping techniques are applied to different portions of the detected object (such as lane lines) identified in different elements of the time series to map different portions of the object to precise three-dimensional locations including elevation. The set of mapped three-dimensional locations represents the ground truth of the object, such as segments of lane lines captured within the time series. In some embodiments, the localization and mapping techniques produce a precise set of points, such as a set of points corresponding to different points along the lane line. This set of points can be converted into a more efficient format, such as splines or parametric curves. In some embodiments, the ground truth is determined to detect objects in three-dimensional space, such as lane lines, drivable spaces, traffic controls, vehicles, etc.

[0047] In some embodiments, ground truth is determined to predict semantic labels. For example, a detected vehicle may be labeled as being in the left or right lane. In some embodiments, a detected vehicle may be labeled as being in a blind spot, as a vehicle to be yielded to, or as having another appropriate semantic label. In some embodiments, based on the determined ground truth, vehicles are assigned to roads or lanes in the map. As an additional example, the determined ground truth may be used to label traffic lights, lanes, drivable space, or other features that assist autonomous driving.

[0048] In some embodiments, the relevant data is depth (or distance) data of the detected objects. By associating distance data with objects identified in the time series of the elements, a machine learning model can be trained to estimate target distances using the relevant distance data as ground truth for the detected objects. In some embodiments, the distance is for detected objects such as obstacles, barriers, moving vehicles, stationary vehicles, traffic control signals, pedestrians, etc.

[0049] At 307, training data is encapsulated. For example, elements of the time series are selected and associated with ground truth determined at 305. In various embodiments, the selected elements are earlier elements in the time series. The selected elements represent sensor data input to the machine learning model, and the ground truth represents the prediction result. In various embodiments, training data is encapsulated and prepared as training data. In some embodiments, the training data is encapsulated as training, validation, and testing data. Based on the determined ground truth and the selected elements of the time series, training data can be encapsulated to train a machine learning model to identify lane lines, predicted vehicle paths, speed limits, vehicle insertions, object distances, and / or drivable space, as well as other useful features for autonomous driving. The encapsulated training data can then be used to train the machine learning model.

[0050] Figure 4 A flowchart illustrating an embodiment of the process for training and applying a machine learning model for autonomous driving. In some embodiments, using Figure 4 The process involves collecting and retaining sensor and mileage data for training machine learning models for autonomous driving. In some embodiments, autonomous driving control is implemented on vehicles with autonomous driving enabled. Figure 4 The process. For example, sensor and mileage data can be collected immediately after disengaging from autonomous driving, while the vehicle is driven by a human driver, and / or simultaneously in autonomous driving mode. In some embodiments, using Figure 1 Deep learning systems achieve this through Figure 4 The described technology. In some embodiments, Figure 4 Part of the process in Figure 2At positions 207, 209, and / or 211, the process of applying a machine learning model for autonomous driving is executed.

[0051] At 401, sensor data is received. For example, a vehicle equipped with sensors captures sensor data and provides it to a neural network running on the vehicle. In some embodiments, the sensor data may be visual data, ultrasonic data, LiDAR data, or other suitable sensor data. For example, images are captured from a high dynamic range forward-facing camera. As another example, ultrasonic data is captured from a lateral ultrasonic sensor. In some embodiments, the vehicle is equipped with multiple sensors for capturing data. For example, in some embodiments, eight ambient cameras are fixed to the vehicle and provide 360-degree visibility around the vehicle with a range of up to 250 meters. In some embodiments, the camera sensors include a wide forward-facing camera, a narrow forward-facing camera, a rear-view camera, a forward-view side camera, and / or a rear-view side camera. In some embodiments, ultrasonic and / or radar sensors are used to capture ambient details. For example, twelve ultrasonic sensors may be fixed to the vehicle to detect both hard and soft objects. In some embodiments, forward-facing radar is utilized to capture data about the surrounding environment. In various embodiments, radar sensors are able to capture the surrounding situation even in the presence of heavy rain, fog, dust, and other vehicles. Various sensors are used to capture the environment around the vehicle, and the captured data is provided for deep learning analysis.

[0052] In some embodiments, sensor data includes mileage data, which includes the vehicle's position, orientation, changes in position, and / or changes in orientation. For example, position data is captured and associated with other sensor data captured during the same time frame. As an example, position data captured while capturing image data is used to associate position information with the image data.

[0053] At 403, the sensor data is preprocessed. In some embodiments, one or more preprocessing passes may be performed on the sensor data. For example, the data may be preprocessed to remove noise, correct alignment problems, and / or blur, etc. In some embodiments, one or more different filtering passes may be performed on the data. For example, high-pass filtering may be performed on the data and low-pass filtering may be performed on the data to separate the different components of the sensor data. In various embodiments, the preprocessing step performed at 403 is optional and / or may be incorporated into the neural network.

[0054] At point 405, deep learning analysis of the sensor data is initiated. In some embodiments, deep learning analysis is performed on the sensor data that has been optionally preprocessed at point 403. In various embodiments, neural networks (such as convolutional neural networks (CNNs)) are used to perform the deep learning analysis. In various embodiments, the machine learning model is used... Figure 2 The process involves offline training and deployment on vehicles to perform inferences based on sensor data. For example, the model can be trained to appropriately identify road lane lines, obstacles, pedestrians, moving vehicles, parked vehicles, drivable spaces, etc. In some embodiments, multiple trajectories of lane lines are identified. For example, several potential trajectories of lane lines are detected, each with a corresponding probability of occurrence. In some embodiments, the predicted lane line is the one with the highest probability of occurrence and / or the highest associated confidence value. In some embodiments, the predicted lane line from deep learning analytics is required to exceed a minimum confidence threshold. In various embodiments, the neural network includes multiple layers, including one or more intermediate layers. In various embodiments, sensor data and / or the results of deep learning analytics are retained and transmitted at 411 for use in automatically generating training data.

[0055] In various embodiments, deep learning analytics are used to predict additional features. These predicted features can be used to assist autonomous driving. For example, detected vehicles can be assigned to lanes or roads. As another example, it can be determined whether a detected vehicle is in a blind spot, is a vehicle that should be yielded to, is a vehicle in the adjacent left lane, is a vehicle in the adjacent right lane, or has another suitable attribute. Similarly, deep learning analytics can identify traffic lights, drivable space, pedestrians, obstacles, or other suitable features for driving.

[0056] At 407, the results of the deep learning analysis are provided to the vehicle control. For example, the results are provided to the vehicle control module to control the vehicle for autonomous driving and / or to achieve autonomous driving functionality. In some embodiments, the results of the deep learning analysis at 405 are passed through one or more additional deep learning iterations using one or more different machine learning models. For example, the predicted path of lane lines may be used to determine lanes, and the determined lanes are used to determine drivable space. The drivable space is then used to determine the vehicle's path. Similarly, in some embodiments, predicted vehicle insertions are detected. The determined path of the vehicle takes into account the predicted insertions to avoid potential collisions. In some embodiments, various outputs of the deep learning are used to construct a three-dimensional representation of the vehicle environment for autonomous driving, which includes the predicted path of the vehicle, identified obstacles, traffic control signals including speed limit indicators, etc. In some embodiments, the vehicle control module uses the determined results to control the vehicle along the determined path. In some embodiments, the vehicle control module is Figure 1 The vehicle control module 109.

[0057] At point 409, the vehicle is controlled. In some embodiments, a vehicle control module (such as...) is used. Figure 1The vehicle control module 109 controls the vehicle to activate autonomous driving. Vehicle control can adjust the vehicle's speed and / or steering, for example, to maintain the vehicle at an appropriate speed in the lane, taking into account its surrounding environment. In some embodiments, the results are used to adjust the vehicle when an adjacent vehicle is expected to merge into the same lane. In various embodiments, using the results of deep learning analysis, the vehicle control module determines the appropriate way to operate the vehicle, for example, at an appropriate speed along a determined path. In various embodiments, the results of vehicle control (such as changes in speed, application of braking, adjustments to steering, etc.) are retained and used to automatically generate training data. In various embodiments, vehicle control parameters are retained and transmitted at 411 for automatically generating training data.

[0058] At 411, sensor data and related data are transmitted. For example, sensor data received at 401, along with the results of deep learning analysis at 405 and / or vehicle control parameters used at 409, are transmitted to a computer server for the automatic generation of training data. In some embodiments, the data is a time series, and various collected data are correlated together via the computer server. For example, mileage data is correlated with captured image data to generate ground truth. In various embodiments, the collected data is wirelessly transmitted from the vehicle to a training data center, for example via WiFi or a cellular connection. In some embodiments, metadata is transmitted along with the sensor data. For example, metadata may include the time of day, timestamp, location, vehicle type, vehicle control and / or operating parameters such as speed, acceleration, braking, whether autonomous driving is enabled, steering angle, mileage data, etc. Additional metadata includes the time since the last previous sensor data was transmitted, vehicle type, weather conditions, road conditions, etc. In some embodiments, the transmitted data is anonymized, for example by removing the vehicle's unique identifier. As another example, data from similar vehicle models is merged to prevent the identification of individual users and their vehicle usage.

[0059] In some embodiments, data is transmitted only in response to triggers. For example, in some embodiments, an incorrect prediction triggers the transmission of sensor data and related data to automatically collect data to create an example set of warp-programmed policies for improving predictions of deep learning networks. For example, a prediction performed at 405 related to whether the vehicle attempted to merge is determined to be incorrect by comparing the prediction with the observed actual result. Data associated with the incorrect prediction, including sensor data and related data, is then transmitted and used to automatically generate training data. In some embodiments, triggers can be used to identify specific scenarios, such as sharp turns, road junctions, lane merging, sudden stops, or another suitable scenario where additional training data is useful but may be difficult to collect. For example, triggers may be based on sudden deactivation or disengagement of autonomous driving characteristics. As another example, vehicle operating properties, such as changes in speed or acceleration, can form the basis of triggers. In some embodiments, predictions with accuracy below a certain threshold trigger the transmission of sensor data and related data. For example, in some scenarios, predictions may not have a Boolean correct or incorrect result and are instead evaluated by determining a value for the accuracy of the prediction.

[0060] In various embodiments, sensor and related data are captured within a time period, and the entire time series of the data is transmitted together. The time period can be configured and / or based on one or more factors, such as vehicle speed, distance traveled, speed variations, etc. In some embodiments, the sampling rate of the captured sensor data and / or related data is configurable. For example, the sampling rate increases at higher speeds, during sudden braking, during sudden acceleration, during sharp turns, or in another suitable scenario where additional fidelity is required.

[0061] Figure 5 This is an illustration of an example of an image captured from a vehicle's sensors. In the example shown, Figure 5 The image includes image data 500 captured from a vehicle traveling in a lane between two lane lines. The location of the vehicle and sensor used to capture the image data 500 is indicated by label A. Image data 500 is sensor data and can be captured from a camera sensor, such as a forward-facing camera of the vehicle while driving. Image data 500 captures portions of lane lines 501 and 511. As lane lines 501 and 511 approach the horizon, they curve to the right. In the example shown, lane lines 501 and 511 are visible but become increasingly difficult to detect as they curve away from the location of the camera sensor. A white line drawn on top of lane lines 501 and 511 approaches the detectable portions of lane lines 501 and 511 from the image data 500 without any additional input. In some embodiments, the detectable portions of lane lines 501 and 511 can be detected by segmenting the image data 500.

[0062] In some embodiments, markers A, B, and C correspond to different locations on the road and to different times in the time series. Marker A corresponds to the time and location of the vehicle when image data 500 is captured. Marker B corresponds to the location on the road before marker A and the time after the time of marker A. Similarly, marker C corresponds to the location on the road before marker B and the time after the time of marker B. As the vehicle travels, a time series of sensor data and related data is captured, passing through the locations of markers A, B, and C (from marker A to marker C) and while traveling. The time series includes elements captured at the locations (and times) of markers A, B, and C. Marker A corresponds to the first element of the time series, marker B corresponds to the middle element of the time series, and marker C corresponds to the middle (or potentially the last) element of the time series. At each marker, additional data, such as vehicle mileage data, is captured at the marker location. Depending on the length of the time series, additional data or less data is captured. In some embodiments, a timestamp is associated with each element of the time series.

[0063] In some embodiments, ground truth values ​​(not shown) for lane lines 501 and 511 are determined. For example, using the process disclosed herein, the locations of lane lines 501 and 511 are identified by identifying different portions of lane lines 501 and 511 from different elements of the time series of the elements. In the example shown, portions 503 and 513 are identified using image data 500 and related data (such as mileage data) acquired at location A at time. Portions 505 and 515 are identified using image data (not shown) and related data (such as mileage data) acquired at location B at time. Portions 507 and 517 are identified using image data (not shown) and related data (such as mileage data) acquired at location C at time. By analyzing the time series of the elements, the locations of different portions of lane lines 501 and 511 are identified, and ground truth values ​​can be determined by combining the different identified portions. In some embodiments, a portion is identified as a point along each section of the lane line. In the example shown, only three portions of each lane line (parts 503, 505, and 507 of lane line 501 and parts 513, 515, and 517 of lane line 511) are highlighted to explain the process, but additional portions can be captured in the time series to determine the position of the lane lines at higher resolution and / or with greater accuracy.

[0064] In various embodiments, the locations of captured lane lines 501 and 511 in the image data closest to the sensor's location are determined with high accuracy. For example, the locations of portions 503 and 513 are identified with high accuracy using image data 500 labeled A and related data (such as mileage data). The locations of portions 505 and 515 are identified with high accuracy using images labeled B and related data. The locations of portions 507 and 517 are identified with high accuracy using images labeled C and related data. By utilizing time series of elements, the locations of various portions of lane lines 501 and 511 captured by the time series can be identified in three dimensions with high accuracy and used as the basis for ground truth values ​​of lane lines 501 and 511. In various embodiments, the determined ground truth values ​​are associated with selected elements of the time series (such as image data 500). The ground truth values ​​and selected elements can be used to create training data for predicting lane lines. In some embodiments, the training data is created automatically and without human labeling. Training data can be used to train machine learning models to predict the three-dimensional trajectory of lane lines from captured image data (such as image data 500).

[0065] Figure 6 This is an illustration of an example of predicting a 3D trajectory from an image captured by a vehicle's sensors, showing lane markings. In the example shown, Figure 6 The image includes image data 600 captured from a vehicle traveling in a lane between two lane lines. The positions of the vehicle and sensors used to capture the image data 600 are indicated by label A. In some embodiments, label A corresponds to... Figure 5 The location of marker A is the same as the location of the image data 600. Image data 600 is sensor data and can be captured from a camera sensor, such as a forward-facing camera of a vehicle while driving. Image data 600 captures portions of lane lines 601 and 611. As lane lines 601 and 611 approach the horizon, they curve to the right. In the example shown, lane lines 601 and 611 are visible but become increasingly difficult to detect because the lane lines curve away from the location of the camera sensor and recede into the distance. The red line drawn on top of lane lines 601 and 611 is the predicted 3D trajectory of lane lines 601 and 611. Using the process disclosed herein, image data 600 is used as input to a trained machine learning model to predict the 3D trajectory. In some embodiments, the predicted 3D trajectory is represented as a 3D parameterized spline or another parameterized form of the representation.

[0066] In the example shown, portion 621 of lane lines 601 and 611 is the distant portion of lane lines 601 and 611. The three-dimensional location (i.e., longitude, latitude, and altitude) of portion 621 of lane lines 601 and 611 is determined with high accuracy using the process disclosed herein and is included in the predicted three-dimensional trajectory of lane lines 601 and 611. Using a trained machine learning model, the three-dimensional trajectory of lane lines 601 and 611 can be predicted using image data 600 without requiring location data at the location of portion 621 of lane lines 601 and 611. In the example shown, image data 600 is captured at the location and time of marker A.

[0067] In some embodiments, Figure 6 The label A corresponds to Figure 5 The predicted 3D trajectories of lane lines 601 and 611, labeled A, were determined using only image data 600 as input to a trained machine learning model. By training the machine learning model using ground truth, the 3D trajectories of lane lines 601 and 611 were predicted with high accuracy, even for distant portions of the lane lines (such as portion 621). This ground truth was determined using time-series images and related data, including... Figure 5 The elements obtained at the positions marked A, B, and C. Although the image data is 600 and... Figure 5 Image data 500 is relevant, but trajectory prediction does not require image data 600 to be included in the training data. By training on sufficient training data, lane lines can even be predicted for newly encountered scenarios. In various embodiments, the predicted 3D trajectories of lane lines 601 and 611 are used to maintain the localization of vehicles within the detected lane lines and / or to autonomously navigate vehicles along the detected lanes along the predicted lane lines. By predicting 3D lane lines, the performance, safety, and accuracy of navigation are significantly improved.

[0068] Although the foregoing embodiments have been described in detail for clarity, the invention is not limited to the details provided. Many alternative ways of implementing the invention exist. The disclosed embodiments are illustrative and not restrictive.

Claims

1. A method comprising: Sensor data associated with time points within a time period is obtained by one or more processors; The one or more processors determine ground truth values ​​based on the sensor data, and the ground truth values ​​are associated with a three-dimensional representation of the features; as well as The machine learning model is trained by the one or more processors using a training dataset, which includes a portion of the sensor data captured at time points within the said time period that are associated with the three-dimensional representation of the feature. The machine learning model is trained to output an identifier of the three-dimensional representation of the feature based on input from sensor data associated with time points within different time periods.

2. The method of claim 1, wherein the three-dimensional representation of the feature is associated with lane lines.

3. The method of claim 2, wherein the sensor data comprises a plurality of images captured at the time points within the said time period, and The lane lines are depicted by a set of images from the plurality of images.

4. The method of claim 3, wherein images from the set of images depicting the lane lines are selected for training the machine learning model based on measurements associated with the relevance of the images with respect to the remaining images in the set of images depicting the portion of the lane lines.

5. The method of claim 2, wherein the three-dimensional representation of the feature reflects the trajectory of the lane line.

6. The method of claim 1, wherein the three-dimensional representation of the feature reflects the path associated with the vehicle.

7. The method of claim 6, wherein the sensor data comprises a plurality of images captured at the time points within the said time period, and The machine learning model is trained to output the identifier of the path based on individual images of the second vehicle separated from the plurality of images.

8. The method of claim 6, wherein at the time when the sensor data is captured by a sensor of a different vehicle located in the second lane, the vehicle is in a first lane adjacent to the second lane.

9. The method of claim 1, wherein the training dataset further comprises scenario data describing the real-world environment surrounding the sensors of the vehicle that capture the sensor data.

10. A system comprising one or more processors, said one or more processors being configured to: Obtain sensor data associated with time points within a given period; Ground truth values ​​are determined based on the sensor data, and these ground truth values ​​are correlated with a three-dimensional representation of the features; and A machine learning model is trained using a training dataset, which includes a portion of the sensor data captured at time points within the said time period that are associated with the three-dimensional representation of the feature. The machine learning model is trained to output an identifier of the three-dimensional representation of the feature based on input from sensor data associated with time points within different time periods.

11. The system of claim 10, wherein the three-dimensional representation of the feature is associated with lane lines.

12. The system of claim 11, wherein the sensor data comprises a plurality of images captured at the time points within the said time period, and The lane lines are depicted by a set of images from the plurality of images.

13. The system of claim 12, wherein images from the set of images depicting the lane lines are selected for training the machine learning model based on measurements associated with the correlation of the images with respect to the remaining images in the set of images depicting the portion of the lane lines.

14. The system of claim 11, wherein the three-dimensional representation of the feature reflects the trajectory of the lane line.

15. The system of claim 10, wherein the three-dimensional representation of the feature reflects the path associated with the vehicle.

16. The system of claim 15, wherein the sensor data comprises a plurality of images captured at the time points within the said time period, and The machine learning model is trained to output the identifier of the path based on individual images of the second vehicle separated from the plurality of images.

17. The system of claim 15, wherein at the time when the sensor data is captured by a sensor of a different vehicle located in the second lane, the vehicle is in a first lane adjacent to the second lane.

18. The system of claim 10, wherein the training dataset further comprises scenario data describing the real-world environment surrounding the sensors of the vehicle that capture the sensor data.

19. A non-transitory computer-readable storage medium comprising computer instructions that, when executed by one or more processors, cause the one or more processors to: Obtain sensor data associated with time points within a given period; Ground truth values ​​are determined based on the sensor data, and these ground truth values ​​are correlated with a three-dimensional representation of the features; and A machine learning model is trained using a training dataset, which includes a portion of the sensor data captured at time points within the said time period that are associated with the three-dimensional representation of the feature. The machine learning model is trained to output an identifier of the three-dimensional representation of the feature based on input from sensor data associated with time points within different time periods.

20. The non-transient computer-readable storage medium of claim 19, wherein the three-dimensional representation of the feature is associated with a lane line.

Images