Generating ground truth for machine learning from time series elements

Abstract

It provides an improved process for generating training data with accurately labeled features. [Solution] Sensor data containing a group of time series elements is received. A training dataset is determined by determining the corresponding ground truth for at least selected time series elements within the group of time series elements. The corresponding ground truth is based on multiple time series elements within the group of time series elements. A processor is used to train a machine learning model using the training dataset.

Description

Background Art

[0001] [Cross - reference to Related Applications] This application is a continuation of U.S. Patent Application No. 16 / 265729, filed on February 1, 2019, entitled "GENERATING GROUND TRUTH FOR MACHINE LEARNING FROM TIME SE RIES ELEMENTS", claims the priority thereof, and the entire disclosure thereof is incorporated herein by reference.

Summary of the Invention

Problems to be Solved by the Invention

[0002]

Brief Description of the Drawings

[0003] ​​​​​​​​​​​​​​​​​​​Various embodiments of the present invention are disclosed in the following detailed description and accompanying drawings.

[0004] [Figure 1] A block diagram showing one embodiment of a deep learning system for autonomous driving.

[0005] [Figure 2] This flowchart illustrates one embodiment of the process for training and applying machine learning models for autonomous driving.

[0006] [Figure 3] This flowchart illustrates one embodiment of the process of creating training data using time-series elements.

[0007] [Figure 4] This flowchart illustrates one embodiment of the process for training and applying machine learning models for autonomous driving.

[0008] [Figure 5] This figure shows an example of an image captured by a vehicle sensor.

[0009] [Figure 6] This figure shows an example of an image captured by a vehicle sensor that has a predicted 3D trajectory of the lane line. [Modes for carrying out the invention]

[0010] The present invention relates to processes, apparatus, systems, material compositions, and computer-readable storage media. Implemented computer program products, and / or processors, e.g., processors Instructions stored in memory connected to the system and / or instructions provided by memory This can be implemented in various ways, including a processor configured to run it. Therefore, these implementation forms, or any other forms that the present invention may take, can be referred to as the technology. This is possible. Generally, the order of the steps of the disclosed process can be changed within the scope of the present invention. Unless otherwise specified, components such as processors or memories described as being configured to perform tasks are general components that are temporarily configured to perform tasks at a given time, or may be implemented as specific components manufactured to perform tasks. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data such as computer program instructions. A detailed description of one or more embodiments of the present invention is provided below together with the accompanying drawings showing the principles of the present invention. The present invention is described in relation to such embodiments, but the present invention is not limited to any particular embodiment. The scope of the present invention is limited only by the claims, and the present invention includes numerous alternative forms, variations, and equivalents. To provide a complete understanding of the present invention, numerous specific details are set forth in the following description. These details are provided for illustrative purposes only, and the present invention may be practiced without some or all of these specific details in accordance with the claims. Technical material known in the technical field related to the present invention is not described in detail so as not to unnecessarily obscure the present invention. Machine learning training techniques for generating highly accurate machine learning results are disclosed. Data captured by sensors on a vehicle is used to capture the vehicle's environment and vehicle operation parameters.

[0011]

[0012] is used to create a training dataset. For example, sensors attached to a vehicle capture data such as image data of the road and surrounding environment on which the vehicle is traveling. Sensor data can capture vehicle lane lines, vehicle lanes, other vehicle traffic, obstacles, traffic control signs, etc. Odometry and other similar sensors capture vehicle motion parameters such as speed, steering, orientation, change in direction, change in position, change in altitude, change in speed, etc. The captured dataset is sent to a training server to create a training dataset. The training dataset is used to train a machine learning model that generates highly accurate machine learning results. In some embodiments, time-series captured data is used to generate training data. For example, ground truth is determined based on a group of time-series elements and is associated with a single element from the group. As an example, a series of images over a period such as 30 seconds are used to determine the actual path of the vehicle lane line over the period during which the vehicle is traveling. The vehicle lane line is determined by using the most accurate image of the vehicle lane over that period. Different parts (or positions) of the lane line can be identified from different image data in the time series. As the vehicle travels along the lane line, more accurate data is captured for different parts of the lane line. In some examples, blocked portions of the lane appear when the vehicle is traveling along, for example, a hidden curve or over the crest of a hill. The most accurate parts of the lane line from each image in the time series can be used to identify the lane line over the entire group of image data. Image data of the far lane line ​​​​​​​​​​ The image data is usually less detailed than the image data of the lane lines near the vehicle. By capturing time-series image data as the vehicle travels along the lane, the corresponding lane line can be identified. Accurate image data and corresponding odometry data are collected for all parts of the object.

[0013] In some embodiments, the three-dimensional representation of features such as lane lines is ground truth It is created from a group of time-series elements corresponding to the ground true. S is a subset of time-series elements such as a single image frame from a group of captured image data. It is associated with a set. For example, the first image in an image group is represented in three-dimensional space. It is associated with the ground truth of the lane line. The ground truth is the image It is determined based on the group, but the selected first frame and ground true The data is used to create training data. For example, using only a single image of a car Training data is created to predict the 3D representation of both lanes. In some embodiments, This means that any element or group of elements in a time-series element group is in ground truth. It is associated with and used to create training data. For example, ground truth This can be applied to the entire video sequence used to create training data. As another example, time The middle or last element of a group of sequential elements is associated with ground truth. This is used to create training data.

[0014] In various embodiments, the selected image and ground truth are lane lines, Applicable to various features such as vehicle path prediction including adjacent vehicles, object depth, and traffic signs. For example, a series of images of vehicles in adjacent lanes can be used to predict the vehicle's path. It is used to capture time-series images taken by adjacent vehicles and the actual route. Then, a single image of the captured group and the actual route is used to predict the vehicle's path. This can be used as training data for an autonomous vehicle. Using this information, an adjacent vehicle can be trained to be an autonomous vehicle. It is also possible to predict whether or not an adjacent vehicle will cut in on the route. For example, route prediction can predict whether or not an adjacent vehicle will cut in on the route. It can predict whether an autonomous vehicle will merge in front of it. The autonomous vehicle can predict the possibility of a collision. It can be controlled to minimize the impact. For example, autonomous vehicles can prevent collisions. Slow down, adjust the vehicle's speed and / or steering to prevent collisions with adjacent vehicles and / or Or, for example, initiating a warning to the occupants of an autonomous vehicle and / or changing lanes. Yes, it is possible. In various embodiments, the ability to accurately infer route predictions, including vehicle route prediction, is possible. This will significantly improve the safety of conventional vehicles.

[0015] In some embodiments, a trained (pre-trained) machine learning model determines the lane lines. It is used to predict a three-dimensional representation of one or more features for autonomous driving. For example, by segmenting an image of lane lines, two-dimensional lines can be obtained from image data. Instead of identifying the line, the elements of the time series and the odometry data corresponding to the time series. A 3D representation is generated using the following: The 3D representation is generated using lane line detection and corresponding This includes advanced changes that significantly improve the accuracy of lane and identified drivable path detection. In some embodiments, the lane line is one or more splines or another para It is represented using a metered representation format. A piecewise polynomial is used to represent the lane lines. Using this formula significantly reduces the computational resources required to evaluate three-dimensional objects. This reduction in computing resources does not significantly sacrifice the precision of representation, and improves processing speed and To address the improvement of efficiency. In various embodiments, lane lines, in particular, include curves of the lane lines. The expression can be represented using a piecewise polynomial, a 3D set of points, or another suitable representation. For example, a piecewise polynomial is a group of elements captured over time using sensor data. The actual lane lines are interpolated using high-precision sections of the lane lines identified from the data. do.

[0016] In some embodiments, sensor data is received. The sensor data includes images (videos) (and / or still images, etc.), radar, audio, LiDAR, inertial, odometry, position It may include positional and / or other forms of sensor data. The sensor data is time-series Includes groups of column elements. For example, a group of time series elements could represent data on vehicles over a certain period of time. It can include a group of images captured from a camera sensor. In some embodiments, The training dataset contains at least one selected time series element within a group of time series elements. Therefore, based on multiple time series elements within a group of time series elements, the corresponding ground to It is determined by determining the looseness, for example, the ground truth is The most relevant element for each element in a group of time series elements that includes the preceding and / or succeeding time series elements within the loop. This is determined by examining the related parts. In some scenarios, the preceding and / or The car is one in which only the later time-series elements disappear first around the curve and only appear in the later time-series elements. Includes data not present in previous time-series elements, such as both lane lines. Determined ground Dotrus provides a 3D representation of vehicle lane lines, predicts vehicle paths, or other similar predictions. It is possible to measure. An element of a group of time-series elements is selected and associated with ground truth. The selected elements and ground truth are part of the training dataset. In some embodiments, the processor uses the training dataset to create a machine learning model. It is used to train the system. For example, the training dataset is used for autonomous driving or driving of vehicles. Used to train machine learning models for inferring features used in assistive behaviors. Using a trained machine learning model, the neural network determines the vehicle lane and driving. Movable space, objects (e.g., pedestrians, stationary vehicles, moving vehicles, etc.), weather (e.g., rain, hail) (e.g., fog), traffic control objects (e.g., traffic lights, traffic signs, road signs), traffic patterns Features related to autonomous driving, such as these, can be inferred.

[0017] In some embodiments, the system includes a processor and memory coupled to the processor. The system includes the following: The processor processes image data based on images captured by the vehicle's cameras. It is configured to receive. For example, a camera sensor mounted on a vehicle receives the vehicle's environment Capture images of the subject. The camera can be a front-facing camera, a pillar camera, or another appropriately positioned camera. It may also be a camera. The image data captured from the camera is processed by the vehicle's GPU or A It is processed using a processor such as an I processor. The data is sent to a trained machine learning model that has been trained to predict the 3D trajectory of the vehicle lane. It is used as the basis for input. For example, image data is used to train a system to predict vehicle lanes. It is used as input to the neural network. The machine learning model detects Infer the 3D trajectory of the lane. The image shows the lane and non-lane segments of the 2D image. Instead of segmenting, a 3D representation is inferred. In some embodiments, 3D The original representation can describe a curve using a spline, parametric curve, or in 3D. This is another way of expressing it. In some embodiments, the three-dimensional trajectory of the vehicle lane is used to automatically guide the vehicle. Provided when controlling. For example, a 3D trajectory is provided for the lane lines and the corresponding drivability. It is used to determine the power space.

[0018] Figure 1 is a block diagram showing one embodiment of a deep learning system for autonomous driving. The layered learning system is used for the autonomous driving and / or driver assistance operations of the vehicle, as well as for autonomous driving. To collect and process data for training machine learning models for dynamic driving, It includes various components that can be used together. In various embodiments, deep learning systems The device is installed in the vehicle. Data from the vehicle is used for the autonomous driving functions of the vehicle or other similar vehicles. It can be used to train and improve.

[0019] In the illustrated example, the deep learning system 100 includes a sensor 101 and an image preprocessor 103. , deep learning network 105, artificial intelligence (AI) processor 107, vehicle control module A deep learning network including L109 and network interface 111. In various embodiments, different components are connected in a communicative manner. For example, sensors Sensor data from 101 is supplied to the image preprocessor 103. The sensor data processed by the sensor 103 is used by a deep learning processor running on the AI ​​processor 107. It is supplied to network 105. A deep learning network running on AI processor 107. The output of 105 is supplied to the vehicle control module 109. In various embodiments, the vehicle The control module 109 is connected to the vehicle's operation, such as vehicle speed, braking, and / or steering. This is followed by controlling the vehicle's operation. In various embodiments, sensor data and / or mechanical data are used. The learning results are sent to the remote server via the network interface 111. This can be done. For example, training to improve the performance, comfort, and / or safety of vehicles. To collect data, sensor data is transmitted via network interface 111. It can be sent to a remote server. In various embodiments, the network interface Among other reasons, Face 111 communicates with remote servers and makes phone calls. Send and / or receive text messages and collect sensor data based on vehicle operation. Used for transmission. In some embodiments, the deep learning system 100 is required Depending on the implementation, it may include additional or fewer components. For example, several implementations In terms of form, the image preprocessor 103 is an optional component. As another example, In one embodiment, before the output is provided to the vehicle control module 109, a deep learning network is used. Post-processing components (not shown) are used to perform post-processing on the output of the twerk 105. It is used.

[0020] In some embodiments, the sensor 101 includes one or more sensors. In the configuration, the sensor 101 is mounted on the vehicle at different locations on the vehicle, and / or They may be oriented in one or more different directions. For example, sensor 101 may be oriented in front of the vehicle. , on the sides, rear, and / or roof, etc., facing forward, backward, sideways, etc. It may be attached. In some embodiments, the sensor 101 has a high dynamic range. It may also be an image sensor such as a camera. In some embodiments, the sensor 101 is non-visual. Includes sensors. In some embodiments, the sensor 101 is, among other things, radar, audio This includes, for example, LiDAR, inertial, odometry, position, and / or ultrasonic sensors. In that embodiment, the sensor 101 is mounted on a vehicle having a vehicle control module 109. It is not possible. For example, sensor 101 may be attached to an adjacent vehicle, and / or It may be attached to roads or the environment, and deep learning for capturing sensor data. It is included as part of the system. In some embodiments, the sensor 101 detects when the vehicle is moving. Includes one or more cameras that capture the road surface. For example, one or more forward-facing cameras. The and / or pillar camera captures the lane markings of the lane the vehicle is traveling in. As another example, a camera can detect a vehicle attempting to cut into a lane in which another vehicle is traveling. It detects adjacent vehicles, including the vehicle itself. Additional sensors detect odometry, position, and / or vehicle trajectory. It captures vehicle control information, including information about the tracks. Sensor 101 captures still images and / or It can include both image sensors capable of capturing video. The data is collected over a certain period. A sequence of data captured over a period of time can be captured. For example... For example, the image of the lane markings will be displayed on the vehicle for a period of 15 seconds or another appropriate period of time. It may be captured along with the dome data. In some embodiments, the sensor 101 is located in the vehicle. Global Positioning System (GPS) sensors for determining both locations and / or changes in location Includes position sensors such as S.

[0021] In some embodiments, the image preprocessor 103 processes the sensor data of the sensor 101. Used to preprocess the sensor. For example, using the image preprocessor 103, Preprocess the data, split the sensor data into one or more components, and / or One or more components can be post-processed. In some embodiments, the image The reprocessor 103 includes a graphics processing unit (GPU), a central processing unit (CPU), and An image signal processor, or a dedicated image processor. In various embodiments, an image The preprocessor 103 is a tone mapper processor that handles high dynamic range data. In some embodiments, the image preprocessor 103 is an artificial intelligence (AI) preprocessor. It is implemented as part of the processor 107. For example, the image preprocessor 103 is an AI preprocessor. It may be a component of the reprocessor 107. In some embodiments, the image preprocessor You can use SA103 to normalize or transform images. For example Images captured with a fisheye lens may be distorted, so the image preprocessor 103 It can be used to transform images and remove or correct distortions. Several embodiments Therefore, noise, distortion, and / or blur are removed or reduced during the preprocessing step. In various embodiments, the images are adjusted or adjusted to improve the results of machine learning analysis. It is normalized. For example, the white balance of an image is normalized, in particular for daylight, sunny, cloudy, and twilight. It is adjusted to take into account different lighting operating conditions such as sunrise, sunset, and nighttime conditions. ru.

[0022] In some embodiments, the deep learning network 105 uses lane markers, lanes, and controls. The driving environment is used to determine available space, obstacles, and / or potential vehicle paths. A deep learning network used to determine vehicle control parameters, including analysis. This is a work. For example, the deep learning network 105 is trained with input such as sensor data. The output of this convolutional neural network is then provided to the vehicle control module 109. It may also be an artificial neural network such as a CNN. For example, the output is: It can include at least a three-dimensional representation of the marker. As another example, the output is less This could include potential vehicles that may merge into the vehicle lane. In this embodiment, the deep learning network 105 receives at least sensor data as input. I believe it. Additional inputs include vehicle specifications such as the surrounding environment of the vehicle and / or the vehicle's operating characteristics. Scene data can include descriptions of rainfall, wet roads, snowfall, etc. The environment surrounding vehicles includes muddy areas, high-density traffic, highways, cities, and school zones. It may include scene tags to describe the scene. In some embodiments, a deep learning network The output of K105 is the three-dimensional trajectory of the vehicle's lane. In some embodiments, The output of the layered learning network 105 is a potential vehicle interrupt. For example, deep learning network Twerk 105 identifies adjacent vehicles that are likely to enter the lane ahead of the vehicle.

[0023] In some embodiments, the artificial intelligence (AI) processor 107 is a deep learning network This is a hardware processor for executing 105. In some embodiments, A The I processor 107 processes sensor data using a convolutional neural network (CNN). It is a dedicated AI processor for performing inference using [this method]. AI processor 107 is, It can be optimized for the bit depth of the sensor data. In some embodiments, The AI ​​processor 107 performs convolution, dot product, vector, and / or matrix operations, among other things. It is optimized for deep learning operations such as neural network operations, including several real In this configuration, the AI ​​processor 107 uses a graphics processing unit (GPU) to perform It is equipped. In various embodiments, when the AI ​​processor 107 is executed, the AI ​​processor The system then performs deep learning analysis on the received input sensor data and uses it for autonomous driving. A memory configured to provide instructions to the AI ​​processor that determine the machine learning results. They are combined. In some embodiments, the AI ​​processor 107 combines the data with the training data. It is used to process sensor data in preparation for making it available.

[0024] In some embodiments, the vehicle control module 109 is an artificial intelligence (AI) processor It processes the output of 107 and uses it to convert the output into vehicle control actions. In this configuration, the vehicle control module 109 is used to control the vehicle for autonomous driving. In various embodiments, the vehicle control module 109 controls the vehicle's speed, acceleration, Steering, braking, etc. can be adjusted. For example, in some embodiments, the vehicle control motor The Joule 109 controls the vehicle to maintain its position in the lane and move the vehicle to another lane. Used to merge vehicles and adjust vehicle speed and lane configuration to account for merging vehicles, etc. It will be done.

[0025] In some embodiments, the vehicle control module 109 controls brake lights, turn signals, and Used to control vehicle lighting such as headlights. In some embodiments, the vehicle The control module 109 controls the vehicle's sound system, audio alarm playback, and microphones. Used to control vehicle audio states such as activating the on and horn. In some embodiments, the vehicle control module 109 detects potential collisions or intents. Warning systems to notify the driver and / or passengers of driving events such as approaching a designated destination. It is used to control a notification system, including a notification system. In some embodiments, The vehicle control module 109 is used to adjust the vehicle's sensors, such as the sensor 101. For example, the vehicle control module 109 can be used to change the orientation, output resolution and / Alternatively, you can change the format type, increase or decrease the capture rate, or change the captured dynamic range. Adjustments, camera focus adjustments, sensor activation and / or deactivation, etc., one or more of these. The parameters of the sensor can be changed. In some embodiments, the vehicle control module Use 109 to change the filter's frequency range, feature and / or edge detection. The image preprocessor 103 adjusts parameters, channels, and bit depth, etc. The parameters can be changed. In various embodiments, the vehicle control module 109 It is used to implement autonomous driving and / or driver assistance controls for vehicles. In this embodiment, the vehicle control module 109 uses a processor coupled with memory. It is implemented in some embodiments. In some embodiments, the vehicle control module 109 is a collection of applications. Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), or other suitable processing devices It is implemented using hardware.

[0026] In some embodiments, the network interface 111 includes voice data. A communication interface for sending and / or receiving data. Various implementations In this configuration, the network interface 111 is connected to the remote server via an interface connection. Next, you can connect and make voice calls, and send and / or receive text messages. It transmits sensor data and a deep learning network including an updated machine learning model. Receive updates to the network and check environmental conditions including weather conditions and forecasts, and traffic conditions. Includes a cellular or wireless interface for searching. For example, a network interface. Face 111 consists of a sensor 101, an image preprocessor 103, and a deep learning network 1 05, instructions and / or instructions of the AI ​​processor 107 and / or the vehicle control module 109 Alternatively, it can be used to receive updates on operating parameters. Deep learning network The machine learning model of 105 is uploaded using network interface 111. It is possible to date. As another example, using network interface 111 The image prep software includes the firmware and / or image processing parameters of the sensor 101. The operating parameters of the Rossesser 103 can be updated. As another example... Using the network interface 111, a crypt is deployed to train a machine learning model. It is possible to send real-time training data to a remote server.

[0027] Figure 2 shows one example of the process for training and applying machine learning models for autonomous driving. This is a flowchart illustrating the implementation configuration. For example, input data including sensor and odometry data The data is received, processed, and used to create training data for training machine learning models. In one embodiment, the sensor data is image data captured via the autonomous driving system. This corresponds to the following: In some embodiments, sensor data is used to manually disengage autopilot by the user. It supports sensor data captured based on specific use cases such as several. In this embodiment, the process involves a machine learning model for the deep learning system 100 shown in Figure 1. Used for creation and deployment.

[0028] In 201, training data is prepared. In some embodiments, image data and The system receives sensor data, including odometry data, and creates a training dataset. The data can include still images and / or videos from one or more cameras. It is possible. Additional sensors such as radar, LiDAR, and ultrasound are available to provide relevant sensor data. The following sensors can be used. In various embodiments, the sensor data is used in the sensor data To help identify the characteristics of the data, it is paired with corresponding odometry data. For example, For example, using position and position change data, lane lines, traffic control signals, objects, etc. The location of relevant features within the sensor data can be identified. In some embodiments, Sensor data is a time-series element and is used to determine ground truth. Next, the ground truth of the group is such as the first frame of the image data. Associated with a subset of the time series. Selected elements of the time series and ground truth The data is used to prepare the training data. In some embodiments, the training data It identifies only features from sensor data such as lane lines, vehicle paths, and traffic patterns. The prepared training data is used to train a machine learning model. Data for verification and testing may be included. In various embodiments, sensors The data may be in different formats. For example, sensor data may be still images, moving images, etc. It may also be images, audio, etc. Odometry data includes applied acceleration, applied Braking, applied steering, vehicle position, vehicle orientation, change in vehicle position, change in vehicle orientation Vehicle operation parameters such as transformation may be included. In various embodiments, training data is used for training It is curated and annotated to create a dataset. In some embodiments, part of the preparation of the training data may be performed by a human curator. In various embodiments, a portion of the training data is automatically generated from data captured from the vehicle. This significantly reduces the effort and time required to build robust training datasets. In some embodiments, the data format is used in the deployed deep learning application. It is compatible with machine learning models used in the application. In various embodiments, training data The data includes validation data to test the accuracy of the trained model.

[0029] In step 203, the machine learning model is trained. For example, the data prepared in step 201 A machine learning model is trained using convolutional genes. In some embodiments, the model uses convolutional genes. These are neural networks such as CNNs. The model includes multiple hidden layers. In some embodiments, the neural network It can include multiple layers, including multiple convolutional layers and pooling layers. In this implementation, the training model uses a validation dataset created from the received sensor data. It is verified using this method. In some embodiments, the machine learning model is derived from a single input image. It is trained to predict the 3D representation of a feature. For example, the 3D representation of a lane line is a This can be inferred from images captured by the camera. As another example, From the image, the predicted path of adjacent vehicles is predicted, including whether or not a vehicle is attempting to merge. ru.

[0030] In 205, trained machine learning models are deployed. For example, trained machine learning The learning model updates deep learning networks such as the deep learning network 105 in Figure 1. It is installed in the vehicle as a kit. In some embodiments, wireless updates are used. This is used to install a newly trained machine learning model. In some embodiments, The update uses a wireless network such as Wi-Fi or a cellular network. This is a firmware update that is transmitted. In some embodiments, a new machine The learning model may be installed when the vehicle is serviced.

[0031] In 207, sensor data is received. For example, sensor data is received from one of the vehicles. Alternatively, it is captured by multiple sensors. In some embodiments, the sensor is sensor 10 in Figure 1. The answer is 1. The sensor is a fisheye camera mounted behind the windshield, attached to the pillar. It may include image sensors such as attached forward-facing or side-facing cameras, or rear-facing cameras. In various embodiments, sensor data is input to a machine learning model trained in 203. It is in a format that can be used as such, or it will be converted to that format. For example, sensor data is The data may be raw or processed image data. In some embodiments, the data is From ultrasonic sensors, radar, LiDAR sensors, microphones, or other suitable technologies This is the captured data. In some embodiments, the sensor data is processed during the preprocessing step. The images are then preprocessed using an image preprocessor such as the image preprocessor 103 in Figure 1. For example, an image may be normalized to remove distortion, noise, etc.

[0032] In step 209, the trained machine learning model is applied. For example, trained in step 203 The machine learning model is applied to the sensor data received at 207. Several implementations In this case, the application of the model is to deep learning networks such as the deep learning network 105 in Figure 1. This is executed by an AI processor, such as AI processor 107 in Figure 1. In various embodiments, by applying a trained machine learning model, the lane lines Three-dimensional representations of features such as a moving vehicle are identified and / or predicted. Two splines are inferred to represent the lane lines of the lane. As another example, adjacent cars The predicted paths of adjacent vehicles are inferred, including whether both vehicles are likely to cut into the current lane. In various embodiments, by applying machine learning models, vehicles, obstacles, and The system identifies elements such as traffic control signals, map features, object distances, speed limits, and drivable space. In some embodiments, features are identified in three dimensions.

[0033] In 211, the autonomous vehicle is controlled. For example, one or more autonomous driving functions This is achieved by controlling various aspects of the vehicle. Examples include the steering, speed, and acceleration of the vehicle. Controlling the degree and / or braking, maintaining the position of the vehicle in the lane, and other vehicles Maintain the vehicle's position relative to both and / or obstacles, and notify or warn the occupants. This may include providing, etc. Based on the analysis performed in 209, the steering of the vehicle The speed is controlled to keep the vehicle between two lane lines. For example, left and right Lane lines are predicted, and the corresponding vehicle lanes and drivable spaces are identified. In this embodiment, a vehicle control module such as the vehicle control module 109 shown in Figure 1 controls the vehicle. do.

[0034] Figure 3 shows a flowchart illustrating one embodiment of the process of creating training data using time-series elements. This is an example. For instance, a time-series element consisting of sensor and odometry data represents a vehicle. It is collected from and used to automatically create training data. The process in Figure 3 automatically labels the training data with the corresponding ground truth. Used to resolve. Results corresponding to a time series are associated with the elements of the time series. The results and selected elements are packaged as training data to predict future outcomes. In various embodiments, the sensors and associated data are used with the deep learning system shown in Figure 1. It is captured using the sensor 1 in Figure 1. For example, in various embodiments, the sensor data is captured using the sensor 1 in Figure 1. Captured from 01. In some embodiments, the process in Figure 3 is performed at 201 in Figure 2. In some embodiments, the process shown in Figure 3 is performed if the existing prediction is incorrect or This is done to automatically collect data when improvements can be made. For example, if the vehicle is autonomous To determine whether an object is cutting in the vehicle's path, predictions are made by the autonomous vehicle. It waits for a certain period of time, analyzes the captured sensor data, and then determines whether the prediction is correct or incorrect. It is possible to determine whether or not. In some embodiments, it is possible to determine that the prediction can be improved. A decision is made. If the prediction was incorrect, or if it could be improved, the plan in Figure 3 is used. An example of applying the process to data related to prediction to improve machine learning models. You can create a set of selected items.

[0035] In 301, time-series elements are received. In various embodiments, the elements are captured by the vehicle. Sensor data, such as image data, is captured and sent to the training server. and are captured over a certain period of time to create time-series elements. The elements are timestamps to maintain the order of the elements. As time progresses, later events in the time series are consequences of earlier elements in the time series. It is used to aid in prediction. For example, time series signal to merge, accelerate, and self It can detect vehicles in adjacent lanes that are positioned near the nearby lane line. It is possible to use the entire time series and the results to determine when a vehicle merged into a shared lane. This can be done. The result is a selected time series, such as one of the initial images in the time series. It can be used to predict when vehicles will merge based on the factors. Another example is The time series then captures the curve of the lane line. The time series is derived from only a single element of the time series. It captures various dips, curves, bumps, etc. in the lane that are not obvious. In this context, the element is sensor data in a format that the machine learning model uses as input. For example, The sensor data may be raw or processed image data. Several implementations In terms of form, the data is transmitted via ultrasonic sensors, radar, LiDAR sensors, or other appropriate technologies. This data was captured from [source].

[0036] In various embodiments, the time series associates a timestamp with each element of the time series. It is organized by and For example, a timestamp is at least one element in the time series It is associated with the original data. The timestamp is time-series data in related data such as odometry data. It can be used to calibrate the primes. In various embodiments, the length of the time series is 10 seconds, 30 seconds It can be a fixed length of time, such as seconds or another appropriate length. The length of time is configurable. It is possible. In various embodiments, the time series is based on the vehicle's speed, such as the average speed of the vehicle. This can be done. For example, at slower speeds, the length of the time series can be increased to achieve the same speed. Using a shorter time duration per degree allows for longer travel distances than is possible. The data can be captured. In some embodiments, the number of elements in the time series is configurable. For example, the number of elements can be based on the distance traveled. For example, over a certain period of time, A faster vehicle contains more elements in its time series than a slower vehicle. The additional elements enhance the fidelity of the captured environment and improve the accuracy of the predicted machine learning results. It can be made possible. In various embodiments, the number of elements is such that the sensor captures data. By adjusting the frames per second and / or breaking unnecessary intermediate frames It is adjusted by discarding.

[0037] In 303, data related to time-series elements is received. In various embodiments, Related data is received by the training server along with the elements received in 301. Several implementations In terms of form, the relevant data is vehicle odometry data: position, orientation, change in position, direction. Changes in time series and / or other relevant vehicle data are used to identify specific elements. The positional data of the features can be labeled. For example, the time series of elements in a lane line can be labeled. By examining the lines, it is possible to label them with very precise location. The lane line closest to the vehicle camera is always accurate and closely related to the vehicle's position. On the other hand, determining the XYZ position of the line furthest from the vehicle is difficult. The side section is closed off (for example, behind a bend or hill) and / or precisely It may be difficult to capture them (for example, due to distance or lighting). The data related to the element is labeled with the parts of the identified features in the time series that were identified with high accuracy. Used to remove. In various embodiments, a threshold is used to remove the identified portion of a feature. Determine whether to associate (part of the lane line, etc.) with the relevant data. For example, high The parts of the lane line identified with high accuracy (such as the parts closest to the vehicle) are associated with relevant data. The portion of the lane line identified as having an accuracy below the threshold (such as the portion far from the vehicle). It is not associated with the relevant data of that element. Instead, it is associated with a subsequent element with a higher probability. Other time-series elements and their associated data are used, Related data includes the output of the deep learning network 105 in Figure 1, and other neural networks. This is the output of the vehicle control module 10 in Figure 1. In some embodiments, the relevant data is output from the vehicle control module 10 in Figure 1. This is the output of a vehicle control module such as 9. Related data includes speed, speed change, acceleration, This includes vehicle operation parameters such as changes in acceleration, steering, changes in steering, braking, and changes in braking. This can be done. In some embodiments, the relevant data can be used to estimate the distance of objects such as obstacles. This is radar data for that purpose.

[0038] In some embodiments, the data related to time-series elements includes map data. For example In 303, offline data such as road and / or satellite-level map data Data is received. Map data includes roads, vehicle lanes, intersections, speed limits, and school zones. It can be used to identify features such as vehicle lanes. For example, map data can be used to identify vehicle lanes. The route can be described. As another example, map data is related to various roads on the map. The assigned speed limit can be described.

[0039] In various embodiments, data related to time-series elements is associated with a timestamp. It is organized by associating it with data. Corresponding elements from time series and related data. Timestamps can be used to synchronize two datasets. In this embodiment, the data is synchronized at the time of capture. For example, when each element of the time series is captured, The corresponding set of related data is captured and stored along with the time-series elements. Various embodiments So, the time period for the related data is configurable and / or matches the time series period of the element. In some embodiments, the relevant data is sampled at the same rate as the time-series elements. It can be done.

[0040] In 305, the ground truth is determined for the time series. Various embodiments In this context, time series are machine learning features. e) is analyzed to determine the ground truth related to the lane line. The lane line is identified from the time series corresponding to the ground truth of that lane line. As an example, a graph of the path of a moving object (e.g., vehicle, pedestrian, bicycle, animal, etc.) The undtruth is the identified path of a moving object detected from a time series. In some embodiments, when a mobile vehicle enters the lane of an autonomous vehicle over a period of time, In some embodiments, the moving vehicle is annotated as an interrupting vehicle. A trajectory is represented as a three-dimensional representation, such as a three-dimensional trajectory. For example, a lane line The ground truth associated with the ground truth is a three-dimensional parameterized spline or curve. It can be represented as a line. As another example, the predicted path of a detected vehicle is determined, and the 3D trajectory is determined. It is represented as a trace. The predicted route is used to determine whether the vehicle is merging into the occupied space. It can be used for this purpose. In various embodiments, by examining the time series elements only, Ground truth can be determined. For example, by analyzing only a subset of the time series. This could leave a portion of the lane line blocked. By extending this analysis, the blocked sections of the lane lines become clear. Furthermore, Data captured near the end of the series provides more detail about the parts of the lane lines that are further away. To capture more accurately (for example, with higher fidelity). Furthermore, related data is closer. Because it is based on data captured (both distance and time), the relevant data is also more accurate. Yes. In various embodiments, different parts of an object are mapped to precise three-dimensional positions, including altitude. To do this, detected objects such as lane lines identified by elements in different time series of elements Simultaneous localization and mapping technology is applied to different parts of the body. A set of three-dimensional positions is an object such as a segment of a lane line captured over time. Represents the ground truth of the body. In some embodiments, localization and mapping The scribing technology involves precise point sets, for example, point sets corresponding to different points along a vehicle lane line. It brings about a convergence. The set of points is more efficient than a spline curve or a parametric curve. It can be converted to a format. In some embodiments, ground truth This involves detecting objects such as lane lines, drivable space, traffic control systems, and vehicles in three dimensions. It will be decided by [the end].

[0041] In some embodiments, ground truth is determined to predict semantic labels. For example, a detected vehicle may be labeled as being in the left lane or the right lane. This is possible. In some embodiments, if a detected vehicle is in a blind spot, it will yield the right of way. They can be labeled as vehicles that should be, or with other appropriate semantic labels. In this embodiment, the vehicle is located on a map of roads or It is assigned to a lane. As an additional example, using the determined ground truth This involves labeling traffic lights, lanes, drivable space, or other features that support autonomous driving. It is possible.

[0042] In some embodiments, the relevant data is the depth (or distance) data of the detected object. Therefore, by associating distance data with objects identified in the time series of elements, The machine learning model uses the ground truth as the relevant distance for the detected object. By using the data, it can be trained to estimate object distances. In several embodiments, the distance is defined as an obstacle, barrier, moving vehicle, stationary vehicle, traffic control signal, walking This is the distance to the detected object, such as a person.

[0043] In step 307, the training data is packaged. For example, time series elements are selected. , associated with the ground truth determined in 305. In various embodiments The elements selected are the initial elements in the time series. The selected elements are used in the machine learning model. The input sensor data is represented, and ground truth represents the predicted result. In various embodiments, the training data is packaged and prepared as training data. In some embodiments, training data is packed into training, validation, and test data. It is caged. Based on the determined ground truth and selected time series elements. Among other useful features for autonomous driving, the training data includes lane lines and vehicle data. Identify predicted routes, speed limits, vehicle interruptions, object distances, and / or drivable space. It can be packaged to train machine learning models for this purpose. The collected training data then becomes available for training machine learning models.

[0044] Figure 4 shows one embodiment of the process for training and applying a machine learning model for autonomous driving. This is a flowchart illustrating the process shown in Figure 4. In some embodiments, the process in Figure 4 is used for autonomous driving. Collect and store sensor data and odometry data for training machine learning models. It is used for the purpose of enabling automatic driving control. In some embodiments, the process shown in Figure 4 is used for the purpose of enabling automatic driving control. Whether or not, autonomous driving is implemented in vehicles where it is enabled. For example, sensors and Odometry data is obtained when the vehicle is driven by a human driver immediately after autonomous driving is deactivated. It can be collected while the vehicle is in operation and / or while it is in autonomous driving mode. In that embodiment, the technique described by Figure 4 uses the deep learning system of Figure 1 It is implemented. In some embodiments, part of the process shown in Figure 4 is performed on a machine for autonomous driving. As part of the process of applying the learned model, see Figures 207, 209, and / or 2 It will be executed in version 11.

[0045] In 401, sensor data is received. For example, a vehicle equipped with a sensor, It captures data and provides sensor data to a neural network operating on the vehicle. In some embodiments, sensor data includes visual data, ultrasonic data, and LiDAR data. This could be t, or other suitable sensor data. For example, high dynamic range forward-facing. Images are captured from the camera. As another example, ultrasonic data is captured from a sideways ultrasonic sensor. The data is captured. In some embodiments, the vehicle has multiple sensors for capturing data. They are installed. For example, in some embodiments, eight surround cameras are installed in the vehicle. It can be mounted and provides a 360-degree view of the area around the vehicle with a range of up to 250 meters. In some embodiments, the camera sensor includes a wide-angle front camera, a narrow-angle front camera, and a rear-view camera. Includes front-view cameras, front-view side cameras, and / or rear-view side cameras. In the embodiment, ultrasonic and / or radar sensors are used to capture details of the surroundings. Example For example, 12 ultrasonic sensors are attached to the vehicle to detect both hard and soft objects. It is possible to use forward radar to obtain data on the surrounding environment. In some embodiments, forward radar is used to obtain data on the surrounding environment. It captures. In various embodiments, the radar sensor also detects heavy rain, fog, dust, and other vehicles. Nevertheless, it can capture details of the surroundings. Various sensors capture the environment around the vehicle. The captured data is used for capture and provided for deep learning analysis.

[0046] In some embodiments, the sensor data includes the vehicle's position, orientation, change in position, and / Or it includes odometry data including changes in orientation, etc. For example, if position data is captured, It is associated with other sensor data captured within the same time frame. For example, an image. Location data captured during image acquisition is used to associate location information with image data. I can stay.

[0047] In 403, the sensor data is preprocessed. In some embodiments, the sensor data One or more preprocessing passes can be performed on the data. For example, the data is no Pre-processed to remove noise and correct alignment issues and / or blurring, etc. It may be. In some embodiments, one or more different filtering paths may be used. This is performed on the data. For example, to separate different components of sensor data, the data is processed In contrast, a high-pass filter can be applied to the data, and a low-pass filter can be applied to the data. In various embodiments, the preprocessing steps performed in 403 are optional, and It may be incorporated into a neural network.

[0048] In step 405, deep learning analysis of the sensor data is initiated. In some embodiments, Deep learning analysis is performed on sensor data that has been selectively preprocessed in 403. In various embodiments, deep learning analysis is performed using convolutional neural networks (CNNs). It is executed using neural networks such as the following. In various embodiments, machine learning models Dell uses the process shown in Figure 2 to train offline and perform inferences on sensor data. It is deployed on the vehicle for execution. For example, the model includes road lane lines, obstacles, and pedestrians. They may be trained to appropriately identify people, moving vehicles, parked vehicles, drivable spaces, etc. In some embodiments, multiple trajectories of the lane line are identified. For example, the lane line Several potential trajectories are detected, and each trajectory has a corresponding probability of occurrence. In some embodiments, the predicted lane line is the one with the highest probability of occurrence and / or This is the lane line with the highest associated confidence value. In some embodiments, deep learning The predicted lane lines from the analysis must exceed a minimum confidence threshold. Various embodiments In this context, the neural network includes multiple layers, each containing one or more hidden layers. In various embodiments, sensor data and / or the results of deep learning analysis are used as training data. It is held for automatic generation of the data and transmitted at 411.

[0049] In various embodiments, deep learning analysis is used to predict additional features. The predicted features can be used to assist autonomous driving. For example, the detected vehicle It can be assigned to a lane or road. As another example, a detected vehicle can be placed in a blind spot. There is a vehicle that should yield the right of way, a vehicle in the lane to the left, and a vehicle in the lane to the right. It can be determined that it has, or another appropriate attribute. Similarly, deep learning analysis can determine that Identifying the vehicle number, drivable space, pedestrians, obstacles, or other appropriate features for driving. It is possible.

[0050] In 407, the results of deep learning analysis are provided to vehicle control. For example, the results are automatically To control the vehicle for driving and / or to implement autonomous driving functions, Provided to the control module. In some embodiments, the results of deep learning analysis in 405 This involves using one or more different machine learning models to perform one or more additional deep learning Passing through a designated path. For example, determining a vehicle lane using a predicted path of the lane lines. This can be done, and the drivable space is determined using the determined vehicle lane. Then, drivable The space is used to determine the vehicle's path. Similarly, in some embodiments, A vehicle interruption is detected. The determined vehicle path is used to avoid a potential collision. For the purpose of considering the predicted interrupts, in some embodiments, various outputs of deep learning are This includes the predicted vehicle path, identified obstacles, identified traffic control signals including speed limits, etc. Used to construct a 3D representation of the vehicle environment for autonomous driving. Several implementations In this configuration, the vehicle control module is determined to control the vehicle along a determined path. The results obtained are used. In some embodiments, the vehicle control module is the vehicle control shown in Figure 1. This is module 109.

[0051] In 409, the vehicle is controlled. In some embodiments, autonomous driving is activated. The vehicle is controlled using a vehicle control module such as the vehicle control module 109 shown in Figure 1. Vehicle control, for example, takes the surrounding environment into consideration to maintain the vehicle in the lane at an appropriate speed. Therefore, the vehicle's speed and / or steering can be adjusted. In some embodiments The results are used to adjust vehicles by predicting that adjacent vehicles will merge into the same lane. In various embodiments, the vehicle control module uses the results of deep learning analysis to perform the following actions: For example, determining the appropriate way to operate a vehicle along a predetermined route at an appropriate speed. In various embodiments, the results of vehicle control, such as changes in speed, application of braking, and steering adjustments, are retained. This is used for the automatic generation of training data. In various embodiments, vehicle control parameters The data is retained for the automatic generation of training data and transmitted at 411.

[0052] At 411, sensor data and related data are transmitted. For example, received at 401. The sensor data obtained was used in the deep learning analysis in 405 and / or in 409. Along with vehicle control parameters, the data is sent to a computer server for automatic generation of training data. In some embodiments, the data is time-series data, and various collected data These are associated together by a computer server. For example, odometry data is linked It is associated with captured image data to generate round truth. In this implementation, the collected data is transmitted from the vehicle, for example, via Wi-Fi or cellular connection. It is transmitted wirelessly to the training data center. In some embodiments, metadata is transmitted to the sensor data It is sent along with the metadata. For example, metadata may include time, timestamp, location, and vehicle type. This includes data such as type, speed, acceleration, braking, whether autonomous driving is enabled, steering angle, and odometer readings. It can include vehicle control and / or operational parameters such as TA. Additional metadata This includes the time since the last previous sensor data was transmitted, the type of vehicle, weather conditions, and road conditions. This includes circumstances, etc. In some embodiments, the data transmitted may include, for example, a vehicle's unique identifier. Anonymization is achieved by removing certain information. As another example, data from similar vehicle models is anonymized. The data is merged in a way that does not identify individual users or their vehicle usage.

[0053] In some embodiments, data is sent only in response to a trigger. For example, several In that embodiment, incorrect predictions are used as an example to improve the predictions of the deep learning network. Sensor data to automatically collect data to create a regulated set Triggers the transmission of data and related information, such as whether a vehicle is attempting to merge. The predictions made in 405 in relation to this are made by comparing the predictions with the observed actual results. It is determined to be incorrect. Next, the sensor data associated with the incorrect prediction and Data containing related information is sent and used to automatically generate training data. In some embodiments, triggers are used for sharp curves, road junctions, lane merges, and sudden events. Stopping or finding another suitable scenario where additional training data would be useful and difficult to collect. It is possible to identify specific scenarios such as O. For example, the trigger could be the suddenness of the autonomous driving function. This can be based on the natural cessation or release. Another example is a change in velocity or a change in acceleration. Vehicle operating characteristics such as transformation can form the basis for the trigger. In some embodiments, Predictions with accuracy below a certain threshold trigger the transmission of sensor data and related data. For example, in certain scenarios, the prediction may not have a Boolean true or false result. Instead, it is evaluated by determining the accuracy value of the prediction.

[0054] In various embodiments, sensor data and related data are captured over a period of time. The entire time-series data is transmitted together. The period includes vehicle speed, distance traveled, and speed change. It can be composed and / or based on one or more factors such as the following. In that embodiment, the sampling rate of the captured sensor data and / or related data The sampling rate can be set. For example, the sampling rate can be set to high speed, during sudden braking, and during sudden acceleration. It is increased during high speed, during sharp steering, or in other appropriate scenarios where additional fidelity is required.

[0055] Figure 5 shows an example of an image captured by a vehicle sensor. In the example shown, Figure 5 The image is image data 50 captured from a vehicle traveling in the lane between two lane lines. Including 0. The positions of the vehicle and sensors used to capture image data 500 are, Represented by Bell A. Image data 500 is sensor data, taken in front of the vehicle while driving. It can be captured by camera sensors such as orientation cameras. Image data 500 is lane Capture the sections of lines 501 and 511. Lane lines 501 and 511 are lane lines. Lines 501 and 511 curve to the right as they approach the horizontal line. In the illustrated example, the lane Lines 501 and 511 are visible, but they are far away from the camera sensor's position. As it curves, detection becomes increasingly difficult. Above lane lines 501 and 511 The white lines drawn on them are from image data 500 to lane lines 501 and 5 without any additional input. Approximate the 11 detectable portions. In some embodiments, lane line 501 and The detected portion 511 is identified by segmenting image data 500. It is possible.

[0056] In some embodiments, labels A, B, and C are located at different positions and times on the road. Corresponds to different times in the column. Label A is the time of the vehicle when image data 500 was captured. Corresponds to time and position. Label B is the position on the road ahead of the position of label A, It corresponds to the position at a time after the time of label A. Similarly, label C corresponds to the position of label B. This is the position on the road ahead of the location, corresponding to the position at a time after the time of label B. As the vehicle moves, the positions of labels A, B, and C change (from label A to label C). The vehicle then passes through label C, capturing time-series data of sensor data and related data during the journey. The sequence includes elements captured at the positions (and times) of labels A, B, and C. Label A corresponds to the first element of the time series, label B corresponds to the middle element of the time series, and label C This corresponds to the intermediate (or potentially last) element in the time series. For each label, Additional data, such as vehicle odometry data at the location, is captured. Depending on the length of the time series. Then, additional or less data is captured. In some embodiments, times Each stamp is associated with a specific element in the timeline.

[0057] In some embodiments, the ground truth of lane lines 501 and 511 ( (Not shown) is determined. For example, using the process disclosed herein, LaneRa The positions of in 501 and 511 are from different elements of the time series, lane line 501 It is identified by identifying the different parts of 511. In the illustrated example, part Minutes 503 and 513 show image data 500 acquired at the location and time of label A. Identified using related data (such as odometry data). Parts 505 and 515 This includes image data (not shown) and related data acquired at the location and time of label B. Identified using odometry data, etc. Parts 507 and 517 of label C Image data (not shown) and related data (odometry data) acquired at location and time. Identified using (e.g., Ta). By analyzing the time series elements, lane line 50 The positions of the different parts 1 and 511 are identified, and the different identified parts are combined. The ground truth can be determined by the following. In some embodiments, the part is These are identified as points along each part of the line. In the illustrated example, to illustrate the process... Only three parts of each lane line are highlighted (part of lane line 501). 503, 505, and 507 and parts 513, 515, and lane line 511 (517) determines the position of the lane lines with higher resolution and / or higher precision. To that end, additional parts may be captured over time.

[0058] In various embodiments, within the image data capturing lane lines 501 and 511, The position of the part closest to the position of the sensor is determined with high precision. For example, parts 503 and 5 Location 13 is for image data 500 and related data (such as odometry data) for label A. High-precision identification is performed using [this method]. The positions of parts 505 and 515 are based on the image data of label B. It is identified with high accuracy using the data and related information. The locations of parts 507 and 517 are Label C is identified with high accuracy using image data and associated data. Time series elements are By using this, various aspects of lane lines 501 and 511 captured in chronological order can be obtained. The position of the relevant part is identified in 3D with high precision, and the ground of lane lines 501 and 511 It can be used as the basis for the dotrus. In various embodiments, the determined Ground truth is associated with selected time-series elements such as 500 image data points. Ground truth and selected elements are used to predict lane lines. It can be used to create training data. In some embodiments, the training data is human It is automatically created without labeling. The training data consists of 500 image data, such as capture images. To train a machine learning model to predict the 3D trajectory of lane lines from image data. It can be used for this purpose.

[0059] Figure 6 shows an image captured from a vehicle sensor with the predicted 3D trajectory of the lane line. This figure shows an example. In the example shown, the image in Figure 6 shows the lane between the two lane lines. Includes image data 600 captured from a vehicle traveling along the route. The location of the vehicle and sensor used is indicated by label A. Morphologically, label A corresponds to the same position as label A in Figure 5. Image data 600 is a sensor It is a data source and may be captured by camera sensors such as a forward-facing camera of the vehicle while driving. Image data 600 captures the portion of lane lines 601 and 611. Lane line 6 Lane lines 01 and 611 curve to the right as lanes 601 and 611 approach the horizon. It curves. In the illustrated example, lane lines 601 and 611 are visible, but As they move away from the camera sensor and curve into the distance, detection becomes increasingly difficult. The red lines drawn above lane lines 601 and 611 are lane lines 601 and 6 These are 11 predicted 3D trajectories. Using the process disclosed herein, 3D The trajectory was predicted using image data 600 as input to a trained machine learning model. In some embodiments, the predicted 3D trajectory is a 3D parameterized spline. Alternatively, it can be expressed as another parameterized representation.

[0060] In the illustrated example, portion 621 of lane lines 601 and 611 is a distance away from each other. Part of lane lines 601 and 611. Part 6 of lane lines 601 and 611 The three-dimensional positions (i.e., longitude, latitude, and altitude) of 21 are as disclosed herein. The predicted 3D trajectories of lane lines 601 and 611 are determined with high precision using Seth. It is included. Using a trained machine learning model, and using image data 600, and Without requiring positional data at the location of part 621 of line lines 601 and 611, The three-dimensional trajectories of lane lines 601 and 611 can be predicted. In the illustrated example, Image data 600 is captured at the location and time of label A.

[0061] In some embodiments, label A in Figure 6 corresponds to label A in Figure 5, and lane line 6 The predicted 3D trajectories of 01 and 611 are used as input to a trained machine learning model using image data. Determined using only -600. Obtained at the positions of labels A, B, and C in Figure 5. Ground toe determined using time-series image data and related data that include the elements By training a machine learning model using Ruth, lanes 601 and 61 The 3D trajectory of 1 can be predicted with high accuracy, even for distant lane lines such as section 621. The image data 600 and the image data 500 in Figure 5 are related, but the trajectory prediction is... This does not require that image data 600 be included in the training data. With sufficient training data Through training, it becomes possible to predict lane lines even in newly encountered scenarios. In various embodiments, the predicted three-dimensional trajectories of lane lines 601 and 611 are detected. To maintain the position of the vehicle within the lane line and / or to detect the predicted lane line It is used to autonomously navigate vehicles along the designated lanes. By predicting the line of sight, the performance, safety, and accuracy of navigation can be greatly improved. It will be done well.

[0062] The embodiments described above have been explained in some detail to clarify understanding, but the present invention The details provided are not limited to those disclosed. There are many alternative ways to carry out the invention. The embodiments described are illustrative and not limiting.

Claims

1. A step of receiving sensor data that includes a group of time-series elements, For at least selected time series elements within the group of time series elements, the time series Determine the corresponding ground truth based on multiple time-series elements within a group of elements. The steps include determining the training dataset, which involves, A processor is used to train a machine learning model using the aforementioned training dataset. The steps, A method that includes this.

2. Claim 1, wherein the corresponding ground truth is associated with the vehicle lane line. Method of description.

3. Different parts of the vehicle lane line can be identified from different elements of the aforementioned time-series element group. The method according to claim 2, further comprising the step of doing the following.

4. The plurality of time series elements recognize the position of the vehicle lane line within the selected time series element. The method according to any one of claims 1 to 3, used for separating.

5. Each element of the aforementioned group of time-series elements is an image associated with a corresponding timestamp. The method according to any one of claims 1 to 4, including the method described in any one of claims 1 to 4.

6. The corresponding ground truth is associated with the multiple time-series elements. The method according to any one of claims 1 to 5, determined using Tridata.

7. The odometry data includes vehicle position data and vehicle orientation data, as described in claim 6. The method.

8. The odometry data identifies a first change in vehicle position and a second change in vehicle orientation. The method according to claim 6.

9. The corresponding ground truth includes a three-dimensional trajectory, any one of claims 1 to 8 The method described in section [section number].

10. The method according to claim 9, wherein the three-dimensional trajectory is represented as a parametric curve.

11. The corresponding ground truth includes a first sensor that captured the sensor data. Any of claims 1 to 10, associated with the predicted path of a second vehicle different from the vehicle in question. The method described in item 1.

12. Claim 11, the second vehicle is identified as entering the lane of the first vehicle. Methods used.

13. Claims 1 to 12, wherein the corresponding ground truth is associated with the distance of an object. The method described in any one of the items.

14. The person according to claim 13, wherein the object is an obstacle, a moving vehicle, a stationary vehicle, or a barrier. Law.

15. The distance of the object is determined based on radar data associated with the plurality of time-series elements. The method according to claim 13.

16. The sensor data is collected over a set period of time, according to any of claims 1 to 15. The method described in any one of the items.

17. The number of elements included in the group of time-series elements is based on the distance traveled, as in claim 1. The method described in any one of items 16.

18. The number of elements included in the group of time-series elements is based on the average vehicle speed, as in claim 1. The method described in any one of items 17.

19. A computer program product, wherein the computer program product is non-temporary It is materialized in a computer-readable memory medium, Receive sensor data containing a group of time-series elements, For at least selected time series elements within the group of time series elements, the time series Determine the corresponding ground truth based on multiple time-series elements within a group of elements. This includes determining the training dataset, A processor is used to train a machine learning model using the aforementioned training dataset. A computer program product that includes computer instructions for doing so.

20. Processor and A memory coupled to the aforementioned processor, which, when executed, causes the processor to: The system receives sensor data that includes a group of time-series elements. For at least selected time series elements within the group of time series elements, the time series Determine the corresponding ground truth based on multiple time-series elements within a group of elements. This includes determining the training dataset, The machine learning model is trained using the aforementioned training dataset. A memory configured to provide instructions to the processor, A system equipped with these features.

Images