Vehicle Perception System with Time Tracker

Abstract

Described herein are techniques for a perception system for a vehicle that can detect and track objects in an environment. The perception system can include a machine learning model that includes one or more different parts, such as different components, sub-processes, etc. In an example, the techniques can include training a machine learning model end-to-end such that an output of a first part of the machine learning model is tuned for use as an input to another part of the machine learning model. Additionally or alternatively, the perception system described herein can utilize time data to track objects in the vehicle's environment and associate the tracking data with a particular object in the environment detected by the machine learning model. That is, the architecture of the machine learning model can include both a detection part and a tracking part in the same loop.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This patent application claims priority to U.S. Patent Application No. 17 / 542,352, filed December 3, 2021, which is incorporated by reference in its entirety herein. [Background technology]

[0002] Autonomous vehicles utilize various systems, methods, and apparatus to traverse an environment. For example, an autonomous vehicle may utilize one or more machine learning models to navigate an environment in which vehicles, people, buildings, and other objects may be present. In many cases, an autonomous vehicle utilizes specific machine learning models for specific tasks and operations. Additionally, output from one machine learning model of an autonomous vehicle may be used as input to another machine learning model of the autonomous vehicle. [Brief description of the drawings]

[0003] The detailed description will be set forth with reference to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different drawings indicates similar or identical components or features.

[0004] [Figure 1] FIG. 1 is a flow diagram illustrating an example process in accordance with the techniques described herein for determining tracked object data using a machine learning model. [Diagram 2] FIG. 2 illustrates a detailed example of a machine learning model for determining tracked object data based on input sensor data. [Diagram 3] FIG. 3 illustrates a detailed example of the detection and tracking components of a machine learning model in accordance with some of the techniques disclosed herein. [Figure 4]FIG. 4 is a flowchart illustrating an example method associated with training a machine learning model end-to-end such that an output of a first portion of the machine learning model is conditioned for use as an input to a second portion of the machine learning model. [Diagram 5] FIG. 5 is a flow chart illustrating an example method associated with a machine learning model for determining tracked object data used by a vehicle. [Figure 6] FIG. 6 is a block diagram illustrating an example system that may be used to implement the techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] The present disclosure is directed, among other things, to techniques for improving a vehicle's perception system by training a first portion of the perception system (e.g., a first machine learning model, a first component, etc.) at least in part with the output of a second portion of the perception system (e.g., a second machine learning model, a second component, etc.) such that the output of the first portion is adjusted for use as an input to the second portion. In other words, rather than training each portion of the perception system individually, the present disclosure describes techniques in which different portions of the perception system are trained together. By training the perception system end-to-end (e.g., training the first portion based on the output of the second portion, or training otherwise separate portions together) and including temporal information in the input, the output of the first portion can be adjusted for use as an input to the second portion, resulting in improved accuracy and reliability for the output of the overall perception system.

[0006] Additionally, this application is directed to an improved architecture for a vehicle perception system that tracks objects within the vehicle's environment and utilizes temporal data to associate the tracking data with specific objects within the environment. That is, rather than training a first machine learning model of the perception system to detect objects and separately training a second machine learning model of the perception system to track the objects, the perception system architecture is configured to output tracked object data based on input sensor data. That is, the perception system may include a first part and a second part that together output tracked object data based on input sensor data.

[0007] For example, a perception system (e.g., a machine learning model) may be configured such that in a first operation, object data is determined for an object detected in an environment, where the object data indicates a current position of the object. In a second operation, the perception system may determine an estimated current position of the object based on stored tracking data associated with the object. In a third operation, the perception system may associate the object data with the stored tracking data as the tracked object data based at least in part on the current position being within a threshold distance of the estimated current position. The perception system may output the tracked object data, which may be used by other components and / or systems of the vehicle to, among other things, make predictions about the detected objects, determine a trajectory for the vehicle to traverse the environment, etc.

[0008] By way of example and not limitation, methods according to various techniques described in this disclosure may include techniques for receiving sensor data representative of an environment in which a vehicle is operating. In some examples, the vehicle may include one or more sensors, such as an image sensor (e.g., a camera), a lidar sensor, a radar sensor, a time-of-flight sensor, an environmental sensor, etc. Thus, the sensor data representative of the environment may include image data, lidar data, radar data, time-of-flight data, etc. Additionally, in some examples, the sensor data may be associated with a first instance of time, such as a current time, which may be referred to herein as "t0."

[0009] In some examples, the sensor data may be input to a machine learning model configured for use in the vehicle. In some examples, the machine learning model may be part of a perception system of the vehicle configured to detect objects, track detected objects, and / or make certain predictions about detected and / or tracked objects. In some examples, the perception system may include one or more machine learning models. Additionally or alternatively, the perception system may include a single machine learning model that includes multiple stages, sub-processes, or components. That is, the single machine learning model may be configured to determine a single output for a single input, even though different stages of the machine learning model may determine intermediate outputs.

[0010] In various examples, the machine learning model may include an object detection portion (e.g., a first stage, sub-process, component, etc.) trained to generate or otherwise determine object data associated with one or more objects detected in the environment. That is, the object detection portion of the machine learning model may analyze or process the sensor data using one or more algorithms to detect the presence of an object in the environment and / or determine object data associated with the object. In some examples, the object data may include one or more bounding boxes associated with each object in the environment. Each bounding box may indicate information associated with a particular object at a first time instance (t0) at which the sensor data was generated / captured. For example, the bounding box may indicate, among other things, the location of the object in the environment at t0, the orientation of the object at t0, the size of the object, etc. Additionally or alternatively, the object data may indicate a classification and / or type of object, such as whether the object is a mobile dynamic object (e.g., a vehicle, a motorcycle, a bicycle, a pedestrian, an animal, etc.) and / or a static object (e.g., a building, a road surface, a tree, a sign, a barrier, a curb, a parked vehicle, etc.). Additionally or alternatively, the object data may indicate other information associated with the object, such as the velocity of the object at t0, a confidence level associated with the object, etc.

[0011] In some examples, the machine learning model may include an object tracking portion. 0-NThe object tracking portion may be configured to perform a number of sub-processes to determine a trajectory traversed by the detected object during a period of time t -t0, where "N" represents any integer equal to or greater than 1. For example, the object tracking portion may include, among others, a position estimation sub-process, a track position association sub-process, a trajectory estimation sub-process, and / or a track storage sub-process. In an example, the object tracking portion may receive as input the object data determined by the object detection portion, as well as t 0-N -t 0-1 The tracking device may receive stored trajectory data indicative of a trajectory traversed by an object during a period of time and output tracked object data as described herein.

[0012] In some examples, the machine learning model may determine an estimated position of the object in the environment at time t0 during the position estimation subprocess. Additionally or alternatively, the machine learning model may determine an estimated orientation of the object, an estimated size of the object, an estimated classification of the object, etc., as well as any other features that may be indicated in the object data. In examples, the machine learning model determines the estimated position of the object at t0 based at least in part on stored tracking data associated with the object. The stored tracking data may be stored in a time domain. 0-N -t 0-1 For example, the stored tracking data may include multiple frames of sensor data, tracking data, tracked object data, etc., each frame representing a different time instance (e.g., t 0-1 , t 0-2 , t 0-3 , t 0-4 , t 0-5The machine learning model may determine an estimated position of the object at a current instance of time t0. In some examples, the machine learning model may determine multiple estimated positions of different objects in the environment. In some examples, a period of time (e.g., 100 ms, 200 ms, 500 ms, etc.) may separate each instance of time. Using the stored frames of tracking data, the machine learning model may determine an estimated position of the object at a current instance of time t0. In some examples, the machine learning model determines multiple estimated positions of different objects in the environment.

[0013] In some examples, the machine learning model may associate particular object data with particular track data based at least in part on the object's location corresponding to the object's estimated location during the tracking-location association sub-process. In other words, the machine learning model may determine that the object's location corresponds to the estimated location as indicated in the object data. For example, the machine learning model may be trained to determine that the location corresponds to the estimated location based at least in part on one or more of the following: the distance between the location and the estimated location, the object's orientation and the object's estimated orientation, the object's size and the object's estimated size, the object's classification and the object's predicted classification, etc. In some examples, associations between tracks and associated objects may be created for multiple objects and trajectories at one time.

[0014] In some examples, the machine learning model can determine tracked object data associated with a particular object during the trajectory estimation sub-process, where the tracked object data includes the object data and a time period that includes the current time instance (e.g., t 0-N In an example, the machine learning model determines the tracked object data based on an association between the particular object data and the particular tracking data. For example, the machine learning model may determine the position of the object at t0 and the trajectory traversed by the object at t1-t2. 0-1 Based at least in part on the prior location of the object at t 0-1-t0 may determine a trajectory of the object. In some examples, the machine learning model may determine tracked object data for multiple objects during the trajectory estimation subprocess.

[0015] In some examples, the machine learning model may store the trajectory of the object in memory accessible to the machine learning model during the track storage sub-process. In some examples, this may be the long short-term memory of the machine learning model. In some examples, the machine learning model may run continuously as the vehicle traverses the environment, so that the machine learning model can continue to update each stored trajectory of the object as that object's trajectory progresses.

[0016] In some examples, the techniques may also include receiving an output from the machine learning model that includes at least tracked object data, i.e., the tracked object data received as output from the machine learning model may indicate, for each object in the environment, object data associated with that object (e.g., bounding box, classification, speed, confidence, etc.) and a trajectory traversed by that object through the environment.

[0017] In some examples, the vehicle may be controlled based at least in part on the tracked object data. For example, at least a portion of the tracked object data may be used as an input to another system associated with the vehicle, such as a forecasting system, a planning system, or the like.

[0018] As mentioned above, the present disclosure is also directed to techniques for end-to-end training of machine learning models that include multiple parts (e.g., stages, sub-processes, components, etc.). When a machine learning model is trained end-to-end, the output of the machine learning model is used to train individual parts of the machine learning model that may otherwise exist as independent machine learning models. In this manner, intermediate outputs of some of the machine learning models may be specifically tailored for use as inputs by other parts of the machine learning model.

[0019] By way of example and not limitation, a method associated with end-to-end training of a machine learning model may include techniques for receiving sensor data representative of a vehicle traversing an environment. In some examples, the sensor data may include stored logged data associated with the vehicle. The sensor / logged data may be image data, lidar data, radar data, time-of-flight data, etc. In some examples, the sensor data may be a time-ordered collection of image frames representing sensor data associated with the environment, such that a first frame represents the environment at a first time and a second frame represents the environment at a second time.

[0020] In some examples, the techniques may also include receiving ground truth data associated with objects in the environment represented in the sensor data. In some examples, the ground truth data may include tracked object data associated with the objects. Thus, the ground truth data may include bounding boxes associated with the objects, trajectories traversed by the objects, classifications of the objects, and the like. In some examples, the ground truth data may be determined by human labelers, trained machine learning models, and the like. In some examples, the ground truth data may include a time-ordered collection of image frames representing the sensor data associated with the environment, each frame annotated to indicate information about an object in the environment at a respective time, the information including bounding boxes, trajectories, classifications, and the like.

[0021] In some examples, the sensor data may be input to a machine learning model. In various examples, the machine learning model may include a first portion (e.g., an object detection portion) configured to determine object data associated with the object based at least in part on the sensor data. Thus, the object data may include at least a predicted bounding box associated with the object. Further, in some examples, the machine learning model may include a second portion (e.g., an object tracking portion) configured to determine tracking data associated with movement of the object through the environment based at least in part on the object data determined by the first portion. In some examples, the tracking data may include a predicted trajectory traversed by the object through the environment.

[0022] In some examples, the technique may include receiving an output from the machine learning model. The output may include predicted tracked object data including, among other things, a predicted bounding box and a predicted trajectory. Based on the output, the ground truth data and the predicted tracked object data may be compared to determine whether a difference exists between the ground truth data and the predicted tracked object data. In some examples, if the difference is determined to meet or exceed a threshold difference, parameters of the machine learning model may be altered to minimize the difference. In some examples, parameters of one or more portions of the machine learning model may be altered. For example, parameters of a first portion of the machine learning model may be altered to adjust the object data to be used by a second portion of the machine learning model such that the difference between the ground truth data and the predicted tracked object data is minimized. Additionally or alternatively, parameters of the second portion of the machine learning model may be altered to help minimize the difference. In some examples, altering the parameters of the machine learning model may include retraining the first portion or the second portion based on the difference, updating software of the first portion or the second portion, etc.

[0023] The techniques described herein can improve the performance of vehicle perception systems and / or machine learning models. By utilizing the temporal history of object trajectories when making associations between detected objects and stored tracking data, the machine learning model / perception system is less vulnerable to object occlusion and data switching (e.g., associating the wrong track with the wrong object). Furthermore, by training the perception system model end-to-end according to the techniques disclosed herein, intermediate outputs from one part of the model can be aligned with other parts of the model, ultimately leading to a better overall output. In the context of autonomous vehicles, the techniques described herein provide more accurate machine learning models, which can enhance the safety of autonomous vehicles by making more accurate predictions, such as improving safe decision-making and minimizing the risk of adverse events. These and other improvements will be readily apparent to those skilled in the art.

[0024] The techniques described herein can be implemented in several ways. Examples are presented below with reference to the figures. Although an exemplary implementation is described in the context of an autonomous vehicle, the methods, apparatus, and components described herein can be applied to a variety of components (e.g., sensor components or robotic platforms) and are not limited to autonomous vehicles. For example, the techniques can be utilized in an aviation or marine context, or in other machine learning and training contexts. Furthermore, the techniques described herein can be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

[0025] 1 is a pictorial flow diagram illustrating an example process 100 in accordance with the techniques described herein for determining tracked object data using a machine learning model. In an example, a vehicle 102 may be associated with one or more sensor components. In at least one example, the sensor components may capture sensor data 106 associated with an environment 104 surrounding the vehicle 102. The environment 104 may include one or more objects, such as object 108(1), which is a vehicle, and object 108(2), which is a pedestrian. Objects 108(1) and 108(2) may be collectively referred to herein as “object 108.” In at least one example, the sensor components may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), wheel encoders, microphones, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), time-of-flight (ToF) sensors, etc. The sensor component can generate sensor data 106 associated with the environment 104, which may include, but is not limited to, lidar data, radar data, ultrasonic transducer data, sonar data, position data (e.g., GPS, compass, etc.), pose data, inertial data (e.g., inertial measurement unit data, accelerometer data, magnetometer data, gyroscope data, etc.), image data (e.g., RGB data, IR data, intensity data, depth data, etc.), wheel encoder data, microphone data, environmental sensor data (e.g., temperature sensor data, humidity sensor data, light sensor data, pressure sensor data, etc.), ToF sensor data, etc. In some examples, the sensor data 106 may be in the form of multiple snapshots (e.g., frames) that can be input as a collection to the machine learning model 112 so that the movement of an object can be tracked over time. In some examples, the sensor data 106 may be a time-ordered collection of frames representing the environment 104 over a period of time.For example, the sensor data 106 may be a collection of five frames representing frames in chronological order, than may be updated in a rolling manner (eg, new frames are added to the collection and old frames are removed).

[0026] In at least one example, the sensor data 106 may be transmitted to one or more computing devices 110 associated with the vehicle 102. The one or more computing devices 110 may be on-board the vehicle 102 or may be remotely accessible to the vehicle 102 via a network connection. The one or more computing devices 110 may execute the machine learning model 112 using resources (e.g., computation, memory, processing cores, etc.) of the computing device 110. In one example, the machine learning model 112 is associated with a perception system of the vehicle 102. The machine learning model may include a detection component 114 and a tracking component 122.

[0027] 1, the sensor data 106 is forwarded as an input to the machine learning model 112. Based at least in part on the sensor data 106, the detection component 114 of the machine learning model 112 can output object data 116. In the illustrated example, the object data 116 includes top-down data 118 representing a view of the environment 104 from a top-down (e.g., bird's-eye) perspective. However, this is not intended to be so limited, and the object data 116 may include other representations of data. In some examples, the object data 116 may include bounding boxes associated with the objects 108 detected in the environment 104 by the detection component 114, such as a bounding box 120(1) associated with the object 108(1) and a bounding box 120(2) associated with the object 108(2).

[0028] The object data 116, in some examples, may be used in whole or in part as an input to a tracking component 122 of the machine learning model 112. Although not shown in FIG. 1, in some examples, the tracking component 122 may also receive as an input stored tracking data associated with the objects 108(1) and 108(2), which will be described in further detail below. Based at least in part on the object data 116 and / or the stored tracking data, the tracking component 122 may determine, and the machine learning model 112 may output, tracked object data 124. In some examples, the tracked object data 124 may include top-down data 118 representing a view of the environment 104 from a top-down perspective, where the movement data indicates movement of the object. In some examples, in addition to including object data 116 (e.g., bounding boxes 120(1) and 120(2)), tracked object data 124 may also include trajectories traversed by objects within environment 104, such as trajectory 126(1) traversed by object 108(1) and trajectory 126(2) traversed by object 108(2).

[0029] In some examples, the tracking component 122 may be configured to perform multiple sub-processes to determine the trajectory 126 traversed by the detected object 108 during a period prior to the current time. For example, the tracking component 122 may perform a position estimation sub-process, a track position association sub-process, a trajectory estimation sub-process, and / or a track storage sub-process, as described above and herein. In some examples, these sub-processes may be stored as separate components of the tracking component 122 that execute independently of one another, or may be considered as logical operations associated with how the tracking component 122 of the machine learning model 112 determines the tracked object data 124.

[0030] In some examples, during the position estimation subprocess, tracking component 122 can determine an estimated position of objects 108(1) and 108(2) within environment 104 at a current time t0. Additionally or alternatively, tracking component 122 can determine an estimated orientation of objects 108(1) and 108(2), an estimated size of objects 108(1) and 108(2), an estimated classification of objects 108(1) and 108(2), and the like, as well as any other characteristics that may be indicated in object data 116 as described above and herein. In examples, tracking component 122 determines the estimated positions of objects 108(1) and 108(2) at t0 based at least in part on stored tracking data associated with objects 108(1) and 108(2). The stored tracking data may be stored in a time domain such as t0. 0-N -t 0-1 For example, the stored tracking data may include multiple previous frames of sensor data 106, tracking data, tracked object data 124, etc., each frame representing a different time instance (e.g., t 0-1 , t 0-2 , t 0-3 , t 0-4 , t 0-5 Using the stored frames of tracking data, tracking component 122 can determine an estimated position of objects 108(1) and 108(2) at a current instance of time t0.

[0031] In some examples, during the track position association subprocess, tracking component 122 can associate particular object data 116 (e.g., bounding boxes 120(1) and 120(2)) with particular track data (e.g., trajectories 126(1) and 126(2), respectively) based at least in part on the positions of objects 108(1) and 108(2) that correspond to the estimated positions of objects 108(1) and 108(2). In other words, tracking component 122 can determine that the positions of objects 108(1) and 108(2), as shown in object data 116, correspond to the estimated positions.

[0032] In some examples, the tracking component 122 can determine tracked object data 124 associated with a particular object 108(1) or 108(2) during the trajectory estimation sub-process, where the tracked object data 124 is associated with the object data 116 and a current instance of time (e.g., t 0-N 1 includes trajectories 126(1) and 126(2) traversed by objects 108(1) and 108(2) during a period including (from -t0).

[0033] In some examples, the tracking component 122 or the machine learning model 112 may store the trajectories 126(1) and 126(2) of the objects 108(1) and 108(2) in a memory accessible to the machine learning model 112 and / or the tracking component 122 during the track storage sub-process. In some examples, the memory may include a long short-term memory of the machine learning model 112 and / or the tracking component 122. In some examples, the machine learning model 112 may run continuously as the vehicle 102 traverses the environment 104, such that the tracking component 122 can continue to update the stored respective trajectories of the objects 108(1) and 108(2) as they progress through the environment 104.

[0034] The tracked object data 124 may, in some examples, be forwarded to a prediction component 128 executing on the computing device 110. In one example, the prediction component 128 includes one or more machine learning models trained to make predictions about objects in the environment 104. Thus, the prediction component 128 may determine prediction data 130 associated with the objects based at least in part on some or all of the tracked object data 124. In some examples, the prediction data 130 may include top-down data 118 representing a view of the environment 104 from a top-down perspective. Additionally or alternatively, the prediction data 130 may indicate one or more predictions associated with the objects. For example, predictions 132(1) and 132(2) associated with objects 108(1) and 108(2), respectively, include trajectories that the objects 108(1) and 108(2) are predicted to traverse within the environment 104 during a future time period.

[0035] In some examples, the predictive data 130 may be forwarded as an input to a planner component 134 of the vehicle 102 executing on the computing device 110. In one example, the planner component 134 includes one or more machine learning models and / or other algorithms configured to determine a planned trajectory for the vehicle 102 through the environment 104. Thus, the planner component 134 may determine a planned trajectory for the vehicle 102 based at least in part on the tracked object data 124 and / or the predictive data 130.

[0036] In various examples, any one of the sensor data 106, object data 116, tracked object data 124, or prediction data 130 may be formatted as a multi-channel image, and individual channels of the multi-channel image may represent a selection of information. For example, a first channel of the multi-channel image may include bounding box(es) associated with the object(s), a second channel of the multi-channel image may include speed information associated with the object(s), and a third channel of the multi-channel image may include environmental data (e.g., surface or lane markings, traffic light information, etc.). Furthermore, in some examples, multiple different channels of the multi-channel image may include some of the same information. For example, the first channel and the second channel of the multi-channel image may each include bounding boxes associated with the same object in the environment. The channels may be represented, for example, as colors and / or layers in a three-dimensional image stack.

[0037] 2 illustrates example details 200 of a machine learning model 112 for determining tracked object data 124 based on input sensor data 106. The sensor data 106 may be image data, lidar data, radar data, and / or time-of-flight data associated with an environment in which a vehicle is operating. At "1", a detection component 114 of the machine learning model 112 receives the sensor data 106 and determines object data 116 associated with an object at a current time t0. The object data 116 may be forwarded to a track position association and trajectory estimation component 212 of the machine learning model 112. In some examples, the track position association and trajectory estimation component 212 may be part of a tracking component 122 described herein.

[0038] At "2", the localization component 202 may receive tracking data 206 associated with an object in an environment and determine estimated object data 210 associated with the object at a current time t0 based on the tracking data 206. The localization component 202 may receive the tracking data 206 from a memory 204 of the machine learning model 112, which may include a long short-term memory. The tracking data 206 may include one or more frames 208(1)-208(N), where N may represent any integer greater than or equal to 2, of sensor data, tracking data, and the like. In some examples, a first frame 208(1) is a frame 208(2) that is associated with an object in an environment. 0-1 and the second frame indicates the trajectory and / or position of the object in the environment at t 0-2 t, and the Nth frame 208(N) may represent the trajectory and / or position of an object in the environment at t 0-N The position estimation component 202 may utilize the multiple frames 208 to determine estimated object data 210 at the current time t0. The estimated object data 210 may be forwarded to a track position association and trajectory estimation component 212.

[0039] At "3", the track position association and trajectory estimation component 212 can determine tracked object data at a current time t0 based at least in part on the object data 116 and the estimated object data 210. For example, the track position association and trajectory estimation component 212 can determine that a position of an object shown in the object data 116 corresponds to an estimated position of an object shown in the estimated object data 210. Based on the corresponding positions, the track position association and trajectory estimation component 212 can determine which tracks correspond to which objects and then associate the stored tracking data with the objects. Additionally, the track position association and trajectory estimation component 212 can determine that at least one track corresponds to a current time t0. 0-1Based on the object's position at t0 and the object's current position at t0, a trajectory traversed by the object can be estimated. This estimated trajectory may be included in the tracked object data 124. Thus, for each object 108 represented in the tracked object data 124, a bounding box 120 associated with the object 108 may be represented, as well as a trajectory 126 traversed by the object 108.

[0040] At "4", the tracked object data 124 at t0 is stored in the memory 204. In this manner, the tracked object data 124 at t0 is stored at a future time t 0+1 The tracked object data 124 may be utilized by position estimation component 202 to estimate the object data of objects 108(1) and 108(2). Further, at "5", tracked object data 124 is forwarded to prediction component 128 to predict future trajectories and other behaviors of objects 108(1) and 108(2).

[0041] 3 illustrates example details 300 of the detection component 114 and tracking component 122 of a machine learning model in accordance with some of the techniques disclosed herein. As shown, the detection component 114 may include a two-dimensional convolutional neural network (2D CNN) backbone 302 that may determine the top-down data 118 and a detection head 306 that may determine the object data 116. The tracking component 122 may include a bilinear linear interpolation component 308, a finite difference component 310, a multi-layer perception (MLP) component 312, a long short-term memory (LSTM) component 316, an MLP refinement component 320, a bipartite matching and single object tracking (SOT) component 322, and a non-maximum suppression (NMS) component 326.

[0042] A bilinear linear interpolation component 308 receives the top-down data 118 and the tracking data 206 and can determine objects in the environment that are new observations (e.g., objects for which there is no tracking data 206), or vice versa. For example, if the tracking data indicates that there are three objects in the environment, but the top-down data 118 indicates only two objects, the bilinear linear interpolation component 308 can be utilized to determine why there is no matching tracking data (e.g., a new object in the environment, an object leaving the environment, etc.).

[0043] The LSTM component 316 can receive data and other information associated with the objects / features, such as feature tensors, finite differences, and outputs from the MLP component 312, and determine one or more trajectory-level representations 318. The MLP refinement component 320 can utilize the trajectory-level representations 318 to determine scores, position offsets, and other data 328.

[0044] The bipartite matching and SOT component 322 can perform one-to-one matching between tracked objects as shown in the tracking data 206 and new observations of objects included in the object data 116. That is, the bipartite matching and SOT component 322 can determine one or more current tracks 324 associated with objects (e.g., tracked object data 124) in the environment.

[0045] The NMS component 326 can utilize the current track 324 and the score, position offset, and other data 328 to determine a refined current track 330. The refined current track(s) can be transferred and stored with the other tracking data 206. In this manner, the refined current track 330 can be used to determine a trajectory level representation 318 and / or additional current tracks 324 at future times.

[0046] 4 and 5 are flow charts illustrating an exemplary method of presenting various user interfaces on a display associated with monitoring a vehicle. The method illustrated in FIG. 4 and FIG. 5 is described with reference to one or more of the vehicles, systems, and / or user interfaces described in FIG. 1-3 for convenience and ease of understanding. However, the method illustrated in FIG. 4 and FIG. 5 is not limited to being performed with the vehicles, systems, and / or user interfaces described in FIG. 1-3, but may be implemented using any of the other vehicles, systems, and user interfaces described in this application, as well as vehicles, systems, and user interfaces other than those described herein. Furthermore, the vehicles, systems, and user interfaces described herein are not limited to performing the method illustrated in FIG. 4 and FIG. 5.

[0047] Methods 400 and 500 are illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as limiting, and any number of the described blocks may be combined in any order and / or in parallel to implement a process. In some embodiments, one or more blocks of a process may be omitted entirely. Additionally, methods 400 and 500 may be combined with each other or in other ways, in whole or in part.

[0048] 4 is a flowchart illustrating an example method 400 associated with training a machine learning model end-to-end such that an output of a first portion of the machine learning model is conditioned for use as an input to a second portion of the machine learning model. The method 400 begins at operation 402, which includes receiving sensor data associated with a vehicle traversing an environment. For example, the computing device 110 may receive sensor data 106 associated with an environment 104 surrounding the vehicle 102. In some examples, the environment may include one or more objects, such as objects 108(1) and 108(2). In some examples, the sensor data may include, but is not limited to, lidar data, radar data, ultrasonic transducer data, sonar data, position data (e.g., GPS, compass, etc.), pose data, inertial data (e.g., inertial measurement unit data, accelerometer data, magnetometer data, gyroscope data, etc.), image data (e.g., RGB data, IR data, intensity data, depth data, etc.), wheel encoder data, microphone data, environmental sensor data (e.g., temperature sensor data, humidity sensor data, light sensor data, pressure sensor data, etc.), time of flight (ToF) sensor data, etc. In some examples, the sensor data may be actual sensor data (e.g., generated by a sensor in the vehicle) or simulated sensor data.

[0049] At operation 404, the method 400 includes receiving ground truth data associated with an object in the environment. For example, the computing device 110 may receive the ground truth data. In some examples, the ground truth data may include tracked object data associated with the object. Thus, the ground truth data may include a bounding box associated with the object, a trajectory traversed by the object, a classification of the object, etc. In some examples, the ground truth data may be determined by a human labeler, a trained machine learning model, etc.

[0050] At operation 406, the method 400 includes inputting the sensor data into a machine learning model. For example, the computing device 110 can input the sensor data 106 into the machine learning model 112. At operation 408, the method 400 includes receiving an output from the machine learning model including the first data determined by the first portion of the machine learning model and the second data determined by the second portion of the machine learning model. For example, the output may include tracked object data 124 including the first data (e.g., object data determined by the detection component 114) and the second data (e.g., trajectory data determined by the tracking component 122). That is, the output may include at least a predicted bounding box associated with the object and a predicted trajectory traversed by the object. As described herein, in an example, the second portion of the machine learning model may determine the second data based at least in part on the first data determined by the first portion of the machine learning model.

[0051] At operation 410, the method 400 includes determining a difference between the output and the ground truth data. In some examples, the difference between the output and the ground truth data may be a difference between a trajectory and a predicted trajectory, a difference between a bounding box and a predicted bounding box, etc. At operation 412, the method 400 includes determining whether the difference exceeds a threshold difference. If the difference does not exceed (e.g., is less than) the threshold difference, the method 400 proceeds to operation 414. However, if the difference meets or exceeds the threshold difference, the method 400 proceeds to operation 416.

[0052] At operation 414, the method 400 includes performing a first action. In some examples, the first action may include determining that the machine learning model is a trained machine learning model. Additionally or alternatively, the first action may include transmitting the machine learning model to the vehicle for use by the vehicle to traverse the environment. In some examples, the first action may include validating performance of the machine learning model in a simulation or with additional sensor data.

[0053] At operation 416, the method 400 includes modifying parameters of the first portion of the machine learning model. For example, parameters of the detection component 114 or the tracking component 122 may be modified, as well as or alternatively to the sub-processes performed by those components. In some examples, modifying the parameters may include modifying software components of the first portion of the model or retraining the first portion of the model based at least in part on the difference. In this manner, the first data determined by the first portion of the model may be adjusted such that the first data is adjusted for use as an input for the second portion of the machine learning model in future predictions.

[0054] 5 is a flow chart illustrating an example method 500 associated with a machine learning model that determines tracked object data for use by a vehicle. The method 500 begins at operation 502, which includes inputting sensor data representative of an environment in which the vehicle is operating into the machine learning model, the sensor data being associated with a current time. For example, the computing device 110 may input sensor data 106 associated with an environment 104 surrounding the vehicle 102 into the machine learning model 112. In some examples, the environment represented in the sensor data 106 may include one or more objects, such as objects 108(1) and 108(2). In some examples, the sensor data may include, but is not limited to, lidar data, radar data, ultrasonic transducer data, sonar data, position data (e.g., GPS, compass, etc.), pose data, inertial data (e.g., inertial measurement unit data, accelerometer data, magnetometer data, gyroscope data, etc.), image data (e.g., RGB data, IR data, intensity data, depth data, etc.), wheel encoder data, microphone data, environmental sensor data (e.g., temperature sensor data, humidity sensor data, light sensor data, pressure sensor data, etc.), time of flight (ToF) sensor data, etc. In some examples, the sensor data may be actual sensor data (e.g., generated by a sensor in the vehicle) or simulated sensor data.

[0055] At operation 504 of method 500, the machine learning model 112 can determine object data associated with an object in the environment, the object data indicating a location of the object in the environment. For example, the detection component 114 of the machine learning model 112 can determine object data 116 associated with an object 108 in the environment 104. In some examples, the object data may include top-down data 118 that represents a view of the environment from a top-down (e.g., bird's-eye) perspective. In some examples, the object data 116 may include bounding boxes associated with objects detected in the environment 104 by the detection component 114, such as bounding box 120(1) associated with object 108(1) and bounding box 120(2) associated with object 108(2) as shown in FIG. 1 .

[0056] At operation 506 of method 500, the machine learning model 112 may determine an estimated position of an object within the environment based at least in part on the stored tracking data 206 associated with the object. For example, the tracking component 122 may determine an estimated position of an object within the environment based at least in part on the stored tracking data 206. In some examples, the position estimation component 202, which may be a component or a sub-process of the tracking component 122, may determine the estimated position of the object. In some examples, determining the estimated position may include determining an estimated position of a bounding box associated with the object.

[0057] At operation 508 of method 500, the machine learning model 112 can associate the object data with the tracking data based at least in part on the location and the estimated location as tracked object data. For example, the tracking component 122 can associate the object data 116 with the tracking data 206 as tracked object data 124. In some examples, a track position association component and / or a trajectory estimation component, which may be components or sub-processes of the tracking component 122, can associate the tracked object data. In some examples, the association of the object data with the stored tracking data may be based at least in part on the proximity between the location and the estimated location.

[0058] At operation 510 of method 500, the machine learning model 112 may update the stored tracking data based at least in part on the tracked object data. For example, the tracking component 122 of the machine learning model 112 may update the stored tracking data. In some examples, the updated stored tracking data 206 may be stored in a memory, such as a long short-term memory, accessible to the machine learning model 112 and / or the tracking component 122.

[0059] At operation 512 of the method 500, the machine learning model 112 may output the tracked object data. At operation 514, the method 500 includes controlling operation of the vehicle. For example, operation of the vehicle may be controlled based at least in part on the tracked object data 124 received from the machine learning model 112. In some examples, a prediction component 128 of the vehicle 102 may utilize the tracked object data 124 to make one or more predictions regarding objects in the environment, such as predicted future trajectories, and the vehicle may be controlled at least in part based on the predictions. Additionally or alternatively, a planner component 134 of the vehicle 102 may utilize the tracked object data 124 and / or predictions based on the tracked object data to control operation of the vehicle. For example, the planner component 134 may determine a trajectory for the vehicle 102 to traverse an environment based at least in part on the tracked object data or the predictions.

[0060] 6 is a block diagram illustrating an example system that may be used to implement the techniques described herein. In at least one example, a vehicle 602 may correspond to vehicle 102 described above with reference to FIGS. 1-5 and may include one or more vehicle computing devices 604 (which may correspond to computing device 110), one or more sensor systems 606, one or more emitters 608, one or more communication connections 610, at least one direct connection 612, and one or more drive systems 614.

[0061] In at least one example, the vehicle 602 is an autonomous vehicle configured to operate according to a Level 6 classification issued by the National Highway Traffic Safety Administration, which may represent a vehicle capable of performing all safety-critical functions for an entire journey without the expectation of a driver (or passenger) controlling the vehicle at all times. In such an example, the vehicle 602 may be configured to control all functions from starting to stopping, including all parking functions, and thus may be unmanned. This is merely an example, and the components and methods described herein may be incorporated into any ground, air, or water vehicle, including vehicles that require manual control at all times by a driver, to those that are partially or fully autonomously controlled. That is, in the described example, the vehicle 602 is an autonomous vehicle, but the vehicle 602 may be any other type of vehicle. Although only a single vehicle 602 is shown in FIG. 6, in practical applications, the exemplary system 600 may include multiple vehicles, and in some examples, may include a fleet of vehicles.

[0062] The vehicle computing device 604 may include a processor 616 and a memory 618 communicatively coupled to the processor 616. In the illustrated example, the memory 618 of the vehicle computing device 604 stores a localization component 620, a perception component 622, a prediction component 130, a planner component 134, and one or more system controllers 624.

[0063] In at least one example, and as described above, the localization component 620 can determine a pose (position and orientation) of the vehicle 602 relative to a local and / or global map based at least in part on sensor data received from the sensor component 606 and / or map data associated with a map of the environment in which the vehicle 602 is operating. In at least one example, the localization component 620 can include or be associated with a calibration component capable of substantially simultaneously performing calibrating operations (determining various intrinsic and extrinsic parameters associated with any of the one or more sensor components 606), positioning operations, and mapping operations.

[0064] In at least one example, the perception component 622 can perform object detection, segmentation, and / or classification based at least in part on the sensor data received from the sensor component 606. In at least one example, the perception component 622 can receive raw sensor data (e.g., from the sensor component 606). In at least one example, the perception component 622 can receive the sensor data and utilize one or more processing algorithms and / or machine learning models to perform object detection, segmentation, and / or classification with respect to the object(s) identified in the sensor data. In some examples, the perception component 622 can associate a boundary region (e.g., a bounding box or otherwise instance segmentation) with the identified object and can associate a confidence score associated with the classification of the identified object with the identified object.

[0065] In examples, the perception component 622 can include the machine learning model 112 as well as the tracking data 206. The machine learning model 112, or at least the tracking component of the machine learning model 112, can utilize the tracking data 206 to estimate the position of objects in the environment, as described herein. In some examples, the tracking data 206 is stored in a memory, such as a long short-term memory.

[0066] The prediction component 130 may receive sensor data from the sensor component 606, map data, and / or perception data (e.g., processed sensor data) output from the perception component 622, and may output predictions associated with one or more objects in the environment of the vehicle 602. The predictions may include predicted trajectories associated with objects in the environment in which the vehicle 602 is operating. As described herein, the prediction component may also receive tracked object data 124 output by the machine learning model 112.

[0067] The planner component 134 may receive data, information, and / or output from the localization component 620, the perception component 622, the prediction component 130, and map data, and may generate one or more proposed vehicle operations (e.g., proposed trajectories). Additional details of usable localization, perception, prediction, and / or planner components may be found in U.S. Pat. No. 9,612,123, issued Apr. 4, 2017, and U.S. Pat. No. 10,363,390, issued Jul. 16, 2019, the entire contents of both of which are incorporated herein by reference in their entirety and for all purposes. In some examples (e.g., when the vehicle 602 is not an autonomous vehicle), one or more of the aforementioned components may be omitted from the vehicle 602. In at least one example, the planner component 134 receives tracked object data 124 from the machine learning model 112 of the perception component 622.

[0068] In at least one example, the vehicle computing device 604 can include one or more system controllers 624, which can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 602. These system controllers 624 can communicate with and / or control systems corresponding to the drive system 614 and / or other systems of the vehicle 602.

[0069] Although the above components are shown as "on-board" the vehicle 602, in other implementations, the components may be located and / or accessible remotely to the vehicle 602. For example, the components may be located remotely on a computing device 628 and accessible to the vehicle 602 via one or more networks 626. Additionally, while the components are described above as "components," such components may include one or more components and / or modules that may be part of a system to perform the operations attributed to each component.

[0070] In at least one example, the localization component 620, the perception component 622, the prediction component 130, and the planner component 134 can process data as described above and transmit their respective outputs to the computing device 628 over the network 626. In at least one example, the localization component 620, the perception component 622, the prediction component 130, and the planner component 134 can transmit their respective outputs to the computing device 628 at a particular frequency, such as after a predetermined period of time, in near real-time, or the like.

[0071] In at least one example, the sensor component 606 can include a lidar sensor, a radar sensor, an ultrasonic transducer, a sonar sensor, a position sensor (e.g., GPS, compass, etc.), an inertial sensor (e.g., inertial measurement unit, accelerometer, magnetometer, gyroscope, etc.), a camera (e.g., RGB, IR, intensity, depth, etc.), a wheel encoder, an audio sensor, an environmental sensor (e.g., temperature sensor, humidity sensor, light sensor, pressure sensor, etc.), a ToF sensor, etc. The sensor component 606 can provide input to the vehicle computing device 604. In an example, the sensor component 606 can pre-process at least a portion of the sensor data before transmitting the sensor data to the vehicle computing device 604. In at least one example, the sensor component 606 can transmit the sensor data to the computing device 628 over the network 626 in near real-time at a particular frequency after a predetermined period of time has elapsed.

[0072] Vehicle 602 may also include one or more emitters 608 that emit light and / or sound, as described above. Emitters 608 in this example include interior audio and visual emitters to communicate with occupants of vehicle 602. By way of example, and not by way of limitation, interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. Emitters 608 in this example may also include exterior emitters. By way of example, and not by way of limitation, exterior emitters in this example may include light emitters (e.g., indicator lights, signs, light arrays, etc.) for visually communicating with pedestrians, other drivers, other nearby vehicles, etc., one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians, other drivers, other nearby vehicles, etc., and the like. In at least one example, emitters 608 may be positioned at various locations on the exterior and / or interior of vehicle 602 .

[0073] Vehicle 602 may also include communications connections 610 that enable communication between vehicle 602 and other local or remote computing devices, such as computing device 628 as well as other remote or local computing devices. For example, communications connections 610 may facilitate communication with other local computing devices on vehicle 602 and / or drive system 614. Communications connections 610 may also enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Communications connections 610 may also enable vehicle 602 to communicate with remote teleoperation computing devices or other remote services.

[0074] The communication connection 610 may include physical and / or logical interfaces for connecting the vehicle computing device 604 to another computing device or network, such as the network 626. For example, the communication connection 610 may enable Wi-Fi based communications over frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth, or any suitable wired or wireless communication protocol that allows each computing device to interface with other computing devices.

[0075] The direct connection 612 can directly connect the drive system 614 to other systems of the vehicle 602. In at least one example, the vehicle 602 can include a drive system 614. In some examples, the vehicle 602 can have a single drive system 614. In at least one example, if the vehicle 602 has multiple drive systems 614, each drive system 614 can be located at either end of the vehicle 602 (e.g., at the front and the rear, etc.). In at least one example, the drive system 614 can include sensor components to detect conditions surrounding the drive system 614 and / or the vehicle 602. By way of example and not limitation, the sensor components can include wheel encoders (e.g., rotary encoders) that sense the rotation of the wheels of the drive system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) that measure the position and acceleration of the drive system, cameras or other image sensors, ultrasonic sensors that acoustically detect objects in the vicinity of the drive system, lidar sensors, radar sensors, etc. Some sensors, such as wheel encoders, can be unique to the drive system 614. In some cases, sensor components on drive system 614 may overlap or complement corresponding systems on vehicle 602 (e.g., sensor component 606).

[0076] The drive system 614 can include many vehicle systems including a high voltage battery, a motor to propel the vehicle 602, an inverter to convert direct current from the battery to alternating current for use by other vehicle systems, a steering system (which can be electric) including a steering motor and a steering rack, a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for braking force distribution to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights to illuminate the vehicle's exterior environment), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, the drive system 614 can include a drive system controller that can receive and pre-process data from sensor components and control the operation of various vehicle systems. In some examples, the drive system controller can include a processor and a memory communicatively coupled to the processor. The memory can store one or more components for performing various functions of the drive system 614. Additionally, drive systems 614 also include communication connections that enable each drive system to communicate with other local or remote computing devices.

[0077] 6, the vehicle computing device 604, the sensor components 606, the emitters 608, and the communication connections 610 are shown as being onboard the vehicle 602. However, in some examples, the vehicle computing device 604, the sensor components 606, the emitters 608, and the communication connections 610 may be implemented outside the actual vehicle (i.e., not onboard the vehicle 602).

[0078] As described above, the vehicle 602 can transmit data to the computing device 628 via the network 626. In some examples, the vehicle 602 can transmit raw sensor data to the computing device 628. In other examples, the vehicle 602 can transmit processed sensor data and / or representations of the sensor data to the computing device 628 (e.g., data output from the localization component 620, the perception component 622, the prediction component 130, the planner component 134, a machine learning model, etc.). In some examples, the vehicle 602 can transmit the sensor data to the computing device 628 at a particular frequency in near real-time after a predetermined period of time has elapsed.

[0079] Computing device 628 may receive data (raw or processed) from vehicle 602 and / or other data collection devices, as well as data from one or more third party sources and / or systems. In at least one example, computing device 628 may include a processor 630 and a memory 632 communicatively coupled to processor 630. In the illustrated example, memory 632 of computing device 628 stores a training component 634 and one or more machine learning models 636.

[0080] The training component 634 includes functionality for training one or more machine learning models 636, which may include the machine learning models 112. The training component 634 may include or store training data 638 for use in training the machine learning models 636. The training data 638 may include log data 640 associated with the vehicle traversing an environment. In some examples, the log data 640 may be sensor data representative of the environment in which the vehicle 602 was operating. That is, in some examples, the log data 640 may include image data, lidar data, radar data, etc.

[0081] The training data 638 may also include ground truth data 642. In some examples, for each instance of the log data 640 used to train the machine learning model 636, a corresponding instance of the ground truth data 642 may be stored. In some examples, the ground truth data 642 may include tracked object data associated with an object. Thus, the ground truth data 642 may include a bounding box associated with the object, a trajectory traversed by the object, a classification of the object, etc. In some examples, the ground truth data 642 may be determined by a human labeler, a trained machine learning model, etc.

[0082] The processor 616 of the vehicle 602 and the processor 630 of the computing device 628 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processors 616 and 630 may comprise one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or any other portion of a device that processes electronic data and converts it to other electronic data that may be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement encoded instructions.

[0083] Memories 618 and 632 are examples of non-transitory computer readable media. Memory 618 and memory 632 may store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functions attributed to the various systems described herein. In various implementations, memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, the ones illustrated in the accompanying drawings are merely examples relevant to the description herein.

[0084] 6 is depicted as a distributed system, it should be noted that in some examples, components of the vehicle 602 may be associated with the computing device 628 and / or components of the computing device 628 may be associated with the vehicle 602. That is, the vehicle 602 may perform one or more of the functions associated with the computing device 628, and vice versa.

[0085] Additionally, while the vehicle computing device 604 and the computing device 628 are shown as including multiple components, in some examples, such components may be associated with more or fewer individual components. For example, the localization component 620, the perception component 622, the prediction component 130, and / or the planner component 134 may be combined into a single component. That is, although shown as separate components, any one or more of the components may be combined.

[0086] Example clauses A. one or more processors; When executed by the one or more processors, the system comprises: receiving sensor data representative of objects in an environment in which the vehicle is operating, the sensor data being associated with a current time; inputting the sensor data into a machine learning model configured for use with the vehicle, the machine learning model comprising: determining object data including at least a bounding box associated with an object in an environment based at least in part on the sensor data, the bounding box indicating a position of the object in the environment at the current time; determining an estimated position of an object within the environment at the current time based at least in part on the sensor data and at least in part on tracking data determined from additional sensor data received by the machine learning model prior to the current time; associating the object data with tracking data as tracked object data based at least in part on the estimated position; and receiving an output from the machine learning model including at least the tracked object data; controlling the vehicle based at least in part on the tracked object data; One or more non-transitory computer-readable media storing instructions for performing operations including: A system equipped with.

[0087] B. The machine learning model is updating the tracking data such that a trajectory intersects the location based at least in part on associating the object data with the tracking data; storing the updated tracking data in a memory accessible to the machine learning model; The system of paragraph A, further configured to perform:

[0088] Associating the object data with the tracking data comprises: the distance between the location and the estimated location is less than a threshold distance; the detected classification of the object corresponds to the tracked classification of the object; and the difference between the detected size of the object and the tracked size of the object is less than a threshold difference; and The system of paragraph A or paragraph B, based at least in part on at least one of:

[0089] D. The system of any one of paragraphs A to C, wherein the output received from the machine learning model further indicates a future position of the object.

[0090] E. The system of any one of paragraphs A-D, wherein the additional sensor data includes time-ordered collection of frames of sensor data representing the object over a period of time prior to the current time.

[0091] F. receiving sensor data associated with a vehicle operating within an environment at a first time; inputting the sensor data into a machine learning model, the machine learning model comprising: determining a position of an object within the environment based at least in part on the sensor data; determining an estimated position of the tracked object within the environment based at least in part on a trajectory traversed by the tracked object; associating the object with the tracked object based at least in part on the location and the estimated location; and receiving an indication of the location and the trajectory as output from the machine learning model; controlling the vehicle based at least in part on the output; and The method includes:

[0092] G. The method of paragraph F, wherein the position and the estimated position are associated with the first time, and the tracked object traversed the trajectory prior to the first time.

[0093] H. The method of paragraph F or G, wherein additional sensor data represents the object over a period of time prior to the first time.

[0094] I. The method of any one of paragraphs F-H, wherein the machine learning model is configured to classify an object as at least one of a vehicle, a pedestrian, or a cyclist.

[0095] J. The machine learning model is updating the trajectory traversed by the tracked object such that the trajectory intersects the location based at least in part on associating the object with the tracked object; storing the updated trajectory in a memory accessible to the machine learning model; The method of any of paragraphs F-I, further configured to:

[0096] K. The method of any one of paragraphs F-J, wherein associating the object with the tracked object is further based at least in part on a feature distance between a first feature associated with the object and a second feature associated with the tracked object.

[0097] L. The method of any one of paragraphs F-K, wherein associating the object with the tracked object is further based at least in part on a determination that the detected classification of the object corresponds to the tracked classification of the tracked object.

[0098] M. The method of any of paragraphs F-L, wherein associating the object with the tracked object is further based at least in part on determining that the position is less than a threshold distance from the estimated position.

[0099] N. The method of any one of paragraphs F-M, wherein output from the machine learning model further includes a bounding box associated with the object, the bounding box indicating at least one of a position of the object in the environment relative to the vehicle, a size associated with the object, and an orientation associated with the object.

[0100] O. The method of any one of paragraphs F-N, wherein the sensor data includes at least one of image data, lidar data, radar data, or time-of-flight data.

[0101] P. The method of any one of paragraphs F-O, wherein the machine learning model includes a memory configured to store prior information about the object.

[0102] Q. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to: receiving sensor data associated with a vehicle operating within an environment at a first time; inputting the sensor data into a machine learning model, the machine learning model comprising: determining a position of an object within the environment based at least in part on the sensor data; determining an estimated position of the tracked object within the environment based at least in part on a trajectory traversed by the tracked object; associating the object with the tracked object based at least in part on the location and the estimated location; and receiving an indication of the location and the trajectory as output from the machine learning model; controlling the vehicle based at least in part on the output; and One or more non-transitory computer-readable media for causing a computer to perform operations including:

[0103] R. The one or more non-transitory computer-readable media of paragraph Q, wherein the position and the estimated position are associated with a first time, and the tracked object traversed the trajectory prior to the first time.

[0104] S. The one or more non-transitory computer-readable media of paragraph Q or R, wherein additional sensor data represents the object over a period of time prior to the first time.

[0105] T. The machine learning model is updating the trajectory traversed by the tracked object such that the trajectory intersects the location based at least in part on associating the object with the tracked object; storing the updated trajectory in a memory accessible to the machine learning model; One or more non-transitory computer-readable media of any of paragraphs Q-S, further configured to:

[0106] U. One or more processors; one or more non-transitory computer-readable media storing instructions; A system comprising: The instructions, when executed by the one or more processors, cause the system to: receiving sensor data comprising a first time-ordered set of frames representative of a vehicle traversing an environment; receiving ground truth data associated with an object in the environment represented in the sensor data, the ground truth data including a second time-ordered set of frames indicating at least a bounding box associated with the object, a trajectory traversed by the object, and an identifier associating the bounding box with the trajectory; inputting the sensor data into a machine learning model associated with the vehicle; receiving an output from the machine learning model, the output including at least a predicted bounding box associated with the object and a predicted trajectory traversed by the object; determining a difference between the ground truth data and the output; training the machine learning model end-to-end such that a discrepancy between the ground truth data and the output is minimized based at least in part on the discrepancy meeting or exceeding a threshold discrepancy; A system for causing an operation including

[0107] V. The system of paragraph U, wherein the sensor data includes at least one of image data, lidar data, radar data, or time-of-flight data.

[0108] W. The system of any one of paragraphs U-V, wherein the machine learning model includes a first portion and a second portion, the first portion configured to determine the predicted bounding box and the second portion configured to determine the predicted trajectory based at least in part on the predicted bounding box, and training the machine learning model end-to-end includes training the first portion of the machine learning model such that an output of the first portion of the machine learning model is conditioned for use as an input to the second portion of the machine learning model, and the output of the first portion includes at least a predicted bounding box.

[0109] X. The system of any one of paragraphs U-W, wherein a first frame in the set of time-ordered frames shows a first position of an object in the environment at a current time, and a second frame in the set of time-ordered frames shows a second position of the object along the trajectory at a previous time.

[0110] Y. Receiving sensor data associated with a vehicle traversing an environment, the sensor data representing objects within the environment over a period of time; receiving ground truth data associated with the sensor data, the ground truth data indicating at least a bounding box indicating a position of an object within the environment; inputting the sensor data into a machine learning model; receiving an output from the machine learning model including at least a predicted bounding box indicating a predicted location of an object within the environment; determining a difference between the location and the predicted location; modifying parameters of the machine learning model to minimize the variance based at least in part on the variance; and The method includes:

[0111] Z. The method of paragraph Y, wherein altering parameters of the machine learning model includes training a machine learning model end-to-end such that a first portion of the machine learning model and a second portion of the machine learning model are each trained based at least in part on the output.

[0112] AA. The method of any one of paragraphs Y-Z, wherein the machine learning model includes a first portion and a second portion, the first portion configured to determine a predicted bounding box, and the second portion configured to determine a predicted trajectory based at least in part on the predicted bounding box.

[0113] BB. The method of any one of paragraphs Y-AA, wherein the difference is a first difference, the ground truth data further indicates a trajectory traversed by the object, and the output further includes a predicted trajectory to be traversed by the object, the method further includes determining a second difference between the trajectory and the predicted trajectory, and altering a parameter of the machine learning model is further based at least in part on the second difference.

[0114] CC. The method of any one of paragraphs Y-BB, wherein the difference is a first difference, the bounding box further indicates an orientation of the object, and the predicted bounding box further indicates a predicted orientation of the object, the method further including determining a second difference between the orientation of the object and the predicted orientation of the object, and altering parameters of the machine learning model is further based at least in part on the second difference.

[0115] DD. The method of any one of paragraphs Y through CC, wherein modifying parameters of the machine learning model includes updating a software component of the machine learning model based at least in part on the difference.

[0116] EE. The method of any one of paragraphs Y-DD, wherein the sensor data includes at least one of image data, lidar data, radar data, or time-of-flight data.

[0117] FF. The method of any one of paragraphs Y-EE, wherein the object is at least one of another vehicle, a pedestrian, or a cyclist.

[0118] GG. The method of any one of paragraphs Y to FF, wherein the ground truth data further includes an identifier that associates a trajectory traversed by the object with a bounding box.

[0119] HH. The method of any one of paragraphs Y-GG, further comprising modifying parameters of the machine learning model and then sending the machine learning model to the vehicle for use in traversing the environment.

[0120] II. The method of any one of paragraphs Y-HH, wherein the machine learning model includes a first portion configured to determine a predicted bounding box indicating a position of an object within the environment, and a second portion configured to determine a predicted position of the object within the environment based at least in part on tracking data associated with the object, associate the predicted bounding box with the tracking data based at least in part on the position and the predicted position, and determine a predicted trajectory traversed by the object based at least in part on the position and the tracking data.

[0121] JJ. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to: receiving sensor data associated with a vehicle traversing an environment, the sensor data representing objects within the environment over a period of time; receiving ground truth data associated with the sensor data, the ground truth data indicating at least a bounding box indicating a position of an object within the environment; inputting the sensor data into a machine learning model; receiving an output from the machine learning model including at least a predicted bounding box indicating a predicted location of an object within the environment; determining a difference between the location and the predicted location; modifying parameters of the machine learning model to minimize the variance based at least in part on the variance; and One or more non-transitory computer-readable media for causing a computer to perform operations including:

[0122] KK. One or more non-transitory computer-readable media of paragraph JJ, wherein altering parameters of the machine learning model includes training the machine learning model end-to-end such that a first portion of the machine learning model and a second portion of the machine learning model are each trained based at least in part on the output.

[0123] LL. One or more non-transitory computer-readable media described in any one of paragraphs JJ-KK, wherein the machine learning model includes a first portion and a second portion, the first portion configured to determine a predicted bounding box and the second portion configured to determine a predicted trajectory based at least in part on the predicted bounding box.

[0124] MM. One or more non-transitory computer-readable media of any one of paragraphs JJ-LL, wherein the difference is a first difference, the ground truth data further indicates a trajectory traversed by the object, and the output further includes a predicted trajectory to be traversed by the object, and the method further includes determining a second difference between the trajectory and the predicted trajectory, and altering parameters of the machine learning model is further based at least in part on the second difference.

[0125] NN. One or more non-transitory computer-readable media described in any one of paragraphs JJ-MM, wherein modifying parameters of the machine learning model includes updating software components of the machine learning model based at least in part on the differences.

[0126] Although the example clauses set forth above are described with respect to one particular implementation, it should be understood in the context of this document that the contents of the example clauses may be implemented via methods, devices, systems, computer-readable media, and / or other implementations.

[0127] Conclusion Having described one or more examples of the technology described herein, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0128] In describing the embodiments, reference is made to the accompanying drawings which form a part hereof, and which show, by way of example, specific embodiments of the claimed subject matter. It is to be understood that other embodiments may be used and that modifications or alterations, such as structural changes, may be made. Such embodiments, modifications or alterations do not necessarily depart from the intended scope of the claimed subject matter. Although steps herein may be shown in a particular order, in some cases the order may be changed such that certain inputs are provided at different times or in a different order without changing the functionality of the systems and methods described. The disclosed procedures may also be performed in different orders. The various calculations described herein need not be performed in the order disclosed, and other examples using alternative orders of calculations may be easily implemented. Calculations may not only be rearranged, but may also be decomposed into sub-calculations which achieve the same results.

Claims

1. one or more processors; When executed by the one or more processors, the system: receiving sensor data representative of objects in an environment in which the vehicle is operating, the sensor data being associated with a current time; inputting the sensor data into a machine learning model configured for use with the vehicle, the machine learning model comprising: determining object data including at least a bounding box associated with an object in the environment based at least in part on the sensor data, the bounding box indicating a position of the object in the environment at the current time; determining an estimated position of an object within the environment at the current time based at least in part on the sensor data and at least in part on tracking data determined from additional sensor data received by the machine learning model prior to the current time; associating the object data with tracking data as tracked object data based at least in part on the estimated position; and receiving an output from the machine learning model that includes at least the tracked object data; controlling the vehicle based at least in part on the tracked object data; one or more non-transitory computer-readable media storing instructions for performing operations including: A system with.

2. The machine learning model is updating the tracking data so that a trajectory intersects the location based at least in part on associating the object data with the tracking data; storing the updated tracking data in a memory accessible to the machine learning model; The system of claim 1 , further configured to:

3. Associating the object data with the tracking data includes: the distance between the location and the estimated location is less than a threshold distance; the detected classification of the object corresponds to the tracked classification of the object; and the difference between the detected size of the object and the tracked size of the object is less than a threshold difference; 3. The system of claim 1 or claim 2, further based at least in part on at least one of:

4. The system of claim 1 , wherein the output received from the machine learning model further indicates a future position of the object.

5. The system of claim 1 , wherein the additional sensor data comprises a time-ordered collection of frames of sensor data representing the object over a period of time prior to the current time.

6. receiving sensor data associated with a vehicle operating in an environment at a first time; inputting the sensor data into a machine learning model, the machine learning model comprising: determining a position of an object within the environment based at least in part on the sensor data; determining an estimated position of a tracked object within the environment based at least in part on a trajectory traversed by the tracked object, the trajectory being determined based at least in part on additional sensor data received by the machine learning model prior to the first time; associating the object with the tracked object based at least in part on the location and the estimated location; and receiving an indication of the location and the trajectory as output from the machine learning model; controlling the vehicle based at least in part on the output; A method comprising:

7. The method of claim 6 , wherein the position and the estimated position are associated with the first time, and the tracked object traversed the trajectory before the first time.

8. The method of claim 6 or claim 7, wherein the additional sensor data represents the object over a period of time prior to the first time.

9. The method of claim 6 , wherein the machine learning model is configured to classify an object as at least one of a vehicle, a pedestrian, or a cyclist.

10. The machine learning model is updating the trajectory traversed by the tracked object so that the trajectory intersects the location based at least in part on associating the object with the tracked object; storing the updated trajectory in a memory accessible to the machine learning model; The method of claim 6 further configured to:

11. 7. The method of claim 6, wherein associating the object with the tracked object is further based at least in part on a feature distance between a first feature associated with the object and a second feature associated with the tracked object.

12. The method of claim 6 , wherein associating the object with the tracked object is further based at least in part on determining that the detected classification of the object corresponds to the tracked classification of the tracked object.

13. The method of claim 6 , wherein associating the object with the tracked object is further based at least in part on determining that the location is less than a threshold distance from the estimated location.

14. 7. The method of claim 6, wherein output from the machine learning model further includes a bounding box associated with the object, the bounding box indicating at least one of a position of the object in an environment relative to the vehicle, a size associated with the object, and an orientation associated with the object.

15. The sensor data is Image data, Lidar data, radar data, or time-of-flight data; The method of claim 6.

Images