A tool for off-line perceptual component assessment

Abstract

A computer-implemented method comprising: obtaining point-wise data indicative of point-wise measurements of a state of an object detected by an object detection system for a plurality of time steps; obtaining runtime data indicative of a runtime estimate of the state of the object for the plurality of time steps from a runtime model; processing the point-wise data with a benchmark model to determine a benchmark estimate of the state of the object for the plurality of time steps; evaluating a metric that measures deviation between the runtime estimate and the benchmark estimate of the state of the object for the plurality of time steps; and updating the runtime model based on the evaluation of the metric.

Description

[Background technology]

[0001] An autonomous vehicle typically includes various sensors and on-board data processing systems that enable it to detect entities, such as other vehicles, in its vicinity and estimate state variables associated with those entities, such as position, size, orientation, and speed. By tracking an individual entity, information derived from previous estimates of its state variables can be used to refine a current estimate of its state variables.

[0002] The details will now be described with reference to the accompanying drawings, in which the same reference numbers in different drawings indicate similar or identical items or features. [Brief description of the drawings]

[0003] [Figure 1] FIG. 1 is a schematic diagram of a system for benchmarking runtime models for deployment on a vehicle. [Diagram 2] FIG. 1 is a schematic diagram of a system for assisting in labeling training data for machine learning models. [Diagram 3] 1 shows a block diagram of an exemplary system for implementing the techniques described herein. [Figure 4] 1 shows a first tracking hypothesis for an entity in the vicinity of an autonomous vehicle. [Diagram 5] 13 illustrates a second tracking hypothesis for an entity in the vicinity of the autonomous vehicle. [Figure 6] 13 shows precision and recall curves for a runtime model and a benchmark model before and after an update to an inference system, according to an example. [Figure 7] 1 shows a flowchart illustrating a method for benchmarking a runtime model, according to an example. [Figure 8] 1 shows a flowchart illustrating a method for annotating data for training a machine learning model, according to an example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] The present disclosure relates to methods and systems for estimating the state of an entity, such as a dynamic entity in the vicinity of an autonomous vehicle. In this context, a dynamic entity (sometimes referred to as an agent) may include a vehicle or other object that is movable in an environment, and may be distinguished from a static object by its movable nature. The state of an entity refers to one or more properties associated with the entity at a given time. The state may include dynamic properties that are expected to vary over time, including values ​​of one or more kinematic variables, such as the entity's position and / or orientation, as well as time derivatives of these quantities, such as rotational speed and / or rate. Alternatively, or in addition, the state of an entity may include static properties that are expected to remain constant over time, including geometric properties of the entity, such as the entity's dimensions, size, and / or shape. The state of an entity may be represented as a vector with components that represent the respective state variables. In the context of an entity in the vicinity of an autonomous vehicle, the vector may represent, for example, planar (horizontal) position, planar velocity, yaw, and yaw rate. Additionally or alternatively, other variables, such as vertical position, pitch, or roll, may be included in the state along with their associated time derivatives.

[0005] The autonomous vehicle may include one or more sensors and an object detection system for detecting entities in the vicinity of the autonomous vehicle. The autonomous vehicle may further include an on-board perception component that may be configured to estimate a state of the detected entity using sensor data collected from one or more sensors at a given time point or within a narrow time window (such as a few milliseconds) around a given time point. An estimate of the state derived from data associated with a given time point may be referred to as a point-wise measurement of the state.

[0006] Point-wise measurements of the state may be subject to various sources of error and noise, which may include observation noise caused by inherent uncertainties associated with the sensor(s), occlusion errors caused by entities being fully or partially occluded from one or more sensors, and kinematic noise caused by unpredictable motion of the sensor(s), e.g., vibrations caused by uneven road surfaces. The impact of such errors on the point-wise measurements of the state may be mitigated using information derived from one or more previous estimates of the state. In particular, by tracking individual entities over time, a noise filter may be applied recursively to point-wise measurements of the entity's state at a series of time steps to determine run-time estimates of these states, thereby mitigating at least some of the sources of error that may affect the point-wise measurements. To facilitate this, the on-board perception component of the autonomous vehicle may include a data association model for associating instances of the entity detected at different time steps, along with a noise filter.

[0007] Prior to deployment of the data association model and / or noise filter, values ​​of associated parameters of the data association model and / or noise filter may be determined based on physical considerations (e.g., based on kinematic equations of motion and / or known uncertainties associated with one or more sensors) or based on empirical data. For example, values ​​of associated parameters may be determined using online or offline machine learning methods. A difficulty with the latter approach is that ground truth states of entities observed in the environment are rarely available unless simulations are used, but such simulations may not accurately represent the various sources of noise and error present in a real physical environment. More generally, ground truth data may be difficult to obtain for machine learning tasks related to the control of an autonomous vehicle because the process of labeling data that covers a sufficiently diverse set of scenarios is very time-consuming and resource-intensive.

[0008] In consideration of the above problems, the present disclosure provides an offline tool configured to process point-wise measurements of states of detected entities over a period of time to determine offline estimates of the states of those entities. The offline estimates may be determined using an offline model and may more accurately represent the true states of the entities than runtime estimates determined using a runtime model. Thus, the offline estimates may be treated as pseudo ground truth values ​​of those states in situations where a sufficient amount / diversity of ground truth data is not readily available.

[0009] In contrast to runtime models, offline models of the present disclosure may not be suitable for execution in an online setting, for example, on board an autonomous vehicle. In particular, to determine an offline estimate of an entity's state at a given time step, an offline model may be configured to use information from time steps later than the given time step. This is in contrast to a runtime model that may only use information from previous time steps. A model that requires information from future time steps will necessarily operate with a temporal lag, which may be undesirable in the context of a runtime model in an autonomous vehicle or other settings where decisions need to be made with minimal delay. Furthermore, execution of an offline model may require significantly more processing than execution of a runtime model, which may result in unacceptable levels of lag and / or prevent the offline model from being able to execute in real-time or near real-time. In some cases, the computational cost of executing an offline model may be prohibitively high for execution by a computer system on board an autonomous vehicle.

[0010] Examples of noise filters suitable for implementation onboard an autonomous vehicle may include Kalman-type filters (such as Kalman, extended Kalman, or unscented Kalman filters), although other filters such as particle filters and Gaussian process-based filters may additionally or alternatively be employed. In this manner, if noise that was uncorrelated from one time step to the next can be reasonably modeled, point-wise measurements of a state may be refined based on information derived from point-wise measurements of previous states (and optionally one or more earlier states), such that various types of noise may be filtered out of the measurements. Such filters may be used in combination with a robust loss function that reduces the sensitivity of the output of the noise filter to outliers, allowing for further refinement.

[0011] To apply a noise filter to a given sequence of point-wise measurements, instances of an entity detected at different time steps may first be associated with one or more "tracks" using a data association model, and the noise filter may be applied to instances associated with a common track. Entities associated with tracks using a data association model may be referred to as track entities or track objects. The data association model aims to accurately associate instances of the same entity at different time steps with a common track. If an instance of an entity cannot be associated with an existing track (e.g., because the entity has just entered the range of a sensor and / or object detection system), the data association model may create a new track or reject the instance as a false positive. In a multi-entity or multi-agent setting (which is common in driving environments), the data association task may not have a unique solution. For example, if an entity becomes occluded for one or more time steps, it may be difficult to determine whether a subsequent detection corresponds to the same entity or to an entirely new entity. To determine whether an instance of an entity detected at a given time step should be associated with an existing track or whether a new track should be started, the data association model may use an estimate of the state from the previous time step. For example, the mean and covariance of noise filtered estimates from previous time steps may be used to define a "gating" region of the state space where a new measurement may be determined to correspond to the same entity. Further criteria may be introduced to resolve ambiguities, for example when multiple measurements appear within a gating region of the state space. Such criteria may, for example, use classification and / or bounding box losses to resolve such ambiguities. Examples of data association models include network flow-based models, Markov Chain Monte Carlo (MCMC) data association, joint probability data association, and multidimensional assignment.

[0012] As described above, the combination of the data association model and the filtering model may be implemented in combination as a runtime (or "online") model, for example, to be executed by computing hardware on board the autonomous vehicle. The runtime model may be configured to process point-wise measurements of states of entities detected in the vicinity of the autonomous vehicle to generate runtime estimates of the states of the entities, which may be used to predict trajectories of the entities and plan what actions, if any, the autonomous vehicle should take depending on the predicted trajectories of the entities. The runtime model thereby enables the autonomous vehicle to take actions that are less adversely affected by noise than if the point-wise measurements were used directly.

[0013] The performance of the data association model and the filtering model (collectively referred to as the runtime model) may be affected by the values ​​of several associated parameters. For example, the data association model may include a parameter to control whether a new track is created for a given detection of an entity (as opposed to classifying it as a false positive) and / or whether a detected entity is associated with an existing track. The filtering model may include parameters that characterize various type(s) of noise / error associated with the point-wise measurements. In the case of a Kalman-type filter, the point-wise measurements z of a state at a given time step t t Let x be the true state at time step t. t As noisy observations of z t =g(X t )+v t where g is the measurement operator and v t is the measurement noise that captures the random error associated with the measurement of the state x at time step t. tis the state x at the previous time step t-1 by applying a linear or nonlinear state transition operator f, where f represents the time evolution of the state. t-1 In some examples, at least some components of the state, such as the dimensions of the entities, are expected to be constant over time, while other components, such as the positions of the entities, may vary over time. The time evolution of the state is governed by the state transition noise w t This state transition noise is assumed to be influenced by x t =f(x t-1 )+w t Depending on the filtering model, the operators f and / or g may each have a known parametric form (e.g., based on known equations of motion), in which case the parameters of the operators f and / or g may be the parameters of the runtime model. Alternatively, the operators f and / or g may be modeled in a non-parametric manner, e.g., governed by a latent Gaussian process, in which case the hyper-parameters of the latent Gaussian process may be the parameters of the runtime model. The measurement noise v t and state transition noise w t may be modeled as Gaussian noise parameterized by respective covariance matrices, in which case the components of these covariance matrices may also be parameters of the runtime model.

[0014] The state transition operator f may depend on the classification of the entities, reflecting the fact that different entities may be subject to different dynamics. For example, a vehicle with four-wheel steering may be able to move (e.g., drift) in a direction different from the direction the vehicle is facing, while a vehicle with two-wheel steering may only be able to move in the direction the vehicle is facing (unless a situation is identified that may cause the vehicle to lose traction with the road).

[0015] While the above run-time model may include a noise filter, the offline model of the present disclosure may include a smoother such as a Rauch-Tung-Striebel (RTS) smoother, a two-filter smoother, a sequence importance resampling smoother, a Rao-Blackwellized particle smoother, or a grid-based smoother. For a given noise filter (such as a Kalman-type filter), the associated smoother may be constructed by applying recursion in a reverse time direction to update and improve the estimate from the filtering model. In this manner, information from future time steps may be used to significantly reduce the uncertainty of the estimate for a given time step. By incorporating information from future time steps, certain hypotheses regarding the state may be ruled out. The smoother may be a fixed-lag smoother configured to estimate the state of the entity for a predetermined number of time steps prior to the latest available time step. Alternatively, the smoother may be a fixed-point smoother configured to estimate the state of the entity at a fixed time step using information from an increasing number of future time steps. Alternatively, the smoother may be a fixed interval smoother configured to estimate the state of the entity over a fixed interval, given point-wise measurements of the state of the entity over the same interval. The smoother may be an optimal smoother in the Bayesian sense (a.k.a. Bayesian optimal), i.e., given a filtering model, the smoother may determine the most likely value of the state at each time step, given the information available to the smoother. The optimal smoothing formula is available for all Kalman-type filters and may be evaluated using a series of recursive calculations in reverse time. Effectively, the smoother may take the information captured at a sequence of time steps, solve a Bayesian optimization problem, and give the most likely configuration of the state over that sequence of time steps (in other words, the joint distribution of the state with the highest likelihood). In an offline setting, the parameters associated with the smoother (e.g., the measurement noise v t and state transition noise w tThe components of the covariance matrix associated with θ (θ,θ) may be optimized or learned from the data, either for individual tracks or globally for all tracks in a given time window, so that their state estimates can be smoothed and Bayes-optimal with respect to the selected filtering model.

[0016] The smoother may be a Gaussian process-based smoother. This type of smoother may be associated with a Gaussian process-based filter and may not assume a parametric form of the underlying state transition operator f and measurement operator g, but instead may associate each of these operators with one or more Gaussian processes that use Bayesian inference to determine the posterior distribution. For a particular Gaussian process model, exact inference may be performed, allowing the joint posterior distribution of the states and their uncertainties over the time step sequence to be determined in a closed form. Alternatively, approximate inference may be used, for example, using sampling techniques or sparse approximations of the underlying Gaussian process. In either case, the hyperparameters of the Gaussian process may be optimized using maximum a posteriori (MAP) estimation, maximum likelihood estimation, evidence maximization, or sampling. Gaussian process inference, including hyperparameter optimization, may be computationally very expensive and not suitable for use in an online setting.

[0017] The offline model may include a multiple hypothesis batch tracking model. Over a time step sequence, the multiple hypothesis tracking model may be configured to build and update a so-called track tree that encapsulates one or more tracking hypotheses. The track tree may have nodes corresponding to detections of one or more entities at each of the time step sequences and branches connecting detections in the hypothesis tracks (or trajectories) between time steps. At a given time step, a new track tree may be built for each entity detected at that time step, taking into account the possibility that the detection corresponds to a new entity that falls within the detection range. Existing track trees may also be updated with detections from a given time step. In particular, an existing track tree may be extended by adding any new detections that are consistent with existing nodes in the track tree as separate branches. A new detection at a given time step may be considered consistent with an existing node in the track tree if a point-wise measurement of its state (or simply another subset of its positions or state variables) is within a predefined metric distance D (e.g., a predefined Mahalanobis distance) of the state (or subset of states) predicted for that time step according to a filtering model applied to the existing nodes, or in other words, if the measurement is within a region of state space according to a noise filtered estimate of the state at the previous time step. Further branches may be added to the track tree to account for occlusions, or other missing detections of entities at a given time step.

[0018] The track tree may be pruned at some or all time steps according to one or more pruning criteria, and the track hypotheses remaining after pruning may be scored depending on their Bayesian likelihood derived from, for example, a filtering model and / or other factors such as bounding box loss and / or classification loss. After pruning, a configuration of track hypotheses (i.e., global hypotheses) with the highest overall scores may be determined by solving a discrete optimization problem (e.g., a maximum weighted independent set (MWIS) optimization problem), and the determined configuration may be identified as the correct global hypothesis at a given time step, such that each track in the identified global hypothesis may be determined to correspond to multiple instances of a common entity.

[0019] In an online setting, multiple hypothesis tracking models are constrained by the need to prune the track tree quickly to prevent an exponential growth in the number of track hypotheses, as well as the fact that the model can only use information from previous time steps. Either or both of these constraints can lead to missing true track configurations, especially if the parameters of the tracking model are not properly tuned. Pruning may be performed using various techniques, such as N-scan pruning, where a branch corresponding to a global hypothesis at time step k is traced backwards to its corresponding node at time step kN (for a given parameter value N), thereby removing subtrees that diverge from the global hypothesis at that node. In this way, track tree branches that are far from the global hypothesis are removed. In practice, data association ambiguities up to time step kN are resolved by looking ahead a window of N frames. Other examples of pruning criteria include pruning track trees with branches that exceed a given value B, for example by keeping only the branches with the highest scores B. In an online setting, values ​​for parameters D, B, and / or N (and possibly other parameters depending on the data association model) may be selected to satisfy a trade-off between accuracy and computational cost. For example, selecting larger values ​​of D, B, and / or N reduces the probability of missing a true global hypothesis, but increases the computational cost and run-time of the data association model. Therefore, online settings may be restricted to using relatively small values ​​for these parameters, which may limit the practicality of multiple hypothesis tracking models for use in online settings.

[0020] In this setting, the offline model may utilize a multiple hypothesis tracking model (or any other suitable type of tracking model) in an offline manner. As described above, the offline model is not restricted to using information from past time steps, and temporal lag may not be a primary consideration as in the case of the online model. Lifting these restrictions may enable the use of parameter values ​​that result in a highly accurate data association model (e.g., high values ​​of D, B and / or N of the multiple hypothesis batch tracking model described above). Furthermore, the data association model may be run in forward and / or backward time directions, which may further improve accuracy. In certain examples, a smoother (e.g., one of the smoothers described above) may be run for each candidate track at each time step, which may result in more accurate estimates of the Bayesian likelihood and more accurate determination of the correct track configuration. In other examples, the offline model may forego the use of a track tree altogether, and may, for example, perform a full search for track configurations within a fixed time horizon. Although the number of track configurations may grow exponentially with the size of the time horizon, limiting the size of the time horizon allows all track configurations to be considered within a reasonable time scale in an offline setting. In this example, a smoother may be applied to each track in each candidate track configuration, allowing accurate Bayesian likelihoods to be determined for each track for data association purposes. Such methods may be applied, for example, using a sliding time window.

[0021] By using a Bayesian optimal smoother and an accurate data association model such as a multiple hypothesis batch tracking model, the offline model can generate a global Bayesian optimal set of estimates of the states of entities detected within a given time window. Estimates such as these may be treated as pseudo ground truth values ​​of the states, for example in settings where ground truth data is unavailable or insufficient for a given task.

[0022] To expand on the above, FIG. 1 illustrates an example of using an offline model as described above to generate pseudo ground truth data for benchmarking a runtime model for deployment in an autonomous vehicle. FIG. 1 illustrates an autonomous vehicle 100 having an on-board perception component (not shown) configured to process data captured by on-board sensors 102 to detect and classify entities in the vicinity of the vehicle and estimate states of the detected entities. In this example, three dynamic entities 104a, 104b, 104c (collectively referred to as entities 104) are detected in the vicinity of the vehicle 100 and classified as vehicles, and the on-board perception component determines point-wise measurements of the state of the entities 104 (including position, velocity, yaw, and yaw rate, represented by the arrows below the entities 104) at a sequence of time steps where the entities 104 are within range of the sensors 102. The on-board perception system is further configured to process the point-wise measurements using the runtime model described above to determine a runtime estimate of the state of the entities 104. These runtime estimates are communicated to an on-board prediction component and an on-board planning component, which together determine the actions to be taken by the drive system of vehicle 100. It should be noted that the on-board perception component is for determining the state of objects or entities other than vehicle 100. Vehicle 100 may further include a self-localization component, which, in contrast, is for determining the position and / or orientation of vehicle 100 itself.

[0023] During operation, the on-board perception components of the vehicle 100 may generate point-wise data 106 indicative of point-wise measurements of states of the entities 104 at different time steps. The point-wise data 106 may further include metadata related to downstream processing of the point-wise measurements, such as timestamps and classifications of the entities 104. The point-wise data 106 may be stored in any suitable format by one or more memory devices on-board the vehicle 100, for example in a log file and / or a relational database. Optionally, raw sensor data from which the point-wise measurements are derived may also be stored in association with the point-wise data 106. In an example of the invention, the vehicle 100 provides the point-wise data 106 to the remote system 108, for example by transmission over a network (not shown) using wired and / or wireless communication means. The point-wise data 106 may be provided to the remote system 108 in a streaming or batch manner, either periodically (e.g., hourly, daily, weekly, etc.), or when certain conditions are met, such as when a suitable wired or wireless connection to a network is established and / or when a certain amount of point-wise data 106 has been generated. The vehicle 100 may additionally or alternatively provide the remote system 108 with raw sensor data and / or runtime estimates determined using the runtime model.

[0024] The remote system 108 includes an offline perception component 110 that has similar functionality to the on-board perception components of the vehicle 100. The offline perception component 110 includes a runtime model 112. Although the runtime model 112 is stored and executed on the remote system 108, the runtime model 112 may be suitable for being stored and executed on-board a vehicle, such as the autonomous vehicle 100. The runtime model 112 may be an instance of the same runtime model stored on-board the vehicle 100, or may differ from the runtime model stored on-board the vehicle 100, for example, by having different parameter values ​​and / or by implementing different data association and / or filtering models.

[0025] The offline perception component 110 may be configured to use the runtime model 112 to process the point-wise data 106 received from the vehicle 100, together with point-wise data received from other (possibly many) vehicles, to generate runtime estimates 114 of states of entities detected by those vehicles. The runtime model 112 may be configured to associate a given instance of an entity detected at a given time step with further instances detected at previous time steps according to a data association model. The runtime model 112 may be configured to filter point-wise measurements of states associated with a given instance as a function of runtime estimates of states associated with the further instances at previous time steps according to a filtering model. In this manner, the runtime model iteratively generates runtime estimates 114 of states of detected entities using information available in the runtime configuration (i.e., without using information from future time steps). It should be noted that in some examples, the offline perception component 110 may be omitted and the perception component may receive runtime estimates 114 directly from one or more vehicles running an instance of the common runtime model.

[0026] The remote system 108 further includes a benchmark component 116. The benchmark component 116 may be configured to process the point-wise data 106 received from the vehicle 100, along with point-wise data received from other (possibly many) vehicles, using a benchmark model 118 to generate a benchmark estimate 120 of the state of the entities detected by those vehicles. The benchmark model 118 may include an offline data association model and a smoother, e.g., as described above, and accordingly may generate a more accurate estimate of the state of the entities than the runtime model 112. In particular, the benchmark model 118 may use information from future time steps to determine the benchmark estimate at a given time step. Furthermore, in some cases, the benchmark model 118 may arrive at a different track hypothesis than the runtime model 112, as shown in the following example.

[0027] 4 shows a top-down view of a scene including an autonomous vehicle 100. In this scene, the autonomous vehicle 100 is traveling behind a bus 402, such that there are occlusion regions 404 of the scene (as shown between the solid lines in FIG. 4) that are hidden from at least a portion of the sensors on the vehicle 100. Using the sensors 102 and an on-board data processing system, the autonomous vehicle 100 detects instances of entities classified as “cars” over a set of time steps t a , t b , t c , t d (positions for the autonomous vehicle 100 are shown simultaneously as instances 406a, 406b, 406c, 406d, and 406e in FIG. 4). In this example, the time t c and d The spacing between adjacent time steps in each pair is equal, except for the spacing between time steps t a , t b , t c , t dAt each of the time steps, the autonomous vehicle 100 determines point-wise measurements of states associated with the instances detected at that time step and transmits point-wise data 106 indicative of the point-wise measurements at the time step to a remote system 108.

[0028] The offline perception component 110 processes the point-wise data 106 using a runtime model 112 and computes the time step t a , t b , t c , t d The runtime model 112 determines runtime estimates of states associated with the instances 406a, 406b, 406c, 406d detected in the time step. As described above, the runtime model 112 includes a data association model and a filtering model. In this example, the data association model associates each of the instances 406a, 406b, 406c, 406d with a common track (shown by the dashed curved line) corresponding to the hypothesis that all of the instances 406a, 406b, 406c, 406d are instances of the same car. The uncertainty of the filtered estimates of states is represented by the dashed bounding boxes around the instances 406a, 406b, 406c, 406d. It is observed that the uncertainty in the states associated with the instances 406a, 406b, 406c is relatively low, but the uncertainty in the state associated with the instance 406d is higher. This is because the interval that has elapsed since the previous time step is large, and the longer the time interval between measurements, the greater the uncertainty under the filtering model.

[0029] The benchmark component 116 processes the point-wise data 106 using the benchmark model 118 and calculates the time step t a , t b , t c , t d5, the benchmark model 118 associates each of the instances 406a, 406b, 406c, 406d with a common track, but starts a new track for the instance 406d in response to the hypothesis that the instance 406d is a different car than the instances 406a, 406b, 406c. In this case, the benchmark model 118's hypothesis corresponds to the ground truth, while the runtime model 112's hypothesis is inaccurate (the dashed instances 408a, 408b show the ground truth positions of the two cars at a time step when the two cars are occluded). Furthermore, it is observed that the uncertainty in the state associated with the instances is consistently lower for the benchmark model 118 than for the runtime model 112, and even for the instance 406d that starts a new track. The benchmark model 118 is able to correctly resolve the tracking ambiguity and determine a more accurate estimate of the state. This is at least in part because the benchmark model 118 can leverage information from future time steps, while the runtime model is constrained to use information from past time steps. The benchmark model 118 may further use a different data association model than the runtime model 112.

[0030] The offline perception component 110 and the benchmark component 116 may send the runtime estimates 114 and the benchmark estimates 120 to the model evaluator and updater 122, optionally with similar data generated from other sources (e.g., many other autonomous vehicles). The model evaluator and updater 122 is configured to process the runtime estimates 114 and the benchmark estimates 120 to generate data corresponding to an updated runtime model 126. In particular, the model evaluator and updater 122 may be configured to evaluate a metric 124 that measures the deviation between the runtime estimates 114 and the corresponding benchmark estimates 120, and to update the runtime model 112 in response to the evaluation of the metric. By treating the benchmark estimates 120 as a pseudo ground truth, the metric 124 may be used to measure the performance of the runtime model 112. Because the performance of the runtime model 112 may be affected by various factors, as described below, the benchmark component 116 and the model evaluator and updater 122 may be run separately on different sets of point-wise data 106, resulting in multiple versions of the updated runtime model 126.

[0031] The runtime model 112 may be executed differently in different environmental conditions, e.g., different times of day (corresponding to different lighting conditions), different driving environments (such as urban or rural environments), and / or different weather conditions. Thus, components of the remote system 108 may be executed separately using the point-wise data 106 for each different environmental condition to generate different updated runtime models 126 appropriate for the different environmental conditions. For example, in dark or snowy conditions, measurement noise may be greater and it may be more difficult for the runtime model 112 to associate object instances with a given track, in which case a different threshold value may be appropriate for object association.

[0032] The runtime model 112 may be executed differently when used with different perception components or different versions of perception components. Thus, components of the remote system 108 may be executed separately using point-wise data 106 generated by different perception components or different versions of perception components, resulting in different updated runtime models 126 appropriate for each different perception component of the different versions of perception components. For example, different values ​​and thresholds of measurement noise may be appropriate for object association depending on the accuracy of the object detection model used in generating the point-wise data 106.

[0033] Metric 124 may measure pairwise deviations between the runtime estimates of the state and the benchmark estimates. For example, metric 124 may depend on the metric distance between the runtime estimates of the state of the entity detected at a given time step and the benchmark estimates of the state of the entity. Metric 124 may be, for example, a function of L1 loss, smoothed L1 loss, L2 loss, or any other suitable pairwise distance measure. By summing or otherwise combining these losses over multiple detections in a given period, metric 124 may measure the performance of the runtime model 112 over that period. Metric 124 may additionally or alternatively measure pairwise deviations between the uncertainty estimates between the runtime estimates of the state and the benchmark estimates. For example, for each detection at each time step, the Kalman-type filter (and corresponding smoother) may generate a posterior covariance matrix of the state, which may be considered as an estimate of the predictive accuracy of the runtime estimate of the state. Metric 124 may then depend on the metric distance between the posterior covariance of the runtime estimate and the posterior covariance of the corresponding benchmark estimate. By measuring the deviation between a state estimate and an associated uncertainty estimate, metric 124 may measure the effectiveness of runtime model 112 in filtering out noise in the state measurements and in estimating the confidence that can be attributed to the resulting estimate. Quantifying uncertainty is important in safety-critical environments, such as driving environments, because certain actions are only taken if there is sufficient confidence in the perception of the environment.

[0034] Instead of measuring pairwise deviations, metric 124 may measure the deviation between a joint probability distribution of runtime estimates 114 and benchmark estimates 120. For example, metric 124 may depend on the Kullback-Leibler (KL) divergence of a first probability distribution from a second probability distribution, where the first (or second) probability distribution may be the joint probability distribution of the runtime estimates (defined by the runtime estimates and their corresponding posterior covariances) and the second (or first) probability distribution may be the joint probability distribution of the corresponding benchmark estimates. The joint distribution may be obtained over multiple detections, e.g., within a given time window, optionally over multiple time steps.

[0035] As explained above, the runtime model 112 may perform less accurate data association than the benchmark model 118. The metric 124, which measures the deviation between the runtime estimates 114 and the benchmark estimates 120, and optionally their corresponding uncertainty estimates, may automatically capture situations where the tracking predictions diverge between the runtime model 112 and the benchmark model 118, as a result of the different tracking predictions using information from different previous states to filter noise. In the example of FIG. 4, the runtime estimate of the state associated with instance 406d is derived from the runtime estimate of the state associated with instance 406c. In contrast, the benchmark estimate of the state associated with instance 406d does not depend on the state associated with instance 406c. Even if the filtering model is very accurate, these different tracking predictions are likely to cause differences between the runtime and benchmark estimates. In this way, the metric 124 may capture deviations between the two data association models when tracking ambiguities. In other cases, it may be preferable for the metric 124 to compare the runtime and benchmark estimates only if they produce the same tracking predictions, for example. In this manner, the metric 124 may measure the accuracy of the filtering model as opposed to the combined effect of the data association model and the filtering model.

[0036] In addition to or instead of measuring the deviation between state estimates, metric 124 may explicitly measure the deviation between tracking predictions made by runtime model 112 and benchmark model 118. For example, metric 124 may count the number of times that runtime model 112 and benchmark model 118 diverge from each other within a given time frame. Alternatively, the data association problem for a given time step may be viewed as a classification problem, for example, where each detected instance is classified as either a new entity, a false positive, or an entity seen in a previous time step, in which case metric 124 may measure the precision and recall of this classification problem (treating the output of benchmark model 118 as ground truth). Alternatively, metric 124 may measure the precision and recall on the cardinality (i.e., the number of entities estimated to be present) of a set of entities over one or more time steps.

[0037] The model evaluator and updater 122 is configured to generate data corresponding to an updated runtime model 126 in response to an evaluation of the metrics 124. The updated runtime model 126 may depend on an evaluation of the metrics 124 aggregated across multiple data sources, such as multiple autonomous vehicles including the vehicle 100. The model evaluator and updater 122 may determine, for example, based on the evaluation of the metrics 124, that the runtime model 112 needs to be updated because one or more aspects of the performance of the runtime model 112 differs from that of the benchmark model 120 by more than a threshold amount. The model evaluator and updater 122 may determine that a data association portion of the runtime model 112 needs to be updated and / or that a filtering portion of the runtime model 112 needs to be updated. The model evaluator and updater 122 may update the indicated aspect(s) of the runtime model 112. The updated runtime model 126 may then be evaluated against the benchmark model 118 to determine whether the updates were effective in improving the runtime model 112. Updating the runtime model 112 may include, for example, replacing a data association model and / or a filtering model with a different data association model and / or filtering model.

[0038] Updating the runtime model 112 may include updating values ​​of one or more parameters associated with the data association model and / or the filtering model. Updating may include sampling new values ​​of the one or more parameters using a sampling technique such as random search, grid search, or MCMC sampling. The sampling may be performed iteratively in a manner that optimizes the value of the one or more parameters with respect to the evaluation of the metric 124. For example, the updating may use MCMC sampling where the sampling distribution is conditioned on the evaluation of the metric 124. In some examples, Bayesian optimization may be performed, providing a principled framework for dealing with the exploration / exploitation dilemma encountered when optimizing parameter values. A suitable Bayesian optimization method may employ a surrogate function, such as a Gaussian process, a Bayesian neural network, or other probabilistic function or process, to predict the evaluation of the metric 124 for a given set of parameter values, and then derive an acquisition function from the surrogate function, e.g., based on an entropy search or expected improvement, and sample parameter values ​​based on the acquisition function. The acquisition function may be configured to automatically balance exploration and exploitation (e.g., moving from exploration in early stages of an experiment toward exploitation in later stages of an experiment). In other examples, the sampling of parameter values ​​may be performed by a reinforcement learning agent.

[0039] Alternatively, or in addition, values ​​of one or more parameters of the runtime model 112 may be optimized using gradient-based optimization, such as using stochastic gradient descent or any of its variants. Gradient-based optimization may be suitable for situations in which the metric 124 is differentiable with respect to one or more parameters of the runtime model 112.

[0040] In some examples, the remote system 108 may have access to ground truth data of the tracks and / or states of the detected entities. In this case, the model updater 122 may further use the ground truth data to update the runtime model 112. For example, the model updater 122 may use the ground truth data to measure the accuracy of the benchmark model 118 and the accuracy of the runtime model 112, enabling the model updater 112 to determine whether inaccuracies in the track determinations of states and / or runtime estimates of those states determined by the runtime model 112 are caused by deficiencies in the runtime model 112 (in which case the runtime model 112 is expected to be significantly worse than the benchmark model 118) and / or inaccuracies in the point-wise data 106 (in which case the output of the benchmark model 118 is expected to differ significantly from the ground truth).

[0041] In the example of FIG. 6, the top box 600a shows a first precision and recall curve 602 and a second precision and recall curve 604 for the task of estimating cardinality, where the first curve 602 corresponds to the benchmark model 118 and the second curve 604 corresponds to the runtime model 112. For both models, precision and recall are measured against ground truth and aggregated from multiple data sources. It is observed that the area under the second curve 604 is significantly smaller than the area under the first curve 602, indicating that the benchmark model 118 outperforms the runtime model 112 in this task. In this example, the model updater 122 updates the parameter values ​​of the runtime model 112 in response to a comparison between the first precision and recall curve 602 and the second precision and recall curve 604. For example, the model updater 112 may perform MCMC sampling, where the sampling distribution is conditioned on the difference between the first precision and recall curve 602 and the second precision and recall curve 604, e.g., the difference between the areas of the curves. 6 shows a first precision and recall curve 602 and a third precision and recall curve 606, where the third curve 606 corresponds to the updated runtime model 112. It is observed that the third curve 606 is closer to the first curve 602 than the second curve 604, indicating that the runtime model 112 has improved as a result of the update.

[0042] While the system 108 is described above as being remote from the vehicle 100, in other examples, the vehicle may have an on-board component for implementing benchmark models and generating update data to update the runtime models for the vehicle. This allows a vehicle to adapt its own runtime models based on data generated in its own environment, but may not benefit from data generated by other vehicles.

[0043] FIG. 7 illustrates an example of a computer-implemented method 700 that may be performed by a computing system, such as the remote system 108 of FIG. 1. The method 700 includes, at 702, obtaining point-wise data indicative of point-wise measurements of a state of an object detected by an object detection system over a number of time steps. The object may be, for example, an object in the vicinity of an autonomous vehicle or other type of vehicle. The object detection system may be part of a perception component mounted on the vehicle, in which case the point-wise data may be received from the perception component mounted on the vehicle. Alternatively, the point-wise data may be determined remotely from the vehicle, for example by processing raw sensor data or logged data received from the vehicle.

[0044] The method 700 proceeds, at 704, by obtaining runtime data indicative of a runtime estimate of a state of the object for a number of time steps. The runtime data is generated by a runtime model, which may be implemented as part of a perception system onboard the vehicle or may be implemented remotely from the vehicle. The runtime data may be generated by recursively processing the point-wise data, as described elsewhere in this disclosure. The runtime model may include, for example, an online data association model and a filtering model.

[0045] The method 700 proceeds, at 706, by processing the point-wise data to determine benchmark estimates of the object's state for a number of time steps. The benchmark data is generated by a benchmark model, which may be an offline system that is remote from a source of the point-wise data. The benchmark model may include determining a benchmark estimate of the object's state at a given time step based on point-wise measurements of the object's state at the given time step and a number of additional time steps, which may include at least one time step later than the given time step. In this manner, at least, the benchmark model may be contrasted with a runtime model. The benchmark model may include, for example, an offline data association model and a smoother.

[0046] The method 700 proceeds, at 708, by evaluating a metric that measures the deviation between a run-time estimate of an object's state and a benchmark estimate for multiple time steps.

[0047] The method 700 concludes at 710 by updating the runtime model based on the evaluation of the metrics. The updated runtime model may optionally be used to generate further runtime data, so that the method 700 returns to 704 and continues iteratively until a stopping condition, such as a convergence condition, is met or a predetermined number of iterations have been performed.

[0048] As a further illustration, Figure 2 shows an example in which an offline model such as that described above is used to assist in labeling data, e.g., to label training data for a machine learning model. Examples of techniques for assisting in labeling data can be found, for example, in U.S. Patent Application No. 17 / 538,909, filed November 30, 2021, and entitled "Generating and Training Object Detection Models for Autonomous Vehicles," the contents of which are incorporated by reference in their entirety and for all purposes.

[0049] FIG. 2 illustrates an autonomous vehicle 200 equipped with sensors 202 for detecting and determining point-wise measurements of states of dynamic entities, including vehicles 204a, 204b, 204c. The autonomous vehicle 200 of FIG. 2 may be functionally identical to the autonomous vehicle 100 of FIG. 1 and is similarly configured to transmit point-wise data 206 to a remote system 208. The point-wise data 206 is indicative of point-wise measurements of states of entities detected in the vicinity of the autonomous vehicle 200 at different time steps. In this example, the vehicle 200 further provides sensor data 209 captured by the on-board sensors 202 to the remote system 208. The sensor data 209 may be raw sensor data and / or processed sensor data and includes data from which a visual representation of the environment surrounding the vehicle 200 can be derived. In this example, the vehicle 200 provides the point-wise data 206 to the remote system 208, although in other examples, the remote system 208 may instead derive the point-wise data 206 from the sensor data 209.

[0050] The sensor data 209 may include sufficient input (test) data for machine learning models (e.g., neural network models) configured to perform object detection and / or other tasks related to controlling an autonomous vehicle, including, but not limited to, semantic segmentation, instance segmentation, object classification, and object tracking. Machine learning models for these purposes are typically trained using supervised learning based on labeled training data. The process of obtaining such labeled training data that covers a sufficient variety of scenarios can be very time-consuming and resource-intensive, and traditionally involves a human user manually applying labels or annotations. A label is metadata associated with an input data item (e.g., an image) that can be compared to the output of a machine learning model during supervised learning. In this context, a label may include, for example, a bounding box, a boundary contour for semantic or instance segmentation, a class label, a tracking prediction, etc., for a particular type of entity identified in the input data item.

[0051] To aid in the data labeling process, the remote system 208 includes a proposal component 210 configured to process the point-wise data 206 using a proposal model 212 to generate proposed annotations 214 for the sensor data 209. The proposal model 212 may include an offline data association model and a smoother, e.g., as described above, and may accordingly determine a refined estimate of the state associated with the detected instances at a given time step based on the accurate track predictions. The proposal model 212 may further include trained machine learning models and / or heuristic models for performing tasks such as object detection and / or other tasks related to the control of an autonomous vehicle. The proposed annotations 214 may depend on the track predictions and / or refined estimates of the state. For example, the proposal model 212 may be configured to determine a common class label for instances of entities associated with a common track based on confidence levels associated with class labels of instances on the track. In this way, the common class label may propagate along the track even if some instances (e.g., in the case of partial occlusion) have a low confidence level. On the other hand, changes in class labels on a predicted track may cause the proposal model 212 to reevaluate the tracking predictions, as described above with respect to resolving tracking ambiguities. More generally, the proposal model 212 may be configured to enforce or encourage continuity of proposed annotations on a given track, for example, by ensuring that bounding boxes or boundary contours associated with different instances of an entity do not imply changes in the size or shape of the entity. In another example, the proposal model 212 may be configured to determine the position and / or orientation of a detected instance based on refined estimates of states associated with instances on the same track. This allows, for example, to determine accurate top-down bounding boxes for all instances on a track.In this way, the proposal model 212 can use information from multiple instances associated with a common track to determine a proposed annotation that is accurate and robust to noise.

[0052] The remote system 208 includes a user interface 216, which may include a combination of hardware and software components that enable the user 218 to interact with the remote system 208. The user interface 216 may include one or more displays, one or more input devices, and rendering software and drivers associated with these devices. The user interface 216 is configured to render a visual representation of the environment derived from the sensor data 209, such as an image or video representing the environment at one or more time steps as viewed by the user 218. The visual representation may be an image-based representation based on image data captured by the sensor 202 (e.g., a camera). In other examples, the visual representation may include a non-image-based visual representation, such as a visual representation of a lidar point cloud, radar or sonar return signals, or the like. In some cases, the user interface 216 may present one or more visual representations based on a combination of different sensor modalities of the same environment, such as a visual image and a corresponding lidar point cloud. The visual representation may be rendered from the vehicle's perspective and / or from a variety of other angles, such as a top-down view of the environment.

[0053] The user interface 216 is further configured to render a visual representation of one or more proposed annotations 214, e.g., overlaid on top of a visual representation of the environment. The visual representation of the proposed annotation(s) 214 may include one or more of a proposed bounding box, a proposed boundary contour for semantic or instance segmentation, and / or text or a symbol indicating a proposed class label for the entity. The user interface 216 further enables a user 218 to modify the proposed annotation(s) 214 or otherwise provide user-approved annotation(s) 220 using one or more input devices presented with the proposed annotation(s) 214. For example, the user interface 216 may render a visual representation of the proposed class labels for the detected objects and provide a means for the user 218 to either approve the class labels or to reject the class labels. If the user 218 rejects a class label, the user interface 216 may present alternative class labels (e.g., in order of decreasing confidence as determined by the proposed model 212). The user 218 may then select the user-approved class label from the list of alternative class labels. In another example, the user interface 216 may present multiple options of proposed annotations (e.g., multiple proposed class labels) and allow the user 216 to select one of the proposed annotations, in which case the selected annotation may become the user-approved annotation. In another example, the visual representation of the proposed annotation may include a proposed bounding box (e.g., a bounding box of the entity viewed from the viewpoint of the vehicle 200) or a top-down bounding box (e.g., if the visual representation of the environment is a top-down representation).If the user 218 determines that the proposed model 212 has incorrectly determined the size, shape, and / or location of the entity, the user interface 216 may enable the user to determine a user-approved bounding box that more accurately corresponds to the boundaries of the entity by dragging the corners of the bounding box to new locations. Alternatively, the user interface 216 may enable the user, having viewed the proposed bounding box, to draw or otherwise define a new user-approved bounding box. If the user 218 determines that the proposed annotation corresponds to a false positive, the user interface 216 may enable the user to delete the proposed annotation.

[0054] In response to receiving a different user-approved annotation for the same instance of the entity than the proposed annotation, the suggestion component 210 may be configured to update the proposed annotation for one or more further instances detected at the respective different time steps. For example, the user 218 may specify a class label for a given instance detected at a given time step. The given instance may be associated with a track having a sequence of further instances according to the data association model. The suggestion component 210 may thus update the proposed class label of any instance associated with the same track to match the class label specified by the user 218. The user 218 may then be presented with a visual representation of the environment at a later or earlier time step along with the updated proposed class labels for the instances on the track. The update of the proposed annotation in this manner may depend on a confidence value associated with the original class label and / or the updated class label for the further instance. For example, if the original class label and the updated class label for one of the further instances are assigned similar confidence levels, the suggestion component 210 may perform the update as described, but if the original class label has a significantly higher confidence level than the updated class label for the further instance, the suggestion component 210 may refrain from performing the update. The suggestion component 210 may be configured to update bounding boxes, boundary contours, or other proposed annotations in a similar manner. For example, if the user 218 changes the size and / or shape of a bounding box or contour for a given instance, the suggestion component 210 may change the size and / or shape of the corresponding bounding boxes or contours for further instances on the same track.

[0055] The user interface 216 may be configured to present a video stream representation of the environment over a sequence of time steps along with the corresponding annotations. The user 218 may be provided with a means to pause or rewind the video stream to a selected time step and provide a user-approved annotation for that time step. Once an annotation is corrected at a given time step, the correction may be propagated to other instances associated with the same track, as described above, providing an intuitive and time-efficient way for a user to correct annotations across several time steps.

[0056] The remote system 208 generates labeled training data 222 based at least in part on the user-approved annotations 220 and the sensor data 209. The labeled training data 220 may further include proposed annotations 214 that have been explicitly or implicitly accepted by the user 218. The remote system 208 may use the labeled training data 222 for supervised training of a machine learning model 224. The machine learning model 224 may be suitable for use in controlling an autonomous vehicle and may be the same or different from the machine learning model implemented by the proposed model 212 to generate the proposed annotations 214. The machine learning model 224 may be, for example, a runtime model suitable for use onboard an autonomous vehicle. As an offline model, the proposed model 212 may be able to perform a given task more accurately than the runtime machine learning model 224. Thus, the combination of the proposed annotations 214 generated by the proposed model 212 and the user-approved annotations 220 provided by the user 218 may represent a sufficiently good approximation of the ground truth data for effective training of the machine learning model 224.

[0057] The machine learning model 224 may be trained using labeled training data aggregated from multiple sources, for example, based on sensor data received from many autonomous vehicles. Additionally, the task of generating user-approved annotations may be shared among many users accessing the remote system 208 or using other systems, for example, in exchange for monetary compensation. The methods and techniques described herein may greatly improve the speed and accuracy with which labeled training data can be generated.

[0058] Figure 8 illustrates an example of a computer-implemented method 800 that may be performed by a computing system, such as remote system 208 of Figure 2. The method 800 includes, at 802, determining an estimate of a state of the object at a first time step based on point-wise measurements of the state of the object at a first time step and point-wise measurements of the state of the object at a number of additional time steps. The estimate of the state may be determined using an offline model described herein, including, for example, a data association model and a smoother.

[0059] The method 800 proceeds, at 804, by generating a proposed annotation associated with the object at the first time step. The proposed annotation may be generated using an estimate of the state of the object determined at 802.

[0060] The method 800 proceeds at 806 by rendering, via a user interface, a visual representation of the environment including the object at the first time step and a visual representation of the proposed annotations. The visual representation of the environment and point-wise measurements of the state of the object at the first time step may be derived from common sensor data.

[0061] The method 800 proceeds, at 808, by receiving user input via a user interface, the user input indicating a user-approved annotation associated with the object at the first time step. As described above, the user-approved annotation may be an approval or confirmation of the proposed annotation, or may be a modified annotation that differs from the proposed annotation.

[0062] The method 800 concludes at 810 by generating training data for a machine learning model for use in controlling an autonomous vehicle based at least in part on the user-approved annotations. The training data may include an input portion based on sensor data (e.g., images) that derives point-wise measurements of an object's state at a first time step, and labels based on the user-approved annotations.

[0063] FIG. 3 illustrates a block diagram of an example system 300 for implementing the techniques described herein. In some examples, the system 300 may include a vehicle 302, which may correspond to the vehicle 100 of FIG. 1 and / or the vehicle 200 of FIG. 2. In some examples, the vehicle 302 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the US National Highway Traffic Safety Administration, which describes a vehicle that can perform all safety-critical functions during the entire journey without the driver (or passenger) being expected to control the vehicle at all times. However, in other examples, the autonomous vehicle 302 may be a fully or partially autonomous vehicle having any other level or classification. Additionally, in some examples, the techniques described herein may be used by non-autonomous vehicles.

[0064] The vehicle 302 may include vehicle computing device(s) 304, one or more sensor systems 306, one or more emitters 308, one or more communication connections 310, at least one direct connection 312 (e.g., for physically coupling the vehicle 302 to exchange data and / or provide power), and one or more drive systems 314.

[0065] In some examples, the sensor(s) 306 may include light detection and ranging (LIDAR) sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., Global Positioning System (GPS), compass, etc.), inertial sensors (e.g., Inertial Measurement Unit (IMU), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., Red-Green-Blue (RGB), Infrared (IR), intensity, depth, time of flight, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. The sensor(s) 808 may include multiple instances of each of these or other types of sensors. For example, the LIDAR sensors may include individual LIDAR sensors located at the corners, front, rear, sides, and / or top of the vehicle 302. As another example, the cameras may include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle 302. The sensor(s) 306 may provide input to the vehicle computing device(s) 304.

[0066] Vehicle 302 may also include emitter(s) 308 for emitting light and / or sound, as described above. Emitter(s) 308 in this example may include interior audio and visual emitter(s) for communicating with occupants of vehicle 302. By way of example and not limitation, interior emitter(s) may include speakers, lights, indicator lights, display screens, touch screens, haptic emitter(s) (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), etc. Emitter(s) 308 in this example may also include exterior emitter(s). By way of example and not limitation, the external emitter(s) in this example include lights or other indicators of vehicle action (e.g., indicator lights, indicator lights, light arrays, etc.) for signaling direction of travel, and one or more audio emitter(s) (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which may include acoustic beam steering technology.

[0067] Vehicle 302 may also include communication connection(s) 310 enabling communication between vehicle 302 and one or more other local or remote computing device(s). For example, communication connection(s) 310 may facilitate communication with other local computing device(s) on vehicle 302 and / or drive system(s) 314. Communication connection(s) 308 may also additionally or alternatively enable vehicle 302 to communicate with other nearby computing device(s) (e.g., other nearby vehicles, traffic signals, etc.). Communication connection(s) 310 may additionally or alternatively enable vehicle 302 to communicate with computing device 336.

[0068] The vehicle computing device(s) 304 may include one or more processors 316 and a memory 318 communicatively coupled to the one or more processors 316. In the depicted example, the memory 318 of the vehicle computing device(s) 304 stores a localization component 320, an on-board perception component 322 including a data association model 324 and a filtering model 326, one or more system controllers 328, and a planning component 330. While shown in FIG. 3 as resident in the memory 318 for illustrative purposes, it is contemplated that the localization component 320, the perception component 322, the one or more system controllers 328, and / or the planning component 330 may additionally or alternatively be accessible (e.g., stored remotely) to the vehicle 302.

[0069] In some examples, the perception component 322 may include functionality for performing object detection, semantic segmentation, instance segmentation, and / or classification. In some examples, the perception component 322 may generate processed sensor data indicative of the presence of an entity in proximity to the vehicle 302 and / or a classification of the entity type (e.g., automobile, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional or alternative examples, the perception component 322 may provide processed sensor data indicative of one or more characteristics associated with the detected entity (e.g., tracked object) and / or the environment in which the entity is located. In some examples, the characteristics associated with the entity may include, but are not limited to, x-position (global and / or local position), y-position (global and / or local position), z-position (global and / or local position), orientation (e.g., roll, pitch, yaw), entity type (e.g., classification), velocity of the entity, acceleration of the entity, range (size) of the entity, etc. Characteristics associated with an environment may include, but are not limited to, the presence of another entity in the environment, the state of another entity in the environment, the time of day, the day of the week, the season, weather conditions, an indication of darkness / light, etc.

[0070] In at least one example, vehicle computing device(s) 304 can include one or more system controllers 324 that can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 302. The system controller(s) 324 can communicate with and / or control corresponding systems of the drive system(s) 314 and / or other components of the vehicle 302.

[0071] The system controller(s) 324 may be communicatively coupled to one or more sensors of the vehicle sensor system(s) 306. As a non-limiting example, the sensors may detect the presence of objects in the vehicle's environment and / or determine attributes of those objects. The system controller(s) 324 may also trigger activation of safety systems of the vehicle 302 if it is determined that the safety systems should be activated. For example, the system controller(s) 324 may command an airbag control unit to deploy one or more airbags or send a signal to a tensioner configured to adjust the tension of one or more restraint devices. Other safety systems are known and may be activated. In other embodiments, the system controller 324 may command the activation of multiple safety systems. In some embodiments, some or all of the functions of the system controller 324 may be performed remotely from the vehicle 302, for example, on a remote server associated with dispatch or headquarters of the vehicle 302, or in the cloud. In other embodiments, some or all of the functions of the system controller(s) 324 may be performed in the vehicle 302 to minimize any delays that may result from transmitting data between local.

[0072] The drive system(s) 314 may include many vehicle systems, including a high voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the exterior surroundings of the vehicle), as well as one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, the drive system(s) 314 may include a drive system controller that may receive and pre-process data from the sensor(s) and control the operation of various vehicle systems. In some examples, the drive system controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more modules for performing various functions of the drive system(s) 314. Additionally, the drive system(s) 314 may also include one or more communication connection(s) that enable the respective drive system to communicate with one or more other local or remote computing device(s).

[0073] In some examples, the vehicle 302 may transmit operational data, including raw or processed sensor data from the sensor system(s) 306, over the network(s) 334 to one or more computing device(s) 336. In other examples, the vehicle 302 may transmit processed operational data and / or representations of the operational data to the computing device(s) 336 at a particular frequency, such as after a predetermined period of time, in near real-time, etc. In some cases, the vehicle 302 may transmit the raw or processed operational data to the computing device(s) 336 as one or more log files.

[0074] The one or more computing device(s) 336 may include one or more processors 338 and a memory 340 communicatively coupled to the one or more processors 338. The memory 340 may store data defining the offline model 342, as described elsewhere in this disclosure. The computing device(s) 336 may also include a user interface 346 to enable user input regarding assistance with the labeling function, as described elsewhere in this disclosure.

[0075] In some cases, some or all aspects of the components discussed herein may include any model, algorithm, and / or machine learning algorithm. For example, some of the component(s) in memory 318 may be implemented as a neural network. As can be understood in the context of the present disclosure, a neural network may be trained using machine learning where values ​​for the network's parameters may be automatically determined from data during the training process, rather than being explicitly programmed by a human programmer.

[0076] Illustrative clauses A: A system including one or more processors and one or more computer readable media storing computer executable instructions, the computer executable instructions, when executed by the one or more processors, causing the one or more processors to obtain point-wise data indicative of point-wise measurements of a state of an object detected by an object detection system mounted on a vehicle for a plurality of time steps, and recursively processing the point-wise data with a runtime model to determine a runtime estimate of the state of the object for the plurality of time steps, wherein recursively processing with the runtime model includes determining the runtime estimate of the state of the object at the first time step based on the point-wise measurements of the state of the object at a first time step and the runtime estimate of the state of the object at a second time step; the second time step being earlier than the first time step; processing the point-wise data with a benchmark model to determine a benchmark estimate of the state of the object for the plurality of time steps, the processing with the benchmark model including determining the benchmark estimate of the state of the object at the first time step based on the point-wise measurements of the state of the object at the first time step and at a plurality of further time steps, the plurality of further time steps including time steps later than the first time step; evaluating a metric that measures deviation between the runtime estimate of the state of the object and the benchmark estimate for the plurality of time steps; and updating the runtime model based on the evaluation of the metric.

[0077] B: The system of clause A, configured to transmit data indicative of the updated runtime model to an autonomous vehicle via the data interface.

[0078] C: The system of any one of clauses A and B, wherein the state of the object includes at least one of position, velocity, yaw, and yaw rate.

[0079] D: The system described in any of clauses A to C, wherein processing the point-wise data using the benchmark model includes associating a first instance of the object detected at the first time step with each further instance of the object detected at the multiple further time steps, the associating including determining a multiple candidate track configurations comprising candidate groupings of instances of the object detected at the multiple time steps, determining a most likely track configuration among the multiple candidate track configurations, and associating the first instance of the object with each further instance of the object in accordance with the determined most likely track configuration.

[0080] E: The system described in clause D, wherein the runtime model includes one or more thresholds for controlling association of object instances detected at each of the different time steps, and updating the runtime model includes determining updated values ​​for the one or more thresholds.

[0081] F: A computer-implemented method comprising: obtaining point-wise data indicative of point-wise measurements of a state of an object detected by an object detection system for a plurality of time steps; obtaining runtime data indicative of a runtime estimate of the state of the object for the plurality of time steps from a runtime model; processing the point-wise data with a benchmark model to determine a benchmark estimate of the state of the object for the plurality of time steps; evaluating a metric that measures deviation between the runtime estimate and the benchmark estimate of the state of the object for the plurality of time steps; and updating the runtime model based on the evaluation of the metric.

[0082] G. The computer-implemented method of claim F, further comprising transmitting data indicative of the updated runtime model to an autonomous vehicle.

[0083] H: The computer-implemented method of any one of clauses F-G, wherein the state of the object includes at least one of position, velocity, yaw, and yaw rate.

[0084] I: The computer-implemented method of any of clauses F-H, wherein the runtime model generates the runtime data using a recursive filter having an associated process noise covariance and an associated observation noise covariance, and updating the runtime model includes updating at least one of the associated process noise covariance and the associated observation noise covariance.

[0085] J: recursively processing the point-wise data by the runtime model to determine the runtime estimate of the state of the object for the plurality of time steps, wherein recursively processing by the runtime model includes determining the runtime estimate of the state of the object at the first time step based on the point-wise measurements of the state of the object at a first time step and the runtime estimate of the state of the object at a second time step, the second time step being earlier than the first time step.

[0086] K: The computer-implemented method of any of clauses F-J, wherein the processing with the benchmark model includes determining the benchmark estimate of the state of the object at a first time step based on the point-wise measurements of the state of the object at the first time step and a plurality of further time steps, the plurality of further time steps including time steps slower than the first time step.

[0087] L: The computer-implemented method of claim K, wherein the processing with the benchmark model includes processing the point-wise measurements of the state at the first time step and the plurality of further time steps through a smoother.

[0088] M: The computer-implemented method of clause K or L, wherein the processing with the benchmark model includes associating a first instance of the object detected at the first time step with each further instance of the object detected at the multiple further time steps.

[0089] N: The computer-implemented method of clause M, wherein associating the first instance of the object detected at the first time step with each of the further instances of the object detected at the multiple further time steps includes determining a multiple of candidate track configurations comprising candidate groupings of instances of the object detected at the multiple time steps, determining a most likely track configuration among the multiple of candidate track configurations, and associating the first instance of the object with the each of the further instances of the object in accordance with the determined most likely track configuration.

[0090] O: The computer-implemented method of clause N, further comprising: determining a respective likelihood value for each of the plurality of candidate track configurations, and determining the most likely track configuration comprises selecting the candidate track configuration determined to have the highest likelihood value.

[0091] P: The computer-implemented method of any of clauses L-N, wherein the runtime model includes one or more thresholds for controlling association of instances of the object detected at each of the different time steps, and updating the runtime model includes determining updated values ​​for the one or more thresholds.

[0092] Q: The computer-implemented method of any of clauses F-P, wherein obtaining the point-wise data includes receiving the point-wise data from a computer system installed in a vehicle.

[0093] R: The computer-implemented method of any of clauses F-Q, wherein the metric measures the divergence between a probability distribution associated with the runtime estimate of the object for the multiple time steps and a probability distribution associated with the benchmark estimate of the kinematic state of the object for the multiple time steps.

[0094] S: obtaining ground truth data indicative of ground truth values ​​of the state of the object for the plurality of time steps, wherein the metric measures the deviation between the runtime estimate of the state of the object and the benchmark estimate of the state of the object by referencing the ground truth values ​​of the state of the object.

[0095] T: One or more non-transitory computer-readable media storing instructions executable by one or more processors, the instructions, when executed, causing the one or more processors to perform operations including obtaining point-wise data indicative of point-wise measurements of a state of an object detected by an object detection system for a plurality of time steps; obtaining runtime data indicative of a runtime estimate of the state of the object for the plurality of time steps from a runtime model; processing the point-wise data with a benchmark model to determine a benchmark estimate of the state of the object for the plurality of time steps; evaluating a metric that measures deviation between the runtime estimate and the benchmark estimate of the state of the object for the plurality of time steps; and updating the runtime model based on the evaluation of the metric.

[0096] Although the exemplary clauses above are described with respect to one particular embodiment, it should be understood in the context of this specification that the contents of the exemplary clauses can also be implemented by a method, device, system, computer readable medium, and / or other embodiment. Furthermore, any of Examples A-T can be implemented alone or in combination with any other one or more of Examples A-T.

Claims

1. 1. A system including one or more processors and one or more computer-readable media storing computer-executable instructions, The computer-executable instructions, when executed by the one or more processors, cause the one or more processors to: obtaining point-wise data indicative of point-wise measurements of a state of an object detected by an object detection system mounted on a vehicle for a plurality of time steps; recursively processing the point-wise data with a runtime model to determine a runtime estimate of the state of the object for the plurality of time steps, where recursively processing with the runtime model includes determining the runtime estimate of the state of the object at the first time step based on the point-wise measurements of the state of the object at a first time step and the runtime estimate of the state of the object at a second time step, where the second time step is earlier than the first time step; processing the point-wise data with a benchmark model to determine benchmark estimates of the state of the object for the plurality of time steps, the processing with the benchmark model including determining the benchmark estimate of the state of the object at the first time step based on the point-wise measurements of the state of the object at the first time step and a plurality of further time steps, the plurality of further time steps including time steps later than the first time step; evaluating a metric that measures deviation between the runtime estimate and the benchmark estimate of the state of the object for the plurality of time steps; updating the runtime model based on the evaluation of the metrics; and The system performs an operation including:

2. 2. The system of claim 1, configured to transmit data indicative of the updated runtime model to an autonomous vehicle via the data interface.

3. The system of claim 1 , wherein the state of the object includes at least one of a position, a velocity, a yaw, and a yaw rate.

4. said processing with said benchmark model including associating a first instance of the object detected at said first time step with each further instance of the object detected at said plurality of further time steps; The linking step comprises: determining a plurality of candidate track configurations comprising candidate groupings of instances of objects detected at the plurality of time steps; determining a most likely track configuration among the plurality of candidate track configurations; associating the first instance of the object with each of the further instances of the object in accordance with the determined most likely track configuration; The system of claim 1 , comprising:

5. the runtime model includes one or more thresholds for controlling association of object instances detected at each of the different time steps; The system of claim 4 , wherein updating the runtime model comprises determining updated values ​​for the one or more thresholds.

6. obtaining point-wise data indicative of point-wise measurements of states of objects detected by the object detection system for a plurality of time steps; obtaining runtime data from a runtime model indicative of runtime estimates of the state of the object for the plurality of time steps; processing the point-wise data with a benchmark model to determine benchmark estimates of the state of the object for the plurality of time steps; evaluating a metric that measures deviation between the runtime estimate and the benchmark estimate of the state of the object for the plurality of time steps; updating the runtime model based on the evaluation of the metrics; and 4. A computer-implemented method comprising:

7. The computer-implemented method of claim 6 , comprising transmitting data indicative of the updated runtime model to an autonomous vehicle.

8. The computer-implemented method of claim 6 , wherein the state of the object includes at least one of a position, a velocity, a yaw, and a yaw rate.

9. the runtime model generates the runtime data using a recursive filter having an associated process noise covariance and an associated observation noise covariance; The computer-implemented method of claim 6 , wherein updating the runtime model comprises updating at least one of the associated process noise covariance and the associated observation noise covariance.

10. 7. The computer-implemented method of claim 6, wherein the processing with the runtime model includes determining the runtime estimate of the state of the object at a first time step based on the point-wise measurements of the state of the object at the first time step and the runtime estimate of the state of the object at a second time step, the second time step being earlier than the first time step.

11. 7. The computer-implemented method of claim 6, wherein the processing with the benchmark model includes determining the benchmark estimate of the state of the object at a first time step based on the point-wise measurements of the state of the object at the first time step and multiple further time steps, the multiple further time steps including time steps slower than the first time step.

12. The computer-implemented method of claim 11 , wherein the processing with the benchmark model comprises processing the point-wise measurements of the state at the first time step and the plurality of further time steps through a smoother.

13. 12. The computer-implemented method of claim 11, wherein the processing with the benchmark model includes associating a first instance of the object detected at the first time step with each further instance of the object detected at the multiple further time steps.

14. Associating the first instance of the object detected at the first time step with the respective further instances of the object detected at the plurality of further time steps comprises: determining a plurality of candidate track configurations comprising candidate groupings of instances of objects detected at the plurality of time steps; determining a most likely track configuration among the plurality of candidate track configurations; associating the first instance of the object with each of the further instances of the object in accordance with the determined most likely track configuration; 13. The computer-implemented method of claim 12, comprising:

15. determining a respective likelihood value for each of the plurality of candidate track configurations; The computer-implemented method of claim 13 , wherein determining the most likely track configuration comprises selecting the candidate track configuration determined to have the highest likelihood value.

16. the runtime model includes one or more thresholds for controlling association of object instances detected at each of the different time steps; The computer-implemented method of claim 12 , wherein updating the runtime model comprises determining updated values ​​for the one or more thresholds.

17. The computer-implemented method of claim 6 , wherein obtaining the point-wise data comprises receiving the point-wise data from a vehicle-mounted computer system.

18. 7. The computer-implemented method of claim 6, wherein the metric measures a divergence between a probability distribution associated with the runtime estimates of the states of the object for the multiple time steps and a probability distribution associated with the benchmark estimates of the states of the object for the multiple time steps.

19. obtaining ground truth data indicative of ground truth values ​​of the states of the object for the plurality of time steps; 7. The computer-implemented method of claim 6, wherein the metric measures the deviation between the runtime estimate and the benchmark estimate of the state of the object by referencing the ground truth value of the state of the object.

20. One or more non-transitory computer-readable media storing instructions executable by one or more processors, The instructions, when executed, cause the one or more processors to: obtaining point-wise data indicative of point-wise measurements of states of objects detected by the object detection system for a plurality of time steps; obtaining runtime data from a runtime model indicative of runtime estimates of the state of the object for the plurality of time steps; processing the point-wise data with a benchmark model to determine benchmark estimates of the state of the object for the plurality of time steps; evaluating a metric that measures deviation between the runtime estimate and the benchmark estimate of the state of the object for the plurality of time steps; updating the runtime model based on the evaluation of the metrics; and The one or more non-transitory computer-readable media configured to cause operations to be performed, including:

Images