A tool for offline perceptual component assessment

Abstract

A computer-implemented method comprising: obtaining point-wise data indicative of point-wise measurements of a state of an object detected by an object detection system for a plurality of time steps; obtaining runtime data indicative of a runtime estimate of the state of the object for the plurality of time steps from a runtime model; processing the point-wise data with a benchmark model to determine a benchmark estimate of the state of the object for the plurality of time steps; evaluating a metric that measures deviation between the runtime estimate and the benchmark estimate of the state of the object for the plurality of time steps; and updating the runtime model based on the evaluation of the metric.

Description

[Background technology]

[0001] An autonomous vehicle typically includes various sensors and on-board data processing systems that can detect entities, such as other vehicles, in its vicinity and estimate state variables associated with those entities, such as position, size, orientation, and velocity. By tracking an individual entity, information derived from previous estimates of its state variables can be used to refine a current estimate of its state variables.

[0002] The details will now be described with reference to the accompanying drawings, in which the same reference numbers in different drawings indicate similar or identical items or features. [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 is a schematic diagram of a system for benchmarking runtime models for deployment on vehicles. [Figure 2] FIG. 1 is a schematic diagram of a system for assisting in labeling training data for machine learning models. [Figure 3] 1 shows a block diagram of an exemplary system for implementing the techniques described herein. [Figure 4] 1 shows a first tracking hypothesis for an entity near an autonomous vehicle. [Figure 5] 10 shows a second tracking hypothesis for an entity near the autonomous vehicle. [Figure 6] 10 shows the precision and recall curves of the runtime model and benchmark model before and after updating the inference system, according to an example. [Figure 7] 1 shows a flowchart illustrating a method for benchmarking a runtime model, according to an example. [Figure 8] 1 shows a flowchart illustrating a method for annotating data for training a machine learning model, according to an example. DETAILED DESCRIPTION OF THE INVENTION

[0004] The present disclosure relates to methods and systems for estimating the state of entities, such as dynamic entities, in the vicinity of an autonomous vehicle. In this context, dynamic entities (sometimes referred to as agents) may include vehicles or other objects that are movable within an environment and may be distinguished from static objects by their mobility within the environment. The state of an entity refers to one or more properties associated with the entity at a given time. This state may include dynamic properties that are expected to vary over time, including values ​​of one or more kinematic variables, such as the entity's position and / or orientation, as well as time derivatives of these quantities, such as rotational speed and / or rotation rate. Alternatively, or in addition, the state of an entity may include static properties that are expected to remain constant over time, including geometric properties of the entity, such as the entity's dimensions, size, and / or shape. The state of an entity may be represented as a vector including components representing each state variable. In the context of an entity in the vicinity of an autonomous vehicle, the vector may represent, for example, planar (horizontal) position, planar velocity, yaw, and yaw rate. Additionally or alternatively, other variables, such as vertical position, pitch, or roll, along with their associated time derivatives, may be included in the state.

[0005] An autonomous vehicle may include one or more sensors and an object detection system for detecting entities in the vicinity of the autonomous vehicle. The autonomous vehicle may further include an on-board perception component that may be configured to estimate the state of the detected entity using sensor data collected from one or more sensors at a given time point or within a narrow time window (e.g., a few milliseconds) around the given time point. An estimate of the state derived from data associated with a given time point may be referred to as a point-wise measurement of the state.

[0006] Point-wise measurements of state may be susceptible to various sources of error and noise, which may include observation noise caused by inherent uncertainties associated with the sensor(s), occlusion error caused by entities being fully or partially occluded from one or more sensors, and kinematic noise caused by unpredictable motion of the sensor(s), such as vibrations caused by uneven road surfaces. The impact of such errors on point-wise measurements of state can be mitigated using information derived from one or more previous state estimates. In particular, by tracking individual entities over time, a noise filter can be recursively applied to point-wise measurements of the entity's state at a series of time steps to determine runtime estimates of these states, thereby mitigating at least some of the sources of error that may affect the point-wise measurements. To facilitate this, an on-board perception component of an autonomous vehicle may include a noise filter as well as a data association model for associating instances of entities detected at different time steps.

[0007] Prior to deployment of the data association model and / or noise filter, values ​​of associated parameters of the data association model and / or noise filter may be determined based on physical considerations (e.g., based on kinematic equations of motion and / or known uncertainties associated with one or more sensors) or based on empirical data. For example, values ​​of associated parameters may be determined using online or offline machine learning methods. A difficulty with the latter approach is that ground truth states of observed entities in the environment are rarely available unless simulations are used, and such simulations may not accurately represent the various sources of noise and error present in a real physical environment. More generally, obtaining ground truth data for machine learning tasks related to the control of an autonomous vehicle can be difficult because the process of labeling data that covers a sufficiently diverse set of scenarios is very time-consuming and resource-intensive.

[0008] In consideration of the above problems, the present disclosure provides an offline tool configured to process point-wise measurements of states of detected entities over a period of time to determine offline estimates of the states of those entities. The offline estimates may be determined using an offline model and may more accurately represent the true states of the entities than runtime estimates determined using a runtime model. Thus, the offline estimates may be treated as pseudo-ground truth values ​​of those states in situations where a sufficient amount / diversity of ground truth data is not readily available.

[0009] In contrast to runtime models, offline models of the present disclosure may not be suitable for execution in an online setting, for example, onboard an autonomous vehicle. In particular, to determine an offline estimate of an entity's state at a given time step, an offline model may be configured to use information from time steps later than the given time step. This is in contrast to runtime models, which may only use information from earlier time steps. Models that require information from future time steps will necessarily operate with a temporal lag, which may be undesirable in the context of runtime models in autonomous vehicles or other settings where decisions must be made with minimal delay. Furthermore, executing an offline model may require significantly more processing than executing a runtime model, which may introduce unacceptable levels of lag and / or prevent the offline model from being capable of executing in real time or near real time. In some cases, the computational cost of executing an offline model may be prohibitively high for execution by a computer system onboard an autonomous vehicle.

[0010] Examples of noise filters suitable for implementation onboard an autonomous vehicle may include Kalman-type filters (such as a Kalman filter, an extended Kalman filter, or an unscented Kalman filter), although other filters, such as particle filters and Gaussian process-based filters, may additionally or alternatively be employed. In this way, if noise that is uncorrelated from one time step to the next can be reasonably modeled, point-wise measurements of a state may be refined based on information derived from point-wise measurements of previous states (and optionally one or more earlier states), such that various types of noise may be filtered out of the measurements. Such filters may be used in combination with a robust loss function that reduces the sensitivity of the noise filter's output to outliers, enabling further refinement.

[0011] To apply a noise filter to a given sequence of point-wise measurements, entity instances detected at different time steps may first be associated with one or more "tracks" using a data association model, and the noise filter may be applied to instances associated with a common track. Entities associated with tracks using a data association model may be referred to as track entities or track objects. The data association model aims to accurately associate instances of the same entity at different time steps with a common track. If an entity instance cannot be associated with an existing track (e.g., because the entity recently entered the range of a sensor and / or object detection system), the data association model may create a new track or reject the instance as a false positive. In multi-entity or multi-agent settings (which are common in driving environments), the data association task may not have a unique solution. For example, if an entity becomes occluded for one or more time steps, it may be difficult to determine whether subsequent detections correspond to the same entity or an entirely new entity. To determine whether an entity instance detected at a given time step should be associated with an existing track or whether a new track should be started, the data association model may use state estimates from previous time steps. For example, the mean and covariance of noise-filtered estimates from previous time steps may be used to define a "gating" region of the state space where a new measurement may be determined to correspond to the same entity. Further criteria may be introduced to resolve ambiguities, for example, when multiple measurements appear within a gating region of the state space. Such criteria may, for example, use classification and / or bounding box losses to resolve such ambiguities. Examples of data association models include network flow-based models, Markov Chain Monte Carlo (MCMC) data association, joint probability data association, and multidimensional assignment.

[0012] As described above, the combined data association model and filtering model may be implemented in combination as a runtime (or "online") model, for example, to be executed by computing hardware onboard an autonomous vehicle. The runtime model may be configured to process point-wise measurements of the states of entities detected in the vicinity of the autonomous vehicle to generate runtime estimates of the entities' states, which may be used to predict the entities' trajectories and plan what actions, if any, the autonomous vehicle should take in response to the entities' predicted trajectories. The runtime model thereby enables the autonomous vehicle to take actions that are less adversely affected by noise than if the autonomous vehicle were to use the point-wise measurements directly.

[0013] The performance of the data association model and the filtering model (collectively referred to as the runtime model) may be affected by the values ​​of several associated parameters. For example, the data association model may include parameters to control whether a new track is created for a given detection of an entity (as opposed to classifying it as a false positive) and / or whether a detected entity is associated with an existing track. The filtering model may include parameters to characterize various types of noise / error associated with the point-wise measurements. In the case of a Kalman-type filter, the point-wise measurements z of the state at a given time step t t Let x be the true state at time step t. t As noisy observations of z t =g(X t )+v t where g is the measurement operator and v t is the measurement noise that captures the random error associated with the measurement of the state x at time step t. tis the state x at the previous time step t-1 by applying a linear or nonlinear state transition operator f, where f represents the time evolution of the state. t-1 In some instances, at least some components of the state, such as the dimensions of the entity, are expected to be constant over time, while other components, such as the position of the entity, may change over time. The time evolution of the state is governed by the state transition noise w t This state transition noise is assumed to be influenced by x t =f(x t-1 )+w t The filtering model captures random (uncorrelated) fluctuations in the system dynamics, such as: Depending on the filtering model, the operators f and / or g may each have a known parametric form (e.g., based on known equations of motion), in which case the parameters of the operators f and / or g may be parameters of the runtime model. Alternatively, the operators f and / or g may be modeled in a non-parametric manner, e.g., governed by a latent Gaussian process, in which case the hyperparameters of the latent Gaussian process may be parameters of the runtime model. The measurement noise v t and state transition noise w t may be modeled as Gaussian noise parameterized by respective covariance matrices, in which case the elements of these covariance matrices may also be parameters of the runtime model.

[0014] The state transition operator f may depend on the classification of the entities, reflecting the fact that different entities may be subject to different dynamics. For example, a vehicle with four-wheel steering may be able to move (e.g., drift) in a direction different from the direction the vehicle is facing, while a vehicle with two-wheel steering may only be able to move in the direction the vehicle is facing (unless a situation is identified that may cause the vehicle to lose traction with the road).

[0015] While the runtime model described above may include a noise filter, the offline model of the present disclosure may include smoothers such as a Rauch-Tung-Striebel (RTS) smoother, a two-filter smoother, a sequence importance resampling smoother, a Rao-Blackwellized particle smoother, or a grid-based smoother. For a given noise filter (e.g., a Kalman-type filter), the associated smoother may be constructed by applying recursion in reverse time to update and refine estimates from the filtering model. In this manner, information from future time steps may be used to significantly reduce the uncertainty of estimates at a given time step. By incorporating information from future time steps, certain hypotheses regarding the state may be ruled out. The smoother may be a fixed-lag smoother configured to estimate the state of an entity a predetermined number of time steps prior to the latest available time step. Alternatively, the smoother may be a fixed-point smoother configured to estimate the state of an entity at a fixed time step using information from an increasing number of future time steps. Alternatively, the smoother may be a fixed-interval smoother configured to estimate the state of an entity over a fixed interval, given point-wise measurements of the entity's state over the same interval. This smoother may be an optimal smoother in the Bayesian sense (a.k.a., Bayesian optimal), i.e., given a filtering model, the smoother may determine the most likely value of the state at each time step, given the information available to the smoother. The optimal smoothing formula is available for all Kalman-type filters and may be evaluated using a series of recursive calculations in reverse time. Effectively, the smoother may take the information captured over a sequence of time steps, solve a Bayesian optimization problem, and give the most likely configuration of the state over that sequence of time steps (in other words, the joint distribution of the state with the highest likelihood). In an offline setting, the parameters associated with the smoother (e.g., the measurement noise v t and state transition noise w tThe components of the covariance matrix associated with σ (σ,σ) may be optimized or learned from the data, either for individual tracks in a given time frame or globally for all tracks, so that their state estimates can be smoothed and Bayes-optimal with respect to the selected filtering model.

[0016] The smoother may be a Gaussian process-based smoother. This type of smoother may be associated with a Gaussian process-based filter and may not assume a parametric form for the underlying state transition operator f and measurement operator g, but instead may associate each of these operators with one or more Gaussian processes that determine the posterior distribution using Bayesian inference. For a particular Gaussian process model, exact inference may be performed, allowing the joint posterior distribution of states and their uncertainties over a time step sequence to be determined in closed form. Alternatively, approximate inference may be used, for example, using sampling techniques or sparse approximations to the underlying Gaussian process. In either case, the hyperparameters of the Gaussian process may be optimized using maximum a posteriori (MAP) estimation, maximum likelihood estimation, evidence maximization, or sampling. Gaussian process inference, including hyperparameter optimization, may be computationally very expensive and not suitable for use in an online setting.

[0017] The offline model may include a multiple-hypothesis batch tracking model. Over a time step sequence, the multiple-hypothesis tracking model may be configured to build and update a so-called track tree that encapsulates one or more tracking hypotheses. The track tree may have nodes corresponding to the detection of one or more entities in each of the time step sequences and branches connecting the detections in the hypothesis tracks (or trajectories) between time steps. For a given time step, a new track tree may be built for each entity detected in that time step, considering the possibility that the detection corresponds to a new entity falling within the detection range. Existing track trees may also be updated with detections from a given time step. In particular, an existing track tree may be extended by adding any new detections consistent with existing nodes in the track tree as separate branches. A new detection at a given time step may be considered consistent with an existing node in the track tree if the point-wise measurement of its state (or simply its position or another subset of state variables) is within a predetermined metric distance D (e.g., a predetermined Mahalanobis distance) of the state (or subset of states) predicted for that time step according to a filtering model applied to the existing node, or in other words, if the measurement is within a region of state space according to a noise-filtered estimate of the state at the previous time step. Further branches may be added to the track tree to account for occlusions or other missing detections of entities at a given time step.

[0018] The track tree may be pruned at some or all time steps according to one or more pruning criteria, and the track hypotheses remaining after pruning may be scored depending on their Bayesian likelihood derived from, for example, a filtering model and / or other factors such as bounding box loss and / or classification loss. After pruning, a configuration of track hypotheses (i.e., global hypotheses) with the highest overall score may be determined by solving a discrete optimization problem (e.g., a maximum weighted independent set (MWIS) optimization problem), and the determined configuration may be identified as the correct global hypothesis at a given time step, such that each track in the identified global hypothesis may be determined to correspond to multiple instances of a common entity.

[0019] In an online setting, multiple hypothesis tracking models are constrained by the need to quickly prune the track tree to prevent the number of track hypotheses from exponentially increasing, as well as the fact that the model can only use information from previous time steps. Either or both of these constraints can result in the loss of the true track configuration, especially if the tracking model parameters are not properly tuned. Pruning may be performed using various techniques, such as N-scan pruning, in which a branch corresponding to a global hypothesis at time step k is traced backward to its corresponding node at time step kN (for a given parameter value N), thereby removing subtrees that diverge from the global hypothesis at that node. In this way, track tree branches that are far from the global hypothesis are removed. In practice, data association ambiguities up to time step kN are resolved by looking ahead a window of N frames. Other examples of pruning criteria include pruning track trees with branches exceeding a predetermined value B, for example, by retaining only the branches with the highest scores B. In an online setting, the values ​​of the parameters D, B, and / or N (and possibly other parameters depending on the data association model) may be selected to satisfy a trade-off between accuracy and computational cost. For example, selecting larger values ​​of D, B, and / or N reduces the probability of missing a true global hypothesis, but increases the computational cost and runtime of the data association model. Therefore, online settings may be restricted to using relatively small values ​​for these parameters, which may limit the practicality of multiple hypothesis tracking models for use in online settings.

[0020] In this setting, the offline model may utilize a multiple hypothesis tracking model (or any other suitable type of tracking model) in an offline manner. As described above, the offline model is not restricted to using information from past time steps, and temporal lag may not be a primary consideration, as is the case with online models. Lifting these restrictions may enable the use of parameter values ​​that result in highly accurate data association models (e.g., high values ​​of D, B, and / or N in the multiple hypothesis batch tracking model described above). Furthermore, the data association model may be run in forward and / or backward time directions, which may further improve accuracy. In certain examples, a smoother (e.g., one of the smoothers described above) may be run for each candidate track at each time step, which may result in more accurate Bayesian likelihood estimates and more accurate determination of the correct track configuration. In other examples, the offline model may forego the use of a track tree entirely, e.g., performing an exhaustive search for track configurations within a fixed time horizon. Although the number of track configurations can grow exponentially with the size of the time horizon, limiting the size of the time horizon allows all track configurations to be considered within a reasonable time scale in an offline setting. In this example, a smoother can be applied to each track in each candidate track configuration, allowing accurate Bayesian likelihoods to be determined for each track for data association purposes. Such methods can be applied, for example, using a sliding time window.

[0021] By using a Bayesian optimal smoother and an accurate data association model such as a multiple hypothesis batch tracking model, the offline model can generate a global Bayesian optimal set of estimates of the states of entities detected within a given time window. These estimates may be treated as pseudo-ground truth values ​​of the states, for example, in settings where ground truth data is unavailable or insufficient for a given task.

[0022] To expand on the above, FIG. 1 illustrates an example of using an offline model such as that described above to generate pseudo ground truth data for benchmarking a runtime model for deployment in an autonomous vehicle. FIG. 1 illustrates an autonomous vehicle 100 having an on-board perception component (not shown) configured to process data captured by on-board sensors 102 to detect and classify entities near the vehicle and estimate the state of the detected entities. In this example, three dynamic entities 104a, 104b, and 104c (collectively referred to as entities 104) are detected near the vehicle 100 and classified as vehicles, and the on-board perception component determines point-wise measurements of the state of the entities 104 (including position, velocity, yaw, and yaw rate, represented by arrows below the entities 104) at a sequence of time steps during which the entities 104 are within range of the sensors 102. The on-board perception system is further configured to process the point-wise measurements using the runtime model described above to determine a runtime estimate of the state of the entities 104. These runtime estimates are communicated to an on-board prediction component and an on-board planning component, which together determine actions to be taken by the drive system of vehicle 100. Note that the on-board perception component is for determining the state of objects or entities other than vehicle 100. Vehicle 100 may further include a localization component, which, in contrast, is for determining the position and / or orientation of vehicle 100 itself.

[0023] During operation, the on-board perception components of the vehicle 100 may generate point-wise data 106 indicative of point-wise measurements of the state of the entities 104 at different time steps. The point-wise data 106 may further include metadata related to downstream processing of the point-wise measurements, such as timestamps and classifications of the entities 104. The point-wise data 106 may be stored in any suitable format by one or more memory devices onboard the vehicle 100, for example, in a log file and / or a relational database. Optionally, the raw sensor data from which the point-wise measurements are derived may also be stored in association with the point-wise data 106. In the present example, the vehicle 100 provides the point-wise data 106 to the remote system 108, for example, by transmission over a network (not shown) using wired and / or wireless communication means. The point-wise data 106 may be provided to the remote system 108 in a streaming or batch manner, either periodically (e.g., hourly, daily, weekly, etc.), or when certain conditions are met, such as when a suitable wired or wireless connection to a network is established and / or when a certain amount of point-wise data 106 has been generated. The vehicle 100 may additionally or alternatively provide the remote system 108 with raw sensor data and / or runtime estimates determined using a runtime model.

[0024] Remote system 108 includes an offline perception component 110 that has similar functionality to the on-board perception component of vehicle 100. Offline perception component 110 includes a runtime model 112. Although runtime model 112 is stored and executed on remote system 108, runtime model 112 may be suitable for being stored and executed onboard a vehicle, such as autonomous vehicle 100. Runtime model 112 may be an instance of the same runtime model stored onboard vehicle 100, or may differ from the runtime model stored onboard vehicle 100, for example, by having different parameter values ​​and / or by implementing different data association models and / or filtering models.

[0025] The offline perception component 110 may be configured to use a runtime model 112 to process point-wise data 106 received from the vehicle 100, along with point-wise data received from other (possibly many) vehicles, to generate runtime estimates 114 of states of entities detected by those vehicles. The runtime model 112 may be configured to associate a given instance of an entity detected at a given time step with further instances detected at previous time steps according to a data association model. The runtime model 112 may be configured to filter point-wise measurements of states associated with the given instance depending on runtime estimates of states associated with the further instances at previous time steps according to a filtering model. In this manner, the runtime model iteratively generates runtime estimates 114 of states of detected entities using information available in the runtime configuration (i.e., without using information from future time steps). Note that in some examples, the offline perception component 110 may be omitted, and the perception component may receive runtime estimates 114 directly from one or more vehicles running an instance of a common runtime model.

[0026] The remote system 108 further includes a benchmark component 116. The benchmark component 116 may be configured to process the point-wise data 106 received from the vehicle 100, along with point-wise data received from other (possibly many) vehicles, using a benchmark model 118 to generate a benchmark estimate 120 of the state of entities detected by those vehicles. The benchmark model 118 may include an offline data association model and a smoother, e.g., as described above, and may accordingly generate a more accurate estimate of the state of the entities than the runtime model 112. In particular, the benchmark model 118 may use information from future time steps to determine the benchmark estimate at a given time step. Furthermore, in some cases, the benchmark model 118 may arrive at a different track hypothesis than the runtime model 112, as shown in the following example.

[0027] 4 shows a top-down view of a scene including an autonomous vehicle 100. In this scene, the autonomous vehicle 100 is traveling behind a bus 402, such that there is an occlusion region 404 of the scene (as shown between the solid lines in FIG. 4) that is hidden from at least a portion of the sensors on the vehicle 100. Using sensors 102 and an on-board data processing system, the autonomous vehicle 100 detects instances of entities classified as "cars" over a set of time steps t a , t b , t c , t d (positions for the autonomous vehicle 100 are shown simultaneously as instances 406a, 406b, 406c, 406d, and 406e in FIG. 4). In this example, the time t is twice as long as if there had been no detection of entities (other than the bus 402) in the intervening time steps. c and t d The spacing between each pair of adjacent time steps is equal, except for the spacing between time steps t a , t b , t c , t dAt each of the time steps, autonomous vehicle 100 determines point-wise measurements of the state associated with the instance detected at that time step and transmits point-wise data 106 indicative of the point-wise measurements at the time step to remote system 108.

[0028] The offline perception component 110 processes the point-wise data 106 using a runtime model 112 and computes the time step t a , t b , t c , t d The runtime model 112 determines runtime estimates of states associated with the detected instances 406a, 406b, 406c, and 406d. As described above, the runtime model 112 includes a data association model and a filtering model. In this example, the data association model associates each of the instances 406a, 406b, 406c, and 406d with a common track (shown by the dashed curve), which corresponds to the hypothesis that all of the instances 406a, 406b, 406c, and 406d are instances of the same car. The uncertainty in the filtered estimates of states is represented by the dashed bounding boxes around the instances 406a, 406b, 406c, and 406d. It is observed that the uncertainty in the states associated with the instances 406a, 406b, and 406c is relatively low, while the uncertainty in the state associated with the instance 406d is higher. This is because the interval elapsed since the previous time step is large, and the longer the time interval between measurements, the greater the uncertainty under the filtering model.

[0029] The benchmark component 116 processes the point-wise data 106 using the benchmark model 118 and calculates the time step t a , t b , t c , t d5, the benchmark model 118 associates each of the instances 406a, 406b, 406c, and 406d with a common track, but initiates a new track for the instance 406d, corresponding to the hypothesis that the instance 406d is a different vehicle from the instances 406a, 406b, and 406c. In this case, the benchmark model 118's hypothesis corresponds to the ground truth, while the runtime model 112's hypothesis is inaccurate (the dashed instances 408a and 408b indicate the ground truth positions of the two vehicles at the time step when both vehicles are occluded). Furthermore, it is observed that the uncertainty in the state associated with the instances is consistently lower for the benchmark model 118 than for the runtime model 112, and even for the instance 406d that initiates a new track. The benchmark model 118 is able to correctly resolve the tracking ambiguity and determine a more accurate estimate of the state. This is at least in part because the benchmark model 118 can leverage information from future time steps, while the runtime model is constrained to use information from past time steps. The benchmark model 118 may also use a different data association model than the runtime model 112.

[0030] The offline perception component 110 and the benchmark component 116 may send the runtime estimates 114 and the benchmark estimates 120, optionally using similar data generated from other sources (e.g., many other autonomous vehicles), to a model evaluator and updater 122. The model evaluator and updater 122 is configured to process the runtime estimates 114 and the benchmark estimates 120 to generate data corresponding to an updated runtime model 126. In particular, the model evaluator and updater 122 may be configured to evaluate metrics 124 that measure deviations between the runtime estimates 114 and the corresponding benchmark estimates 120 and to update the runtime model 112 in response to the evaluation of the metrics. By treating the benchmark estimates 120 as a pseudo-ground truth, the metrics 124 may be used to measure the performance of the runtime model 112. Because the performance of the runtime model 112 can be affected by various factors, as described below, the benchmark component 116 and the model evaluator and updater 122 may be run separately on different sets of point-wise data 106, resulting in multiple versions of the updated runtime model 126.

[0031] The runtime model 112 may run differently in different environmental conditions, such as different times of day (corresponding to different lighting conditions), different driving environments (such as urban or rural environments), and / or different weather conditions. Thus, components of the remote system 108 may run separately using the point-wise data 106 for each different environmental condition to generate different updated runtime models 126 appropriate for the different environmental conditions. For example, in dark or snowy conditions, measurement noise may be greater, making it more difficult for the runtime model 112 to associate object instances with a given track, in which case a different threshold for object association may be appropriate.

[0032] The runtime model 112 may execute differently when used with different perception components or different versions of perception components. Thus, components of the remote system 108 may execute differently using point-wise data 106 generated by different perception components or different versions of perception components, resulting in different updated runtime models 126 appropriate for each different perception component of the different versions of perception components. For example, different values ​​and thresholds of measurement noise may be appropriate for object association depending on the accuracy of the object detection model used in generating the point-wise data 106.

[0033] Metric 124 may measure pairwise deviations between runtime estimates of the state and benchmark estimates. For example, metric 124 may depend on the metric distance between the runtime estimate of the entity's state detected at a given time step and the benchmark estimate of the entity's state. Metric 124 may be a function of, for example, L1 loss, smoothed L1 loss, L2 loss, or any other suitable pairwise distance measure. By summing or otherwise combining these losses across multiple detections within a given time period, metric 124 may measure the performance of the runtime model 112 over that time period. Metric 124 may additionally or alternatively measure pairwise deviations between uncertainty estimates between the runtime estimate of the state and the benchmark estimate. For example, for each detection at each time step, the Kalman-type filter (and corresponding smoother) may generate a posterior covariance matrix of the state, which may be considered an estimate of the predictive accuracy of the runtime estimate of the state. Metric 124 may then depend on the metric distance between the posterior covariance of the runtime estimate and the posterior covariance of the corresponding benchmark estimate. By measuring the deviation between a state estimate and an associated uncertainty estimate, metric 124 may measure the effectiveness of runtime model 112 in filtering out noise in the state measurements and in estimating the confidence that can be attributed to the resulting estimate. Quantifying uncertainty is important in safety-critical environments, such as driving environments, because certain actions are taken only if there is sufficient confidence in the perception of the environment.

[0034] Instead of measuring pairwise deviations, metric 124 may measure the deviation between the joint probability distribution of runtime estimates 114 and benchmark estimates 120. For example, metric 124 may depend on the Kullback-Leibler (KL) divergence of a first probability distribution from a second probability distribution, where the first (or second) probability distribution may be the joint probability distribution of the runtime estimates (defined by the runtime estimates and their corresponding posterior covariances), and the second (or first) probability distribution may be the joint probability distribution of the corresponding benchmark estimates. The joint distribution may be obtained over multiple detections, e.g., within a given time frame, optionally over multiple time steps.

[0035] As explained above, runtime model 112 may perform less accurate data associations than benchmark model 118. Metric 124, which measures the deviation between runtime estimates 114 and benchmark estimates 120 and, optionally, their corresponding uncertainty estimates, may automatically capture situations in which tracking predictions diverge between runtime model 112 and benchmark model 118 because different tracking predictions result in different tracking predictions and use information from different prior states to filter noise. In the example of FIG. 4 , the runtime estimate of the state associated with instance 406d is derived from the runtime estimate of the state associated with instance 406c. In contrast, the benchmark estimate of the state associated with instance 406d is independent of the state associated with instance 406c. Even if the filtering model is highly accurate, these different tracking predictions are likely to result in differences between the runtime and benchmark estimates. In this way, metric 124 may capture deviations between two data association models when tracking ambiguities. In other cases, it may be preferable for metric 124 to compare runtime and benchmark estimates only if they produce the same tracking predictions, for example. In this way, the metric 124 may measure the accuracy of the filtering model as opposed to the combined effect of the data association model and the filtering model.

[0036] In addition to, or instead of, measuring the deviation between state estimates, metric 124 may explicitly measure the deviation between tracking predictions made by runtime model 112 and benchmark model 118. For example, metric 124 may count the number of times runtime model 112 and benchmark model 118 diverge from each other within a given time frame. Alternatively, the data association problem for a given time step may be viewed as a classification problem, where each detected instance is classified as either a new entity, a false positive, or an entity seen in a previous time step, in which case metric 124 may measure the precision and recall of this classification problem (treating the output of benchmark model 118 as ground truth). Alternatively, metric 124 may measure the precision and recall of the cardinality of a set of entities (i.e., the number of entities estimated to be present) over one or more time steps.

[0037] The model evaluator and updater 122 is configured to generate data corresponding to an updated runtime model 126 in response to an evaluation of the metrics 124. The updated runtime model 126 may depend on an evaluation of the metrics 124 aggregated across multiple data sources, such as multiple autonomous vehicles including the vehicle 100. The model evaluator and updater 122 may determine, for example, based on the evaluation of the metrics 124, that the runtime model 112 needs to be updated because one or more aspects of the performance of the runtime model 112 differ from that of the benchmark model 120 by more than a threshold amount. The model evaluator and updater 122 may determine that a data association portion of the runtime model 112 needs to be updated and / or that a filtering portion of the runtime model 112 needs to be updated. The model evaluator and updater 122 may update the indicated aspect(s) of the runtime model 112. The updated runtime model 126 may then be evaluated against the benchmark model 118 to determine whether the updates were effective in improving the runtime model 112. Updating the runtime model 112 may include, for example, replacing a data association model and / or a filtering model with a different data association model and / or a filtering model.

[0038] Updating the runtime model 112 may include updating values ​​of one or more parameters associated with the data association model and / or the filtering model. Updating may include sampling new values ​​of the one or more parameters using a sampling technique such as random search, grid search, or MCMC sampling. The sampling may be performed iteratively in a manner that optimizes the value of the one or more parameters with respect to the evaluation of the metric 124. For example, the updating may use MCMC sampling, in which the sampling distribution is conditioned on the evaluation of the metric 124. In some examples, Bayesian optimization may be performed, providing a principled framework for addressing the exploration / exploitation dilemma encountered when optimizing parameter values. A suitable Bayesian optimization method may employ a surrogate function, such as a Gaussian process, a Bayesian neural network, or other probabilistic function or process, to predict the evaluation of the metric 124 for a given set of parameter values, and then derive a gain function from this surrogate function, e.g., based on entropy search or expected improvement, and sample parameter values ​​based on this gain function. The gain function may be configured to automatically balance exploration and exploitation (e.g., moving from exploration in early stages of an experiment toward exploitation in later stages of an experiment). In other examples, the sampling of parameter values ​​may be performed by a reinforcement learning agent.

[0039] Alternatively, or in addition, the values ​​of one or more parameters of the runtime model 112 may be optimized using gradient-based optimization, such as using stochastic gradient descent or any of its variants. Gradient-based optimization may be suitable for situations in which the metric 124 is differentiable with respect to one or more parameters of the runtime model 112.

[0040] In some examples, the remote system 108 may have access to ground truth data for the tracks and / or states of the detected entities. In this case, the model updater 122 may further use the ground truth data to update the runtime model 112. For example, the model updater 122 may use the ground truth data to measure the accuracy of the benchmark model 118 and the accuracy of the runtime model 112, enabling the model updater 112 to determine whether inaccuracies in the track determinations of states and / or runtime estimates of those states determined by the runtime model 112 are caused by defects in the runtime model 112 (in which case the runtime model 112 is expected to be significantly worse than the benchmark model 118) and / or inaccuracies in the point-wise data 106 (in which case the output of the benchmark model 118 is expected to differ significantly from the ground truth).

[0041] In the example of FIG. 6 , the top box 600 a shows a first precision-recall curve 602 and a second precision-recall curve 604 for the task of estimating cardinality, where the first curve 602 corresponds to the benchmark model 118 and the second curve 604 corresponds to the runtime model 112. For both models, precision and recall are measured against ground truth and aggregated from multiple data sources. It is observed that the area under the second curve 604 is significantly smaller than the area under the first curve 602, indicating that the benchmark model 118 outperforms the runtime model 112 for this task. In this example, the model updater 122 updates the parameter values ​​of the runtime model 112 in response to a comparison between the first precision-recall curve 602 and the second precision-recall curve 604. For example, the model updater 112 may perform MCMC sampling, where the sampling distribution is conditioned on the difference between the first precision-recall curve 602 and the second precision-recall curve 604, e.g., the difference between the areas of the curves. 6 shows a first precision and recall curve 602 and a third precision and recall curve 606, where the third curve 606 corresponds to the updated runtime model 112. It is observed that the third curve 606 is closer to the first curve 602 than the second curve 604, indicating an improvement in the runtime model 112 as a result of the update.

[0042] Although the system 108 is described above as being remote from the vehicle 100, in other examples, the vehicle may have on-board components for implementing benchmark models and generating update data to update the runtime model for the vehicle. This allows the vehicle to adapt its own runtime model based on data generated in its own environment, but may not benefit from data generated by other vehicles.

[0043] FIG. 7 illustrates an example of a computer-implemented method 700 that may be performed by a computing system, such as the remote system 108 of FIG. 1 . The method 700 includes, at 702, obtaining point-wise data indicative of point-wise measurements of the state of an object detected by an object detection system over a plurality of time steps. The object may be, for example, an object in the vicinity of an autonomous vehicle or other type of vehicle. The object detection system may be part of a perception component mounted on the vehicle, in which case the point-wise data may be received from the perception component mounted on the vehicle. Alternatively, the point-wise data may be determined remotely from the vehicle, for example, by processing raw sensor data or logged data received from the vehicle.

[0044] The method 700 proceeds, at 704, by obtaining runtime data indicative of a runtime estimate of the object's state for multiple time steps. The runtime data is generated by a runtime model, which may be implemented as part of a perception system onboard the vehicle or may be implemented remotely from the vehicle. The runtime data may be generated by recursively processing point-wise data, as described elsewhere in this disclosure. The runtime model may include, for example, an online data association model and a filtering model.

[0045] The method 700 proceeds, at 706, by processing the point-wise data to determine benchmark estimates of the object's state for multiple time steps. The benchmark data is generated by a benchmark model, which may be an offline system remote from the source of the point-wise data. The benchmark model may include determining the benchmark estimate of the object's state at a given time step based on point-wise measurements of the object's state at the given time step and multiple additional time steps, which may include at least one time step later than the given time step. In this manner, at least, the benchmark model may be contrasted with a runtime model. The benchmark model may include, for example, an offline data association model and a smoother.

[0046] The method 700 proceeds, at 708, by evaluating a metric that measures the deviation between a runtime estimate of the object's state and a benchmark estimate for multiple time steps.

[0047] The method 700 concludes at 710 by updating the runtime model based on the evaluation of the metrics. The updated runtime model may optionally be used to generate further runtime data, so that the method 700 returns to 704 and continues iteratively until a stopping condition, such as a convergence condition, is met or a predetermined number of iterations have been performed.

[0048]

[0023] By way of further illustration, Figure 2 shows an example in which such an offline model is used to assist in data labeling, e.g., to label training data for a machine learning model. Examples of techniques for assisting in data labeling can be found, for example, in U.S. Patent Application No. 17 / 538,909, filed November 30, 2021, and entitled "Generating and Training Object Detection Models for Autonomous Vehicles," the contents of which are incorporated herein by reference in their entirety for all purposes.

[0049] FIG. 2 illustrates an autonomous vehicle 200 equipped with sensors 202 for detecting and determining point-wise measurements of states of dynamic entities, including vehicles 204a, 204b, and 204c. The autonomous vehicle 200 of FIG. 2 may be functionally identical to the autonomous vehicle 100 of FIG. 1 and is similarly configured to transmit point-wise data 206 to a remote system 208. The point-wise data 206 indicates point-wise measurements of states of entities detected near the autonomous vehicle 200 at different time steps. In this example, the vehicle 200 also provides sensor data 209 captured by the onboard sensors 202 to the remote system 208. The sensor data 209 may be raw sensor data and / or processed sensor data and includes data from which a visual representation of the environment surrounding the vehicle 200 can be derived. While in this example, the vehicle 200 provides the point-wise data 206 to the remote system 208, in other examples, the remote system 208 may instead derive the point-wise data 206 from the sensor data 209.

[0050] The sensor data 209 may include sufficient input (test) data for machine learning models (e.g., neural network models) configured to perform object detection and / or other tasks related to controlling an autonomous vehicle, including, but not limited to, semantic segmentation, instance segmentation, object classification, and object tracking. Machine learning models for these purposes are typically trained using supervised learning based on labeled training data. The process of obtaining such labeled training data that covers a sufficiently diverse range of scenarios can be very time-consuming and resource-intensive and traditionally involves a human user manually applying labels or annotations. Labels are metadata associated with input data items (e.g., images) that can be compared to the output of machine learning models during supervised learning. In this context, labels may include, for example, bounding boxes, boundary contours for semantic or instance segmentation, class labels, tracking predictions, etc., for particular types of entities identified in the input data items.

[0051] To assist in the data labeling process, the remote system 208 includes a proposal component 210 configured to process the point-wise data 206 using a proposal model 212 and generate proposed annotations 214 for the sensor data 209. The proposal model 212 may include, for example, an offline data association model and a smoother, as described above, and may accordingly determine refined estimates of states associated with detected instances at a given time step based on accurate track predictions. The proposal model 212 may further include trained machine learning models and / or heuristic models for performing tasks such as object detection and / or other tasks related to controlling an autonomous vehicle. The proposed annotations 214 may depend on the track predictions and / or refined estimates of states. For example, the proposal model 212 may be configured to determine a common class label for instances of entities associated with a common track based on confidence levels associated with the class labels of instances on the track. In this way, the common class label may propagate along the track even if some instances (e.g., in the case of partial occlusion) have low confidence levels. On the other hand, changes in class labels on a predicted track may cause the proposal model 212 to reevaluate the tracking prediction, as described above with respect to resolving tracking ambiguities. More generally, the proposal model 212 may be configured to enforce or encourage continuity of proposed annotations on a given track, for example, by ensuring that bounding boxes or boundary contours associated with different instances of an entity do not imply changes in the size or shape of the entity. In another example, the proposal model 212 may be configured to determine the position and / or orientation of detected instances based on refined estimates of states associated with instances on the same track. This may enable, for example, determining accurate top-down bounding boxes for all instances on a track.In this way, the proposal model 212 can use information from multiple instances associated with a common track to determine a proposed annotation that is accurate and robust to noise.

[0052] The remote system 208 includes a user interface 216, which may include a combination of hardware and software components that enable a user 218 to interact with the remote system 208. The user interface 216 may include one or more displays, one or more input devices, and rendering software and drivers associated with these devices. The user interface 216 is configured to render a visual representation of the environment derived from the sensor data 209, such as an image or video representing the environment as viewed by the user 218 at one or more time steps. The visual representation may be an image-based representation based on image data captured by the sensors 202 (e.g., a camera). In other examples, the visual representation may include a non-image-based visual representation, such as a visual representation of a lidar point cloud, radar or sonar return signals, or the like. In some cases, the user interface 216 may present one or more visual representations based on a combination of different sensor modalities of the same environment, such as a visual image and a corresponding lidar point cloud. The visual representation may be rendered from the vehicle's perspective and / or from various other angles, such as a top-down view of the environment.

[0053] The user interface 216 is further configured to render a visual representation of one or more proposed annotations 214, e.g., overlaid on top of a visual representation of the environment. The visual representation of the proposed annotation(s) 214 may include one or more of a proposed bounding box, a proposed boundary contour for semantic or instance segmentation, and / or text or symbols indicating a proposed class label for the entity. The user interface 216 further enables a user 218 to modify the proposed annotation(s) 214 or otherwise provide user-approved annotation(s) 220 using one or more input devices presented with the proposed annotation(s) 214. For example, the user interface 216 may render a visual representation of the proposed class label for the detected object and provide a means for the user 218 to either approve the class label or reject the class label. If the user 218 rejects a class label, the user interface 216 may present alternative class labels (e.g., in order of decreasing confidence as determined by the proposed model 212). The user 218 may then select a user-approved class label from the list of alternative class labels. In another example, the user interface 216 may present multiple options of proposed annotations (e.g., multiple proposed class labels) and allow the user 216 to select one of the proposed annotations, in which case the selected annotation may become the user-approved annotation. In another example, the visual representation of the proposed annotation may include a proposed bounding box (e.g., a bounding box of an entity viewed from the perspective of the vehicle 200) or a top-down bounding box (e.g., if the visual representation of the environment is a top-down representation).If user 218 determines that proposed model 212 has incorrectly determined the size, shape, and / or location of an entity, user interface 216 allows the user to determine a user-approved bounding box that more accurately corresponds to the entity's boundaries by dragging the corners of the bounding box to new positions. Alternatively, user interface 216 allows the user, having viewed the proposed bounding box, to draw or otherwise define a new user-approved bounding box. If user 218 determines that the proposed annotation corresponds to a false positive, user interface 216 allows the user to delete the proposed annotation.

[0054] In response to receiving a different user-approved annotation for the same instance of an entity than the proposed annotation, the suggestion component 210 may be configured to update the proposed annotation for one or more additional instances detected at a respective different time step. For example, the user 218 may specify a class label for a given instance detected at a given time step. The given instance may be associated with a track having a sequence of additional instances according to a data association model. The suggestion component 210 may therefore update the proposed class label of any instances associated with the same track to match the class label specified by the user 218. The user 218 may then be presented with a visual representation of the environment at a later or earlier time step along with the updated proposed class labels for the instances on the track. Updating the proposed annotation in this manner may depend on confidence values ​​associated with the original class labels and / or the updated class labels for the additional instances. For example, if the original class label and the updated class label for one of the additional instances are assigned similar confidence levels, the suggestion component 210 may perform the update as described, but if the original class label has a significantly higher confidence level than the updated class label for the additional instance, the suggestion component 210 may refrain from performing the update. The suggestion component 210 may be configured to update bounding boxes, boundary contours, or other proposed annotations in a similar manner. For example, if the user 218 changes the size and / or shape of the bounding box or contour of a given instance, the suggestion component 210 may change the size and / or shape of the corresponding bounding boxes or contours for additional instances on the same track.

[0055] The user interface 216 may be configured to present a video stream representation of the environment over a sequence of time steps along with corresponding annotations. A user 218 may be provided with a means to pause or rewind the video stream to a selected time step and provide a user-approved annotation for that time step. Once an annotation is corrected at a given time step, the correction may be propagated to other instances associated with the same track, as described above, providing an intuitive and time-efficient way for a user to correct annotations across several time steps.

[0056] The remote system 208 generates labeled training data 222 based at least in part on the user-approved annotations 220 and the sensor data 209. The labeled training data 220 may further include proposed annotations 214 that have been explicitly or implicitly accepted by the user 218. The remote system 208 may use the labeled training data 222 for supervised training of a machine learning model 224. The machine learning model 224 may be suitable for use in controlling an autonomous vehicle and may be the same as or different from the machine learning model implemented by the proposed model 212 to generate the proposed annotations 214. The machine learning model 224 may be, for example, a runtime model suitable for use onboard an autonomous vehicle. As an offline model, the proposed model 212 may be able to perform a given task more accurately than the runtime machine learning model 224. Thus, the combination of the proposed annotations 214 generated by the proposed model 212 and the user-approved annotations 220 provided by the user 218 may represent a sufficiently good approximation of the ground truth data for effective training of the machine learning model 224.

[0057] The machine learning model 224 may be trained using labeled training data aggregated from multiple sources, for example, based on sensor data received from many autonomous vehicles. Furthermore, the task of generating user-approved annotations may be shared among many users accessing the remote system 208 or using other systems, for example, in exchange for monetary compensation. The methods and techniques described herein may significantly improve the speed and accuracy with which labeled training data can be generated.

[0058] 8 illustrates an example of a computer-implemented method 800 that may be performed by a computing system, such as remote system 208 of FIG. 2. Method 800 includes, at 802, determining an estimate of a state of an object at a first time step based on point-wise measurements of the state of the object at a first time step and point-wise measurements of the state of the object at a plurality of additional time steps. The estimate of the state may be determined using offline models described herein, including, for example, data association models and smoothers.

[0059] The method 800 proceeds, at 804, by generating a proposed annotation to be associated with the object at the first time step. The proposed annotation may be generated using the estimate of the object's state determined at 802.

[0060] The method 800 proceeds at 806 by rendering, via a user interface, a visual representation of the environment including the object at the first time step and a visual representation of the proposed annotations. The visual representation of the environment and point-wise measurements of the state of the object at the first time step may be derived from common sensor data.

[0061] The method 800 proceeds at 808 by receiving user input via a user interface, the user input indicating a user-approved annotation associated with the object at the first time step. As described above, the user-approved annotation may be an approval or confirmation of the proposed annotation, or may be a modified annotation that differs from the proposed annotation.

[0062] The method 800 concludes at 810 by generating training data for a machine learning model for use in controlling the autonomous vehicle based at least in part on the user-approved annotations. The training data may include an input portion based on sensor data (e.g., images) that derive point-wise measurements of the state of the object at the first time step, and labels based on the user-approved annotations.

[0063] FIG. 3 shows a block diagram of an example system 300 for implementing the techniques described herein. In some examples, system 300 may include a vehicle 302, which may correspond to vehicle 100 of FIG. 1 and / or vehicle 200 of FIG. 2. In some examples, vehicle 302 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the U.S. National Highway Traffic Safety Administration, which describes a vehicle that can perform all safety-critical functions during the entire journey without a driver (or passenger) being expected to control the vehicle at all times. However, in other examples, autonomous vehicle 302 may be a fully or partially autonomous vehicle having any other level or classification. Additionally, in some examples, the techniques described herein may be used by non-autonomous vehicles.

[0064] The vehicle 302 may include vehicle computing device(s) 304, one or more sensor systems 306, one or more emitters 308, one or more communication connections 310, at least one direct connection 312 (e.g., for physically coupling the vehicle 302 to exchange data and / or provide power), and one or more drive systems 314.

[0065] In some examples, the sensor(s) 306 may include light detection and ranging (LIDAR) sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., global positioning system (GPS), compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., red-green-blue (RGB), infrared (IR), intensity, depth, time-of-flight, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. The sensor(s) 808 may include multiple instances of each of these or other types of sensors. For example, the LIDAR sensors may include individual LIDAR sensors located at the corners, front, rear, sides, and / or top of the vehicle 302. As another example, the cameras may include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle 302. The sensor(s) 306 may provide input to the vehicle computing device(s) 304 .

[0066] Vehicle 302 may also include emitter(s) 308 for emitting light and / or sound, as described above. Emitter(s) 308 in this example may include interior audio and visual emitter(s) for communicating with occupants of vehicle 302. By way of example and not limitation, interior emitter(s) may include speakers, lights, indicator lights, display screens, touchscreens, haptic emitter(s) (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.), etc. Emitter(s) 308 in this example may also include exterior emitter(s). By way of example and not limitation, the external emitter(s) in this example include lights or other indicators of vehicle action (e.g., indicator lights, indicator lamps, light arrays, etc.) for signaling direction of travel, and one or more audio emitter(s) (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which include acoustic beam steering technology.

[0067] Vehicle 302 may also include communication connection(s) 310 that enable communication between vehicle 302 and one or more other local or remote computing device(s). For example, communication connection(s) 310 may facilitate communication with other local computing device(s) on vehicle 302 and / or drive system(s) 314. Communication connection(s) 308 may also additionally or alternatively enable vehicle 302 to communicate with other nearby computing device(s) (e.g., other nearby vehicles, traffic signals, etc.). Communication connection(s) 310 may additionally or alternatively enable vehicle 302 to communicate with computing device 336.

[0068] Vehicle computing device(s) 304 may include one or more processors 316 and a memory 318 communicatively coupled to the one or more processors 316. In the depicted example, the memory 318 of the vehicle computing device(s) 304 stores a localization component 320, an on-board perception component 322 including a data association model 324 and a filtering model 326, one or more system controllers 328, and a planning component 330. While shown in FIG. 3 as resident in memory 318 for illustrative purposes, it is contemplated that the localization component 320, the perception component 322, the one or more system controllers 328, and / or the planning component 330 may additionally or alternatively be accessible to (e.g., remotely stored on) the vehicle 302.

[0069] In some examples, the perception component 322 may include functionality for performing object detection, semantic segmentation, instance segmentation, and / or classification. In some examples, the perception component 322 may generate processed sensor data indicative of the presence of an entity in proximity to the vehicle 302 and / or a classification of the entity type (e.g., automobile, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional or alternative examples, the perception component 322 may provide processed sensor data indicative of one or more characteristics associated with the detected entity (e.g., tracked object) and / or the environment in which the entity is located. In some examples, the characteristics associated with the entity may include, but are not limited to, x-position (global and / or local), y-position (global and / or local), z-position (global and / or local), orientation (e.g., roll, pitch, yaw), entity type (e.g., classification), entity velocity, entity acceleration, entity range (size), etc. Characteristics associated with an environment may include, but are not limited to, the presence of another entity in the environment, the state of another entity in the environment, the time of day, the day of the week, the season, weather conditions, darkness / light indications, etc.

[0070] In at least one example, vehicle computing device(s) 304 can include one or more system controllers 324 that can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 302. The system controller(s) 324 can communicate with and / or control corresponding systems of the drive system(s) 314 and / or other components of the vehicle 302.

[0071] System controller(s) 324 may be communicatively coupled to one or more sensors of vehicle sensor system(s) 306. As a non-limiting example, the sensors may detect the presence of objects in the vehicle's environment and / or determine attributes of those objects. System controller(s) 324 may also trigger activation of safety systems of vehicle 302 if it determines that activation of the safety systems is necessary. For example, system controller(s) 324 may command an airbag control unit to deploy one or more airbags or send a signal to a tensioner configured to adjust the tension of one or more restraint devices. Other safety systems are known and may be activated. In other embodiments, system controller 324 may command the activation of multiple safety systems. In some embodiments, some or all of the functions of system controller 324 may be performed remotely from vehicle 302, for example, on a remote server associated with dispatch or headquarters of vehicle 302, or in the cloud. In other embodiments, some or all of the functions of the system controller(s) 324 may be performed in the vehicle 302 to minimize any delays that may result from transmitting data between local locations.

[0072] The drive system(s) 314 may include many vehicle systems, including a high-voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the exterior surroundings of the vehicle), and one or more other systems (e.g., a cooling system, a safety system, an on-board charging system, other electrical components such as a DC / DC converter, high-voltage junctions, high-voltage cables, a charging system, a charge port, etc.). Additionally, the drive system(s) 314 may include a drive system controller that may receive and preprocess data from the sensor(s) and control the operation of various vehicle systems. In some examples, the drive system controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more modules for performing various functions of the drive system(s) 314. Additionally, the drive system(s) 314 may also include one or more communication connection(s) that enable the respective drive system to communicate with one or more other local or remote computing device(s).

[0073] In some examples, vehicle 302 may transmit operational data, including raw or processed sensor data from sensor system(s) 306, to one or more computing device(s) 336 over network(s) 334. In other examples, vehicle 302 may transmit processed operational data and / or representations of the operational data to computing device(s) 336 at a particular frequency, such as after a predetermined period of time, in near real time, etc. In some cases, vehicle 302 may transmit raw or processed operational data to computing device(s) 336 as one or more log files.

[0074] The one or more computing device(s) 336 may include one or more processors 338 and a memory 340 communicatively coupled to the one or more processors 338. The memory 340 may store data defining the offline model 342, as described elsewhere in this disclosure. The computing device(s) 336 may also include a user interface 346 to allow user input regarding assistance with the labeling function, as described elsewhere in this disclosure.

[0075] In some cases, some or all aspects of the components discussed herein may include any model, algorithm, and / or machine learning algorithm. For example, some of the component(s) in memory 318 may be implemented as a neural network. As can be understood in the context of the present disclosure, a neural network may be trained using machine learning, in which values ​​for the network's parameters may be determined automatically from data during the training process, rather than being explicitly programmed by a human programmer.

[0076] Illustrative clauses A: A system including one or more processors and one or more computer-readable media storing computer-executable instructions, the computer-executable instructions, when executed by the one or more processors, causing the one or more processors to: obtain point-wise data indicative of point-wise measurements of a state of an object detected by an object detection system mounted on a vehicle for a plurality of time steps; and recursively process the point-wise data with a runtime model to determine a runtime estimate of the state of the object for the plurality of time steps, wherein recursive processing with the runtime model includes determining the runtime estimate of the state of the object at the first time step based on the point-wise measurements of the state of the object at a first time step and the runtime estimate of the state of the object at a second time step; the second time step is earlier than the first time step; processing the point-wise data with a benchmark model to determine a benchmark estimate of the state of the object for the plurality of time steps, the processing with the benchmark model including determining the benchmark estimate of the state of the object at the first time step based on the point-wise measurements of the state of the object at the first time step and a plurality of further time steps, the plurality of further time steps including time steps later than the first time step; evaluating a metric that measures deviation between the runtime estimate and the benchmark estimate of the state of the object for the plurality of time steps; and updating the runtime model based on the evaluation of the metric.

[0077] B: The system of clause A, configured to transmit data indicative of the updated runtime model to an autonomous vehicle via the data interface.

[0078] C: The system of clause A or B, wherein the state of the object includes at least one of position, velocity, yaw, and yaw rate.

[0079] D: The system described in any of clauses A to C, wherein processing the point-wise data using the benchmark model includes associating a first instance of the object detected at the first time step with each further instance of the object detected at the multiple further time steps, and the associating includes determining multiple candidate track configurations including candidate groupings of instances of the object detected at the multiple time steps, determining a most likely track configuration from the multiple candidate track configurations, and associating the first instance of the object with each further instance of the object in accordance with the determined most likely track configuration.

[0080] E: The system described in clause D, wherein the runtime model includes one or more thresholds for controlling association of object instances detected at each different time step, and updating the runtime model includes determining updated values ​​for the one or more thresholds.

[0081] F: A computer-implemented method comprising: obtaining point-wise data indicative of point-wise measurements of a state of an object detected by an object detection system for a plurality of time steps; obtaining runtime data indicative of runtime estimates of the state of the object for the plurality of time steps from a runtime model; processing the point-wise data with a benchmark model to determine benchmark estimates of the state of the object for the plurality of time steps; evaluating a metric that measures deviation between the runtime estimates and the benchmark estimates of the state of the object for the plurality of time steps; and updating the runtime model based on the evaluation of the metric.

[0082] G. The computer-implemented method of clause F, including transmitting data indicative of the updated runtime model to an autonomous vehicle.

[0083] H: The computer-implemented method of clause F or G, wherein the state of the object includes at least one of position, velocity, yaw, and yaw rate.

[0084] I: The computer-implemented method of any of clauses F-H, wherein the runtime model generates the runtime data using a recursive filter having an associated process noise covariance and an associated observation noise covariance, and updating the runtime model includes updating at least one of the associated process noise covariance and the associated observation noise covariance.

[0085] J: recursively processing the point-wise data by the runtime model to determine the runtime estimate of the state of the object for the plurality of time steps, wherein recursive processing by the runtime model includes determining the runtime estimate of the state of the object at the first time step based on the point-wise measurements of the state of the object at a first time step and the runtime estimate of the state of the object at a second time step, wherein the second time step is earlier than the first time step.

[0086] K: The computer-implemented method of any of clauses F-J, wherein the processing with the benchmark model includes determining the benchmark estimate of the state of the object at a first time step based on the point-wise measurements of the state of the object at the first time step and a plurality of further time steps, the plurality of further time steps including time steps later than the first time step.

[0087] L: The computer-implemented method of clause K, wherein the processing with the benchmark model includes processing the point-wise measurements of the state at the first time step and the plurality of further time steps with a smoother.

[0088] M: The computer-implemented method of clause K or L, wherein the processing by the benchmark model includes associating a first instance of the object detected at the first time step with each further instance of the object detected at the plurality of further time steps.

[0089] N: The computer-implemented method of clause M, wherein associating the first instance of the object detected at the first time step with each of the further instances of the object detected at the multiple further time steps includes determining a plurality of candidate track configurations including candidate groupings of instances of the object detected at the multiple time steps, determining a most likely track configuration among the plurality of candidate track configurations, and associating the first instance of the object with each of the further instances of the object in accordance with the determined most likely track configuration.

[0090] O: The computer-implemented method of clause N, comprising determining a respective likelihood value for each of the plurality of candidate track configurations, and determining the most likely track configuration comprises selecting the candidate track configuration determined to have the highest likelihood value.

[0091] P: The computer-implemented method described in any of clauses L to N, wherein the runtime model includes one or more thresholds for controlling association of instances of the object detected at each of the different time steps, and updating the runtime model includes determining updated values ​​for the one or more thresholds.

[0092] Q: The computer-implemented method of any of clauses F to P, wherein obtaining the point-wise data includes receiving the point-wise data from a computer system installed in a vehicle.

[0093] R: The computer-implemented method of any of clauses F-Q, wherein the metric measures the divergence between a probability distribution associated with the runtime estimate of the object for the plurality of time steps and a probability distribution associated with the benchmark estimate of the kinematic state of the object for the plurality of time steps.

[0094] S: obtaining ground truth data indicating ground truth values ​​of the state of the object for the plurality of time steps, wherein the metric measures the deviation between the runtime estimate of the state of the object and the benchmark estimate of the state of the object by referencing the ground truth values ​​of the state of the object.

[0095] T: One or more non-transitory computer-readable media storing instructions executable by one or more processors, the instructions, when executed, causing the one or more processors to perform operations including obtaining point-wise data indicative of point-wise measurements of a state of an object detected by an object detection system for a plurality of time steps; obtaining runtime data indicative of runtime estimates of the state of the object for the plurality of time steps from a runtime model; processing the point-wise data with a benchmark model to determine benchmark estimates of the state of the object for the plurality of time steps; evaluating a metric measuring deviation between the runtime estimate and the benchmark estimate of the state of the object for the plurality of time steps; and updating the runtime model based on the evaluation of the metric.

[0096] Although the exemplary clauses above are described with respect to one particular embodiment, it should be understood in the context of this specification that the subject matter of the exemplary clauses may also be implemented by a method, device, system, computer-readable medium, and / or other embodiment. Furthermore, any of Examples A-T may be implemented alone or in combination with any other one or more of Examples A-T.

Claims

1. 1. A system including one or more processors and one or more computer-readable media storing computer-executable instructions, The computer-executable instructions, when executed by the one or more processors, cause the one or more processors to: acquiring point-wise data indicative of point-wise measurements of states of objects detected by an object detection system mounted on a vehicle for a plurality of time steps; recursively processing the point-wise data with a runtime model to determine runtime estimates of the state of the object for the plurality of time steps, wherein recursively processing with the runtime model includes determining the runtime estimate of the state of the object at the first time step based on the point-wise measurements of the state of the object at a first time step and the runtime estimate of the state of the object at a second time step, the second time step being earlier than the first time step; processing the point-wise data with a benchmark model to determine benchmark estimates of the state of the object for the plurality of time steps, wherein the processing with the benchmark model includes determining the benchmark estimate of the state of the object at the first time step based on the point-wise measurements of the state of the object at the first time step and a plurality of further time steps, the plurality of further time steps including time steps later than the first time step; evaluating a metric that measures deviation between the runtime estimate and the benchmark estimate of the state of the object for the plurality of time steps; updating the runtime model based on the evaluation of the metrics; The system causes the system to perform operations including:

2. The system of claim 1 , configured to transmit data indicative of the updated runtime model to an autonomous vehicle via a data interface.

3. The system of claim 1 , wherein the state of the object includes at least one of a position, a velocity, a yaw, and a yaw rate.

4. the processing with the benchmark model includes associating a first instance of the object detected at the first time step with each further instance of the object detected at the plurality of further time steps; The associating step comprises: determining a plurality of candidate track configurations comprising candidate groupings of object instances detected at the plurality of time steps; determining a most likely track configuration from among the plurality of candidate track configurations; associating the first instance of the object with each of the further instances of the object in accordance with the determined most likely track configuration; The system of claim 1 , comprising:

5. the runtime model includes one or more thresholds for controlling association of object instances detected at each different time step; The system of claim 4 , wherein updating the runtime model includes determining updated values ​​for the one or more thresholds.

6. obtaining point-wise data indicative of point-wise measurements of states of objects detected by the object detection system for a plurality of time steps; obtaining runtime data from a runtime model indicative of runtime estimates of the state of the object for the plurality of time steps; processing the point-wise data with a benchmark model to determine benchmark estimates of the state of the object for the plurality of time steps; evaluating a metric that measures deviation between the runtime estimate and the benchmark estimate of the state of the object for the plurality of time steps; updating the runtime model based on the evaluation of the metrics; 20. A computer-implemented method comprising:

7. The computer-implemented method of claim 6 , comprising transmitting data indicative of the updated runtime model to an autonomous vehicle.

8. The computer-implemented method of claim 6 , wherein the state of the object includes at least one of a position, a velocity, a yaw, and a yaw rate.

9. 7. The computer-implemented method of claim 6, wherein the processing with the runtime model includes determining the runtime estimate of the state of the object at a first time step based on the point-wise measurements of the state of the object at the first time step and the runtime estimate of the state of the object at a second time step, the second time step being earlier than the first time step.

10. 7. The computer-implemented method of claim 6, wherein the processing with the benchmark model includes determining the benchmark estimate of the state of the object at a first time step based on the point-wise measurements of the state of the object at the first time step and multiple further time steps, the multiple further time steps including time steps later than the first time step.

11. The computer-implemented method of claim 10 , wherein the processing with the benchmark model comprises processing the point-wise measurements of the state at the first time step and the plurality of further time steps through a smoother.

12. 11. The computer-implemented method of claim 10, wherein the processing with the benchmark model includes associating a first instance of the object detected at the first time step with each further instance of the object detected at the plurality of further time steps.

13. Associating the first instance of the object detected at the first time step with the respective further instances of the object detected at the plurality of further time steps comprises: determining a plurality of candidate track configurations comprising candidate groupings of object instances detected at the plurality of time steps; determining a most likely track configuration from among the plurality of candidate track configurations; associating the first instance of the object with each of the further instances of the object in accordance with the determined most likely track configuration; 13. The computer-implemented method of claim 12, comprising:

14. determining a respective likelihood value for each of the plurality of candidate track configurations; 14. The computer-implemented method of claim 13, wherein determining the most likely track configuration comprises selecting the candidate track configuration determined to have the highest likelihood value.

15. the runtime model includes one or more thresholds for controlling association of object instances detected at each different time step; The computer-implemented method of claim 11 , wherein updating the runtime model comprises determining updated values ​​for the one or more thresholds.

16. The computer-implemented method of claim 6 , wherein obtaining the point-wise data comprises receiving the point-wise data from a computer system onboard a vehicle.

17. 7. The computer-implemented method of claim 6, wherein the metric measures divergence between a probability distribution associated with the runtime estimate of the state of the object for the plurality of time steps and a probability distribution associated with the benchmark estimate of the state of the object for the plurality of time steps.

18. obtaining ground truth data indicative of ground truth values ​​of the states of the object for the plurality of time steps; 7. The computer-implemented method of claim 6, wherein the metric measures the deviation between the runtime estimate and the benchmark estimate of the state of the object by referencing the ground truth value of the state of the object.

19. one or more non-transitory computer-readable media storing instructions executable by one or more processors, The instructions, when executed, cause the one or more processors to: obtaining point-wise data indicative of point-wise measurements of states of objects detected by the object detection system for a plurality of time steps; obtaining runtime data from a runtime model indicative of runtime estimates of the state of the object for the plurality of time steps; processing the point-wise data with a benchmark model to determine benchmark estimates of the state of the object for the plurality of time steps; evaluating a metric that measures deviation between the runtime estimate and the benchmark estimate of the state of the object for the plurality of time steps; updating the runtime model based on the evaluation of the metrics; The one or more non-transitory computer-readable media causing operations to be performed, including:

Images