Localization Error Monitoring

Abstract

Localization error monitoring using vehicle state information is described. A computing system associated with the vehicle may determine a current vehicle state, such as a location, position, orientation, speed, acceleration, or the like. The vehicle computing system may determine one or more metrics associated with the vehicle state. The metrics may include a variance associated with the state, a residual associated with a measurement corresponding to the state, or a correction factor applied to correct for input errors. The computing system may determine whether the metric exceeds a threshold value. Based on the exceedance of the threshold, the computing system may determine one or more errors associated with the vehicle state. The vehicle computing system may control the vehicle based on the one or more errors.

Description

[Background technology]

[0001] Priority This application claims priority to U.S. patent application Ser. No. 16 / 809,400, entitled “LOCALIZATION ERROR MONITORING,” filed Mar. 4, 2020, the entire contents of which are incorporated herein by reference.

[0002] Localization is the process by which an entity determines its location in an environment, i.e., by localizing, an entity, such as a vehicle, can determine its exact relationship to elements in a map associated with the environment in which the entity is located. When used in robotic applications, such as autonomous vehicles, errors in localization can cause undesirable events.

[0003] The detailed description is set forth with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. Use of the same reference number in different drawings indicates similar or identical components or features. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 illustrates a block diagram of an example localization error monitoring system as described herein. [Diagram 2] FIG. 1 illustrates an example environment for determining how to control a vehicle based at least in part on using output from a vehicle localization component as described herein. [Diagram 3] 4 is an example chart of residuals associated with wheels of a vehicle determined over time as described herein. [Figure 4] FIG. 1 is a block diagram illustrating an example system for performing the techniques described herein. [Diagram 5] FIG. 2 illustrates an example process for controlling a vehicle based on a metric associated with a state of the vehicle as described herein. [Figure 6] FIG. 2 illustrates an example process for controlling a vehicle based on a residual value associated with a state of the vehicle as described herein. [Figure 7] FIG. 2 illustrates an example process for determining an optimized vehicle state as described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Techniques described herein relate to monitoring localization errors associated with an entity, such as, for example, a vehicle. A vehicle computing system of the vehicle may determine a current vehicle state. The vehicle state may include one or more of location, position, orientation, speed, acceleration, or the like. In some examples, the state may include a state of the vehicle itself. In some examples, the state may include a state of a component of the vehicle (e.g., wheel speed, wheel orientation, etc.). The vehicle computing system may determine one or more nominal errors and / or one or more unmodeled errors associated with the vehicle state. The vehicle computing system may control the vehicle based on the nominal error(s) and / or the unmodeled error(s).

[0006] The vehicle may include an autonomous or semi-autonomous vehicle that navigates through an environment utilizing sensor data collected by one or more sensors. The sensor data captured by the vehicle may include data captured by a lidar sensor, a camera sensor, a radar sensor, a time-of-flight sensor, a sonar sensor, an inertial measurement unit (IMU), and the like. In various examples, the vehicle computing system may additionally navigate through the environment based on map data associated with one or more maps. The map(s) may include feature maps (e.g., distinct features associated with the environment), tile maps (e.g., the environment represented in separate regions of data), and the like. In some examples, the vehicle computing system may utilize the map(s) described in U.S. patent application Ser. No. 16 / 022,048, entitled "Multi-Resolution Maps for Localization," filed June 28, 2018, U.S. patent application Ser. No. 16 / 022,106, entitled "Loading Multi-Resolution Maps for Localization," filed June 28, 2018, and U.S. patent application Ser. No. 16 / 147,158, entitled "Condition Dependent Parameters for Large-Scale Localization and / or Mapping," filed September 28, 2018, and U.S. patent application Ser. No. 16 / 722,598, entitled "Maps Comprising Covariances in Multi-Resolution Voxels," filed December 20, 2019, the entire contents of each of which are incorporated herein by reference.

[0007] In various examples, the vehicle computing system may include various localizers configured to receive certain sensor data as input and output the state of the vehicle. The localizers may include lidar localizers, radar localizers, visual localizers, IMU (internal measurement unit) localizers, GPS localizers, and the like, including localization based at least in part on any one or more of the sensor modalities described in detail herein. For example, a lidar localizer may receive sensor data from one or more lidar systems and / or one or more maps associated with the environment. The lidar localizer may process the sensor data and the map(s) to determine the state of the vehicle. As another example, a visual localizer may receive sensor data from one or more cameras and one or more maps. The visual localizer may process the sensor data and the map(s) to determine the state of the vehicle. As yet another example, the IMU localizer may receive sensor data from one or more IMUs and / or one or more odometry sensors (e.g., wheel speed sensors, motor speed sensors, steering angle sensors, etc.) and may determine a state of the vehicle based on the sensor data.

[0008] In existing technology, the vehicle computing system does not include a system for direct measurement of errors in the vehicle state. Instead, the existing system may monitor for general discrepancies in the vehicle's position and orientation, for example across all localization systems. Based on the determined discrepancies, the existing system may decide to abort the vehicle's operation.

[0009] Techniques described herein relate to using metrics output by multiple localizers (e.g., localization systems) and / or vehicle systems to determine one or more errors associated with each localizer and / or vehicle system. In various examples, the localizers (e.g., lidar localizers, visual localizers, IMU localizers, etc.) may output a state of the vehicle to a localization error monitoring component. In some examples, the localization component may determine a global state (e.g., combined state) of the vehicle based on state information received from one or more localizers. The localization component may include a component configured to determine a location, position, orientation, velocity, and / or acceleration of the vehicle, such as described in U.S. Patent Application Serial No. 16 / 712,753, entitled “Localization Error Handling,” filed December 12, 2019, the entire contents of which are incorporated herein by reference. In some examples, the localization component may output global state and / or status information of the vehicle to the localization error monitoring component.

[0010] In some examples, the localization error monitoring component may determine one or more metrics associated with the status information received from each localizer. The localization error monitoring component may determine one or more errors in the vehicle status based on the metrics based on the status information provided by the individual localizers and / or vehicle systems. The localization error monitoring component may provide data associated with the errors and the associated localizers and / or vehicle systems to the planner component. Thus, the planner component may determine an action to be taken by the vehicle based on the extent of the error and / or the particular localizer and / or vehicle system, such as taking the localizer and / or vehicle system offline, slowing down, stopping vehicle operation, and the like. Thus, the techniques described herein enable the vehicle to ensure highly accurate location information by monitoring the metrics and / or enable rapid and accurate error identification in the localization system based on the metrics.

[0011] The metrics may include a variance for each estimated state of the vehicle. The variance for each estimated state may include a confidence determined by each localizer that the estimated state is accurate. In some examples, the variance may be estimated by propagating expected uncertainties from the inputs to the estimates using a linearization system. In some examples, the localization error monitoring component may determine a covariance for each estimated state of the vehicle. In the above example, the localization error monitoring component may monitor the variance across two or more dimensions. For example, the covariance may include the variance across longitudinal and lateral dimensions (e.g., X / Y). As another example, the covariance may include the variance across the pitch, roll, and yaw of the vehicle. In various examples, the localization error monitoring component may compare the variance and / or the covariance to a threshold. The threshold may be determined based on a warning limit for the monitored state, which will be described in greater detail below.

[0012] In some examples, the metrics may include one or more optimization residuals (referred to herein as residuals) associated with a metric corresponding to each estimated state of the vehicle. The residual(s) may represent a difference between an observation (e.g., based on sensor inputs) and an estimate for a quantity of interest (e.g., estimated pitch, roll, yaw, location in a coordinate system and / or dimensions, velocity, etc.). In some examples, the estimate may be based in part on a value associated with a previous time step (e.g., a value associated with a measurement that took 0.1 seconds prior to the current measurement). In the above example, the residual may represent a discrepancy in the measurements. For example, the residual may include a difference between an observed speed at a time and a speed of a previous time step. Based on determining the difference, the localization error monitoring component may determine that a system associated with the speed measurement may be generating an error. In various examples, the localization error monitoring component may compare the residual to a predetermined threshold residual associated with the measurement. In the above example, based on the comparison, the localization error monitoring component may determine the extent (e.g., magnitude) of the error.

[0013] In various examples, the localization error monitoring component may additionally compare the residual to an associated variance and / or covariance. In the examples above, the measurements associated with the state may be compared to the confidence in the state. In some examples, the residual may be normalized by the variance and / or covariance (e.g., corrected to the same scale as the variance and / or covariance). In the examples above, the comparison may result in a whitened residual. In some examples, the localization error monitoring component may determine a white color in the residual. Based on a value associated with the white color in the residual, the localization error monitoring component may determine whether the residual exceeds a threshold. In some examples, the planner component may control the vehicle based on a comparison of the residual values.

[0014] In various examples, the localization error monitoring component and / or the planner component may be configured to assign weights (e.g., 1.2, 1.5, etc.) to the residual(s) associated with different localizers. In some examples, the weights may represent the importance of the localizer to the control of the vehicle. In some examples, the weights may be applied based on one or more environmental considerations (e.g., road conditions, time of day, level of darkness (e.g., light levels), etc.) and / or weather considerations (e.g., temperature, precipitation, fog, etc.). In the above examples, the localization error monitoring component and / or the planner component may be configured to dynamically adjust the weights based on changing environmental and / or weather conditions and assign different importance levels to certain localizers based on the environmental conditions. For example, during daylight hours on a sunny day, visual localizers may be weighted higher than lidar localizers. However, during dark nights, the lidar localizer may be weighted higher than the camera localizer due to reduced visual acuity of the cameras. The weights may be machine learned and / or determined using heuristics based at least in part on driving performance. Driving performance may include speed limitations due to reduced visual acuity based on reduced lighting, sun location, etc.

[0015] In some examples, the weight may increase the value of the residual compared to a threshold. In the above example, based on the importance of the associated system to the control of the vehicle, the planner component may select more conservative actuation with respect to controlling the vehicle based on the residual value. For example, based on a high residual associated with one or more measurements determined by the IMU localizer, e.g., due to the weighted residual, the planner system may decide to stop forward movement of the vehicle.

[0016] In some examples, the localization error monitoring component may identify a high residual associated with a particular localizer and / or a particular vehicle system (e.g., a localization associated with measurement data from one or more wheels, such as a wheel encoder) and may compare the high residual to another residual associated with the same or similar type of localizer and / or the same or similar type of vehicle system (e.g., a redundant localizer, sensor, and / or vehicle component or system). In the above example, the localization error monitoring component may be configured to determine whether the particular localizer and / or the particular system is faulty and / or experiencing a temporary glitch, or whether a sensor input to the particular localizer and / or the particular system is incorrect (e.g., an uncalibrated sensor). For example, a four-wheeled vehicle may include four-wheel speed measurements. The localization error monitoring component may determine that a first wheel speed measurement includes a high residual. The localization error monitoring component may compare a single wheel to three other wheels and determine that a first wheel contains an erroneous measurement, for example due to wheel slip. The localization error monitoring component and / or the planner component may decide to ignore the erroneous measurement for a particular time step. In a similar example, the system may determine that one wheel has a lower pressure than required (causing a discrepancy in the predicted state).

[0017] In some examples, the localization error monitoring component and / or the planner component may monitor the residuals over a period of time. In the above example, based on a determination that a particular localizer and / or a particular system is outputting measurements with high residuals over a threshold period of time, the localization error monitoring component and / or the planner component may determine that the particular localizer and / or the particular system is out of calibration and / or is to be ignored for vehicle control considerations. In some examples, the planner component may determine an action for the vehicle to take based on the particular localizer and / or the particular system and some number of other redundant localizers and / or systems. For example, an IMU localizer may receive data from four different IMU sensors. The IMU localizer may associate four residuals with it for measurements associated with the different IMU sensors. Based on a determination that a first sensor of the four sensors has a residual that is above a threshold value for more than a threshold amount of time, the planner component may ignore the IMU for vehicle control considerations.

[0018] In some examples, the planner component may determine to ignore a sensor input based on a number of redundant systems available and / or determine an action for the vehicle to take based on measurements associated therewith. In some examples, based on a determination that a threshold number of systems are available, the planner component may continue operation of the vehicle. In some examples, based on a determination of an error associated with a system and an insufficient number of alternative systems available, the planner component may determine to abort operation of the vehicle. In the above examples, the planner component may pull the vehicle to one side and / or stop the vehicle to ensure maximum safe operation of the vehicle.

[0019] In some examples, the planner system may maximize safe operation by determining that one or more correction factors associated with the vehicle system are within limits. In the above example, the metric may include a correction factor. The correction factor may include IMU bias, wheel diameter scale factor, and the like. The correction factor may be used to correct for input errors known to follow a predefined model. In some examples, the correction factor may have an associated maximum and / or minimum threshold value. In some examples, the localization error monitoring component may determine whether the applied correction factor exceeds a threshold value (e.g., above a maximum value, below a minimum value). For example, the radius of the wheels of a vehicle may change (e.g., due to wear, pressure / temperature changes, etc.) over the time period of operation of the vehicle. Changes to the radius may not be calibrated prior to operation of the vehicle because the exact temperature, road conditions, etc. are not known prior to release. Wheel scale error may include a determination of how fast the wheels are rotating compared to how fast they should be rotating. The correction factor may account for wheel radius and odometry, e.g., to correct for slight changes in radius and resulting slight changes in wheel speed. Based on a determination that the correction factor exceeds a maximum or minimum threshold, e.g., the wheel radius increases or decreases by more than a threshold amount (e.g., a difference of 0.5-1.0%), the localization error monitoring component may identify an error with respect to the respective wheel.

[0020] In some examples, the localization error monitoring component may identify an error associated with one or more metrics (e.g., variance, residual, correction factor) and may provide the information to the planner component. In the above examples, the error may be based on the associated variance, residual, and / or correction factor being outside of an acceptable error range (e.g., exceeding a threshold). The planner component may determine an action to take based on the identified error. The action may include taking the associated system offline (e.g., ignored for vehicle control action), stopping the vehicle, slowing down to a predetermined speed (e.g., 25 miles per hour, 25 kilometers per hour, etc.) and / or slowing down a predetermined amount (e.g., slowing down to 50% of speed, reducing forward speed by 10 miles per hour, etc.), refraining from certain actions (e.g., limiting speed, limiting turning radius, limiting acceleration, etc.), although any other maneuvers are contemplated. In some examples, the vehicle control action may be determined based on the decision to take the system offline. For example, based on the decision to take the first of three IMUs offline, the planner component may reduce the vehicle's speed to a maximum of 20 kilometers per hour, although this is merely an example and any other systems and / or speeds are contemplated herein.

[0021] In some examples, the actuation may be based on the localizer and / or vehicle system associated with the error. For example, an error associated with an IMU localizer may result in the planner component aborting the operation (e.g., stopping the vehicle). In some examples, the localizer and / or vehicle system may be ranked in a hierarchy, such as based on their importance to vehicle control. In some examples, the hierarchy may be a predetermined hierarchy. In some examples, the hierarchy may be dynamically determined, such as based on environmental considerations. For example, on a rainy day, the planner component may determine to rank errors associated with wheel slip lower than errors associated with other systems (e.g., based on correction factors and / or residuals) based on an increased tendency for wheel slip on wet surfaces. In the above example, the planner component may be configured to ignore temporary wheel slip on a rainy day.

[0022] The techniques described herein provide various technical and / or operational benefits to vehicles, such as, for example, autonomous vehicles or other robotic platforms. For example, by utilizing metrics from individual localizers to determine associated errors, the techniques described herein provide a more sophisticated "check" on localization than is provided by current techniques. That is, the localization error monitoring component provides the planner component with context associated with the localization and identified errors that is not used in current techniques. The context provided to the planner component may enable the planner component to determine whether systems associated with the vehicle (e.g., localizers, sensors, etc.) need to be released and / or whether the vehicle needs to slow down or stop operating (e.g., pull to the side of a highway or traffic congestion area). That is, by leveraging the context made available by accessing the output of the individual localizers, the planner component can continue to control the vehicle (e.g., drive autonomously) even if the localization errors have not yet been resolved (e.g., the metrics include acceptable values).

[0023] In addition to providing more sophisticated "checks," the techniques described herein enable the planner component to more safely handle errors in localization, i.e., by leveraging context made available by accessing the output of a localizer and / or other systems, and by allowing the vehicle to continue driving instead of pulling to the side of a highway or congested traffic area, the techniques described herein enable autonomous vehicles to drive more safely within an environment.

[0024] The techniques described herein can be implemented in a number of ways. Exemplary implementations are provided below with reference to the following drawings. Exemplary implementations are described in the context of an autonomous vehicle; however, the methods, apparatus, and systems described herein can be applied to a variety of systems (e.g., sensor systems or robotic platforms) and are not limited to autonomous vehicles. In addition, the techniques described herein can be used with real data (e.g., captured using sensor(s)), simulated data (e.g., generated by a simulator), or any combination of the two.

[0025] 1 illustrates a block diagram of an exemplary vehicle localization error monitoring system 100 according to an example of the disclosure. In at least one example, the localization error monitoring system 100 may include a vehicle 102, such as an autonomous vehicle, a semi-autonomous vehicle, or a manually controlled vehicle.

[0026] In at least one example, the vehicle 102 may include an autonomous vehicle configured to operate (e.g., drive) within an environment. The autonomous vehicle may be configured to operate in accordance with a Level 5 classification issued by the U.S. National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions throughout its journey, where a driver (or passenger) is not expected to control the vehicle at any time. In the above example, the autonomous vehicle may be configured to control all functions from start to stop, including all parking functions, so that it may not be occupied. This is merely an example, and the systems and methods described herein may be incorporated into any vehicle, including vehicles that must be manually controlled by a driver at all times, to vehicles that are partially or fully autonomously controlled.

[0027] The vehicle 102 may include a localization component 104. The localization component 104 may include functionality to receive sensor data from one or more sensors 106 and map data associated with one or more maps 108 to determine a state of the vehicle 102 (e.g., one or more of x-position, y-position, z-position, x-, y-, z-velocity and / or acceleration, roll, pitch or yaw, roll, pitch or yaw velocity and / or acceleration, etc.). In at least one example, the state of the vehicle 102 may be represented by degrees of freedom (e.g., 12 degrees of freedom, 15 degrees of freedom). The sensors 106 used to determine the state may include, but are not limited to, lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, location sensors (e.g., global positioning system (GPS), compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, lateral speed sensors (e.g., sensors that determine lateral speed based on ground tracking), etc.), cameras (e.g., red-green-blue (RGB), infrared (IR), intensity, depth, etc.), wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), time-of-flight (ToF) sensors, etc. In some examples, the localization component 104 may determine the state of the vehicle 102 based on one or more odometry sensors 110. The odometry sensors may include wheel speed sensors, motor speed sensors, steering angle sensors, and the like.

[0028] In at least one example, the localization component 104 may include and / or request / receive map data associated with map(s) 108 of the environment and may continuously determine the location and / or orientation (e.g., state) of the vehicle within the map(s) 108. The map(s) 108 may be any number of data structures capable of providing information about the environment, such as, for example, but not limited to, topology (e.g., junctions, lanes, merge zones, etc.), streets, mountains, roads, terrain, and the general environment. The map(s) 108 may be associated with a real environment or a simulated environment. In some examples, the map(s) 108 may include tile maps, feature maps, and / or other types of maps, such as those described in the U.S. patent applications discussed above. In some examples, the map(s) 108 may include data regarding drivable surfaces on which the vehicle drives in the environment. In some examples, the data may include road pitch (e.g., grade), road incline, surface type, surface friction, and / or other information associated with the drivable surface on which the vehicle 102 may travel.

[0029] In various examples, the localization component 104 may include an IMU 112 and / or one or more localizers configured to determine a state of the vehicle 102. In the above examples, the IMU 112 and / or each localizer may process sensor data from one or more of the sensors 106 and / or 110 and / or map data from the map(s) 108 to determine at least a portion of the state of the vehicle 102 (or to determine intermediate variables used to determine a state or partial state of the vehicle 102 in other situations). At least a portion of the state of the vehicle 102 may be described herein as pseudo measurements that may be used to determine an overall state (e.g., global state) of the vehicle 102. In an illustrative example, the localization component 104 may include an IMU 112, a lidar localizer 114, a radar localizer 116, a visual localizer 118, and a GPS localizer 120. However, the foregoing is not intended to be limiting, and additional localizers may be contemplated herein. For example, the lidar localizer 114 may receive lidar data from multiple lidar sensors 106 and map data associated with one or more maps 108. Based on the lidar data and map data, the lidar localizer 114 may output an estimated location for the vehicle 102 as part of the state. As another example, the IMU 112 may receive IMU data from one or more IMU sensors 106 and odometry data from one or more odometry sensors 110. The IMU 112 may determine an orientation of the vehicle 102 as part of the state based on sensor data associated with the IMU sensors 106 and the odometry sensor(s) 110.

[0030] In various examples, the localization component 104 may be configured to combine (e.g., fuse) state data from the IMUs 112, and / or 114, 116, 118, and / or 120 to determine a global state (e.g., state, overall condition, etc.) of the vehicle. In some examples, the localization component 104 may fuse measurements from the IMUs 112, and / or localizers 114, 116, 118, and / or 120 into an optimized state of the vehicle 102. In the above examples, the localization component 104 may optimize state measurements associated with the IMUs 112, and / or each of the localizers 114, 116, 118, and / or 120. In at least one example, the IMU 112 and / or at least one of the localizers 114, 116, 118, and / or 120, and / or the localization component 104 may determine an optimized state of the vehicle 102 utilizing simultaneous localization and mapping (SLAM), calibration, localization and mapping, simultaneously (CLAMS), relative SLAM, bundle adjustment, nonlinear least squares optimization, a Bayesian filter (e.g., a Kalman filter), or the like, to receive respective data (e.g., image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, etc.) to accurately determine the location (and / or state) of the vehicle 102.

[0031] In various examples, the localization error monitoring component 124 of the planner component 122 may receive state information from the IMU 112, and / or the localizers 114, 116, 118, and / or 120, and / or global state information (collectively state information) from the localization component 104. The state information may include a current location, position, orientation, velocity, and / or acceleration associated with the vehicle. In some examples, the state information may include one or more metrics associated with the state and / or associated metrics. In at least one example, the localization error monitoring component 124 may be configured to determine an error associated with the state information provided by a particular localizer of the localization component 104. In some examples, the metrics may include a variance (and / or covariance) associated with the state, a residual associated with a measurement corresponding to the state, and / or a correction factor applied to the vehicle system.

[0032] The localization error monitoring component 124 illustrated in FIG. 1 may include a variance component 126. The variance component 126 may be configured to receive state information (e.g., global state) output by the localization component 104 and determine one or more variance and / or covariance values ​​128 associated with the output. The variance value 128 for each estimated state may include a confidence determined by the respective localizer that the estimated state is accurate. In some examples, the variance value 128 may be estimated by propagating expected uncertainties from the inputs to the estimates using a linearization system. In some examples, the variance value 128 may include a covariance associated with the estimated state of the vehicle 102. In the above examples, the variance component 126 may determine the variance value 128 associated with two or more dimensions. For example, the variance value 128 may include a variance across the pitch, roll, and yaw of the vehicle.

[0033] In various examples, the variance component 126 may include one or more variance thresholds 130. In some examples, the variance component 126 may compare the variance and / or covariance to the variance thresholds 130. The variance threshold(s) 130 may be determined based on a warning limit associated with the condition being monitored. In at least one example, the variance threshold(s) 130 may be determined utilizing the following formula:

[0034]

number

[0035] However, AL state is the warning limit for the monitoring state, and Φ -1 is the inverse normal Cumulative Distribution Function (CDF), and P is a given probability associated with the likelihood of getting an error larger than AL at a given time.

[0036] In some examples, the variance threshold(s) 130 may include a maximum variance (or covariance) allowed by the IMU 112 and / or localizers 114, 116, 118, and / or 120 and / or localization component 104. In the above examples, the variance component 126 may compare the output from each of the IMU 112 and / or localizers 114, 116, 118, and / or 120 and / or localization component 104 to determine an error associated with the respective system. In some examples, the variance threshold(s) 130 may be based on a particular IMU and / or localizer 114, 116, 118, and / or 120, one or more sensors 106 and / or 110 associated with the state determination, and / or one or more maps 108 associated with the state determination. In the above examples, the variance threshold(s) 130 may be specific to a particular localizer, sensor, and / or map associated with the determined state. For example, the dispersion component 126 may include a first dispersion threshold 130 associated with the lidar localizer 114 and a second dispersion threshold 130 associated with the visual localizer 118. The dispersion component 126 may apply the first dispersion threshold 130 to the first state information output by the lidar localizer 114 and apply the second dispersion threshold 130 to the second state information output by the visual localizer 118.

[0037] The localization error monitoring component 124 illustrated in FIG. 1 may include a residual component 132 configured to receive state information output by the IMUs and / or localizers 114, 116, 118, and / or 120 and determine one or more residual values ​​134 associated with measurements of the state information. In some examples, the residual values ​​134 may include a priori residuals. In the above examples, the residual values ​​134 may include residuals associated with predicted measurements associated with the IMUs 112 and / or localizers 114, 116, 118, and / or 120. In some examples, the residual value(s) 134 may include residuals associated with measurements captured by at least one of the IMUs 112 and / or localizers 114, 116, 118, and / or 120. The residual value(s) 134 may represent a difference between an observation (e.g., based on sensor inputs) and an estimate for a quantity of interest (e.g., estimated pitch, roll, yaw, location in a coordinate system and / or dimensions, velocity, etc.). In some examples, the estimate may be based in part on a value associated with a previous time step (e.g., a value associated with a measurement that took 0.06 seconds prior to the current measurement), e.g., propagated forward to the current time associated with the observation. In the above example, the estimate may be determined using an integral or filter associated with the time step (e.g., the associated amount of time). In some examples, the residual value(s) 134 may represent a discrepancy in the measurements. For example, the residual value 134 may include a difference between an observed velocity at a time and a velocity at a previous time.

[0038] In various examples, the residual value(s) 134 may include pre-optimization or post-optimization measurements. In various examples, the residual value(s) 134 may be associated with raw and / or processed sensor data received from the sensors 106 and / or 110. In some examples, the residual value(s) 134 may be associated with pseudo-measurements associated with a state of the vehicle 102, such as those determined based on the IMU 112 and / or the individual localizers 114, 116, 118, and / or 120. In some examples, the residual value(s) 134 may be associated with an overall state of the vehicle, such as those determined by the localization component 104.

[0039] In various examples, the residual component 132 may include one or more residual thresholds 136. In some examples, the residual threshold(s) 136 may include control data associated with the IMU 112 and / or the localizers 114, 116, 118, and / or 120, and / or the respective sensors 106 and / or 110. In some examples, the residual threshold(s) 136 may include a maximum residual allowed by the localization component 104. In the above examples, the residual threshold(s) 136 may include a maximum difference between the expected and observed measurements, regardless of the associated sensor 106 and / or 110, and / or the IMU 112 and / or the localizers 114, 116, 118, and / or 120. In some examples, the residual threshold(s) 136 may include individual maximum values ​​associated with particular measurements output by the IMU 112 and / or localizers 114, 116, 118, and / or 120. In the above examples, the residual threshold(s) 136 may be based on the particular measurements associated with the residual values ​​134. In various examples, the residual component 132 may compare the residual values ​​134 to the residual threshold(s) 136 associated with the measurements. In the above examples, based on the comparison, the localization error monitoring component may determine the extent (e.g., magnitude) of the error.

[0040] In various examples, the residual component 132 may be configured to determine an error associated with the IMU 112 and / or localizers 114, 116, 118, and / or 120 and / or sensors 106 and / or 110 based on the residual value(s) 134 and / or based on a comparison to a residual threshold(s) 136. In some examples, the residual component 132 may determine the error based on a single residual value 134 exceeding an associated residual threshold 136. In various examples, in response to determining the error, the input modality may be disregarded in vehicle planning considerations. In some examples, the input modality (e.g., sensors 106 and / or 110, the IMU 112 and / or localizers 114, 116, 118, and / or 120) may be removed from consideration in determining the overall state of the vehicle 102, such as determined by the localization component 104.

[0041] In various examples, after removing a single input modality from consideration, the residual component 132 may determine an error based on a sum of the remaining residual values ​​134 associated with a particular IMU 112 and / or a particular localizer 114, 116, 118, and / or 120 and / or a sensor 106 and / or 110. In the above examples, the residual component 132 may determine whether the sum of the remaining residual values ​​134 is greater than or equal to an overall residual threshold 136 (e.g., a threshold associated with the sum of the residuals). In some examples, based on a determination that the sum of the input residual values ​​134 exceeds the overall residual threshold 136, the residual component 132 may provide an indication of an error to the planner component 122. In some examples, the planner component 122 may determine an action to be taken by the vehicle 102 based on the error.

[0042] In various examples, the variance component 126 and / or the residual component 132 may additionally compare the residual value(s) 134 to the associated variance value(s) 128. In the examples above, the measurement associated with the state may be compared to the confidence in the state. In some examples, the residual value(s) 134 may be normalized (e.g., corrected to the same scale as the variance value(s) 128) by the variance value(s) 128. In the examples above, the comparison may result in a whitened residual. In some examples, the residual component 132 may determine a value associated with the whitened residual. Based on the value associated with the whitened residual, the residual component 132 may determine whether the residual value 134 exceeds a residual threshold 136. In some examples, the residual component 132 may identify errors in the associated IMU 112 and / or localizers 114, 116, 118, and / or 120 and / or sensors 106 and / or 110 based on a comparison of a value associated with white in the residual to a residual threshold 136. In various examples, the localization error monitoring component may provide data associated with the errors to the planner component 122, such as for control considerations of the vehicle 102.

[0043] In various examples, the residual component 132 may be configured to assign weights (e.g., 1.2, 1.5, etc.) to the residual(s) associated with different localizers. In some examples, the weights may represent the importance of the IMU 112 and / or the localizers 114, 116, 118, and / or 120 and / or the associated sensors 106 and / or 110 to the control of the vehicle. In some examples, the weights may represent the confidence in the associated systems of the vehicle 102. The weights may be determined based on testing, experimental data, empirical analysis, machine learning, and the like. In some examples, the weights may be applied based on one or more environmental considerations (e.g., road conditions, time of day, level of darkness, etc.). In the above examples, the residual component 132 may be configured to dynamically adjust the weights based on changing environmental conditions and assign different importance levels to particular localizers based on the environmental conditions. For example, during daylight hours on a sunny day, the visual localizer may be weighted higher than the lidar localizer 114. However, during dark nights, the lidar localizer 114 may be weighted higher than the visual localizer 118 due to reduced visual acuity of the cameras.

[0044] In some examples, the weights may increase the residual value 134 (e.g., actual residual times 1.3, etc.). In the above example, the residual component 132 may determine errors associated with critical systems at lower actual residual values ​​134 to maximize safe operation of the vehicle. For example, based on high residuals in one or more measurements determined by the IMU 112, such as due to the weighted residual values ​​134, the planner system may determine to stop forward movement of the vehicle.

[0045] In some examples, the residual component 132 may identify a high residual associated with a particular IMU 112 and / or a particular localizer 114, 116, 118, and / or 120 and / or a particular sensor 106 and / or 110, and may compare the high residual with another residual value 134 associated with the same or similar type of IMU and / or localizer 114, 116, 118, and / or 120 (or other vehicle components, such as wheels, etc.) and / or the same or similar type of sensor 106 and / or 110 (e.g., redundant localizers and / or sensors). In the above example, the residual component 132 may be configured to determine whether a particular IMU 112 and / or a particular localizer 114, 116, 118, and / or 120, and / or a particular sensor 106 and / or 110 is faulty and / or experiencing a temporary glitch, or whether the sensor input to a particular localizer is incorrect (e.g., an uncalibrated sensor). For example, a four-wheeled vehicle 102 may include four-wheel speed measurements. The residual component 132 may determine that a first wheel speed measurement includes a high residual value 134. The residual component 132 may compare a single wheel to three other wheels and determine that a first wheel includes an erroneous measurement, e.g., due to wheel slippage. The residual component 132 and / or the planner component 122 may determine to ignore the erroneous measurement for a particular time step.

[0046] In some examples, the residual component 132 may monitor the residual value(s) 134 over a period of time. In the above example, based on a determination that a particular IMU 112 and / or a particular localizer 114, 116, 118, and / or 120 and / or a particular sensor 106 and / or 110 is outputting measurements having high residual value(s) 134 for a threshold period of time, the residual component 132 may determine that the particular IMU 112 and / or a particular localizer 114, 116, 118, and / or 120 and / or a particular sensor 106 and / or 110 has lost calibration and / or should be ignored for vehicle control considerations. For example, the IMU 112 may receive data from four different IMU sensors 106. The IMU 112 may associate with it four residual values ​​134 corresponding to measurements associated with the different IMU sensors 106. Based on a determination that a first IMU sensor of the four IMU sensors 106 has a residual value 134 that is above a threshold 136 for more than a threshold amount of time, the residual component 132 may identify an error in the first IMU sensor.

[0047] In some examples, the planner component 122 may determine to ignore sensor 106 and / or 110 inputs and / or determine the vehicle action to take based on measurements associated therewith based on a number of redundant sensors 106 and / or 110 available. In some examples, based on a determination that a threshold number of sensors 106 and / or 110 are available, the planner component 122 may continue vehicle operation. In some examples, based on a determination of errors associated with sensors 106 and / or 110 and an insufficient number of alternative systems available, the planner component 122 may determine to abort vehicle 102 operation. In the above examples, the planner component 122 may pull over and / or stop the vehicle 102 to ensure maximum safe operation of the vehicle 102.

[0048] In some examples, the localization error monitoring system 100 may maximize safe operation by determining that one or more correction factors associated with vehicle systems (e.g., sensors 106 and / or 110, wheels, etc.) are within limits. In the above example, the localization error monitoring component 124 may include a correction factor component 138 configured to determine one or more correction values ​​140 to be applied to one or more vehicle systems. The correction values ​​140 may include IMU bias, map distortion factor, scale factor of wheel diameter scale, and the like. The correction values ​​140 may include values ​​used to correct for input errors known to follow a predefined model. The correction factors 140 may apply relevant measurements (e.g., measurements associated with relevant components (e.g., IMU, wheel odometry sensors, etc.)). In some examples, the correction values ​​140 may have associated maximum and / or minimum correction thresholds 142. In some examples, the correction factor component 138 may determine whether the applied correction value 140 exceeds a correction threshold 142 (e.g., above a maximum value, below a minimum value). For example, the radius of the wheels of the vehicle 102 may change over the time period of operation of the vehicle. Changes to the radius may not be calibrated prior to operation of the vehicle because the exact temperature, road conditions, etc. are not known prior to the release. Wheel scale error may include a determination of how fast the wheels are rotating compared to how fast they should be rotating. The correction value 140 may account for wheel radius and odometry, for example, to compensate for slight changes in radius and resulting slight changes in wheel speed.Based on a determination that the correction value 140 exceeds a correction threshold 142, such as by the radius of the wheel increasing or decreasing by more than a threshold amount (e.g., a difference of 0.5-1.0%), the correction factor component 138 may identify an error with respect to the respective wheel.

[0049] In some examples, the localization error monitoring component 124 may identify errors associated with the IMU 112 and / or one or more localizers 114, 116, 118, and / or 120 and / or one or more sensors 106 and / or 110 based on the metrics evaluated by the variance component 126, the residual component 132, and / or the correction factor component 138. In some examples, the localization error monitoring component 124 may provide error data 144 associated with the errors to the planner component 122. In some examples, the error data 144 may include information corresponding to the IMU 112 and / or localizer(s) 114, 116, 118, and / or 120 and / or sensor(s) 106 and / or 110, the variance value(s) 128, the residual value(s) 134, and / or the correction value(s) 140 associated with the error, the amount by which the aforementioned value(s) 128, 134, and 140 exceed their respective thresholds 130, 136, and / or 142, the number and / or type of redundant systems, weighting values ​​(e.g., importance) of the IMU 112 and / or particular localizer(s) 114, 116, 118, and / or 120 and / or sensor(s) 106 and / or 110, and / or any other data consumed to inform the actuation decisions made by the vehicle.

[0050] The planner component 122 may receive the error data 144 and determine an action for the vehicle 102 to take based on the identified error. The action may include taking the associated IMU 112 and / or localizer(s) 114, 116, 118, and / or 120 and / or sensor(s) 106 and / or 110 offline (e.g., ignored for vehicle control actions), stopping the vehicle 102, slowing down to a predetermined speed and / or slowing down a predetermined amount (e.g., slowing down to 50% of speed, reducing forward speed by 10 miles per hour, etc.), refraining from certain actions (e.g., limiting speed, limiting turning radius, limiting acceleration, etc.), although any other maneuvers are contemplated. In some examples, vehicle control actions may be determined based on a decision to take the IMU 112 and / or localizer(s) 114, 116, 118, and / or 120 and / or sensor(s) 106 and / or 110 offline. For example, based on a decision to take one of the eight lidar sensors offline, the planner component 122 may slow the vehicle speed down to a maximum of 25 miles per hour, although these are merely examples and any other systems and / or speeds are contemplated herein.

[0051] In some examples, actuation may be based on the IMU 112 and / or localizer(s) 114, 116, 118, and / or 120 and / or sensor(s) 106 and / or 110 associated with an error. For example, an error associated with the IMU 112 may result in the planner component 122 aborting the operation (e.g., stopping the vehicle 102). In some examples, the IMU 112 and / or localizer(s) 114, 116, 118, and / or 120 and / or sensor(s) 106 and / or 110 may be positioned in a hierarchy, such as based on importance to vehicle control. In some examples, the hierarchy may be a predetermined hierarchy. In some examples, the hierarchy may be dynamically determined, such as based on environmental considerations. For example, on a rainy day, the planner component 122 may determine to rank errors associated with wheel slip lower than errors associated with other systems (e.g., based on the correction values ​​140 and / or residual values ​​134) based on an increased tendency for wheel slip on wet surfaces. In the above example, the planner component 122 may be configured to ignore temporary wheel slip on a rainy day.

[0052] In various examples, the planner component 122 may be configured to send the error data 144 to a remote computing device 146, such as, for example, via one or more networks 148 (e.g., wired or wireless networks). In some examples, the remote computing device 146 may include a computing device having at least one processor 150 and a memory 152. In some examples, the remote computing device 146 may be associated with a monitoring system and / or a maintenance system that monitors the status of the vehicle 102 and / or various components of the vehicle 102 and / or provides maintenance to various components of the vehicle 102. Thus, the error data 144 may provide the remote computing device 146 and / or a person using the remote computing device 146 with a comprehensive package of information related to the status of the various vehicle components, including information about the errors in the error data 144 and the associated vehicle 102 systems (e.g., the IMU 112 and / or the localizers 114, 116, 118, and / or 120 and / or the sensors 106 and / or 110).

[0053] FIG. 2 illustrates an example environment 200 for determining how to control a vehicle, such as vehicle 102, based at least in part on using output from a localization component, such as localization component 104, of the vehicle described herein.

[0054] In at least one example, the vehicle may include a planner component 202 configured to receive sensor data 204 from a sensor system(s) associated with the vehicle, such as, for example, sensors 106 and / or 110, and / or map data 206 associated with an environment in which the vehicle is located. In some examples, the planner component 202 may be configured to control the vehicle based at least in part on any decisions (e.g., localization, perception, prediction, etc.) made by the sensor data 204, the map data 206, and / or other components associated with the vehicle. In at least one example, an output of the planner component 202 may include one or more instructions for running or otherwise controlling the vehicle within the environment. In some examples, the one or more instructions may include a velocity modality, an acceleration modality, a steering angle modality, a steering modality, etc., for controlling the vehicle for a time period (e.g., a defined number of seconds into the future). That is, in at least one example, the output may include a trajectory and / or a route that the vehicle will drive for a time period.

[0055] In various examples, the planner component 202 may receive a signal from a localization component illustrated at operation 208. The localization component may include one or more localizers, such as one or more IMU localizers, lidar localizers, radar localizers, visual localizers, GPS localizers, and the like. In some examples, the localization component may include a global localizer. In the above example, the global localizer may be configured to combine localization data from the localizer(s) to determine a global state (e.g., location, position, orientation, velocity, acceleration, etc.) of the vehicle at a given time. In some examples, the signal may include global state information associated with portions of the vehicle's state (e.g., as determined by the IMU and / or one or more localizers), pseudo measurements including raw sensor data and / or processed sensor data. The state information may include one or more of x-position, y-position, z-position, x-velocity and / or acceleration, y-velocity and / or acceleration, z-velocity and / or acceleration, roll, pitch, or yaw, roll velocity and / or acceleration, pitch velocity and / or acceleration, or yaw velocity and / or acceleration, and the like. In at least one example, the signal may include an optimized state of the vehicle at a given time.

[0056] In addition, the signal may include one or more metrics associated with the state of the vehicle. The metric(s) may be associated with an individual dimension (e.g., a metric associated with X-position, roll, etc.) and / or across two or more dimensions (e.g., a metric associated with XY velocity, etc.). In at least one example, the metric(s) include a variance associated with the state of the vehicle, a residual associated with a measurement corresponding to the state, and / or a correction factor value applied to correct for input errors associated with the system (e.g., wheel scale error, IMU bias, etc.). Variance as used herein may include a covariance associated with the signal.

[0057] In operation 210, the planner component 202 may determine a variance associated with the signal. The variance may include a confidence determined by the localizer component that the estimated state is accurate. In some examples, the variance may be determined based on a linearization system. In some examples, the variance may include a covariance for each estimated state of the vehicle. In the above examples, the variance may include a confidence of the estimated state across two or more dimensions. For example, the covariance may include a variance across longitudinal and lateral dimensions (e.g., X / Y). As another example, the covariance may include a variance across pitch, roll, and yaw of the vehicle.

[0058] In operation 212, the planner component 202 may determine a residual associated with the signal. The residual may represent a difference between an observation (e.g., based on sensor inputs) and an estimate for a quantity of interest (e.g., estimated pitch, roll, yaw, location in a coordinate system and / or dimensions, velocity, etc.). In some examples, the estimate may be based in part on a value associated with a previous time step (e.g., a value associated with a measurement that took 0.1 seconds prior to the current measurement), such as by propagating the value associated with the previous time step forward to the current time. In the above example, the residual may represent a discrepancy in the measurements. For example, the residual may include a difference between an observed velocity at a time and a velocity of a previous time step. Based on determining the difference, the planner component may determine that a system associated with the velocity measurement may be generating an error. In various examples, the planner component may compare the residual to a predetermined threshold residual associated with the measurement. In the above example, based on the comparison, the planner component may determine the extent (e.g., magnitude) of the error.

[0059] In various examples, the planner component may additionally compare the residuals to the associated variance determined in operation 210. In the examples above, a measurement associated with a state may be compared to a confidence in the state. In some examples, the residuals may be normalized by the variance (and / or covariance). In the examples above, the comparison may result in a whitened residual. In some examples, the planner component may determine a white color in the residuals. In the examples above, the white color in the residuals may be compared to a threshold value described below, such as in operation 216.

[0060] In operation 214, the planner component 202 may determine values ​​associated with correction factors for the signal. The correction factors may include IMU bias, map distortion rate, scale factor of wheel diameter scale, and the like. The correction factors may be used to correct for input errors. In some examples, the correction factors may be applied to input errors that are known to follow a predefined model.

[0061] In operation 216, the planner component 202 may determine whether a value associated with the variance, the residual, or the correction factor is greater than or equal to a threshold value. In some examples, the threshold value may include one or more variance thresholds, residual thresholds, and / or correction thresholds. In some examples, the respective threshold values ​​may be determined based on a localizer associated with the variance, residual, and / or correction threshold. In some examples, the respective threshold values ​​may be determined based on a sensor associated with the variance, residual, and / or correction threshold.

[0062] In some examples, the threshold may include a threshold time period (e.g., a maximum time, 1 second, 0.5 seconds, etc.) during which a value associated with the residual and / or correction factor may be above the threshold. In the above examples, the determination that the residual and / or correction factor exceeds the threshold may be based on a determination that the residual and / or correction factor remained high for a threshold time period or more. For example, a first correction factor associated with a first wheel speed may exceed the value of the threshold correction factor at a first time. At a second time less than the threshold time period after the first time, a second correction factor associated with the second wheel speed may be less than the threshold. The planner component 202 may determine that the error was temporary (e.g., a temporary glitch caused by wheel slippage, or the like) and that the value did not exceed the threshold. However, at a third time, a third correction factor associated with a third wheel speed may exceed the threshold correction factor, and at a fourth time greater than the threshold time period from the third time, a fourth correction factor associated with a fourth wheel speed may also exceed the threshold correction factor. Based on exceeding the threshold time period, the planner component 202 may determine that the correction factor value exceeds a threshold.

[0063] Based on a determination that at least one of the variance, residual, or value does not exceed the threshold ("no" at operation 216), the planner component 202 may control the vehicle based on the signals at operation 218. In some examples, the planner component 202 may determine an initial location of the vehicle based on the signals. In the above example, the planner component 202 may determine a trajectory for the vehicle to follow through the environment based on the initial location. In various examples, the planner component 202 may control the vehicle based on the signals utilizing techniques described in U.S. Patent Application Serial No. 16 / 517,506, entitled "Unstructured Vehicle Path Planner," filed July 19, 2019, and U.S. Patent Application Serial No. 16 / 539,928, entitled "Cost-Based Path Determination," filed August 13, 2019, the entire contents of both applications are incorporated herein by reference.

[0064] Based on a determination that at least one of the variance, residual, or value exceeds a threshold ("yes" at operation 216), the planner component 202 may cause the vehicle to take an alternative action at operation 220. The alternative action may include slowing the vehicle to a predetermined speed (e.g., 17 miles per hour, 25 kilometers per hour, etc.), slowing the vehicle to a fraction of its previous speed (e.g., reducing to 60% of its previous speed), or stopping the vehicle, refraining from certain actions (e.g., limiting speed, limiting turning radius, limiting acceleration, etc.), although any other maneuvers are contemplated. For example, based on a determination that the variance exceeds a threshold, the planner component may decide to stop the vehicle (e.g., pull over to the side of the road).

[0065] In some examples, the alternative action may include comparing data associated with different sensors of the same or similar type. For example, the planner component 202 may determine that the residual associated with the first IMU exceeds a threshold value, but the residual associated with the second and third IMUs does not exceed a threshold value. Based at least in part on the comparison, the planner component may determine to slow the forward movement of the vehicle to a reduced speed (e.g., a predetermined or percentage of the current speed).

[0066] In some examples, the alternative action may include ignoring sensors and / or localizers associated with the threshold exceedance. In some examples, the alternative action may include ignoring a particular degree of freedom (e.g., values ​​in a particular dimension) associated with the exceedance. In the above example, the planner component 202 may determine that a particular degree of freedom is not relevant to control of the vehicle and may remove the associated value (e.g., variance, residual, etc.) from vehicle control considerations. For example, the planner component 202 may determine that a residual associated with the z-height of the vehicle exceeds a threshold value. Based on a determination that the z-height is not relevant to a vehicle traveling on a substantially flat road (e.g., <3% grade, etc.), the planner component 202 may determine to ignore the z-height and associated residual from vehicle control planning considerations. As another example, the planner component 202 may determine that a residual associated with the lateral speed of the vehicle exceeds a threshold value. Based on a determination that the lateral velocity is associated with the vehicle, the planner component 202 may determine to slow the forward speed of the vehicle until the error is resolved.

[0067] In some examples, the alternative action may include sending data associated with the error to a remote computing device, such as, for example, remote computing device 146. In some examples, the remote computing device may be configured to evaluate the error and / or perform diagnostic testing and / or maintenance on one or more vehicle systems. In some examples, the planner component 202 may additionally send fault data to the remote computing device, such as described, for example, in U.S. Patent Application Serial No. 16 / 712,753, which is incorporated by reference herein above. In the above examples, based on the error and / or failure data provided by the planner component 202 (e.g., a vehicle computing system of the vehicle), the remote computing device may identify the particular sensor and / or localizer providing the erroneous data.

[0068] 3 is an example chart 300 of residuals (e.g., residuals 312, 316, etc.) associated with wheels of a vehicle 302 determined over time as described herein. In an illustrative example, the vehicle 302 may include four wheels, a first wheel 304, a second wheel 306, a third wheel 308, and a fourth wheel 310. What has just been stated is not intended to be limiting and the vehicle may include a greater or lesser number of wheels.

[0069] In some examples, a localization error monitoring component, such as a planner component, e.g., planner component 122, or the like, such as localization error monitoring component 124, may be configured to determine one or more residuals associated with the measurements based on the sensor data. In an illustrative example, the measurements may include wheel speed. In other examples, the measurements may include other values ​​associated with state information (e.g., x-position, y-position, z-position, x-velocity, y-velocity, z-velocity, pitch, roll, yaw, associated progress, acceleration, and the like) in any dimension and / or degree of freedom and / or across two or more dimensions or degrees of freedom. The state information may include information associated with the vehicle and / or components of the vehicle (e.g., wheels), as an illustrative example.

[0070] In various examples, the localization error monitoring component may evaluate the residual(s) over time. In the above examples, the error evaluation error for the residual may be based on temporal statistics associated with the residual. In various examples, the temporal statistics may enable the localization error monitoring component to determine whether a particular residual at a particular time was an exception, e.g., based on wheel slippage, or whether an associated sensor and / or system is consistently generating a high residual indicative of a true error in the sensor and / or system. In some examples, the residual may be determined based in part on a comparison of the current wheel speed with the wheel speed of a previous time step, e.g., by integrating the wheel speed of a previous time step or by propagating forward early to an estimated wheel speed for the current time.

[0071] In the illustrative example, the second wheel 306 is rotated at time t 2 may include a residual 312, which is related to the measurement of wheel speed. 1For example, time t 2 In some instances, the correction may be due to slight wheel slippage due to water on the road, hitting a pothole, or the like.

[0072] The residual 312 illustrated in FIG. 3 may include a value less than a residual threshold 314. In various examples, the residual threshold 314 may be determined based on a particular sensor and / or component associated with the measurement. In some examples, the residual threshold 314 may be determined based on a localizer providing the measurement. In various examples, the residual threshold 314 may be determined based on environmental considerations (e.g., road conditions, weather conditions, time of day, level of darkness, etc.). For example, the localization error monitoring component may include a first residual threshold 314 for dry day use, e.g., accounting for increased wheel slip due to wet surfaces, and a second, higher residual threshold 314 for rainy day application. Based at least in part on a determination that the residual 312 is less than the residual threshold 314, the localization error monitoring component may determine that no error exists in the associated sensor and / or associated component.

[0073] In the illustrative example, time t 3Accordingly, the localization error monitoring component may identify a residual 316 corresponding to measurements associated with the first wheel 304 and the third wheel 308. In some examples, the localization error monitoring component may determine that the residual 316 is above a residual threshold 314. In some examples, based on a determination that the residual is above the residual threshold 314, the localization error monitoring component may provide error data to a planner component of the vehicle computing system, such as, for example, alerting the vehicle computing system to a possible error associated with a sensor (e.g., a wheel speed sensor) and / or component (e.g., a wheel).

[0074] In some examples, the first wheel 304 and the third wheel 308 (e.g., parallel wheels) rotate at the same time t 3 Based at least in part on the determination that parallel wheels, such as first wheel 304 and third wheel 308, include the same residual 316 at the same time, the localization error monitoring component may determine that the error associated with the residual 316 exceeding the threshold 314 was a temporary glitch.

[0075] In some examples, the localization error monitoring component may decide to delay providing error data to the planner component. In some examples, the delay may be a time (e.g., t 4), for example, if the residual 316 associated with the first wheel and the third wheel 308 remains above a threshold at time t4, the localization error monitoring component may provide the associated error data to the planner component. The localization error monitoring component, which may be described below, may report the error data based on a determination that the time associated with the residual 316 being above the residual threshold 314 is above a threshold time, such as threshold time 318.

[0076] In the illustrative example, the second wheel 306 and the fourth wheel 310 are rotated at time t 4 The second wheel 306 and the fourth wheel 310 illustrated on the vehicle 302 may include parallel, rear wheels. In some examples, the vehicle 302 may be bi-directional. In the above example, the second wheel 306 and the fourth wheel 310 may be configured to operate as front wheels. The measurements associated with the second wheel 306 and the fourth wheel 310 may be calculated at time t 3 Based at least in part on a determination that the first wheel 304 and the third wheel 308 have substantially the same residuals associated with them at time t as the residuals 316 associated with them at time t, the localization error monitoring component may determine that the error (e.g., the residuals 316 exceed the residual threshold 314) is due to an object encountered on the drivable surface. For example, the front wheels (e.g., the first wheel 304 and the third wheel 308) of the vehicle 302 have substantially the same residuals associated with them at time t 3 At time t 4 At t , the rear wheels (second wheel 306 and fourth wheel 310) of the vehicle 302 may go over the railroad track, causing a resulting residual 316 in the measurements associated with the second wheel 306 and the fourth wheel 310. Substantially the same (e.g., within a threshold difference, <3% difference, etc.) residual 316 may be detected at subsequent times (e.g., t 3and t 4 ) associated with the front and rear wheels, the localization error monitoring component may determine that the error was temporary and / or based on an external condition or event, and should not be reported to the planner component.

[0077] The localization error monitoring component illustrated in FIG. 3 monitors the localization error at time t 8 In some examples, the localization error monitoring component may determine a residual 320 corresponding to a measurement associated with the third wheel 308 at. In some examples, the localization error monitoring component may determine that the residual 320 is above the residual threshold 314. In some examples, the localization error monitoring component may provide error data to the planner component based on a determination that the residual 320 is above the residual threshold 314.

[0078] In some examples, the localization error monitoring component may compare the residual 320 to residuals associated with the first wheel 304, the second wheel 306, and the fourth wheel 310. In some examples, based on the comparison, the localization error monitoring component may determine whether the third wheel 308 is individually experiencing an error, or whether further wheels 304, 306, and / or 310 are also experiencing an error (e.g., as described above in the railroad example). In some examples, based on a determination that the third wheel 308 includes an individual high residual (e.g., error), the localization error monitoring component may send an indication to the planner component that the third wheel 308 should be ignored from vehicle control considerations.

[0079] At subsequent times (e.g., t 9Based on a determination that the measurement corresponding to the third wheel 308 does not include the residual error 320 at time t , the localization error monitoring component may determine that the error was temporary. In some examples, the localization error monitoring component may provide an indication that the error is no longer present and / or that a subsequent measurement associated with the third wheel 308 (e.g., at time t 9 , t 10 ) may be included in vehicle control considerations.

[0080] In an illustrative example, the localization error monitoring component may determine a residual 322 associated with the first wheel 304, the second wheel 306, the third wheel 308, and the fourth wheel 310. The illustrated residuals 322 may include values ​​that are equal to or greater than the residual threshold 314. In various examples, the localization error monitoring component may determine a residual 322 associated with the first wheel 304, the second wheel 306, the third wheel 308, and the fourth wheel 310. 11 The vehicle may determine errors related to the four wheels 304, 306, 308, and 310 and may send associated error data to the planner component. In some examples, the planner component may determine a first action to take, such as slowing down, based on the error.

[0081] In some examples, the localization error monitoring component may determine that at a subsequent time, the measurements associated with the wheels 304, 306, 308, and 310 include substantially the same residual 322. In various examples, the localization error monitoring component may determine that the total time (Δt) associated with the residual is above the threshold time 318. Based on the determination that the total time associated with the residual 322 is above the threshold time 318, the localization error monitoring component may send updated error data to the planner component. In some examples, the planner component may determine a second action for the vehicle to take based on the updated error data. In some examples, the second action may include a conservative action, such as, for example, slowing the vehicle, stopping the vehicle, or the like. In the above example, the planner component may maximize safety during the operation of the vehicle 302. In some examples, the second action may include a more conservative action from the first action. For example, a first act may include slowing the forward speed by 5 miles per hour, and a second act may include slowing the forward speed by another 10 miles per hour.

[0082] FIG. 4 is a block diagram illustrating an example system 400 for performing the techniques described herein. In at least one example, the vehicle 402, which may correspond to the vehicle 102 of FIG. 1 or the vehicle 302 of FIG. 3, may include one or more vehicle computing devices 404, one or more sensor systems 406, one or more emitters 408, one or more communication connections 410, at least one direct connection 412, and one or more drive systems 414. The vehicle 402 described above may be an autonomous vehicle configured to operate in accordance with a Level 5 classification issued by the United States Federal Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions throughout its journey, where the driver (or passenger) is not expected to control the vehicle at any point. In the above example, the vehicle 402 may be configured to control all functions from start to stop, including all parking functions, so that it may not be occupied. What has just been described is merely an example, and the systems and methods described herein may be incorporated into any ground, air, or water vehicle, including vehicles ranging from those that must always be manually controlled by a driver to those that are partially or fully autonomously controlled. That is, in the illustrated example, the vehicle 402 is an autonomous vehicle, however, the vehicle 402 could be any other type of vehicle. Although only a single vehicle 402 is illustrated in FIG. 4, in an actual application, the example system 400 may include multiple vehicles, and in some examples, may include a fleet of vehicles.

[0083] The vehicle computing device(s) 404 may include a processor(s) 416 and a memory 418 communicatively coupled to the processor(s) 416. In the illustrated example, the memory 418 of the vehicle computing device(s) 404 stores a localization component 420 (which may correspond to the localization component 104 described above with reference to FIG. 1), a perception component 422, a prediction component 424, a planner component 426 (which may correspond to the planner component 122 of FIG. 1 and / or the planner component 202 of FIG. 2), and one or more system controllers 428. In addition, the memory 418 may include storage 430, which may store map(s), model(s), previous output, and the like. The map described above may be any number of data structures capable of providing information about the environment, such as, for example, but not limited to, topology (e.g., junctions, lanes, merge zones, etc.), streets, mountains, roads, terrain, and the general environment. The map may be associated with a real or simulated environment. The model(s) may include machine learning models, as described below. In some examples, storage 430 may store previous outputs.

[0084] In at least one example, a localization component 420, such as, for example, the localization component 104, may determine a location, one or more positions, orientation, velocity, and / or acceleration of the vehicle 402 with respect to a local map and / or a global map based at least in part on sensor data received from the sensor system(s) 406 and / or map data associated with the map (e.g., of the map(s)). In at least one example, the localization component 420 may include or be associated with a calibration system capable of performing operations to calibrate (determine various intrinsic and non-intrinsic parameters associated with any one or more of the sensor system(s) 406), localize, and map substantially simultaneously. Additional details associated with the localization component 420 are described above with reference to FIG. 1.

[0085] In at least one example, the perception component 422 may perform object detection, segmentation, and / or classification based at least in part on sensor data received from the sensor system(s) 406. In at least one example, the perception component 422 may receive raw sensor data (e.g., from the sensor system(s) 406). In at least one example, the perception component 422 may receive image data and may utilize one or more image processing algorithms to perform object detection, segmentation, and / or classification for object(s) identified in the image data. In some examples, the perception component 422 may associate a bounding box (or instance segmentation in another context) with the identified object and may associate a confidence score associated with the classification of the identified object with the identified object. In some examples, the object may be colored based on the discovered class when rendered via a display. The perception component 422 may perform similar processing for one or more other modalities.

[0086] The prediction component 424 may receive sensor data from the sensor system(s) 406, map data associated with a map (e.g., of the map(s) that may be in storage 430), and / or perception data (e.g., processed sensor data) output from the perception component 422, and may output a prediction associated with one or more objects in the environment of the vehicle 402. In at least one example, the planner component 426 may take actions, as described above, for the vehicle 402 based, at least in part, on the sensor data received from the sensor system(s) 406, on the map data, and / or on any decisions made by other components of the vehicle 402.

[0087] Additional details of suitable localization, perception, prediction, and / or planning systems may be found in U.S. Pat. No. 9,612,123, issued Apr. 4, 2017, and U.S. Pat. No. 10,353,390, issued Jul. 16, 2019, the entire contents of both of which are incorporated herein by reference. In some instances (e.g., if the vehicle 402 is not an autonomous vehicle), one or more of the aforementioned systems may be omitted from the vehicle 402. While the systems described above are illustrated as "on-board" the vehicle 402, in other implementations the systems may be remotely located and / or remotely accessible with respect to the vehicle 402. Furthermore, while the systems are described above as "systems," the above systems may include one or more components for performing operations ascribed to each system.

[0088] In at least one example, the localization component 420, the perception component 422, the prediction component 424, and / or the planner component 426 may process the sensor data as described above and send their respective outputs over the network(s) 432 to the computing device(s) 434. In at least one example, the localization component 420, the perception component 422, the prediction component 424, and / or the planner component 426 may send their respective outputs to the computing device(s) 434 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0089] In at least one example, the vehicle computing device(s) 404 may include one or more system controllers 428 that may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 402. The just mentioned system controller(s) 428 may communicate with and / or control corresponding systems of the drive system(s) 414 and / or other systems of the vehicle 402.

[0090] In at least one example, the sensor system(s) 406 may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, location sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), wheel encoders, audio sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ToF sensors, etc. The sensor system(s) 406 may provide input to the vehicle computing device(s) 404. In some examples, the sensor system(s) 406 may pre-process at least some of the sensor data before transmitting the sensor data to the vehicle computing device(s) 404. In at least one example, the sensor system(s) 406 may send sensor data over the network(s) 432 to the computing device(s) 434 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0091] Additionally, the vehicle 402 may include one or more emitters 408 to emit light and / or sound as described above. The emitter(s) 408 in the present example include interior audio and visual emitters that communicate with passengers of the vehicle 402. By way of example and not limitation, the interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. Additionally, the emitter(s) 408 in the present example may also include exterior emitters. By way of example and not by way of limitation, the exterior emitters in the just-described examples include light emitters (e.g., indicator lights, signs, light arrays, etc.) that communicate visually with pedestrians, other drivers, other nearby vehicles, etc., one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) that communicate audibly with pedestrians, other drivers, other nearby vehicles, etc. In at least one example, emitter(s) 408 may be placed in various locations around the exterior and / or interior of vehicle 402.

[0092] Additionally, the vehicle 402 may include communication connection(s) 410 enabling communication between the vehicle 402 and other local or remote computing device(s). By way of example, the communication connection(s) 410 may facilitate communication of the vehicle 402 with other local computing device(s) and / or with the drive system(s) 414. Additionally, the communication connection(s) 410 may enable the vehicle to communicate with other nearby computer device(s) (e.g., other nearby vehicles, traffic signals, etc.). Additionally, the communication connection(s) 410 may enable the vehicle 402 to communicate with remote teleoperated computing devices or other remote services.

[0093] The communication connection(s) 410 may include physical and / or logical interfaces for connecting the vehicle computing device(s) 404 to another computing device or to a network, such as, for example, network(s) 432. For example, the communication connection(s) 410 may enable Wi-Fi-based communications, such as, for example, over frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as, for example, BLUETOOTH, or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing device(s).

[0094] The direct connection 412 may directly connect the drive system(s) 414 to other systems of the vehicle 402 .

[0095] In at least one example, the vehicle 402 may include drive system(s) 414. In some examples, the vehicle 402 may have a single drive system 414. In at least one example, if the vehicle 402 has multiple drive systems 414, the individual drive systems 414 may be located at opposing ends of the vehicle 402 (e.g., at the front and rear, etc.). In at least one example, the drive system(s) 414 may include sensor system(s) that detect the status of the drive system(s) 414 and / or the conditions around the vehicle 402. By way of example, and not limitation, the sensor system(s) may include wheel encoder(s) (e.g., rotary encoders) that sense the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) that measure the position and acceleration of the drive module, cameras or other image sensors, ultrasonic sensors that acoustically detect objects around the drive module, lidar sensors, radar sensors, etc. Some sensors, such as wheel encoder(s), may be unique to the drive system(s) 414. Sometimes, the sensor system(s) of the drive system(s) 414 may overlap or supplement a corresponding system (e.g., sensor system(s) 406) of the vehicle 402.

[0096] The drive system(s) 414 may include many vehicle systems, including a high voltage battery, a motor to propel the vehicle 402, an inverter to convert direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and steering rack (which may be electrically powered), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system to distribute braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the exterior surroundings of the vehicle), and one or more other systems (e.g., a cooling system, a safety system, an on-board charging system, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). In addition, the drive system(s) 414 may include a drive module controller capable of receiving and preprocessing data from the sensor system(s) and for controlling the operation of various vehicle systems. In some examples, the drive module controller may include a processor(s) and a memory communicatively coupled to the processor(s). The memory may store one or more modules that perform various functionality of the drive system(s) 414. Furthermore, the drive system(s) 414 also include communication connection(s) that enable each drive module to communicate with other local or remote computing device(s).

[0097] 4, the vehicle computing device(s) 404, the sensor system(s) 406, the emitter(s) 408, and the communication connection(s) 410 are shown onboard the vehicle 402. However, in some examples, the vehicle computing device(s) 404, the sensor system(s) 406, the emitter(s) 408, and the communication connection(s) 410 may be implemented outside the actual vehicle (i.e., not onboard the vehicle 402).

[0098] The vehicle 402 described above may send sensor data over the network(s) 432 to the computing device(s) 434. In some examples, the vehicle 402 may send raw sensor data to the computing device(s) 434. In other examples, the vehicle 402 may send processed sensor data and / or representations of the sensor data (e.g., data output from the localization system 420, the perception component 422, the prediction system 424, and / or the planner component 426) to the computing device(s) 434. In some examples, the vehicle 402 may send sensor data to the computing device(s) 434 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0099] The computing device(s) 434 may receive sensor data (raw or processed) from the vehicle 402 and / or other data collection devices, as well as data from one or more third party sources and / or systems. In at least one example, the computing device(s) 434 may include a processor(s) 436 and a memory 438 communicatively coupled to the processor(s) 436. In the illustrated example, the memory 438 of the computing device(s) 434 stores a training system 440, a map(s) storage 432 (e.g., storing one or more maps), and a historical data storage 444 (e.g., data associated with previous errors). In some examples, one or more of the systems and / or storage repositories may be associated with the vehicle 402 or with other computing device(s) associated with the system 400 instead of or in addition to being associated with the memory 438 of the computing device(s) 434.

[0100] In at least one example, the training system 440 can train data model(s) that can be used for various operations described herein. For example, machine learning algorithms for training the machine learning model(s) may include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatter plot smoothing (LOESS)), example-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative dichotomy 3 (ID3), chi-squared automated interaction detection (CHAID), decision cutoff, conditional decision tree), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average one dependency estimators (AODE), Bayesian belief networks (B NN), Bayesian networks), clustering algorithms (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, backpropagation, Hopfield networks, radial basis function networks (RBFN)), deep learning algorithms (e.g., deep Boltzmann machines (DBM), other deep belief networks (DBN), artificial neural networks (ANN), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), support vector machines, supervised learning, unsupervised learning, semi-supervised learning, etc.The resulting data model(s) may be stored in memory 438 and / or storage 430 of the vehicle 402 and may be accessed in near real-time by one or more components of the vehicle computing device(s) 404.

[0101] The processor(s) 416 of the vehicle 402 and the processor(s) 436 of the computing device(s) 434 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processor(s) 416, 436 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts the electronic data into registers and / or other electronic data that may be stored in memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement encoded instructions.

[0102] Memories 418 and 438 are examples of non-transitory computer-readable media. Memories 418 and 438 may store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods described herein and the functions attributed to the various systems. In various implementations, memories may be implemented using any suitable memory technology, for example, SRAM (Static RAM), SDRAM (Synchronous DRAM), non-volatile / flash memory, or other types of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which those shown in the accompanying drawings are merely examples relevant to the discussion herein.

[0103] 4 is illustrated as a distributed system, it should be noted that in some examples, the systems of the vehicle 402 may be associated with the computing device(s) 434 and / or the systems of the computing device(s) 434 may be associated with the vehicle 402. That is, the vehicle 402 may perform one or more functions associated with the computing device(s) 434, and vice versa.

[0104] 2, 5, and 6 are diagrams illustrating an exemplary process according to an embodiment of the present disclosure. The process just described is illustrated as a logical flow graph, where each operation represents a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media, which, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process. Additionally, operations may be omitted or added while remaining within the scope of the present disclosure.

[0105] 5 illustrates an example process 500 for controlling a vehicle based on a metric associated with a state of the vehicle as described herein. Some or all of the process 500 may be performed by one or more components in FIG. 4 as described herein. For example, some or all of the process 500 may be performed by vehicle computing device(s) 404 and / or computing device(s) 434.

[0106] In operation 502, the process may include receiving sensor data from one or more sensor systems of the vehicle. The sensor data may include data captured by a lidar sensor, a radar sensor, an ultrasonic transducer, a sonar sensor, a location sensor (e.g., a global positioning system (GPS), a compass, etc.), an inertial sensor (e.g., an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a lateral velocity sensor (e.g., a sensor that determines lateral speed based on ground tracking), etc.), a camera (e.g., red-green-blue (RGB), infrared (IR), intensity, depth, etc.), a wheel encoder, an environmental sensor (e.g., a temperature sensor, a humidity sensor, a light sensor, a pressure sensor, etc.), a time-of-flight (ToF) sensor, etc.

[0107] In some examples, the process may additionally include receiving map data from one or more maps stored in the vehicle and / or a remote computing device. The maps referred to above may be any number of data structures capable of providing information about the environment, such as, but not limited to, topology (e.g., junctions, lanes, merge zones, etc.), streets, mountains, roads, terrain, and the general environment. In some examples, the map(s) may include tile maps, feature maps, and / or other types of maps, such as those described in the U.S. patent applications discussed above.

[0108] In operation 504, the processing may include determining a state of the vehicle based at least in part on the sensor data. The state of the vehicle may include the location, position, orientation, speed, and / or acceleration of the vehicle at a given time. The state of the vehicle may include an optimized state of the vehicle based on the raw and / or processed sensor data. In some examples, the optimized state of the vehicle may be determined based on inputs from one or more IMUs and / or one or more localizers, such as, for example, IMU 112 and / or localizers 114, 116, 118, and / or 120. The optimization may include, for example, a nonlinear least squares optimization (although any other optimization mechanism is contemplated) that minimizes any one or more of the residuals defined with respect to the individual components described herein. In some examples, the state of the vehicle may be represented based on one or more degrees of freedom (e.g., 12 degrees of freedom, 15 degrees of freedom). In some examples, the state may include one or more locations (e.g., x-location, y-location, and / or z-location), one or more positions and / or orientations (e.g., pitch, roll, yaw, associated rates), one or more velocities (e.g., x-velocity, y-velocity, z-velocity), accelerations (e.g., x-acceleration, y-acceleration, z-acceleration), or the like. In some examples, the state may be represented across two or more dimensions.

[0109] At operation 506, the process may include determining a metric associated with the state of the vehicle, the metric including at least one of a variance, a residual, or a correction factor. The variance associated with the state may represent a confidence that the estimated state is accurate. In some examples, the variance may be determined based on a linearized system. In some examples, the variance may include a covariance associated with the estimated state of the vehicle. In the above examples, the value associated with the variance may represent a confidence in the state across two or more dimensions.

[0110] The residual may represent the difference between an observed measurement (e.g., based on sensor inputs) and an estimated measurement (value) for a quantity of interest (e.g., estimated pitch, roll, yaw, location in a coordinate system and / or dimensions, velocity, etc.). In some examples, the estimated measurement may be based in part on a value associated with a previous time step (e.g., a value associated with a measurement that took 0.06 seconds prior to the current measurement). In the above example, the residual may represent a discrepancy in the measurements. The correction factor may include a value used to correct for input errors. In some examples, the input errors may include those known to follow a predefined model. The correction factor may include, but is not limited to, IMU bias, wheel diameter scale factor, and the like.

[0111] At operation 508, the processing may include determining whether the metric is greater than or equal to a threshold. In some examples, the threshold may be determined based on the metric. In some examples, the threshold may be determined based on an associated sensor. In some examples, the threshold may be determined based on a localizer associated with the metric. In various examples, the threshold may be determined based on a warning limit associated with the condition being monitored. In some examples, the threshold may be based on one or more environmental considerations.

[0112] In some examples, the determination of whether the metric is at or above the threshold may be based on determining that the metric is above the threshold for a threshold time period (e.g., 0.5 seconds, 1 second, 2 seconds). In the above example, the error determination may be based on a prolonged determination of the erroneous measurement. In some examples, the metric may include a maximum value and a minimum value. In the above example, the metric being at or above the threshold may include determining that at least one of the metrics is at or above a maximum value or at or below a minimum value. In some examples, the maximum and minimum values ​​may represent a percentage of an acceptable correction for the component during vehicle operation. For example, due to the temperature of the operating environment, the radius of the wheels may increase and / or decrease during operation. The vehicle computing system may apply a correction factor to the changing radius of the vehicle. However, the correction factor may be limited to correct a maximum of 0.75% increase (e.g., maximum) and 0.75% decrease (e.g., minimum) of the radius of the vehicle.

[0113] Based on a determination that the metric is below the threshold ("NO" at operation 508), processing may include controlling the vehicle based at least in part on the conditions at operation 510. In some examples, the vehicle control may include maintaining a trajectory associated with the vehicle. In some examples, the vehicle control may include modifying the trajectory, such as based on other (non-condition-based) decisions (e.g., stopping for a stop sign or red light, yielding to pedestrians, etc.).

[0114] Based on a determination that the metric is equal to or greater than the threshold value ("YES" at operation 508), the process may include identifying at least one of the sensors or components associated with the metric at operation 512. In various examples, exceeding the threshold value may provide an indication of an error associated with at least one of the sensors or components.

[0115] At operation 514, the processing may include determining an error associated with at least one of the sensors or components. In some examples, the vehicle computing system may determine a degree of the error (e.g., a large error, a small error, etc.). In various examples, the degree of the error may be based on the amount by which the metric exceeds a threshold.

[0116] In operation 516, processing may include controlling the vehicle based at least in part on the error. Controlling the vehicle based on the error may include stopping the vehicle, slowing down to a predetermined speed, and / or slowing down a predetermined amount (e.g., slowing down to 50% of speed, reducing forward speed by 10 miles per hour, etc.), or the like. In some examples, the decision of what action to take may be based on the extent of the error, the number of redundant systems available, the criticality of associated sensors and / or associated components to vehicle control, and the like.

[0117] 6 illustrates an example process 600 for controlling a vehicle based on a residual value associated with a state of the vehicle as described herein. Some or all of the process 600 may be performed by one or more components in FIG. 4 as described herein. For example, some or all of the process 600 may be performed by vehicle computing device(s) 404 and / or computing device(s) 434.

[0118] At operation 602, the process may include receiving a first measurement associated with a state of the vehicle for a first time period. In some examples, the first measurement may include an average associated with measurements taken over the first time period. In some examples, the vehicle computing device may apply an auto aggressive moving average filter to measurements captured by a sensor and / or to measurements determined by a component of the vehicle, e.g., an IMU, a localizer, or the like. For example, the first measurement may be determined utilizing a function such as, for example, an auto regressive moving average (ARMA) filter.

[0119] In some examples, the first measurement may be associated with a first time. The measurement may be determined based on sensor data received from one or more sensors of the vehicle. In some examples, the state of the vehicle may include a location, position, orientation, velocity, and / or acceleration of the vehicle at a given time. In some examples, the state of the vehicle may be represented based on one or more degrees of freedom (e.g., 12 degrees of freedom, 15 degrees of freedom). In some examples, the state may include one or more locations (e.g., x-location, y-location, and / or z-location), one or more positions and / or orientations (e.g., pitch, roll, yaw, associated rates), one or more velocities (e.g., -x-velocity, y-velocity, z-velocity), accelerations (e.g., x-acceleration, y-acceleration, z-acceleration), or the like. In some examples, the state may be represented across two or more dimensions.

[0120] In operation 604, the process may include determining a first residual associated with the first measurement. The residual may represent a difference between an observed measurement (e.g., based on sensor input) and an estimated measurement of a quantity of interest (e.g., estimated pitch, roll, yaw, location in a coordinate system and / or dimensions, velocity, etc.). In some examples, the estimated measurement may be based in part on a value associated with a previous time step (e.g., a value associated with a measurement that took 0.06 seconds prior to the current measurement). In the above example, the residual may represent a discrepancy in measurements over time.

[0121] In some examples, the vehicle computing system may compare the first residual to a variance associated with the state at the first time, such as by normalizing the residual based on the variance. In the above example, the comparison may result in a whitened residual. In various examples, the first residual may include a value associated with the color white in the whitened residual.

[0122] At operation 606, the processing may include determining whether the first residual is greater than or equal to a threshold. In some examples, the threshold may be determined based on an associated sensor and / or component. In some examples, the threshold may be determined based on a localizer associated with the first residual. In some examples, the threshold may be based on one or more environmental considerations.

[0123] Based on a determination that the first residual is less than the threshold ("NO" at operation 606), processing may include controlling the vehicle based at least in part on the conditions at operation 608. In some examples, the vehicle control may include maintaining a trajectory associated with the vehicle. In some examples, the vehicle control may include modifying the trajectory, such as based on other (non-condition-based) decisions (e.g., stopping for a stop sign or red light, yielding to pedestrians, etc.).

[0124] Based on a determination that the first residual is greater than or equal to the threshold ("YES" at operation 606), the process may include determining, at operation 610, whether the first residual has remained greater than or equal to the threshold for greater than or equal to a threshold amount of time. The threshold amount of time may be determined based on an associated sensor, component, and / or localizer. In some examples, the threshold amount of time may be based on a speed associated with the vehicle, an operating environment (e.g., a highway, a business district, etc.), environmental considerations, weather conditions (e.g., detected rain, snow, ice, etc.), traffic density, and the like.

[0125] Based on a determination that the first residual was greater than or equal to the threshold for the threshold amount of time ("YES" at operation 610), the process may include identifying an error associated with at least one of the sensors or components of the vehicle at operation 612. In some examples, the vehicle computing system may determine an extent of the error, such as, for example, based on the extent to which the threshold is exceeded.

[0126] At operation 614, processing may include controlling the vehicle based at least in part on the error. Controlling the vehicle based on the error may include stopping the vehicle, slowing down to a predetermined speed, and / or slowing down a predetermined amount (e.g., slowing down to 50% of speed, reducing forward speed by 10 miles per hour, etc.), or the like. In some examples, the decision of what action to take may be based on the extent of the error, the number of redundant systems available, the criticality of associated sensors and / or associated components to vehicle control, and the like.

[0127] Based on a determination that the first residual is greater than or equal to the threshold for the threshold time period ("NO" at operation 610), the process may include receiving a second measurement associated with a second state of the vehicle at a second time at operation 616. The second measurement may be associated with the same localizer, sensor, and / or component as the first measurement.

[0128] At operation 618, the processing may include determining a second residual associated with the second measurement. The second residual may represent a difference between the measurement observed at the second time and an estimated measurement of the quantity of interest (e.g., estimated pitch, roll, yaw, location in a coordinate system and / or dimensions, velocity, etc.). In some examples, the estimated measurement may be based in part on a value associated with the first time or at another prior time step. In various examples, the vehicle computing system may determine at operation 606 whether the second residual is greater than or equal to a threshold value.

[0129] 7 illustrates an example process 700 for determining an optimized vehicle state as described herein. Some or all of the process 700 may be performed by one or more components in FIG. 4 as described herein. For example, some or all of the process 700 may be performed by vehicle computing device(s) 404 and / or computing device(s) 434.

[0130] In operation 702, the process may include determining a previous state of the vehicle at a first time. The previous state of the vehicle may include a global state of the vehicle and / or a portion of the state of the vehicle at the first time (e.g., a previous time step).

[0131] At operation 704, the process may include receiving sensor data from one or more sensor systems of the vehicle at a second time (after the first time). The sensor data may include data captured by one or more sensors as described above. In addition, the process may include receiving map data as described above.

[0132] In operation 706, the process may include determining, by a first localizer (or an IMU), at least a first portion of a state of the vehicle at the second time (or in another situation, some intermediate variables for determining at least a portion of a final state). The first portion of the state of the vehicle may include at least one of a location, position, orientation, velocity, and / or acceleration of the vehicle at the second time as determined by the first localizer. In some examples, the first portion of the state of the vehicle may be represented across two or more dimensions.

[0133] At operation 708, the process may include determining, by a second localizer (or an IMU), at least a second portion of the vehicle's state (or, as above, some intermediate variables) at the second time. The second portion of the vehicle's state may include at least one of the location, position, orientation, velocity, and / or acceleration of the vehicle at the second time as determined by the second localizer. In some examples, the second portion of the vehicle's state may be represented across two or more dimensions.

[0134] At either operation 706 and / or 708, residuals may be determined for one or more localizers.

[0135] In operation 710, the process may include determining an optimized vehicle state based at least in part on a previous state of the vehicle at the first time, a first portion of the vehicle state at the second time, and / or a second portion of the vehicle state at the second time.

[0136] In operation 712, the process may include determining at least one metric associated with the optimized vehicle state. The at least one metric may include a value associated with a variance (or covariance), a residual, or a correction factor as described herein.

[0137] Example clauses A: A system including one or more processors and one or more non-transitory computer-readable media that, when executed by the one or more processors, cause the system to perform operations including receiving sensor data from sensors of an autonomous vehicle; determining a state of the autonomous vehicle including at least one of a location, position, orientation, velocity, or acceleration of the autonomous vehicle based at least in part on the sensor data; determining a metric associated with the state of the autonomous vehicle; the metric including at least two of a variance, a residual, or a correction factor, determining that the metric is greater than or equal to a metric threshold; determining an error in the state of the autonomous vehicle based at least in part on the metric being greater than or equal to the metric threshold; and controlling the autonomous vehicle based at least in part on the error.

[0138] B: The system described in paragraph A, wherein the operations further include determining at least one of the vehicle's sensors or components associated with a metric, determining a weighting value associated with the at least one of the sensors or components, and determining a weighted metric by applying the weighting value to the metric, the weighting value being based at least in part on the at least one of the sensors or components relative to control of the vehicle, and determining that the metric is equal to or greater than a threshold value includes determining that the weighted metric is equal to or greater than a threshold value.

[0139] C: The system described in either section A or section B, wherein the metric includes a residual and a variance, and determining that the metric is greater than or equal to a threshold value includes determining an updated metric based at least in part on the residual and the variance, and determining that the updated metric is greater than or equal to a threshold value.

[0140] D: The system of any one of paragraphs A-C, wherein the metric is a first metric including a first residual associated with the state of the autonomous vehicle, and the operation further includes determining a second metric including a second residual associated with the state of the autonomous vehicle, and determining that the metric is greater than or equal to the metric threshold includes determining that a sum of the first metric and the second metric is greater than or equal to the metric threshold.

[0141] E: The system of any one of paragraphs A-D, wherein the metric is a first metric associated with a first time, and the operations further include determining a second metric associated with a second time, determining that the second metric exceeds a metric threshold, and determining that the time between the first time and the second time is greater than or equal to a threshold time, and determining the error is based at least in part on the time being greater than or equal to the threshold time.

[0142] F: A method including: receiving sensor data from a vehicle sensor; determining a vehicle state associated with at least one of the vehicle's location, position, orientation, speed, or acceleration based at least in part on the sensor data; determining a metric associated with the autonomous vehicle state, the metric including at least two of a variance, a residual, or a correction factor, determining that the metric is greater than or equal to a threshold; and determining an error in the vehicle state based at least in part on the metric being greater than or equal to the threshold.

[0143] G: The method described in paragraph F, further including controlling the vehicle based at least in part on the error, where controlling the vehicle includes at least one of stopping the vehicle, reducing a speed of the vehicle, limiting an acceleration of the vehicle, or limiting a steering angle of the vehicle.

[0144] H: A method as described in either of terms F or G, wherein at least two of the variances, residuals, or correction coefficients are based at least in part on a global state of the vehicle, a portion of a state of the vehicle, raw sensor data, or processed sensor data.

[0145] I: The method of any one of paragraphs FH, wherein the metric includes a residual associated with the state of the vehicle, and the method further includes determining that the metric exceeds the threshold for at least the threshold time, and determining the error is based at least in part on the metric exceeding the threshold for at least the threshold time.

[0146] The method of any one of paragraphs F-I, wherein J: the metric is a first metric associated with a first component of the vehicle, the method further including determining a second metric associated with a second component of the vehicle, and determining that the first component and the second component are the same or similar type of components and a difference between the first metric and the second metric is greater than or equal to a threshold, and determining the error is based at least in part on the difference being greater than or equal to the threshold.

[0147] The method of any one of paragraphs FJ, wherein K: metric is a first metric including a first residual associated with the state of the vehicle, the method further includes determining a second metric including a second residual associated with the state of the vehicle, and determining that the metric is greater than or equal to a threshold includes determining that a sum of the first metric and the second metric is greater than or equal to a threshold.

[0148] L: The method of any one of paragraphs FK, wherein the metric includes a residual and a variance, and determining that the metric is greater than or equal to the threshold value includes determining an updated metric based at least in part on the residual and the variance, and determining that the updated metric is greater than or equal to the threshold value.

[0149] M: The method of any one of paragraphs FL, further including determining at least one of the vehicle's sensors or components associated with a metric, determining a weight value associated with the at least one of the sensors or components, the weight value being based at least in part on the at least one of the sensors or components relative to control of the vehicle, and applying the weight value to the metric to determine a weighted metric, wherein determining that the metric is equal to or greater than a threshold includes determining that the weighted metric is equal to or greater than a threshold.

[0150] N: The method of any one of clauses FM, further including determining a first sensor or first component of the vehicle associated with the metric, identifying a second sensor or second component of the vehicle, determining a second metric associated with the second sensor or second component, determining that the second metric is less than a threshold, and controlling the vehicle based at least in part on the second sensor or second component.

[0151] O: The method of any one of paragraphs FN, further including determining at least one of the vehicle's sensors or components associated with a metric, determining a weight value associated with the at least one of the sensors or components, the weight value being based at least in part on the at least one of the sensors or components relative to control of the vehicle, and applying the weight value to the metric to determine a weighted metric, wherein determining that the metric is equal to or greater than a threshold includes determining that the weighted metric is equal to or greater than a threshold.

[0152] P: A system or device including a processor and a non-transitory computer-readable medium storing instructions that, when executed, cause the processor to perform a computer-implemented method described in any one of paragraphs FO.

[0153] Q: A system or device comprising: a processing means; and a storing means connected to the processing means, the storing means including instructions for configuring one or more devices to perform a computer-implemented method described in any one of paragraphs FO.

[0154] R: One or more non-transitory computer-readable media containing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including receiving sensor data from sensors of the vehicle; determining a state of the vehicle associated with at least one of the vehicle's location, position, orientation, speed, or acceleration based at least in part on the sensor data; determining a metric associated with the state of the autonomous vehicle, the metric including at least two of a variance, a residual, or a correction factor, determining that the metric is greater than or equal to a threshold; and determining an error in the state of the vehicle based at least in part on the metric being greater than or equal to the threshold.

[0155] S: One or more non-transitory computer-readable media as described in paragraph R, wherein the operations further include determining at least one of the vehicle's sensors or components associated with a metric, determining a weighting value associated with the at least one of the sensors or components, the weighting value being based at least in part on the at least one of the sensors or components relative to control of the vehicle, and applying the weighting value to the metric to determine a weighted metric, wherein determining that the metric is equal to or greater than a threshold includes determining that the weighted metric is equal to or greater than a threshold.

[0156] T: One or more non-transitory computer-readable media as described in paragraph S, wherein the weighting value is determined based at least in part on at least one of road conditions, time of day, light levels, temperature, or precipitation associated with the road corresponding to the vehicle operation.

[0157] U: One or more non-transitory computer-readable media as described in any one of paragraphs RT, wherein the metric includes a residual and a variance, and determining that the metric is greater than or equal to the threshold value includes determining an updated metric based at least in part on the residual and the variance, and determining that the updated metric is greater than or equal to the threshold value.

[0158] V: One or more non-transitory computer-readable media as described in any one of clauses RU, further including determining a first sensor or first component of the vehicle associated with a metric, identifying a second sensor or second component of the vehicle, determining a second metric associated with the second sensor or second component, determining that the second metric is less than a threshold, and controlling the vehicle based at least in part on the second sensor or second component.

[0159] W: A vehicle including a processor and a non-transitory computer readable medium as described in any one of paragraphs RV connected to the processor.

[0160] X: A system including a processor and a non-transitory computer readable medium as described in any one of paragraphs RV connected to the processor.

[0161] Although the exemplary clauses described above are described with respect to one particular implementation, it should be understood in the context of this document that the contents of the exemplary clauses can also be implemented via a method, device, system, computer-readable medium, and / or another implementation. In addition, any of the examples AX may be implemented alone or in combination with any other one or more of the examples AX.

[0162] termination Although one or more examples of the techniques described herein have been described, various alternatives, additions, permutations and equivalents fall within the scope of the techniques described herein.

[0163] In describing the examples, reference is made to the accompanying drawings, which form a part hereof, which show, by way of example, specific examples of the claimed subject matter. It is understood that other examples can be used and that modifications or alternatives, such as structural changes, can be made. The above examples, modifications, or alternatives do not necessarily depart from the scope of the intended and claimed subject matter. Although steps herein may be given in a certain order, sometimes the order can be changed such that certain inputs are provided at different times or in a different order without changing the functionality of the systems and methods described. Furthermore, the procedures disclosed could be performed in different orders. Furthermore, the various calculations herein need not be performed in the order disclosed, and other examples using alternative orderings of calculations could be implemented without difficulty. In addition to being reordered, the calculations could also be decomposed into sub-calculations that have the same results.

Claims

1. 1. A system comprising: one or more processors; When executed by the one or more processors, the system further comprises: Receiving sensor data from a sensor of the autonomous vehicle; determining a state of the autonomous vehicle based at least in part on the sensor data, the state including at least one of a location, position, orientation, velocity, or acceleration of the autonomous vehicle; determining a metric associated with the state of the autonomous vehicle, the metric including at least two of a variance, a residual, or a correction factor; determining at least one of a sensor or a component of the autonomous vehicle associated with the metric; and determining a weighting value associated with the at least one of the sensors or the components, the weighting value being based at least in part on the at least one of the sensors or the components relative to control of the autonomous vehicle; and applying the weighting values ​​to the metric to determine a weighted metric; determining that the weighted metric is greater than or equal to a metric threshold; determining an error in the state of the autonomous vehicle based at least in part on the weighted metric being equal to or greater than the metric threshold; controlling the autonomous vehicle based at least in part on the error; and One or more non-transitory computer-readable media for causing operations to be performed, including A system comprising:

2. The system of claim 1 , wherein the weighting value is determined based on environmental or weather conditions.

3. the metric includes the residual and the variance; Determining that the metric is greater than or equal to the metric threshold value includes: determining an updated metric based at least in part on the residual and the variance; determining that the updated metric is greater than or equal to the metric threshold; 3. The system according to claim 1 or 2, further comprising:

4. The metric is a first metric including a first residual associated with the state of the autonomous vehicle, and the operation includes: Determining a second metric comprising a second residual associated with the state of the autonomous vehicle. Further comprising:

4. The system of claim 1, wherein determining that the metric is greater than or equal to the metric threshold comprises determining that a sum of a first metric and the second metric is greater than or equal to the metric threshold.

5. The metric is a first metric associated with a first time, and the operation comprises: Determining a second metric associated with a second time; and determining that the second metric exceeds the metric threshold; determining that a time between the first time and the second time is greater than or equal to a threshold time; Further comprising:

5. The system of claim 1, wherein determining the error is based at least in part on the time being greater than or equal to the threshold time.

6. A method performed by a computing system, comprising: receiving sensor data from a sensor of the vehicle; determining a state of the vehicle associated with at least one of a location, position, orientation, velocity, or acceleration of the vehicle based at least in part on the sensor data; determining a metric associated with the state of the vehicle, the metric including at least two of a variance, a residual, or a correction factor; determining at least one of a sensor or a component of the vehicle associated with the metric; determining a weighting value associated with the at least one of the sensors or the components, the weighting value being based at least in part on the at least one of the sensors or the components relative to control of the vehicle; and applying the weighting values ​​to the metric to determine a weighted metric; determining that the weighted metric is greater than or equal to a threshold; determining an error in the state of the vehicle based at least in part on the weighted metric being equal to or greater than the threshold; and 23. A method comprising:

7. and controlling the vehicle based at least in part on the error, the controlling the vehicle comprising: Stopping the vehicle; reducing the speed of said vehicle; limiting the acceleration of the vehicle; or Limiting the steering angle of the vehicle.

7. The method of claim 6, comprising at least one of:

8. The at least two of the variance, the residual, or the correction factor are a global state of the vehicle; a portion of the state of the vehicle; Raw sensor data, or Processed Sensor Data 8. The method of claim 6 or 7, characterized in that it is based at least in part on

9. The metric includes the residual associated with the state of the vehicle, and the method further comprises: determining that the metric has exceeded the threshold for at least a threshold time; Further equipped with 9. The method of claim 6, wherein determining the error is based at least in part on the metric exceeding the threshold for at least the threshold time.

10. The metric is a first metric associated with a first component of the vehicle, and the method includes: determining a second metric associated with a second component of the vehicle, the first component and the second component being the same or a similar type of component; determining that a difference between the first metric and the second metric is greater than or equal to a threshold difference; Further equipped with 10. The method of claim 6, wherein determining the error is based at least in part on the difference being greater than or equal to the threshold difference.

11. The metric is a first metric including a first residual associated with the state of the vehicle, and the method further comprises: Determining a second metric comprising a second residual associated with the state of the vehicle. Further equipped with 11. The method of claim 6, wherein determining that the metric is greater than or equal to the threshold comprises determining that a sum of a first metric and the second metric is greater than or equal to the threshold.

12. The metric includes the residual and the variance, and determining that the metric is greater than or equal to the threshold comprises: determining an updated metric based at least in part on the residual and the variance; determining that the updated metric is greater than or equal to the threshold; 12. The method according to any one of claims 6 to 11, comprising:

13. 13. A method according to any one of claims 6 to 12, characterized in that the weighting values ​​are determined on the basis of environmental or weather conditions.

14. determining a first sensor or a first component of the vehicle associated with the metric; identifying a second sensor or a second component of the vehicle; determining a second metric associated with the second sensor or the second component; and determining that the second metric is less than the threshold; controlling the vehicle based at least in part on the second sensor or the second component; and 14. The method according to any one of claims 6 to 13, further comprising:

15. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 6 to 14.

Images