Perceptual Collision Avoidance

Abstract

A secondary system is a system that operates on a vehicle to avoid a collision when a problem occurs in the primary system. For example, a secondary system may operate independently of the primary system to take over control of the vehicle from the primary system when the secondary system detects a potential collision or when the primary system experiences an error. In an example, the primary system may implement a first technology, such as artificial intelligence (AI) technology, to understand the environment around the vehicle and / or direct the vehicle's movement within the environment. The secondary system may also implement a second technology based on the position, velocity, acceleration, etc. of the vehicle and / or objects around the vehicle.

Description

[Technical field]

[0001] The present disclosure relates to perceptual collision avoidance. [Background technology]

[0002] This patent application claims the benefit of priority to U.S. Application No. 16 / 189,726, filed November 13, 2018, which is incorporated by reference herein in its entirety.

[0003] The safety of vehicle occupants and others in close proximity to the vehicle is of paramount importance. Such safety is often based on accurate detection of potential collisions and timely deployment of safety measures. Although autonomous vehicles are equipped with highly effective collision detection systems, on rare occasions, these systems may fail to operate or be ineffective. For example, a power supply to a component of a system installed in the vehicle may be lost, causing the component or system to shut down, either momentarily or for an extended period of time. In another example, errors may occur at any number of points along the system's relatively long processing pipeline. These issues may cause the system to operate less effectively than desired, which may result in unsafe behavior. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 15 / 622905 [Patent Document 2] US Patent Application Publication No. 15 / 833715 [Patent Document 3] US Patent Application Publication No. 15 / 820245 [Patent Document 4] US Patent Application Publication No. 15 / 622905 [Brief description of the drawings]

[0005] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears, and the use of the same reference number in different drawings indicates similar or identical components or features.

[0006] [Figure 1] 1 illustrates an example of an environment in which the techniques discussed herein may be implemented. [Diagram 2] Illustrates details of the vehicle's primary and secondary systems. [Diagram 3] FIG. 4 is a flow diagram illustrating an example of a process for determining whether to slow down a vehicle. [Figure 4] FIG. 11 is a flow diagram illustrating another example process for determining whether to slow down a vehicle. [Diagram 5] FIG. 1 is a block diagram illustrating an example of a system for implementing the techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The present disclosure is directed to a vehicle including a primary system for controlling the vehicle and a secondary system operating on the vehicle to verify operation of the primary system and to control the vehicle to avoid a collision. For example, the secondary system may operate independently of the primary system to take over control of the vehicle from the primary system when the secondary system detects a potential collision (e.g., a collision missed by the primary system), when an error occurs in the primary system (e.g., a power failure, a hardware or software failure, etc.). In an example, the secondary system may be a system with higher integrity (e.g., more verifiable) and / or less complexity than the primary system. For example, the secondary system may take over control from the primary system when there is a relatively high certainty (e.g., above a threshold) that an event such as a collision will occur. The secondary system may operate independently of the primary system to increase safety for passengers of the vehicle or others in close proximity to the vehicle. In an example, if the primary system fails or determines that a collision is imminent, the secondary system may control the vehicle to slow down along its trajectory or take another maneuver. In at least some examples, the secondary system may communicate with the primary system (eg, by sending information about a potential collision to the primary system).

[0008] In some examples, the secondary system is capable of operating in at least two modes, a first mode of the secondary system verifies the trajectory generated by the primary system, and the secondary system intervenes (e.g., assumes control of the vehicle) when it determines that a collision is imminent, and if the secondary system determines a failure of the primary system (e.g., hardware failure), it enters a second mode and operates to control the vehicle to slow down along the trajectory and bring the vehicle to a safe stop.

[0009] Typically, the primary system may perform processing to control how the vehicle operates within the environment. The primary system may implement various artificial intelligence (AI) techniques, such as machine learning, to understand the environment around the vehicle and / or direct the vehicle to move within the environment. For example, the primary system may implement AI techniques to localize the vehicle, detect objects around the vehicle, segment sensor data, determine object classifications, predict object footprints, generate vehicle trajectories, etc. In an example, the primary system processes data from various types of sensors on the vehicle, such as LIDAR (Light Detection and Ranging) sensors, RADAR sensors, image sensors, depth sensors (Time of flight, Structured light, etc.).

[0010] The secondary system may validate the operation of the primary system and may take over control of the vehicle from the primary system if there is a problem with the primary system. The secondary system may implement probabilistic techniques based on the position, velocity, acceleration, etc. of the vehicle and / or objects around the vehicle. For example, the secondary system may implement one or more probabilistic techniques to independently localize the vehicle (e.g., in the local environment), detect objects around the vehicle, segment sensor data, identify object classifications, predict object footprints, generate vehicle trajectories, etc. In examples, the secondary system processes data from a small number of sensors, such as a subset of the sensor data processed by the primary system. Illustratively, the primary system may process LIDAR data, RADAR data, image data, depth data, etc., and the secondary system may process only LIDAR data and / or RADAR data (and / or time of flight data). However, in other examples, the secondary system may process sensor data from any number of sensors, such as data from each of the sensors, data from as many sensors as the primary system.

[0011] In an example, the secondary system may take over control of the vehicle when it detects a potential collision missed by the primary system. By way of example, the secondary system may receive a trajectory of the vehicle from the primary system. The secondary system may analyze sensor data to identify current locations of objects around the vehicle and predict the future locations of the objects. The secondary system may then determine whether the trajectory of the vehicle is likely to cross the current or predicted location of the object. If so, this may indicate that the primary system failed to detect a potential collision with the object. The secondary system may then cause the vehicle to slow down, stop, or take another maneuver (e.g., change lanes, steer, etc.) to avoid a collision with the object. For example, the secondary system may instruct the vehicle to make a hard stop (e.g., a hard stop maneuver including braking as quickly as possible). In an example, the secondary system may slow down or stop the vehicle when it detects that a collision is imminent (e.g., will occur within a relatively short time) and / or when a collision is detected with a relatively high degree of confidence (e.g., above a threshold).

[0012] Additionally or alternatively, the secondary system may take over control of the vehicle when an error occurs in the primary system. By way of example, the secondary system may determine that the primary system is offline (e.g., the primary system experiences a hardware or software failure, power to the first system is disrupted, etc.), or may determine that the trajectory provided by the primary system does not meet one or more criteria (e.g., the trajectory is different from a previous trajectory, the trajectory does not match the attitude of the vehicle, the trajectory does not match the performance of the vehicle, the trajectory controls the vehicle to operate on a surface classified as undriveable, etc.). In response to such a determination, the secondary system may cause the vehicle to slow down, stop, or perform another maneuver. For example, the secondary system may instruct the vehicle to make a gentle stop (e.g., a gentle stop maneuver including braking at a certain speed). This may avoid a collision with any object because the primary system is not operating normally. In an example, the secondary system may continue to monitor for potential collisions while the vehicle is coming to a gentle stop, and if a potential collision is detected, the secondary system may instruct the vehicle to make a hard stop to avoid the collision.

[0013] As mentioned above, the techniques and / or systems discussed herein may increase the safety of passengers of the vehicle and / or others in the vicinity of the vehicle. For example, by using a secondary system, if a problem occurs in the primary system, the vehicle may safely slow down and / or stop to avoid a collision. In examples, the secondary system may operate relatively independently from the primary system such that other types of evaluations are performed to avoid a collision. For example, the secondary system may independently detect objects in the vicinity of the vehicle or evaluate the trajectory of the primary system based on analysis of sensor data. Further, in examples, the secondary system may be designed as a system with relatively high integrity and / or low processing complexity, with the secondary system focusing on a specific task, such as detecting a problem in the primary system. For example, the secondary system may be designed to process less data than the primary system, include a shorter processing pipeline, operate according to a more easily verifiable technology than that of the primary system, etc.

[0014] The methods, apparatus, and systems described herein may be implemented in many ways. Exemplary implementations are provided below with reference to the following drawings. In some examples, discussed in the context of autonomous vehicles, the methods, apparatus, and systems described herein may be applied to a variety of systems. In another example, the methods, apparatus, and systems may be utilized in an aviation or nautical context. Additionally or alternatively, the techniques described herein may be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination thereof.

[0015] FIG. 1 illustrates an example environment 100 in which the techniques discussed herein may be implemented. The example environment 100 illustrates an autonomous vehicle 102 including sensors 104, a primary system 106, a secondary system 108, and a system controller 110. The primary system 106 and / or the secondary system 108 may process sensor data from the sensors 104. Typically, the primary system 106 may implement artificial intelligence, or AI, techniques 112 to control the autonomous vehicle 102. The secondary system 108 may take over control of the autonomous vehicle 102 if there is an issue with the primary system 106. Typically, the secondary system 108 may implement different techniques than the primary system 106. In an example, the secondary system 108 may implement techniques based on the position, velocity, acceleration, etc. of the autonomous vehicle 102 and / or objects surrounding the vehicle (e.g., Bayesian filtering, such as a Kalman filter, an extended Kalman filter, etc.). The secondary system 108 may send signals to the system controller 110 to control the autonomous vehicle 102, as described in more detail below. Although FIG. 1 shows the secondary system 108 sending signals to the system controller 110, the primary system 106 may additionally or alternatively send signals to the system controller 110 to control the autonomous vehicle 102.

[0016] The sensors 104 may include a variety of sensors, such as Light Detection and Ranging (LIDAR) sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., Global Navigation Satellite Systems (GNSS) including Global Positioning System (GPS), compasses, etc.), inertial sensors (e.g., Inertial Measurement Units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., red-green-blue (RGB), infrared (IR), illuminance, depth, etc.), time of flight, microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.).

[0017] The primary system 106 may control the vehicle during normal operation. In many examples, the primary system 106 may be the main computing system on the autonomous vehicle 102. The primary system 106 may implement AI techniques 112 to understand the environment around the autonomous vehicle 102 and / or direct the autonomous vehicle 102 to navigate within the environment. The AI ​​techniques 112 may include machine learning (also commonly referred to as machine learning models), such as one or more neural networks. The primary system 106 may use the AI ​​techniques 112 to process sensor data 114 received from the sensors 104. The sensor data 114 may include a wide variety of data, such as position data, inertial data, LIDAR data, RADAR data, image data, audio data, environmental data, depth data, etc. For example, the primary system 106 may analyze the sensor data 114, localize the autonomous vehicle 102, detect objects around the autonomous vehicle 102, segment the sensor data 114, determine object classifications, predict object footprints, generate a trajectory 116 for the autonomous vehicle 102, etc. In an example, the trajectory 116 comprises control of the steering angle and / or acceleration of the autonomous vehicle 102. For example, the trajectory 116 may control the path the autonomous vehicle 102 takes during a window of time (e.g., 1 microsecond, half a second, 2 seconds, 10 seconds, etc.). The trajectory 116 may be continually updated over time to control the movement of the autonomous vehicle 102. As shown, the primary system 106 may transmit the trajectory 116 to the secondary system 108.

[0018] The secondary system 108 may take over control of the autonomous vehicle 102 if the primary system 106 experiences a problem. The secondary system 108 may implement secondary techniques (e.g., probabilistic techniques, techniques different from the AI ​​techniques 112, etc.) based on the position, velocity, acceleration, etc. of the autonomous vehicle 102 and / or objects surrounding the autonomous vehicle 102. For example, the secondary system 108 may implement a Kalman filter, an extended Kalman filter (EKG), a particle filter, and / or a tracking algorithm to process sensor data, segment the sensor data, identify object classifications, predict object trajectories, etc. In examples, the secondary system 108 may not perform any type of AI technique. In some examples, the AI ​​techniques 112 may use neural networks, while the secondary system 108 may refrain from using such neural networks. In other examples, the secondary system 108 may perform AI techniques, such as implementing machine learning models different from (or in some examples the same as) the machine learning models implemented by the primary system 106.

[0019] In examples, the secondary system 108 processes sensor data 118 from the sensors 104 and / or trajectories 116 from the primary system 106. In some examples, the sensor data 118 includes a subset of the sensor data 114 processed by the primary system 106. To illustrate, the sensor data 118 may include a particular type of data (e.g., LIDAR, RADAR, Time of flight, etc.). In other examples, the sensor data 118 includes the same or more data as the sensor data 114, such as one or more of position data, inertial data, LIDAR data, RADAR data, image data, audio data, environmental data, and / or depth data.

[0020] In some examples, the secondary system 108 takes over control of the autonomous vehicle 102 when the secondary system 108 detects a predicted collision 120 missed by the primary system 106. In examples, the secondary system 108 may process the sensor data 118 to detect objects around the autonomous vehicle 102 and / or track objects over time. Further, in examples, the secondary system 108 may receive information from the primary system 106 indicative of the footprint of an object detected by the primary system 106 and / or the predicted trajectory of the object. The object footprint may include the path traveled by the object. The object footprint may represent (or be based on) the current or past position, velocity, acceleration, direction and / or path of the object over a period of time (e.g., 5 seconds). The secondary system 108 may maintain a footprint of each object detected around the autonomous vehicle 102.

[0021] The secondary system 108 may analyze the trajectory 116 of the autonomous vehicle 102, the footprint of the object, and / or the trajectory of the object to determine whether the trajectory 116 of the autonomous vehicle 102 and the trajectory of the object intersect. In an example, the secondary system 108 may predict the trajectory of the object using a straight-line approximation. If an intersection is detected, the secondary system 108 may detect a predicted collision 120 that was missed by the primary system 106. In other words, the secondary system 108 may determine that if the autonomous vehicle 102 proceeds along the trajectory 116 (e.g., continues to proceed according to the trajectory 116), the autonomous vehicle 102 will collide with the object. In the example of FIG. 1, the secondary system 108 may determine that the trajectory 122 of the object 124 intersects with the trajectory 116 of the autonomous vehicle 102 (e.g., if the autonomous vehicle 102 continues to proceed along the trajectory 116, the autonomous vehicle 102 will collide with the object 124). Here, object 124 represents a person, but object 124 may represent any type of object, such as another vehicle, a structure (eg, a building, etc.), an animal, a sign, etc.

[0022] If the secondary system 108 detects a predicted crash 120, the secondary system 108 may send a modified trajectory signal 126 to cause the autonomous vehicle 102 to slow down, stop, or perform another maneuver (e.g., change lanes, make an abrupt turn, pull over to the side of the road, etc.). For example, the secondary system 108 may send a modified trajectory signal 126 to the system controller 110 to perform a sudden stop maneuver. The sudden stop maneuver may include braking at a particular speed, such as a predetermined speed, a speed greater than a threshold (e.g., a maximum braking speed), or a speed that is a maximum speed among a plurality of available speeds. In one example, the sudden stop maneuver may include decelerating at a rate of 15 to 20 feet per second until the autonomous vehicle 102 comes to a stop. In an example, the modified trajectory signal 126 may instruct the system controller 110 to control the autonomous vehicle 102 and decelerate along the trajectory 116.

[0023] Additionally, in some examples, the secondary system 108 takes over control of the autonomous vehicle 102 when there is a primary system error 128. In one example, the secondary system 108 may detect a primary system error 128 when the primary system 106 is offline (e.g., the primary system 106 experiences a hardware or software failure, power to the primary system 106 is disrupted, etc.). This may be based on signals from the primary system 106, a component of the primary system 106, and / or a service provider (e.g., a remote device) indicating a hardware or software failure. Additionally or alternatively, this may be based on signals from a battery or other component associated with the autonomous vehicle 102's power supply indicating whether the primary system 106 (or a component of the primary system 106) is powered. In another example, the secondary system 108 may determine a primary system error 128 if the trajectory 116 does not meet one or more criteria, such as the trajectory 116 differs from a previous trajectory by a particular amount, the trajectory 116 does not match the attitude of the autonomous vehicle 102, the trajectory 116 does not match the performance of the autonomous vehicle 102, the trajectory 116 does not control the autonomous vehicle 102 to operate over a surface classified as undriveable, etc. Examples of such processes are described in further detail below.

[0024] If the secondary system 108 detects a primary system error 128, the secondary system 108 may send a modified trajectory signal 126 to cause the autonomous vehicle 102 to slow down, stop, or perform another maneuver. For example, the secondary system 108 may send a modified trajectory signal 126 to the system controller 110 to perform a gradual stopping maneuver. The gradual stopping maneuver may include braking at a specific speed, such as a predetermined speed, a speed less than a first speed (e.g., a maximum braking speed) and greater than a second speed (e.g., a minimum braking speed), a substantially intermediate speed among a plurality of available speeds (e.g., a third speed among five speeds), etc. The rate of deceleration may refer to a unit of measurement over time, such as meters or feet per second. In one example, the gradual stopping maneuver may include decelerating at a rate of 5 to 10 feet per second until the autonomous vehicle 102 comes to a stop. In an example, the modified trajectory signal 126 may be associated with the trajectory 116. For example, the modified trajectory signal 126 may instruct the system controller 110 to decelerate along the trajectory 116 (e.g., slow down along the same course). That is, the secondary system 108 may use steering controls associated with the trajectory 116 while modifying acceleration parameters associated with the trajectory 116 to stop the autonomous vehicle 102.

[0025] In some examples, the secondary system 108 may wait until the last allowed time to transmit the modified trajectory signal 126. To illustrate, at 11:30:40 a.m., the secondary system 108 may determine that if the trajectory 116 is maintained, the autonomous vehicle 102 will collide with an object in 5 seconds (at 11:30:45 a.m.). The secondary system 108 may also determine that the time required to stop the autonomous vehicle 102 (e.g., with a quick stop maneuver) and avoid a collision with the object is 3 seconds (e.g., based on the speed of the vehicle and / or the object). The secondary system 108 may use such information to determine the last allowed time (also referred to as Time to last second braking (TTLSB)) that the autonomous vehicle 102 has to initiate a quick stop maneuver to avoid a collision. In this example, the time to last second braking is 11:30:42 a.m. Now, the secondary system 108 may wait two seconds and determine, at 11:30:42 a.m., whether the primary system 106 has adjusted the trajectory 116 of the autonomous vehicle 102 or whether the autonomous vehicle 102 is still on a collision course with the object. If the autonomous vehicle 102 is still on a collision course with the object, the secondary system 108 may send a change trajectory signal 126 to cause a hard stop maneuver to be performed.

[0026] In some examples, the secondary system 108 may provide a warning signal to the primary system 106 that it will intervene if no action is taken. This may allow the primary system 106 to readjust its trajectory. For example, the secondary system 108 may send a warning signal to the primary system 106 indicating that a potential collision will occur in four seconds, and that the secondary system 108 will intervene in two seconds if the trajectory is not adjusted. In at least some examples, such warning signals may also comprise transmitting the detected objects, predicted object trajectories, predicted collisions, and the like, as determined by the secondary system 108.

[0027] If the secondary system 108 does not detect a predicted collision 120 and / or a primary system error 128, the secondary system 108 may transmit the trajectory 116 to the system controller 110. In other words, the secondary system 108 may cause the autonomous vehicle 102 to proceed along the trajectory 116 generated by the primary system 106.

[0028] In some examples, the primary system 106 may generate multiple trajectories, such as a main trajectory and one or more temporary trajectories. In examples, the secondary system 108 may adjust the main trajectory and / or the temporary trajectory when a predicted collision 120 and / or a primary system error 128 is detected. Exemplary techniques are described in further detail below.

[0029] 1 discusses using a hard stop maneuver when a predicted crash 120 is detected and a soft stop maneuver when a primary system error 128 is detected, such maneuvers may be used in any type of situation. For example, a soft stop maneuver may be used when a predicted crash 120 is detected (as long as this avoids a crash) and / or a hard stop maneuver may be used when a primary system error 128 is detected. Additionally, other types of maneuvers may be used, such as maneuvers such as slowing the autonomous vehicle 102 at any speed, pulling onto the shoulder, steering, or making another turn. In an example, the secondary system 108 may generate a new trajectory that causes the autonomous vehicle 102 to perform the maneuver.

[0030] In some examples, the autonomous vehicle 102 may operate according to different parameters to detect the predicted crash 120 and the primary system error 128. In an example, the secondary system 108 may operate according to a first set of parameters to detect the predicted crash 120 with a relatively high degree of accuracy. Illustratively, the secondary system 108 may detect the predicted crash 120 if an analysis of both LIDAR and RADAR data indicates a crash, if an object footprint is present for more than a certain period of time, if the covariance of the detection is below a threshold, etc. Further, in an example, the secondary system 108 may operate according to a second set of parameters to detect the primary system error 128 with less accuracy. By doing so, the secondary system 108 may initiate a hard stop maneuver if a crash is detected with a relatively high degree of confidence and initiate a soft stop maneuver if a primary system 106 error is detected with a low degree of confidence.

[0031] FIG. 2 is a diagram illustrating details of the primary and secondary systems 106 and 108 of the autonomous vehicle 102 of FIG.

[0032] In this example, the primary system 106 includes a localization component 202, a perception component 204, a prediction component 206, and a planning component 208. The localization component 202, the perception component 204, the prediction component 206, and / or the planning component 208, collectively and / or individually, may include processing pipelines that implement one or more machine learning models, such as neural networks, to accomplish various functions. Each processing pipeline may receive data and provide output. In one example, the perception component 204 implements one or more neural networks in a processing pipeline that performs object detection, segmentation, and / or classification. Each stage of the pipeline may provide some aspect of perception (e.g., object detection, classification, bounding box generation, etc.) that can feed into another stage of the pipeline. In an example, the primary system 106 implements a variety of neural networks that have been trained with thousands or hundreds of thousands of hours of vehicle training data. Furthermore, the variety of neural networks may be configured to perceive a myriad of complex scenarios and control the autonomous vehicle 102 in light of those scenarios. In an example, the primary system 106 uses parallel computing (e.g., with a GPU) and the secondary system 108 does not. In other examples, parallel computing is used in the primary system 106 and / or the secondary system 108.

[0033] As described herein, an exemplary neural network is a biologically derived algorithm that passes input data through a series of connected layers to generate an output. Each layer of a neural network may also constitute another neural network, or may constitute any number of layers (convolutional or not). As understood in the context of this disclosure, a neural network may utilize machine learning, which may refer to a broad classification of these algorithms in which an output is generated based on learned parameters.

[0034] Although discussed in the context of neural networks, any type of machine learning compatible with the present disclosure may be used.For example, machine learning algorithms include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally weighted scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative binary tree 3 (ID3), chi-squared automated interaction detection (CH2), etc.), and so on. AID), decision stumps, conditional decision trees), Bayesian algorithms (e.g., Naïve Bayes, Gaussian Naïve Bayes, Multinomial Naïve Bayes, Average Ordinary Dependent Classifier (AODE), Bayesian Belief Networks (BNN), Bayesian Networks), clustering algorithms (e.g., k-means, k-medians, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning Algorithms (e.g., Perceptron, Backpropagation, Hopfield Networks, Radial Basis Function Network (RBFN), etc.), deep learning algorithms (Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN), Convolutional Neural Networks (CNN), Stacked Auto-Encoders, etc.), dimensionality reduction algorithms (e.g. Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA), etc.), ensemble algorithms (e.g. Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, Stacked Generalization (Blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees (GBRT), Random Forest), SVM (Support Vector Machine), supervised learning, unsupervised learning, semi-supervised learning, etc.Additional example architectures include neural networks such as ResNet60, ResNet101, VGG, DenseNet, and PointNet.

[0035] The localization component 202 may include functionality to receive sensor data 114 from the sensors 104 (not shown in FIG. 2 ) to determine the pose (e.g., position and / or orientation) of the autonomous vehicle 102 and / or a map of the environment surrounding the autonomous vehicle 102. For example, the localization component 202 may determine and / or receive a map of the environment and continuously determine the position of the autonomous vehicle 102 within the map. The map may be comprised of a topological map, an occupancy grid, a point map, a landmark map, a mesh, a graph of pose constraints, and / or any other suitable map. In some examples, the localization component 202 may utilize SLAM (Simultaneous localization and mapping) or CLAMS (calibration, localization and mapping, simultaneously) to receive image data, LIDAR data, RADAR data, IMU data, GPS data, wheel encoder data, etc., to accurately determine the position / pose of the autonomous vehicle 102. The position of the autonomous vehicle 102 may constitute an attitude, including the relative position and / or orientation, local and / or global coordinates (e.g., GPS coordinates) of the autonomous vehicle 102 with respect to points and / or objects in a map. In additional or alternative examples, the orientation may include an indication of the yaw, roll, and / or pitch of the autonomous vehicle 102 with respect to a reference plane and / or points and / or objects in a map. The position and / or orientation together may describe an "attitude."

[0036] In some examples, the localization component 202 may provide data to various components of the autonomous vehicle 102. For example, the localization component 202 may provide the pose of the autonomous vehicle 102 to the planning component 208 to generate a trajectory, as described below. In other examples, the localization component may provide the pose of the autonomous vehicle 102 to other components of the autonomous vehicle 102, such as the perception component 204 and / or the prediction component 206.

[0037] The perception component 204 may include functionality to perform object detection, segmentation, and / or classification. In some examples, the perception component 204 may receive sensor data 114 as input and output primary perception data indicating the presence of an object in proximity to the autonomous vehicle 102 and / or a classification of the object as an object type (e.g., a semantic label such as car, pedestrian, bicycle, building, tree, road surface, curb, sidewalk, unknown, etc.). Additionally or alternatively, the perception component 204 may provide processed sensor data indicative of one or more characteristics associated with the detected object and / or the environment in which the object is located. In some examples, the characteristics associated with the object may include, but are not limited to, x-position (e.g., global position, relative position, etc.), y-position (e.g., global position, relative position, etc.), z-position (e.g., global position, relative position, etc.), orientation, object type (e.g., classification), object velocity, object spread (size), etc. In some examples, the object velocity may be determined from RADAR processing and / or other techniques. Characteristics related to the environment may include, but are not limited to, the presence of other objects in the environment, the state of other objects in the environment, time of day, day of the week, season, weather, darkness / light indications, etc.

[0038] In some examples, the perception component 204 may provide the primary perception data and / or the processed sensor data to various components of the autonomous vehicle 102. For example, the perception component 204 may provide the primary perception data and / or the processed sensor data to the planning component 208. In other examples, the perception component 204 may provide the primary perception data and / or the processed sensor data to other components of the autonomous vehicle 102, such as the localization component 202 and / or the prediction component 206. In examples, the primary perception data and / or the processed sensor data may be a type of primary object list that includes a list of objects and / or characteristics for each of the objects.

[0039] The prediction component 206 may associate footprints with detected objects and / or predict object trajectories. An object footprint may include historical object positions, velocities, accelerations, and / or orientations. In some examples, the prediction component 206 may determine whether to associate a historical object footprint with an object recently detected by the perception component 204. For example, the prediction component 206 may determine whether a recently detected object is within a threshold distance of a previous position of an object associated with the historical footprint, a threshold speed of a previous speed of an object associated with the historical footprint, etc. In some examples, the prediction component 206 may include a machine learning model that predicts object behavior in an environment based on lighting conditions (e.g., turn signal detection), object speed / acceleration, map data (e.g., lane information indicative of reasonable driving behavior), and / or learned object behavior. In some examples, the prediction component 206 may predict one or more predicted object trajectories for a particular object detected by the perception component 204 based on, for example, a probabilistic determination or multi-modal distribution of the object's past, current, and / or predicted position, velocity, acceleration, and / or orientation.

[0040] In some examples, the prediction component 206 may provide data regarding the object's footprint, the predicted object's trajectory (e.g., one or more trajectories and associated uncertainties), and / or other data to various components of the autonomous vehicle 102. For example, the prediction component 206 may provide such data to the planning component 208. In some examples, the prediction component 206 may generate heat maps related to the predicted movement of the object and provide these heat maps to the planning component 208. In other examples, the prediction component 206 may provide data regarding the object to other components of the autonomous vehicle 102, such as the localization component 202 and / or the perception component 204. Additionally, in some examples, the data regarding the object may be provided to the ancillary system 108. In examples, the data regarding the object may be stored in a primary object list.

[0041] In some examples, the perception component 204 and / or the prediction component 206 may implement techniques discussed in U.S. Patent No. 6,399,433, entitled “Voxel Based Ground Plane Estimation and Object Segmentation,” filed June 14, 2017; U.S. Patent No. 6,399,433, entitled “Trajectory Prediction of Third-Party Objects Using Temporal Logic and Tree Search,” filed December 6, 2017; and / or U.S. Patent No. 6,399,433, entitled “Sensor Data Segmentation,” filed November 21, 2017, the contents of all of which are incorporated herein by reference in their entireties.

[0042] The planning component 208 may determine a path and / or trajectory to be followed by the autonomous vehicle 102 to traverse an environment. For example, the planning component 208 may determine various routes and / or trajectories as well as various levels of detail. For example, the planning component 208 may determine a route for traveling from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this discussion, the route may be a sequence of waypoints for traveling between the two locations. As non-limiting examples, the waypoints include streets, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the planning component 208 may generate instructions for guiding the autonomous vehicle 102 along at least a portion of the route from the first location to the second location. In at least one example, the planning component 208 may determine how to guide the autonomous vehicle 102 from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instructions may comprise a trajectory, or a portion of a trajectory.

[0043] In some examples, the planning component 208 may generate the diverse trajectories substantially simultaneously (e.g., within engineering tolerances) according to a receding horizon technique and / or based at least in part on data received from other components of the primary system 106. In some examples, the planning component 208 may perform a cost-based search (or other suitable algorithms for identifying suitable trajectories for controlling the movement of the autonomous vehicle 102) on the diverse trajectories to identify a single trajectory to send to the secondary system 108. In some examples, the planning component 208 may generate additional diverse temporary trajectories and select one of the temporary trajectories to send to the secondary system 108. In some examples, the temporary trajectory may differ from the (primary) trajectory. For example, the trajectory may cause the autonomous vehicle 102 to make a lane change, whereas the temporary trajectory attempts to keep the autonomous vehicle 102 in the same lane. In another example, the trajectory may cause the autonomous vehicle 102 to maintain its current speed and steering angle, whereas the temporary trajectory attempts to pull the autonomous vehicle 102 over to the shoulder of the roadway.

[0044] Although the localization component 202, perception component 204, and prediction component 206 are generally discussed in the context of providing output to each other (or to other components of the primary system 106), any of these components may, in some examples, provide output to the secondary system 108.

[0045] As shown, the sub-system 108 includes an estimation and validation component 210, a data processing component 212, a localization component 214, a perception / prediction component 216, a trajectory validation component 218, a trajectory correction component 220, and a trajectory selection component 222. In examples, the estimation and validation component 210, the data processing component 212, the localization component 214, the perception / prediction component 216, the trajectory validation component 218, the trajectory correction component 220, and / or the trajectory selection component 222 may implement models based on physics and / or statistics. In some examples, such models may incorporate probabilistic models. For example, the sub-system 108 may implement a Kalman filter (also referred to as Linear Quadratic Estimation (LQE)) (e.g., Extended Kalman Filter (EKF), Unscented Kalman Filter (UKF), etc.), a particle filter, etc. By way of example, the perception / prediction component 216 may implement a Kalman filter or a particle filter. In some examples, the ancillary system 108 may be configured to exclude the use of machine learning models or to reduce the number of machine learning models, in contrast to the primary system 106. In other examples, the ancillary system 108 may include one or more machine learning models that are different (or similar or identical) to those of the primary system 106. In some examples, the ancillary system 108 may use different hardware (e.g., processor and memory) than the primary system 106.

[0046] In examples, the ancillary system 108 may be designed to be less computationally intensive and / or operate at a higher integrity level. For example, the processing pipeline of the ancillary system 108 may be simplified by relying on less sensor data, including less complex pipeline components, including fewer pipeline components overall, outputting less data, and / or eliminating and / or limiting the use of machine learning models. In some examples, the ancillary system 108 may be a "high integrity" system by achieving tight operating tolerances and / or having the ability to be verified. In examples, the ancillary system 108 may have a higher level of reliability and / or verifiability than the primary system 106. For example, the output of a subcomponent of the ancillary system 108 may be certified to operate with perfect accuracy or near perfect accuracy (e.g., 99.99% of scenarios or better). In some examples, the ancillary system 108 or a component of the ancillary system 108 may be referred to as a collision avoidance system (CAS). In some examples, the secondary system 108 may be equipped with Automotive Safety Integrity Level (ASIL) D certification.

[0047] The estimation and validation component 210 may determine a heading of the autonomous vehicle 102, a speed of the autonomous vehicle 102, an acceleration of the autonomous vehicle 102, a yaw rate of the autonomous vehicle 102, a yaw angular acceleration, and / or a steering angle of the autonomous vehicle 102. In an example, such information may be based on signals from the system controller 110, the drive system, etc. The estimation and validation component 210 may generate data indicative of one or more of such information and provide the data to various components of the secondary system 108, such as the localization component 214, the perception / prediction component 216, the trajectory validation component 218, the trajectory correction component 220, etc. In an example, the estimation and validation component 210 may provide the data to the primary system 106.

[0048] The localization component 214 may receive sensor data 118 from the sensors 104 (or data processing component 212) to determine one or more of the position and / or orientation (along with pose) of the autonomous vehicle 102. Here, the position and / or orientation may be relative to points and / or objects in an environment in which the autonomous vehicle 102 is located. In an example, the orientation may include an indication of the yaw, roll, and / or pitch of the autonomous vehicle 102 relative to a reference plane and / or relative to a point and / or object. In an example, the localization component 214 may perform less processing (e.g., higher level localization) than the localization component 202 of the primary system 106. For example, the localization component 214 may only determine the pose of the autonomous vehicle 102 relative to objects and / or surfaces detected in the autonomous vehicle 102's surroundings (e.g., local position, rather than global position) rather than determining the pose of the autonomous vehicle 102 relative to a map. Such position and / or orientation may be determined using probabilistic filtering techniques, such as, for example, a Bayesian filter (such as a Kalman filter, an extended Kalman filter, an unscented Kalman filter, etc.) using some or all of the sensor data.

[0049] The localization component 214 may provide the position and / or orientation of the autonomous vehicle 102 to various components of the sub-system 108, such as a perception / prediction component 216, a trajectory verification component 218, and / or a trajectory correction component 220.

[0050] The data processing component 212 may process the sensor data 118 received from the sensor 104 (not shown in FIG. 2 ) to determine processed sensor data. For example, the data processing component 212 may analyze the sensor data 118 to remove noise in the sensor data 118. In some examples, the data processing component 212 may include various components, each processing different types of data. For example, a first data processing component may be implemented to process LIDAR data, while a second data processing component may be implemented to process RADAR data. The different data processing components may operate according to different algorithms. In some examples, the sensor data 118 (processed or unprocessed) may indicate an object response (e.g., that an object has been detected on the ground or in the air). Such a response may be determined by the sensor 104. The data processing component 212 may transmit the sensor data 118 (processed or unprocessed) to other components of the sub-system 108, such as the perception / prediction component 216.

[0051] In some examples, the data processing component 212 may filter the sensor data 118 to generate a reduced amount of data. In one example, the data processing component 212 may remove data from the sensor data 118 that is outside of a certain range. This may be based on the speed or acceleration of the autonomous vehicle 102, the footprint of the object, the speed or acceleration of the object, etc. (e.g., remove a significant amount of data when traveling at a relatively slow speed, remove less data when traveling at a relatively fast speed, etc.). To illustrate, if the autonomous vehicle 102 is traveling at 15 miles per hour and there are no vehicles traveling with the autonomous vehicle 102, the data processing component 212 may remove data from 200 feet away, as this data is likely not necessary to evaluate the trajectory or identify the possibility of an imminent collision. In another example, the data processing component 212 may remove data from the sensor data 118 based on the type of environment in which the autonomous vehicle 102 is located. For example, if the autonomous vehicle 102 is in an urban area having a wide variety of vehicles, roads, road intersections, etc., the data processing component 212 may not need to remove data (or may remove a limited amount) from the sensor data 118. However, if the autonomous vehicle 102 is in a rural or sparsely urban area with relatively fewer vehicles, roads, road intersections, etc., the data processing component 212 may remove more data from the sensor data 118. In an example, reducing the amount of sensor data 118 may allow the ancillary system 108 to operate more quickly (e.g., processing less data may reduce computation time).

[0052] 2, in some examples, the sensor data 118 may be passed to various components of the ancillary system 108 without passing through the data processing component 212. For example, the sensor data 118 may be passed to various components of the ancillary system 108 in a raw format.

[0053] The perception / prediction component 216 may include functionality to perform object detection, object segmentation, object classification, footprint determination, and / or prediction of one or more trajectories for each object (e.g., including uncertainty), etc. In some examples, the perception / prediction component 216 may receive as input data the sensor data 118 (in processed or unprocessed format) from the data processing component 212, the attitude of the autonomous vehicle 102 from the localization component 214, and / or the trajectory from the primary systems 106. Additionally or alternatively, the perception / prediction component 216 may receive as input data from the estimation and validation component 210 data indicative of the heading of the autonomous vehicle 102, the speed of the autonomous vehicle 102, the acceleration of the autonomous vehicle 102, the yaw rate of the autonomous vehicle 102, the yaw angular acceleration, and / or the steering angle of the autonomous vehicle 102. Additionally, in some examples, the perception / prediction component 216 may receive as input data data from the primary system 106, such as a primary object list from the perception component 204 and / or a pose of the autonomous vehicle 102. Such data may be received in some examples when the primary system 106 is online. As described above, the object list from the primary system 106 may include primary perception data, processed sensor data, data regarding object footprints, and the like. In at least some examples, the primary system 106 may transmit object footprint information to the secondary system 108. In such examples, the perception / prediction component 216 may perform data association (e.g., using a probabilistic filter, clustering, nearest neighbor analysis, etc.) to associate the sensor data with the provided footprints.

[0054] The perception / prediction component 216 may process the input data to determine secondary sensory data. The secondary sensory data may indicate the presence of an object proximate to the autonomous vehicle 102, characteristics associated with the object, etc. Here, the characteristics associated with the object may include the position of the object relative to the autonomous vehicle 102, the orientation of the object relative to the autonomous vehicle 102, the velocity of the object, the acceleration of the object, the range of the object (size of the object), the size of the object, etc. In many examples, the characteristics included in the secondary sensory data do not include a classification (e.g., a semantic label) of the object. However, in some examples, the characteristics included in the secondary sensory data may be indicative of a classification of the secondary sensory data.

[0055] The perception / prediction component 216 may also process the input data to determine one or more footprints of the object. As described above, the footprint of the object may include historical position, velocity, acceleration, and / or direction, etc. In one example, the perception / prediction component 216 may determine an object in the environment at a first time, determine an object in the environment at a second time, determine overlap of the object at the first time and the second time, and generate a footprint for the object. In another example, the perception / prediction component 216 may determine a velocity of the object at a first time, compare the predicted movement of the object to captured data of the object at a second time, determine overlap, and generate a footprint for the object. The perception / prediction component 216 may determine a footprint for each object detected in the surroundings of the autonomous vehicle 102. In some examples, the perception / prediction component 216 may determine the object footprint independent of the data received from the primary system 106 (e.g., without using the primary object list received from the primary system 106 and / or the pose of the autonomous vehicle 102), while in other examples, the perception / prediction component 216 may use the data received from the primary system 106.

[0056] The perception / prediction component 216 may also process the input data to determine one or more predicted trajectories for the object. For example, based on the object's current location and the object's speed over a period of several seconds, the perception / prediction component 216 may predict the path the object will travel over the course of several seconds. In some examples, such predicted paths may be constructed using linear assumptions of motion given the position, direction, speed, and / or direction. In other examples, such predicted paths may involve more complex analysis.

[0057] In some examples, the perception / prediction component 216 may determine diagnostic data. The diagnostic data may indicate the state of the input data to the perception / prediction component 216 (e.g., indicating the timing of receiving sensor data 118 from the sensor 104, such as whether the sensor data 118 is older than a threshold time). Additionally, the diagnostic data may indicate the frequency with which the input data is received (e.g., every milliseconds), the latency of the input data (e.g., the delay in the sensor 104 sending a signal in the environment and receiving a returning signal), main loop and output frequency, and / or computer utilization.

[0058] The perception / prediction component 216 may output the secondary sensory data, object-related data (e.g., trajectory, footprint, etc.), and / or diagnostic data to the trajectory verification component 218, the trajectory correction component 220, and / or any other components of the secondary system 108. In an example, the secondary sensory data and / or object data is represented in a secondary object list. For example, the object list may indicate, for each object, the object's characteristics and / or the object's trajectory / footprint. In an example, the perception / prediction component 216 may output the secondary sensory data, object-related data, and / or diagnostic data to the primary system 106.

[0059] For ease of illustration, the perception / prediction component 216 (and other components of the primary system 106 and the ancillary system 108) are illustrated as a single block. However, the perception / prediction component 216 (and / or other components of the primary system 106 and the ancillary system 108) may be implemented as any number of blocks.

[0060] The trajectory validation component 218 may validate the trajectory of the autonomous vehicle 102 received from the primary system 106. For example, the trajectory validation component 218 may determine whether the trajectory is collision-free. Here, the trajectory validation component 218 may compare the trajectory of the autonomous vehicle 102 (e.g., the projected path if the autonomous vehicle 102 were to stay on track) with data received from the perception / prediction component 216 (e.g., secondary perception data, data regarding an object's footprint, etc.). If the comparison indicates that the trajectory of the autonomous vehicle 102 intersects (or the probability of intersecting is equal to or greater than a threshold value for) the object's footprint or predicted trajectory, the trajectory validation component 218 may determine that the trajectory is invalid.

[0061] In some examples where the trajectory of the autonomous vehicle 102 and the footprint or predicted trajectory of an object intersect, the trajectory validation component 218 may determine the time to last second braking (TTLSB) of the object. As described above, the time to last second braking is the last possible time that the autonomous vehicle 102 must initiate a hard stop maneuver to avoid a collision with the object. If the time to last second braking is within a threshold, the trajectory validation component 218 may determine that the trajectory of the autonomous vehicle 102 is invalid. By way of illustration, if the time to last second braking of the object is within 10 milliseconds, the trajectory validation component 218 may determine that the trajectory of the autonomous vehicle 102 is invalid. In an example, the trajectory validation component 218 may evaluate the time to last second braking for each object in the object list.

[0062] In an example, the trajectory verification component 218 may validate the trajectory of the autonomous vehicle 102 by determining whether the trajectory differs from a previous trajectory by more than a certain amount. To illustrate, the trajectory verification component 218 may determine whether the trajectory has changed direction from the previous trajectory by more than a threshold amount (e.g., a change in course / angle by more than a threshold). If there is a change by more than a threshold, the trajectory verification component 218 may determine that the trajectory is invalid. In another example, the trajectory verification component 218 may determine that the trajectory is invalid if the acceleration of the trajectory has changed from the previous trajectory by more than a threshold. In some examples, the trajectory may be compared to various previous trajectories, such as the last three trajectories, to determine whether there is a change by more than a threshold from each of the previous trajectories. The trajectory verification component 218 may store the trajectory over time such that one or more previous trajectories may be referenced.

[0063] In an example, the trajectory validation component 218 may validate the trajectory of the autonomous vehicle 102 by determining whether the trajectory is compatible with the pose of the autonomous vehicle 102. To illustrate, the trajectory validation component 218 may analyze the pose received from the localization component 214. If the trajectory requires a change to the pose of the autonomous vehicle 102 that is incompatible with the current pose of the autonomous vehicle 102, such as requiring the autonomous vehicle 102 to roll into an upside-down position, the trajectory validation component 218 may determine that the trajectory is invalid.

[0064] In an example, the trajectory validation component 218 may validate a trajectory of the autonomous vehicle 102 by determining whether the trajectory has a mismatch with the capabilities of the autonomous vehicle 102. The capabilities may be based on hardware and / or software components of the autonomous vehicle 102 and / or limitations of those components. To illustrate, the trajectory validation component 218 may reference the capabilities of the autonomous vehicle 102, such as a maximum steering angle and an amount of time to change from one steering angle to another. If the trajectory requires a change beyond the maximum steering angle and / or if the trajectory requires changing the steering faster than the amount of time required to adjust the steering angle, the trajectory validation component 218 may determine that the trajectory is invalid.

[0065] In an example, the trajectory validation component 218 may validate the trajectory of the autonomous vehicle 102 by determining whether the trajectory would control the autonomous vehicle 102 to maneuver over a surface that is classified as undrivable. To illustrate, the trajectory validation component 218 may receive data (e.g., ancillary sensory data, etc.) from the perception / prediction component 216. Such data may indicate how smooth a surface is. If the trajectory validation component 218 determines that the surface has a smoothness below or above a smoothness threshold, the trajectory validation component 218 may classify the surface as undrivable. In an alternative example, such a determination may be made by the perception / prediction component 216. In any case, if the trajectory validation component 218 determines that the trajectory is along a surface that is undrivable, the trajectory validation component 218 may determine that the trajectory is invalid.

[0066] If the trajectory validation component 218 determines that the trajectory of the autonomous vehicle 102 is invalid, the trajectory validation component 218 may send a signal to the trajectory correction component 220 to instruct it to control the autonomous vehicle 102 to slow down, stop, or perform another maneuver. In an example, the signal may indicate why the trajectory is invalid (e.g., indicate the type of invalidity). Alternatively, if the trajectory validation component 218 does not detect a problem with the trajectory during one or more of the determinations described above (e.g., the trajectory is not invalid), the trajectory validation component 218 may determine that the trajectory is valid. Now, the trajectory validation component 218 may send a signal to the trajectory selection component 222 indicating that the trajectory is valid and to use the trajectory to control the autonomous vehicle 102.

[0067] In some examples, the trajectory verification component 218 may evaluate the main trajectory of the autonomous vehicle 102 and, if the main trajectory is invalid, evaluate an interim trajectory of the autonomous vehicle 102. For example, upon determining that the main trajectory is invalid using any of the techniques described herein, the trajectory verification component 218 may determine whether the interim trajectory is valid. If the interim trajectory is valid, the trajectory correction component 220 may send a signal to the trajectory selection component 222 to use the interim trajectory. If the interim trajectory is invalid, the trajectory correction component 220 may send a signal to the trajectory correction component 220 to modify the main trajectory (or the interim trajectory).

[0068] In some examples, the trajectory verification component 218 may cause the autonomous vehicle 102 to proceed along the trajectory even if the trajectory is invalid. For example, if the trajectory verification component 218 determines that the size of the object is smaller than a predetermined size (or classified as a particular type of object, if such information is available), the trajectory verification component 218 may send a signal to the trajectory selection component 222 to proceed along the trajectory. To illustrate, the trajectory verification component 218 may determine that the trajectory intersects with the footprint of an object, but that the object is the size of a baseball. Here, it may be safer to continue along the trajectory rather than decelerate, since the autonomous vehicle 102 is more likely to run over the object without colliding.

[0069] Further, in an example, the trajectory verification component 218 may determine whether the primary system 106 is offline. For example, the trajectory verification component 218 may receive signals from the primary system 106, components of the primary system 106, and / or a service provider (e.g., a remote device) periodically, continuously, or at any time indicating a state of the hardware and / or software of the primary system 106. If the signals indicate a fault in the hardware and / or software of the primary system 106, the trajectory verification component 218 may determine that the primary system 106 is offline. Additionally or alternatively, the trajectory verification component 218 may receive signals from a battery or other component associated with the power supply of the autonomous vehicle 102 periodically, continuously, or at any time indicating whether the primary system 106 (or a component of the primary system 106) is powered. If the signals indicate that the primary system 106 (or a component of the primary system 106) is not powered, the trajectory verification component 218 may determine that the primary system 106 is offline.

[0070] If the primary system 106 is offline, the trajectory verification component 218 may send a signal to instruct the trajectory correction component 220 to control the autonomous vehicle 102 to slow down, stop, or perform another maneuver. In an example, the primary system 106 likely is not currently outputting a trajectory, so the trajectory verification component 218 may send a previous trajectory to the trajectory correction component 220 (e.g., the last stored trajectory). Alternatively, if the primary system 106 is online (no issues such as no collisions detected), the trajectory verification component 218 may send a signal to the trajectory selection component 222 to proceed along the trajectory.

[0071] The trajectory correction component 220 may modify the trajectory received from the primary system 106. For example, the trajectory correction component 220 may receive data from the localization component 214, data from the perception / prediction component 216 (e.g., secondary sensory data, data indicative of an object's footprint, etc.), and / or data from the estimation and verification component 210. If the trajectory correction component 220 receives a signal to slow down or stop (or otherwise change the trajectory) from the trajectory verification component 218, the trajectory correction component 220 may process the data to determine a deceleration rate along the trajectory. The trajectory correction component 220 may then send a signal to the trajectory selection component 222 to decelerate along the trajectory (e.g., the modified trajectory). In some examples, such trajectory corrections may also include adjusting steering angles and / or other controls to further adjust the trajectory. In at least some examples, such corrections may be to either the primary trajectory and / or the ad-hoc trajectory received from the primary system 106.

[0072] In one example, if the primary system 106 is offline, the trajectory correction component 220 may send a signal to the trajectory selection component 222 to perform a gradual stopping maneuver. Here, the trajectory correction component 220 may determine a deceleration rate to perform a gradual stopping maneuver based on the current speed of the autonomous vehicle 102. In another example, if the trajectory is invalid for reasons other than a collision, the trajectory correction component 218 may send a signal to the trajectory selection component 222 to perform a gradual stopping maneuver. In yet another example, if the trajectory is invalid due to a detected collision, the trajectory correction component 220 may send a signal to the trajectory selection component 222 to perform a hard stopping maneuver (e.g., to be stopped at a particular speed, which may be a maximum speed). Here, the trajectory correction component 220 may determine a deceleration rate based on the current speed of the autonomous vehicle 102 and / or the distance to the object. The deceleration rate may include stopping the autonomous vehicle 102 before colliding with the object. In yet another example, if the trajectory is invalid due to a detected collision and the time to the final stopping braking is far enough away (e.g., outside of a window of time), the trajectory correction component 220 may send a signal to the trajectory selection component 222 to decelerate and / or provide a warning signal of the detected collision to the primary system 106 (which may include the detected object, the object's location, etc.).

[0073] In some examples, a sudden stop maneuver may take precedence over other types of maneuvers. For example, if a potential collision is detected and a sudden stop maneuver is initiated, the sudden stop maneuver may take precedence over other maneuvers, such as a gradual stop maneuver that has already been initiated. For illustrative purposes, assume that a gradual stop maneuver is initiated and, during the execution of the gradual stop maneuver, an imminent collision is detected. Now, trajectory correction component 220 may initiate a sudden stop maneuver that overrides the gradual stop maneuver.

[0074] In some examples, the trajectory correction component 220 may generate a new trajectory for the autonomous vehicle 102. For example, based on the attitude of the autonomous vehicle 102, sensory data, etc., the trajectory correction component 220 may generate a trajectory that moves the autonomous vehicle 102 onto the shoulder or changes lanes.

[0075] In an example, the orbit selection component 222 may select a trajectory to send to the system controller 110. In one example, if the trajectory verification component 218 sends a signal to the orbit selection component 222 (or sends a trajectory) indicating to use a trajectory from the primary system 106, the orbit selection component 222 may send a signal to the system controller 110 to use the trajectory (or a trajectory). In another example, if the trajectory correction component 220 sends a signal to the orbit selection component 222 to use a corrected trajectory (or sends a corrected trajectory), the orbit selection component 222 may send a signal to the system controller 110 to use the corrected trajectory (or a corrected trajectory). The corrected trajectory may represent a deceleration along the trajectory (e.g., primary or provisional) provided by the primary system 106 and / or represent a different maneuver. In yet another example, the orbit selection component 222 may send a signal to the system controller 110 to use a different trajectory indicated or provided by the trajectory verification component 218 and / or the trajectory correction component 220.

[0076] In an example, the trajectory selection component 222 may send a signal or trajectory to the primary system 106. For example, the trajectory selection component 222 may send a signal indicating that a modified trajectory has been selected. This may inform the primary system 106 that there was a problem with the primary system (e.g., an invalid trajectory, etc.). Additionally, this may assist the primary system 106 in adjusting the course of the autonomous vehicle 102 (e.g., if the modified trajectory indicates that the vehicle is slowing down but not coming to a complete stop).

[0077] The system controller 110 may control the steering, propulsion, braking, safety, emitter, communication, and / or other systems of the autonomous vehicle 102. The system controller 110 may communicate with and / or control systems corresponding to the drive system and / or other components of the autonomous vehicle 102. In some examples, the system controller 110 may convert a trajectory into instructions that can be used by the drive system to cause the autonomous vehicle 102 to traverse the trajectory.

[0078] In some examples, the ancillary system 108 or another system performs ground removal techniques on the sensor data 118. This may occur before the sensor data 118 is used by components of the ancillary system 108. For example, the ancillary system 108 may analyze the LIDAR data or voxel data associated with the LIDAR data to determine whether the LIDAR data or voxels represent ground. In some examples, the ground may correspond to a surface drivable by a vehicle. In some examples, the ancillary system 108 may determine a dot product between a vector in an elevation dimension (e.g., a reference direction) of a device carrying the LIDAR system and a normal vector (as may be determined based on a per-voxel eigenvalue decomposition) expressed in a common coordinate system. In such examples, a dot product that meets or exceeds a threshold, for example, on the order of about 15, may indicate that the voxel does not comprise ground. On the other hand, a dot product less than the threshold may indicate that the voxel comprises ground. Additionally, in some examples, the ancillary system 108 may cluster voxels determined to be locally flat voxels to stretch a surface corresponding to the ground. In examples, once a ground surface is identified in the sensor data, data related to the ground surface may be removed from the sensor data and the remaining data may be processed (e.g., object detection, segmentation, classification, etc.). In some examples, the primary system 106, the ancillary system 108, and / or another system may perform ground plane removal techniques described in U.S. Patent Application Publication No. 2013 / 0133994, entitled “Voxel Based Ground Plane Estimation and Object Segmentation,” filed June 14, 2017, the entire contents of which are incorporated herein by reference.

[0079] The components of the primary system 106 and / or the secondary system 108 may represent hardware and / or software components. In a software context, one or more of the components may include instructions executable by one or more processors. In a hardware context, one or more of the components may be implemented as an integrated circuit (e.g., an application specific integrated circuit (ASIC), etc.), a gate array (e.g., a field programmable gate array (FPGA), etc.), etc. In at least some examples, both the hardware and software of the primary system 106 and the secondary system 108 may differ.

[0080] 3 and 4 show example processes 300 and 400 for performing the techniques described herein. Processes 300 and 400 are illustrated as logical flow graphs, each operation of which represents a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order of operations described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process. Processes 300 and / or 400 may be performed by any device or component, such as the autonomous vehicle 102, a service provider (e.g., a computing device of the service provider), or any other device.

[0081] 3 illustrates a flow diagram of an example process 300 for determining whether to slow down a vehicle. For ease of explanation, the process 300 is described as being performed by a first system 302 and a second system 304. For example, the first system 302 may comprise the primary system 106 of FIGS. 1 and 2, while the second system 304 may comprise the secondary system 108 of FIGS. 1 and 2.

[0082] In examples, the first system 302 may implement a first technique, such as a machine learning model, to perform first object detection, first object segmentation, first object classification, and / or first object tracking / prediction. In examples, the second system 304 may implement a second technique, such as a probabilistic model, to perform second object detection, second object segmentation, second object classification, and / or second object tracking / prediction. In some examples, the second technique implements a Kalman filter and / or a particle filter.

[0083] In operation 306, the first system 302 may receive first sensor data. For example, the first system 302 may receive one or more of LIDAR data, RADAR data, image data, depth data (time of flight, structured light, etc.), etc. from a sensor on the vehicle.

[0084] In operation 308, the first system 302 may determine a trajectory for controlling the vehicle. For example, the first system 302 may analyze the first sensor data to generate a trajectory for controlling the vehicle. In an example, the first system 302 may perform localization, perception, and / or prediction operations to generate the trajectory (which may include both a nominal trajectory and an ad-hoc trajectory).

[0085] In operation 310 , the first system 302 may transmit the trajectory (or multiple trajectories) to the second system 304 .

[0086] In operation 312, the second system 304 may receive second sensor data. For example, the second system 304 may receive one or more of LIDAR data, RADAR data, image data, depth data (Time of flight, Structured light, etc.), etc. from sensors on the vehicle. In some examples, the second sensor may include a subset of the first sensor data, such as LIDAR data, RADAR data, wheel encoder data, IMU data, GPS data, and / or Time of flight data. In other examples, the second sensor data may be the same as the first sensor data or may include more data than the first sensor data.

[0087] In operation 314 , the second system 304 may receive the trajectory (or trajectories) from the first system 302 .

[0088] In operation 316, the second system 304 may determine a probability that the vehicle will collide with the object. For example, based on the trajectory and / or the second sensor data, the second system 304 may determine a probability that the trajectory of the autonomous vehicle will intersect with the trajectory / footprint of the object. As described above, such a determination may be based, for example, on a rectilinear (or more complex) physical model of the object detected by the second system.

[0089] In operation 318, the second system 304 may determine whether the probability meets or exceeds a probability threshold. If the second system 304 determines that the probability meets or exceeds the probability threshold (e.g., the "YES" path), the process 300 may proceed to operation 320. Alternatively, if the second system 304 determines that the probability does not meet or exceed the probability threshold (e.g., the "NO" path), the process 300 may proceed to operation 322. In some examples, operation 318 may not determine a probability at all. In such examples, operation 318 may determine whether the trajectory-following vehicle comes within a threshold distance of any detected objects.

[0090] In operation 322, the second system 304 may determine whether an error has occurred in the first system 302. For example, the second system 304 may determine that an error has occurred in the first system 302 by determining that a signal has not been received from the first system 302 for a threshold amount of time or more, determining a hardware and / or software failure in the first system 302, and / or determining that power to the first system 302 has been disrupted. Additionally or alternatively, the second system 304 may determine that an error has occurred in the first system 302 by determining that the trajectory differs from a previous trajectory by more than a threshold amount, determining that the trajectory is incompatible with the vehicle's attitude, determining that the trajectory is incompatible with the vehicle's capabilities, and / or determining that the trajectory controls the vehicle to operate over a surface classified as undriveable.

[0091] If the second system 304 determines that an error occurred in the first system 302 (e.g., the "YES" path), the process 300 may proceed to operation 320. Alternatively, if the second system 304 determines that an error did not occur in the first system 302 (e.g., the "NO" path), the process 300 may proceed to operation 324.

[0092] In operation 320, the second system 304 may cause the vehicle to perform a maneuver such as slowing down, stopping, changing lanes, stopping, turning, or otherwise altering the trajectory. For example, the second system 304 may send a signal to the system controller to cause the vehicle to decelerate along the trajectory (e.g., the altered trajectory). In some examples, if the second system 304 determines in operation 318 that the probability meets or exceeds the threshold, the second system 304 may decelerate the vehicle according to a hard stop maneuver (e.g., a maximum deceleration rate). Further, in some examples, if the second system 304 determines in operation 322 that an error has occurred in the first system 302, the second system 304 may decelerate the vehicle according to a soft stop maneuver (e.g., a deceleration rate less than the maximum deceleration rate). In some examples, the trajectory may be modified (either to the nominal trajectory or to a temporary trajectory) before proceeding to operation 324.

[0093] In operation 324, the second system 302 may cause the vehicle to move along the trajectory. For example, the second system 302 may send a signal to a system controller to cause the vehicle to move along the trajectory (e.g., without making any corrections to the trajectory).

[0094] 4 illustrates a flow diagram of an example process 400 for determining whether to slow down a vehicle. For ease of explanation, the process 400 is described as being performed by a first system 402 and a second system 404. For example, the first system 402 may comprise the primary system 106 of FIGS. 1 and 2, while the second system 404 may comprise the secondary system 108 of FIGS. 1 and 2.

[0095] In operation 406, the first system 402 may receive first sensor data. For example, the first system 402 may receive one or more of LIDAR data, RADAR data, image data, depth data (time of flight, structured light, etc.), etc. from a sensor on the vehicle.

[0096] In operation 408, the first system 402 may perform a first process using the first sensor data. For example, the first system 402 may implement a first technique, such as a machine learning model (e.g., a neural network), to perform first object detection to detect the object, first object segmentation to segment the first sensor data, first object classification to classify the object, and / or first object tracking / prediction to determine a current and / or predicted trajectory of the object. In this manner, the first system 402 may determine a first detection of the object.

[0097] In operation 410, the first system 402 may determine a trajectory for controlling the vehicle. For example, the first system 402 may generate the trajectory for controlling the vehicle based on the first sensor data and / or the first processing.

[0098] In operation 412 , the first system 402 may transmit the trajectory to the second system 404 .

[0099] At operation 414, the second system 404 may receive second sensor data. For example, the second system 404 may receive one or more of LIDAR data, RADAR data, image data, wheel encoder data, IMU data, GPS data, depth data (Time of flight, Structured light, etc.), etc. from sensors on the vehicle. In some examples, the second sensor data may include a subset of the first sensor data. In other examples, the second sensor data may be the same as the first sensor data or may include more data than the first sensor data.

[0100] At operation 416, the second system 404 may filter the second sensor data. In one example, the second system 404 may remove data from the second sensor data that is outside of a certain range. This may be based on the speed or acceleration of the vehicle, the predicted trajectory of the footprint and / or object, the speed or acceleration of the object, etc. In another example, the second system 404 may remove data from the second sensor data based on the type of environment in which the vehicle is located. In this manner, the second system 404 may filter the second sensor data to determine a subset of the second sensor data.

[0101] In operation 418, the second system 404 may perform a second process using the second sensor data (e.g., in a filtered or unfiltered format). For example, the second system 404 may implement a second technique, such as a Kalman filter and / or a particle filter, to perform second object detection to detect the object, second object segmentation to segment the second sensor data, second object classification to classify the object, and / or second object tracking / prediction to determine the object's current and / or predicted footprint. In this manner, the second system 404 may determine a second detection of the object. In an example, in operation 418, the second system 404 may determine the object's velocity, the object's acceleration, the object's size, the object's direction, etc.

[0102] In operation 420 , the second system 404 may receive the trajectory from the first system 402 .

[0103] In operation 422, the second system 404 may determine whether an error occurred with the first system 402 and / or whether there is a potential collision (e.g., the first system 402 failed to detect a collision). Thus, in operation 422, the second system 404 may determine whether an error occurred. Operation 422 may be based on the second sensor data, the trajectory, and / or the second processing. If the second system 404 determines that an error occurred with the first system and / or a potential collision (the "YES" path), the process 400 may proceed to operation 424. Alternatively, if the second system 404 determines that no error occurred with the first system and / or that there is no potential collision (the "NO" path), the process 400 may proceed to operation 426.

[0104] In operation 424, the second system 404 may determine whether the time to last second braking of the object (TTLSB) is within a threshold time. As discussed above, the time to last second braking is the last possible time that the vehicle needs to initiate a quick stop maneuver to avoid a collision. If the second system 404 determines that the time to last second braking is within the threshold time (e.g., seconds, milliseconds, etc. away from the current time) ("YES" path), the process 400 may proceed to operation 428. Alternatively, if the second system 404 determines that the time to last second braking is not within the threshold time ("NO" path), the process 400 may proceed to operation 430.

[0105] 4, in some examples, operation 424 may not be performed. For example, process 400 may proceed from operation 422 to operation 428 along the "YES" path.

[0106] In operation 428, the second system 404 may cause the vehicle to perform a maneuver such as slowing down, stopping, changing lanes, parking, turning, or other trajectory changes. For example, the second system 404 may send a signal to the system controller to cause the vehicle to slow down along the trajectory (e.g., the changed trajectory). In some examples, if the second system 404 determines a potential collision in operation 422, the second system 404 may slow down the vehicle according to a hard stop maneuver. Further, in some examples, if the second system 404 determines that an error has occurred in the first system 402 in operation 422, the second system 404 may slow down the vehicle according to a gradual stop maneuver. In an example, the second system 404 may slow down the vehicle at a rate that depends on the distance from the vehicle to the object.

[0107] In operation 430, the second system 404 may send a signal to the first system 402. The signal may indicate that the second system 404 should slow down or implement other maneuvers if the current trajectory is not corrected. In an example, the signal may indicate when the time is until the second-to-last braking.

[0108] In operation 426, the second system 404 may cause the vehicle to move along the trajectory. For example, the second system 404 may send a signal to a system controller to cause the vehicle to move along the trajectory (e.g., without making any corrections to the trajectory).

[0109] FIG. 5 illustrates a block diagram of an example system 500 for implementing the techniques described herein. In some embodiments, the system 500 may include a vehicle 502 that may correspond to the autonomous vehicle 102 of FIG. 1. In some examples, the vehicle 502 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle that has the capability to perform all safety-critical functions for the entire trip with no driver (or passenger) expected to control the vehicle at any time. However, in other examples, the autonomous vehicle 502 may be a fully or partially autonomous vehicle having any other level or classification. Additionally, in some examples, the techniques described herein may also be applied in non-autonomous vehicles.

[0110] The vehicle 502 may include a first system 504 (e.g., a first computing device), a second system 506 (e.g., a second computing device), one or more sensors 508, one or more emitters 510, one or more communication connections 512, one or more drive systems 514, and / or a directional connection 516 (e.g., for physically coupling the vehicle 502, exchanging data, and / or providing power).

[0111] In some examples, the sensors 508 may include light detection and ranging (LIDAR) sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., global positioning system (GPS), compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., red-green-blue (RGB), infrared (IR), intensity, depth, time of flight, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), and the like. The sensors 508 may include various examples of each of these or other types of sensors. For example, the LIDAR sensors may include individual LIDAR sensors positioned at the corners, front, rear, sides, and / or top of the vehicle 502. As another example, the cameras may include various cameras positioned at various locations on the exterior and / or interior of the vehicle 502. The sensor 508 may provide input to the first system 504 and / or the second system 506 .

[0112] The vehicle 502 may include emitters 510 for emitting light and / or sound, as described above. The emitters 510 in this example may include internal audio and video emitters for communicating with passengers of the vehicle 502. By way of example and not limitation, the internal emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The emitters 510 in this example may also include external emitters. By way of example and not limitation, the external emitters in this example may include lights (e.g., turn signals, signs, light arrays, etc.) to signal a direction of travel or other indicators of the vehicle's movement, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.), one or more of which may include acoustic beam steering technology, for audio communication with pedestrians or other nearby vehicles.

[0113] Vehicle 502 may also include a communication connection 512 having communication between vehicle 502 and one or more other local or remote computing devices. For example, communication connection 512 may facilitate communication with other local computing devices on vehicle 502 and / or drive system 514. Communication connection 512 may also additionally or alternatively have vehicle 502 communicating with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Communication connection 512 may additionally or alternatively have vehicle 502 communicating with computing device 518.

[0114] The communication connection 512 may include physical and / or logical interfaces for connecting the first system 504 and / or the second system 506 to another computing device or network, such as the network 520. For example, the communication connection 512 may include Wi-Fi-based communications, such as over frequencies defined by the IEEE 800.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol by which each computing device interfaces with other computing devices. In some examples, the first system 504, the second system 506, and / or the sensor 508 may transmit sensor data to the computing device 518 over the network 520, at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0115] In some examples, the vehicle 502 may include a drive system 514. In some examples, the vehicle 502 may have a single drive system 514. In some examples, the drive system 514 may include one or more sensors for detecting conditions surrounding the drive system 514 and / or the vehicle 502. By way of example and not limitation, the sensors of the drive system 514 may include one or more wheel encoders (e.g., rotary encoders) that sense the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) that measure the orientation and acceleration of the drive module, cameras or other image sensors, ultrasonic sensors that acoustically detect objects surrounding the drive module, LIDAR sensors, RADAR sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive system 514. In some cases, sensors on the drive system 514 may overlap or complement corresponding systems (e.g., sensors 508) of the vehicle 502.

[0116] The drive system 514 may include many vehicle systems, including a high voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use in other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the perimeter of the vehicle), and one or more systems (e.g., cooling systems, safety systems, on-board charging systems, and electronic components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, the drive system 514 may include a drive system controller for receiving and preprocessing data from sensors and controlling operation of various vehicle systems. In some examples, the drive system controller may include one or more processors and a memory coupled in communication with the one or more processors. The memory may store one or more modules for performing various functions of the drive system 514. Additionally, drive system 514 may include one or more communication connections through which each drive system can communicate with one or more other local or remote computing devices.

[0117] The first system 504 may include one or more processors 522 and a memory 524 coupled in communication with the one or more processors 522. The second system 506 may include one or more processors 526 and a memory 528 coupled in communication with the one or more processors 526. The computing device 518 may also include a processor 530 and / or a memory 532 coupled in communication with the processor 530. The processors 522, 526, and / or 530 may be any suitable processor to process data and execute instructions for performing operations as described herein. By way of example and not limitation, processors 522, 526, and / or 530 may comprise one or more central processing units (CPUs), graphics processing units (GPUs), integrated circuits (e.g., application specific integrated circuits (ASICs), gate arrays (e.g., field programmable gate arrays (FPGAs)), and / or any other device or portion of a device that processes electronic data and converts that electronic data into other electronic data that may be stored in registers and / or memory.

[0118] The memory 524, 528, and / or 532 may be examples of non-transitory computer-readable media. The memory 524, 528, and / or 532 may store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods and functions attributed to the various systems described herein. In various implementations, the memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or other types of memory having the capacity to store information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which the ones shown in the accompanying drawings are merely examples relevant to the discussion herein.

[0119] In an example, the memory 524 of the first system 504 may store one or more components 534. For example, the first system 504 may correspond to the primary system 106 of Figures 1 and 2 and store the localization component 202, the perception component 204, the prediction component 206, and / or the planning component 208. The processor 522 may execute the one or more components 534 to cause the first system 504 to perform various operations discussed herein.

[0120] In an example, the memory 528 of the second system 506 may store one or more components 536. For example, the second system 506 may correspond to the sub-system 108 of Figures 1 and 2 and store the estimation and verification component 210, the data processing component 212, the localization component 214, the perception / prediction component 216, the trajectory verification component 218, the trajectory correction component 220, and / or the trajectory selection component 222. The processor 526 may execute the one or more components 536 to cause the second system 506 to perform various operations discussed herein.

[0121] 5 as residing in memory 524 and / or memory 528 for illustrative purposes, it is contemplated that component 534 and / or component 536 may additionally or alternatively have access to computing device 518 (e.g., be stored remotely). For example, memory 532 may store a first system component 538 that corresponds to at least a portion of component 534 and / or a second system component 540 that corresponds to at least a portion of component 536. Additionally or alternatively, memory 532 may store one or more maps 542.

[0122] In at least one example, the first system 504 and / or the second system 506 may include one or more system controllers 544, which may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 502. In an example, the system controller 544 may be stored in the memory 524 and / or the memory 528. The system controller 544 may communicate with and / or control corresponding systems of the drive system 514 and / or other components of the vehicle 502. In some examples, the system controller 544 may convert a trajectory generated by the first system 504, generated by the second system 506, and / or selected by the second system 506 into instructions usable by the drive system 514 to cause the vehicle 502 to traverse the trajectory.

[0123] In some examples, the first system 504, the second system 506, the system controller 544, and / or any components thereof may process the sensor data as described above and transmit respective outputs (e.g., at a particular frequency, after a predetermined period of time, in near real-time, etc.) to the computing device 518 via the network 520.

[0124] The first system 504 and / or the second system 506 may store one or more maps 546 that may be used by the vehicle 502 to navigate within the environment. For purposes of this discussion, a map may have any number of data characterized in two, three, or N dimensions and provide information about the environment, such as topology (e.g., intersections), streets, mountain ranges, roads, terrain, and the general environment. In some examples, without limitation, the map may include texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and / or intensities), reflectance information (e.g., specularity information, retroreflectance information, BRDF information, BSSRDF information, etc.). In one example, the map may include a three-dimensional mesh of the environment. In some examples, the map may be stored in a tiled format, with individual tiles of the map representing discrete portions of the environment, and may be loaded into the working memory as needed. In some examples, the map may include an occupancy grid, a point map, a landmark map, and / or a graph of pose constraints. In some examples, the vehicle 502 may be controlled based at least in part on the map 546. That is, the map 546 may be used in combination with a localization component of the first system 504 (and / or, in some examples, a localization component of the second system 506) to determine a position of the vehicle 502, identify objects in the environment, and / or generate a route and / or trajectory for navigating the environment.

[0125] In some examples, some or all aspects of the components discussed herein may include any model, algorithm, and / or machine learning algorithm. For example, the components 534 in memory 524 (and / or memory 528, or 532) may be implemented as a neural network.

[0126] 5 is illustrated as a distributed system, it should be noted that in alternative examples, components of the vehicle 502 may be associated with the computing device 518 and / or components of the computing device 518 may be associated with the vehicle 502. That is, the vehicle 502 may perform one or more functions associated with the computing device 518, or vice versa.

[0127] Examples A. An exemplary vehicle system, comprising: a first system comprising one or more sensors; one or more first processors; and one or more first memories, the memory comprising instructions that, when executed by the one or more first processors, cause the first processor to receive first sensor data from a first subset of the one or more sensors and determine a trajectory for controlling an autonomous vehicle based at least in part on the first sensor data; and a second system comprising one or more second processors and one or more second memories, the memory comprising instructions that, when executed by the one or more second processors, cause the first processor to receive first sensor data from a first subset of the one or more sensors and determine a trajectory for controlling an autonomous vehicle based at least in part on the first sensor data. or a second system comprising: a memory comprising instructions for a second processor to: receive a trajectory from the first system; receive second sensor data from a second subset of the one or more sensors; determine a probability that the autonomous vehicle will collide with an object based at least in part on the trajectory and the second sensor data; determine at least one of an error occurred in the first system or the probability meets or exceeds a probability threshold; and cause the vehicle to slow along the trajectory based at least in part on determining that an error occurred in the first system or the probability meets or exceeds a probability threshold.

[0128] B. The vehicle system of Example A, wherein the first system implements a machine learning model to perform at least one of detecting a first object, segmenting the first object, classifying the first object, or tracking the first object, and the second system implements a probabilistic model to perform at least one of detecting a second object, segmenting the second object, classifying the second object, or tracking the second object.

[0129] C. The vehicle system of Example A or Example B, wherein determining at least one of an error has occurred in the first system or the probability meets or exceeds a probability threshold comprises determining that the probability meets or exceeds a probability threshold, and slowing the autonomous vehicle comprises slowing the autonomous vehicle according to a maximum deceleration rate.

[0130] D. The vehicle system of any of Examples A to C, wherein determining at least one of whether an error occurred in the first system or whether the probability meets or exceeds a probability threshold comprises determining that an error occurred in the first system, and wherein slowing the autonomous vehicle comprises slowing the autonomous vehicle at less than a maximum deceleration rate.

[0131] E. The vehicle system of any of Examples A to D, wherein determining at least one of that an error has occurred in the first system or that the probability meets or exceeds a probability threshold comprises determining that an error has occurred in the first system by at least one of determining that a signal has not been received from the first system for a time period equal to or greater than a threshold, determining at least one of a hardware or software failure in the first system, or determining that power to the first system has been interrupted.

[0132] F. The vehicle system of any of Examples A to E, wherein determining that an error has occurred in the first system or that the probability meets or exceeds a probability threshold comprises determining that an error has occurred in the first system by at least one of the following: determining that the trajectory differs from a previous trajectory by more than a threshold, determining that the trajectory is incompatible with the attitude of the autonomous vehicle, determining that the trajectory is incompatible with the performance of the autonomous vehicle, or determining that the trajectory controls the autonomous vehicle to operate over a surface classified as undriveable.

[0133] G. An exemplary method comprising receiving sensor data from a sensor in a first system associated with a vehicle; determining a first detection of an object in an environment proximate to the vehicle by a first system associated with the vehicle based at least in part on the sensor data; determining a second detection of the object by a second system associated with the vehicle based at least in part on a subset of the sensor data; determining at least one of an error in the first system or a potential collision with the object; and decelerating the vehicle along a trajectory by the second system based at least in part on determining at least one of the error in the first system or the potential collision with the object.

[0134] H. The method of Example G, wherein the first system implements a machine learning model to perform at least one of detecting a first object, segmenting the first object, classifying the first object, or tracking the first object, and the second system implements a probabilistic model to perform at least one of detecting a second object, segmenting the second object, classifying the second object, or tracking the second object.

[0135] I. The method of example G or example H wherein the second system implements at least two of a Kalman filter or a particle filter.

[0136] J. The method of any of Examples G through I, wherein the subset of sensor data comprises at least one of LIDAR data or RADAR data.

[0137] K. The method of any of Examples G through J, wherein determining a second detection of the object comprises determining at least one of an object velocity, an object acceleration, an object size, or an object orientation, and determining at least one of an error in the first system or a potential collision with the object is based on at least one of the object velocity, the object acceleration, the object size, or the object orientation.

[0138] L. The method of any of Examples G through K, wherein determining the second detection of the object comprises determining a current or predicted trajectory of the object, and wherein determining at least one of an error of the first system or a likelihood of a collision with the object is based at least in part on the predicted trajectory of the object.

[0139] M. The method of any of Examples G to L, wherein determining at least one of an error with the first system or a potential collision with the object comprises determining a potential collision with the object, and decelerating the vehicle comprises decelerating the vehicle according to a maximum deceleration rate.

[0140] N. The method of any of Examples G to M, wherein determining at least one of an error of the first system or a collision with an object comprises determining an error of the first system, and decelerating the vehicle comprises decelerating the vehicle according to a deceleration rate that is less than a maximum deceleration rate.

[0141] O. One or more exemplary non-transitory computer readable media storing instructions that, when executed, cause one or more processors to perform operations comprising receiving sensor data from a sensor associated with the vehicle; receiving a trajectory from a first system associated with the vehicle; detecting an object in an environment proximate to the vehicle based at least in part on the sensor data; determining at least one of an error in the first system or a potential collision with the object; and causing the vehicle to perform an operation based at least in part on determining at least one of the error in the first system or a potential collision with the object.

[0142] P. One or more non-transitory computer-readable media of Example O, wherein a first system implements a neural network to perform at least one of detecting a first object, segmenting the first object, classifying the first object, or tracking the first object, and a second system performs at least one of detecting a second object, segmenting the second object, classifying the second object, or tracking the second object.

[0143] Q. One or more non-transitory computer-readable media of any of Example O or Example P, wherein determining at least one of an error in the first system or a potential collision with an object comprises determining a potential collision with an object, and causing the vehicle to perform the maneuver comprises decelerating the vehicle according to a maximum deceleration rate.

[0144] R. The non-transitory computer-readable medium of one or more of Examples O through Q, wherein determining at least one of an error in the first system or a potential collision with the object comprises determining an error in the first system, and causing the vehicle to perform the maneuver comprises decelerating the vehicle according to a deceleration rate that is less than a maximum deceleration rate.

[0145] S. The non-transitory computer readable medium of one or more of Examples O through R, wherein the operation further comprises filtering the sensor data based on at least one of an object velocity, an object acceleration, a vehicle velocity, or a vehicle acceleration to determine a subset of the sensor data, and detecting the object is based at least in part on the subset of the sensor data.

[0146] T. The non-transitory computer readable medium of one or more of Examples O through R, wherein causing the vehicle to perform the operation comprises decelerating the vehicle at a rate based at least in part on a distance from the vehicle to the object.

[0147] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

[0148] The modules described herein represent instructions that may be stored on any type of computer-readable medium and may be implemented in software and / or hardware. All of the methods and processes described above may be embodied in and fully automated through software code modules and / or computer-executable instructions executed by one or more computers or processors, hardware, or some combination thereof. Some or all of the methods may alternatively be embodied in specialized computer hardware.

[0149] In particular, conditional language such as "may," "would," "may," or "would" is understood, within the context, to indicate that a particular example includes a particular feature, element, and / or step, while other examples do not, unless specifically stated otherwise. Thus, such conditional language is not intended to imply that a particular feature, element, and / or step is somehow required for one or more examples, or that one or more examples necessarily include logic for determining, with or without user input or prompting, whether a particular feature, element, and / or step is included or performed in a particular example.

[0150] Conjunction language, such as the phrase "at least one of X, Y, or Z," should be understood to indicate that the item, term, etc. can be either X, Y, or Z, or any combination thereof, including multiples of each element, unless specifically stated otherwise. "a" refers to the singular as well as the plural, unless expressly stated in the singular.

[0151] It should be understood that any routine description, element or block in the flow diagrams described herein and / or depicted in the accompanying figures may represent a module, segment or portion of code that includes one or more computer-executable instructions for implementing a particular logical function or element in the routine. As will be appreciated by those skilled in the art, alternative implementations are within the scope of the examples described herein that may be performed in a different order than that shown or discussed, with elements or functions removed, substantially synchronized, in reverse order, with additional operations, or with the omission of operations, depending on the functionality involved.

[0152] Many variations and modifications may be made to the above-described example, and it should be understood that the elements are among the other acceptable examples. All such modifications and modifications are intended to be included herein within the scope of this disclosure and are protected by the following claims.

[0153] conclusion One or more examples of the technology described herein have been described; however, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0154] In the description of the examples, reference is made to the accompanying figures, which form a part hereof, and which illustrate certain embodiments of the claimed subject matter. It is to be understood that other examples may be used and that modifications or alterations, such as structural changes, may be made. Such examples, modifications or alterations do not necessarily depart from the intended scope of the claimed subject matter. Although the procedures herein may be presented in a certain order, in some cases the order may be changed such that certain inputs are provided at different times or in a different order without changing the functionality of the described systems and methods. Also, the disclosed procedures may be performed in a different order. Furthermore, the various calculations described herein need not be performed in the order disclosed, and other examples using alternative orders of calculations may be readily implemented. In addition to changing the order, the calculations may be decomposed into sub-calculations with the same results.

Claims

1. 1. A method executed by one or more processors, comprising: receiving sensor data from a sensor at a first system associated with the vehicle; determining, by the first system, a first detection of an object in an environment proximate to the vehicle and a classification of the object based at least in part on inputting the sensor data into a machine learning model; determining, by the first system, a trajectory of the vehicle based at least in part on the sensor data and the first detection; and determining a second detection of the object by a second system associated with the vehicle based at least in part on the subset of sensor data; and determining a probability that the vehicle will collide with the object based at least in part on the second detection and the trajectory; and determining, by the second system, at least one of an error in the first system or the probability meeting or exceeding a probability threshold; slowing, by the second system, the vehicle along the trajectory based at least in part on determining at least one of the error in the first system or the probability meeting or exceeding the probability threshold.

2. the first system implements a machine learning model for performing at least one of segmenting the object or tracking the object; The method of claim 1 , wherein the second system implements a probabilistic model for performing at least one of segmenting the object, classifying the object, or tracking the object.

3. The method of claim 1 or 2, wherein the second system implements at least one of a Kalman filter or a particle filter.

4. filtering the sensor data based on at least one of a velocity of the object, an acceleration of the object, a velocity of the vehicle, or an acceleration of the vehicle to determine the subset of sensor data; The method of claim 1 , wherein the detecting the object is based at least in part on the subset of the sensor data.

5. determining the second detection of the object comprises determining at least one of a velocity of the object, an acceleration of the object, a size of the object, or a direction of the object; 5. The method of claim 1, wherein determining at least one of the error of the first system or the probability that the probability threshold is met or exceeded is based on at least one of the velocity of the object, the acceleration of the object, the size of the object, or the orientation of the object.

6. determining the second detection of the object comprises determining a current or predicted trajectory of the object; 5. The method of claim 1, wherein determining at least one of the error of the first system or the probability that the probability threshold meets or exceeds is based at least in part on the predicted trajectory of the object.

7. A method according to any one of claims 1 to 6, wherein decelerating the vehicle comprises decelerating the vehicle according to a maximum deceleration rate.

8. A method according to any one of claims 1 to 6, wherein slowing down the vehicle comprises slowing down the vehicle based at least in part on a distance from the vehicle to the object.

9. At least one non-transitory computer-readable medium, A non-transitory computer readable medium storing instructions that, when executed, cause one or more processors to perform the method of any one of claims 1 to 8.

10. A vehicle, a memory storing instructions that, when executed, cause one or more processors of the second system to perform operations; The operation includes: receiving sensor data from a sensor at a first system associated with the vehicle; determining, by the first system, a first detection of an object in an environment proximate to the vehicle and a classification of the object based at least in part on inputting the sensor data into a machine learning model; determining, by the first system, a trajectory of the vehicle based at least in part on the sensor data and the first detection; and determining a second detection of the object by the second system associated with the vehicle based at least in part on the subset of sensor data; and determining a probability that the vehicle will collide with the object based at least in part on the second detection and the trajectory; and determining, by the second system, at least one of an error of the first system or that the probability meets or exceeds a probability threshold; decelerating, by the second system, the vehicle along the trajectory based at least in part on determining at least one of the error of the first system or the probability meeting or exceeding the probability threshold; A vehicle equipped with:

11. the first system implements a neural network to perform at least one of segmenting the object and tracking the object; The vehicle of claim 10 , wherein the second system performs at least one of: segmenting the object, classifying the object, and tracking the object.

12. A vehicle as described in any one of claims 10 or 11, wherein decelerating the vehicle comprises decelerating the vehicle according to a maximum deceleration rate.

13. A vehicle as described in any one of claims 10 or 11, wherein decelerating the vehicle comprises decelerating the vehicle according to a deceleration rate that is less than a maximum deceleration rate.

14. The operation includes: filtering the sensor data based on at least one of a velocity of the object, an acceleration of the object, a velocity of the vehicle, or an acceleration of the vehicle to determine the subset of sensor data; The vehicle of claim 10 or 13, wherein the detection of the object is based at least in part on the subset of the sensor data.

15. 15. The vehicle of claim 10, wherein the slowing down of the vehicle comprises slowing down the vehicle at a rate based at least in part on a distance from the vehicle to the object.

Images