Collision Avoidance Systems

Abstract

The vehicle includes a primary system for generating data to control the vehicle and a secondary system for verifying collision avoidance data and / or other data. For example, the primary system may localize the vehicle, detect objects around the vehicle, predict object trajectories, and generate a trajectory for the vehicle. The secondary system may localize the vehicle, detect objects around the vehicle, predict object trajectories, and evaluate the trajectory generated by the primary system. Additionally, the secondary system may monitor vehicle components to detect errors. If the secondary system detects an error with the trajectory generated by the primary system and / or an error with a vehicle component, the secondary system may cause the vehicle to take action, such as slowing down, changing lanes, swerving, etc.

Description

[Technical field]

[0001] The present invention relates to a collision avoidance system, and more particularly to a collision avoidance system that controls a vehicle. [Background technology]

[0002] Related Applications This PCT international application claims the benefit of priority to U.S. Patent Application No. 16 / 232,863, filed December 26, 2018, which is incorporated herein by reference.

[0003] The safety of the occupants of the vehicle and other people or objects in the vicinity of the vehicle is of paramount importance. Such safety is often predicated on accurate detection of potential collisions and timely deployment of safety measures. Although autonomous vehicles are often implemented with systems that have highly effective collision detection systems, on rare occasions, these systems may not work or may be ineffective. For example, an error may occur in a relatively long and potentially complex processing pipeline for an on-vehicle system, causing the vehicle to move in an unsafe manner.

[0004] The detailed description will now be described with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different figures indicates similar or identical components or features. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] U.S. Patent Application Serial No. 16 / 189,726 [Patent Document 2] U.S. Patent Application Serial No. 16 / 218,182 [Patent Document 3] U.S. Patent Application Serial No. 15 / 897,028 [Patent Document 4] U.S. Patent Application Serial No. 15 / 947,486 [Patent Document 5] U.S. Patent Application Serial No. 15 / 622,905 [Brief description of the drawings]

[0006] [Figure 1] 1 illustrates an example environment in which the techniques described herein may be implemented. [Diagram 2] 1 depicts exemplary details of a primary and secondary system of a vehicle. [Diagram 3] 1 illustrates an example process for determining whether a trajectory for a vehicle is valid or invalid. [Figure 4] 1 illustrates an example process for determining whether a trajectory for a vehicle is valid or invalid. [Diagram 5] 1 illustrates an example process for determining whether a trajectory for a vehicle is valid or invalid. [Figure 6] 1 illustrates an exemplary hierarchy of states that may be implemented by a secondary system. [Figure 7A] 1 illustrates an exemplary process for performing the techniques described herein. [Figure 7B] 1 illustrates an exemplary process for performing the techniques described herein. [Figure 8] Illustrates example processing that may be performed by various components of the primary and / or secondary systems. [Figure 9] Illustrates example processing that may be performed by various components of the primary and / or secondary systems. [Figure 10] Illustrates example processing that may be performed by various components of the primary and / or secondary systems. [Figure 11] 1 illustrates a block diagram of an example system for implementing the techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The present disclosure is directed to a vehicle including a primary system for generating data to control the vehicle and a secondary system that validates the data and / or other data to avoid collisions. For example, the primary system can localize the vehicle, detect objects around the vehicle, segment sensor data, determine object classifications, predict object trajectories, generate trajectories for the vehicle, etc. The secondary system can independently localize the vehicle, detect objects around the vehicle, predict object trajectories, evaluate trajectories generated by the primary system, etc. In examples, the secondary system can also monitor components of the vehicle to detect errors. If the secondary system detects an error with the trajectory generated by the primary system and / or an error with components of the vehicle, the secondary system can cause the vehicle to perform actions such as slowing down, changing lanes, swerving, etc. In examples, the secondary system can send information (e.g., information regarding a potential collision) to the primary system. In many examples, the techniques discussed herein may be implemented to avoid potential collisions with objects around the vehicle. Although described herein as primary and secondary systems, it should be understood that the described techniques may be implemented in any number of systems and subsystems to verify control and provide high integrity algorithms and redundant processes for safety control.

[0008] The primary system may generally perform processing to control how to move a vehicle within an environment. The primary system may implement various artificial intelligence (AI) techniques, such as machine learning, to understand the environment around the vehicle and / or to instruct the vehicle to move within the environment. For example, the primary system may implement AI techniques to localize the vehicle, detect objects around the vehicle, segment sensor data, determine object classifications, determine object tracks, generate a trajectory for the vehicle, etc. In one example, the primary system generates a primary trajectory for controlling the vehicle and a secondary contingent trajectory for controlling the vehicle, and provides the primary trajectory and the secondary trajectory to the secondary system. The contingent trajectory may control the vehicle to come to a stop and / or to perform another movement (e.g., lane change, etc.).

[0009] A secondary system can generally evaluate the primary system using at least a subset of data (e.g., sensor data) made available to the primary system. The secondary system can use similar techniques to those used in the primary system to verify the output of the primary system and / or use dissimilar techniques to ensure consistency and verifiability of such output. In an example, the secondary system can include a localizer for independently localizing the vehicle by determining the position and / or orientation (together attitude) of the vehicle relative to points and / or objects in the environment in which the vehicle is positioned. The secondary system can also include a perceiver for detecting objects around the vehicle, determining tracks for the objects, predicting trajectories for the objects, etc. The secondary system can include a monitor component for monitoring one or more components of the vehicle to detect errors related to the one or more components. Additionally, the secondary system can include a trajectory manager for evaluating the trajectory of the vehicle provided by the primary system and / or for determining a trajectory to use to control the vehicle using data from the localization component, the perceiver, and / or the monitor component of the secondary system. The secondary system may also include a drive manager (and / or a system controller) for receiving the trajectory from the trajectory manager and controlling the vehicle based on the trajectory.

[0010] To illustrate, the trajectory manager can perform one or more operations to evaluate (or verify) the trajectory provided by the primary system. For example, the secondary system can check to see if the trajectory was generated less than a threshold amount of time ago, whether the trajectory is consistent with the vehicle's current or previous attitude (e.g., the trajectory controls the vehicle to be positioned where it is possible given the vehicle's current attitude), whether the trajectory is compatible with the vehicle's performance (e.g., steering limits, acceleration limits, etc.), etc. Additionally, the trajectory manager can check to see if the trajectory is associated with a collision. For example, the trajectory manager can check to see if the vehicle's trajectory provided by the primary system intersects with the trajectory of an object determined by a perceiver of the secondary system, and whether the object and the vehicle meet at an intersection at the same time (or within a time frame) and / or within a threshold distance from each other. That is, the trajectory manager can determine whether the vehicle will collide with the object if it is maintained along the trajectory provided by the primary system, and / or the associated uncertainty of such an event. Such collision checking may be based on direct kinematic assumptions of the movement and / or on predictions of the motion determined by one or more additional techniques.

[0011] In an example, the trajectory manager can evaluate each trajectory of the vehicle provided by the primary system. For example, the trajectory manager can evaluate a primary trajectory (e.g., the highest level trajectory) generated by the primary system and a secondary trajectory (e.g., the next highest level trajectory) generated by the primary system. The secondary trajectory may be a contingent trajectory that may be used in cases where the primary trajectory is invalid (e.g., does not provide a safe trajectory for the vehicle, is incompatible with the current vehicle state, etc.). Further, in an example, the trajectory manager can generate a tertiary trajectory (e.g., the next highest level trajectory) for the vehicle and evaluate the tertiary trajectory. The tertiary trajectory may be a further contingent trajectory that may be used in cases where the primary and secondary trajectories are invalid. The tertiary trajectory may bring the vehicle to a stop along the secondary trajectory (or the primary trajectory in some cases) or cause the vehicle to perform another movement (e.g., lane change, swerve, etc.). In some examples, the tertiary trajectory may be determined based at least in part on a minimum correction of the secondary trajectory (e.g., correction to longitudinal acceleration, steering angle, lateral acceleration, etc.).

[0012] The trajectory manager may determine a trajectory to use to control the vehicle based on the evaluation of the trajectory. For example, if the state of the primary trajectory indicates that the primary trajectory is associated with no collision or a collision "far" ahead (e.g., the primary trajectory is temporarily valid), the trajectory manager may select the primary trajectory provided by the primary system. Furthermore, if the primary trajectory is associated with a state indicating that a collision is imminent and the secondary trajectory is associated with a state indicating that the secondary trajectory is collision-free, the trajectory manager may select the tertiary trajectory generated by the trajectory manager. Furthermore, if an error is detected by the monitor, the trajectory manager may select the secondary or tertiary trajectory. In any of the above examples, collision-free may refer to the probability of a collision (and / or being within a threshold distance of another object for a time frame) being less than or equal to a threshold probability.

[0013] Upon selecting a trajectory, the trajectory manager can send a signal to the drive manager (and / or system controller) to cause the vehicle to be controlled based on the trajectory. In at least some examples, if there is no collision-free trajectory, the secondary system can cause the vehicle to take one or more actions, such as, for example, engaging the emergency brakes, tensioning the seat belts, etc.

[0014] In an example, the trajectory manager can maintain control of the vehicle based on the selected trajectory until a signal is received to release control from the selected trajectory. For example, the trajectory manager can transition to a lower level trajectory whenever needed and refrain from transitioning back to the higher level trajectory until a signal is received to release control to the higher level trajectory. To illustrate, if the trajectory manager selects a secondary trajectory because the primary trajectory is associated with a collision, control of the vehicle may be maintained along the secondary trajectory (or an even lower level trajectory) until a signal is received from a remote operation system (e.g., a system associated with an operator) to release control to the higher level trajectory. By doing so, the vehicle can avoid frequent changes between trajectories.

[0015] Further, in examples, the trajectory manager can provide messages to the primary system indicating errors with respect to the trajectory provided by the primary system. For example, if the trajectory manager determines that a collision is estimated to occur far enough ahead (e.g., more than a threshold amount of time away) that the vehicle would not need to brake immediately if the vehicle were to proceed along the primary trajectory provided by the primary system, the secondary system can send a message to the primary system to alert the primary system. This can enable the primary system to adjust the primary trajectory before a collision occurs.

[0016] In at least some examples, one or more of the calculations performed by the secondary system (including warning messages, trajectory validation, perception information, tracking information, etc.) may be continuously transmitted to the primary system at one or more frequencies (which may depend on the data being transmitted).

[0017] In examples, the techniques and / or systems discussed herein can enhance the safety of occupants in a vehicle and / or other individuals proximate to the vehicle. For example, the secondary system can detect errors in the trajectory provided by the primary system and / or errors with respect to components of the vehicle and control the vehicle to safely slow down, stop, and / or perform another movement to avoid a collision. In examples, the secondary system can operate relatively independently from the primary system, resulting in another form of evaluation to avoid a collision. For example, the secondary system can independently detect objects proximate to the vehicle, monitor components of the vehicle, and / or evaluate the trajectory generated by the primary system. Furthermore, in examples, the secondary system can be a system of higher integrity (e.g., more verifiable) and / or less complexity than the primary system. For example, the secondary system can be designed to process less data than the primary system, include a short processing pipeline, operate according to techniques that are more easily verifiable than those of the primary system, etc.

[0018] In examples, the techniques discussed in this specification may implement techniques discussed in U.S. Patent Application Publication No. 2015 / 0133634, filed November 11, 2018, entitled “Perception Collision Avoidance,” and / or U.S. Patent Application Publication No. 2015 / 0133634, filed December 12, 2018, entitled “Collision Avoidance System with Trajectory Validation,” the entire contents of both of which are incorporated herein by reference.

[0019] The methods, apparatus, and systems described herein may be implemented in a number of ways. Example implementations are provided below with reference to the following figures. Although discussed in the context of autonomous vehicles, in some examples, the methods, apparatus, and systems described herein may be applied to a variety of systems. In another example, the methods, apparatus, and systems may be utilized in an aviation or nautical context. Additionally or alternatively, the techniques described herein may be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination thereof.

[0020] FIG. 1 illustrates an example environment 100 in which the techniques discussed herein may be implemented. The example environment 100 shows an autonomous vehicle 102 including sensors 104, a primary system 106, a secondary system 108, and a system controller 110. The primary system 106 may generally implement artificial intelligence (e.g., AI techniques 112) to perform various operations, and the secondary system 108 may generally implement different techniques to evaluate the primary system 106 and take over control when needed. For example, the primary system 106 may process sensor data 114 from the sensors 104 to generate a primary trajectory 116 and a secondary trajectory 118. The primary system 106 may send the primary trajectory 116 and the secondary trajectory 118 to the secondary system 108. The secondary system 108 may process sensor data 120 from the sensors 104 to evaluate the primary trajectory 116 and / or the secondary trajectory 118. In examples, the techniques implemented by the secondary system 108 may include techniques similar to those employed by the primary system 106, and the results may be compared to determine similarity. In other examples, the secondary system 108 may use techniques dissimilar to those in the primary system 106, such that similar results may indicate that both systems are functioning correctly. Based on the evaluation, the secondary system 108 may select an output trajectory 122 and send the output trajectory 122 to the system controller 110 to control the autonomous vehicle 102. The output trajectory 122 may include the primary trajectory 116, the secondary trajectory 118, and / or another trajectory, such as a trajectory generated by the secondary system 108 as discussed in detail below. In the example of FIG. 1, the primary trajectory 116 and the secondary trajectory 118 are illustrated as being sent from the primary system 106. However, in some examples, a single trajectory may be sent by the primary system 106, and / or any number of additional trajectories may be sent by additional systems not shown.

[0021] Although many of the techniques are discussed in the context of the secondary system 108 sending signals to the system controller 110 (e.g., sending the output trajectory 122), in some cases the primary system 106 may alternatively or additionally send signals to the system controller 110.

[0022] The sensors 104 may include a variety of sensors such as light detection and ranging (LIDAR) sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., Global Navigation Satellite Systems (GNSS) (including Global Positioning System (GPS)), compasses, etc.), inertial sensors (e.g., Inertial Measurement Units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., red-green-blue (RGB), infrared (IR), luminance, depth, etc.), time of flight sensors, microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc.

[0023] As noted above, the primary system 106 may generally process the sensor data 114, and the secondary system 108 may process the sensor data 120. In some examples, the sensor data 120 includes a subset of the sensor data 114. To illustrate, the sensor data 114 may include data from a wide variety of sensors 104, such as position data, inertial data, LIDAR data, RADAR data, image data, audio data, environmental data, depth data, etc. Meanwhile, the sensor data 120 may include data from a more limited set of sensors 104, such as LIDAR data, RADAR data, and / or time-of-flight data. In other examples, the sensor data 120 includes the same or more data than the sensor data 114, such as one or more of the position data, inertial data, LIDAR data, RADAR data, image data, audio data, environmental data, and / or depth data. In at least some examples, the subsets of sensor data 120 used by the secondary system 108 may be based on one or more processes and may differ from one another, for example. As a non-limiting example, the primary system 106 may use all sensors for localization and object detection, whereas localization in the secondary system 108 may use only IMU and wheel odometry data, and object detection in the secondary system 108 may use only LIDAR data.

[0024] The primary system 106 may control the vehicle during normal operation. In many examples, the primary system 106 may be the main computing system on the autonomous vehicle 102. The primary system 106 may implement AI techniques 112 to understand the environment around the autonomous vehicle 102 and / or instruct the autonomous vehicle 102 to move within the environment. The AI ​​techniques 112 may include machine learning (also commonly referred to as machine-learned models), such as one or more neural networks. For example, the primary system 106 may analyze sensor data 114 to localize the autonomous vehicle 102, detect objects around the autonomous vehicle 102, segment the sensor data 114, determine object classifications, predict object tracks, generate primary trajectories 116 and / or secondary trajectories 118 for the autonomous vehicle 102, etc. 1, the primary system 106 generates a primary trajectory 116 for controlling the autonomous vehicle 102 and generates a secondary trajectory 118 as a contingency (e.g., backup) trajectory to be used in place of the primary trajectory if the primary trajectory 116 is unjustified or otherwise unacceptable, or to be otherwise executed in case of hardware or software failure, etc. In an example, the secondary trajectory 118 can control the autonomous vehicle 102 to decelerate to a stop (e.g., a gentle stop associated with a deceleration rate lower than a maximum deceleration rate possible for the autonomous vehicle 102). In at least some examples, the secondary trajectory 118 can include complex movements (including steering, accelerating, etc.) to facilitate the vehicle 102 moving to the side of the road, avoiding obstacles, or generally increasing the amount of safety for the occupants.

[0025] In examples, the trajectory includes control over the steering angle and / or acceleration of the autonomous vehicle 102. Additionally, the trajectory may include a sequence of time and vehicle states (e.g., attitude) that represent an estimated future path of the autonomous vehicle 102. For example, the trajectory may control the path that the autonomous vehicle 102 will take over a time frame (e.g., 1 millisecond, half a second, 2 seconds, 10 seconds, etc.). The trajectory may be continuously updated over time to control the motion of the autonomous vehicle 102.

[0026] The secondary system 108 can evaluate the primary system 106 and take over control of the autonomous vehicle 102 when the primary system 106 has issues. The secondary system 108 can implement secondary techniques (e.g., probabilistic techniques, techniques different from the AI ​​techniques 112, etc.) based on the position, velocity, acceleration, etc. of the autonomous vehicle 102 and / or objects around the autonomous vehicle 102. For example, the secondary system 108 can implement a Kalman filter, an extended Kalman filter (EKF), a particle filter, and / or a tracking algorithm to process sensor data, segment the sensor data, identify object classifications, predict object trajectories, etc. In examples, the secondary system 108 may not implement any form of AI techniques. In some examples, the AI ​​techniques 112 may use neural networks, while the secondary system 108 may refrain from using such neural networks. In other examples, the secondary system 108 may implement AI techniques, such as by implementing a different (or in some instances the same) machine-learned model than the machine-learned model implemented by the primary system 106. Although a single secondary system 108 is illustrated, in examples, the secondary system 108 may be implemented as multiple systems, such as multiple hardware / software systems. In examples, the secondary system 108 may have a different range of vision than the primary system 106 (e.g., detect and / or process sensor data at a different range than the primary system 106), operate at a different rate of processing than the primary system 106 (e.g., process instructions at a faster (or in some cases slower) rate than the primary system 106), etc.

[0027] In an example, to evaluate the primary system 106, the secondary system 108 can process the sensor data 120 to determine the position and / or orientation (together, the attitude) of the autonomous vehicle 102 relative to points and / or objects in the environment. Additionally, the secondary system 108 can process the sensor data 120 to detect objects around the autonomous vehicle 102, follow the tracks of the objects over time, and / or predict trajectories for the objects. In some examples, the secondary system 108 can receive information from the primary system 106 indicative of tracks of objects detected by the primary system 106 and / or predicted trajectories of the objects. The tracks of the objects can include the path traveled by the object (e.g., previous states-position, orientation, speed, etc., as well as center position, breadth, etc., and / or uncertainties associated therewith). The tracks of the objects can represent (or be based on) the current or previous positions, speeds, accelerations, orientations, and / or heading of the object over a period of time (e.g., 5 seconds). The secondary system 108 can maintain tracks and / or predicted trajectories for each object detected around the autonomous vehicle 102. 1, the secondary system 108 estimates a trajectory 124 for an object 126, where the object 126 represents a person, but the object 126 may represent any type of object, such as another vehicle, a structure (e.g., a building, etc.), an animal, a sign, etc.

[0028] The secondary system 108 may use the attitude, track, and / or predicted trajectory to evaluate the primary trajectory 116 and / or the secondary trajectory 118. The secondary system 108 may perform one or more validation operations to validate a trajectory generated by the secondary system 108, such as the primary trajectory 116, the secondary trajectory 118, and / or a collision avoidance trajectory (e.g., a trajectory that slows the autonomous vehicle 102 to a stop, which may be a modified version of either the primary trajectory 116 or the secondary trajectory 118). In an example, the secondary system 108 may validate one or more of the primary trajectory 116, the secondary trajectory 118, and / or a trajectory generated by the secondary system 108 (e.g., a collision avoidance trajectory) in parallel. Although, any of such trajectories may be validated sequentially or otherwise.

[0029] In an example, the secondary system 108 may perform one or more validation actions to determine whether the trajectory is relatively new, consistent with the state of the autonomous vehicle 102, and / or feasible given the limits of the autonomous vehicle 102. For example, the secondary system 108 may determine whether the trajectory was generated less than a threshold amount of time ago (e.g., indicating that the trajectory is relatively new and should be used), whether the trajectory is consistent with a current or previous position of the vehicle (e.g., the trajectory controls the autonomous vehicle 102 to position itself in a position that is possible given the current position of the autonomous vehicle 102), whether the trajectory is compatible with the performance of the autonomous vehicle 102 (e.g., steering limits, acceleration limits, etc.), etc.

[0030] Further, in examples, the secondary system 108 can perform one or more validation actions to determine whether the trajectory is associated with a collision. For example, the secondary system 108 can determine whether the trajectory provided by the primary system 106 intersects with the trajectory of the object determined by the secondary system 108. That is, the secondary system 108 can determine whether the autonomous vehicle 102 would collide with the object if the autonomous vehicle 102 continued along the trajectory (e.g., detect a collision potentially missed by the primary system 106). In at least some examples, this can include a probability that the vehicle 102 will come within a threshold distance of the object in a time frame falling below a threshold probability. In examples, the secondary system 108 can predict the trajectory of the object using a straight-line approximation.

[0031] Based on the one or more authorization actions, the secondary system 108 can determine a state for the trajectory. For example, the secondary system 108 can determine a valid state for the trajectory if the trajectory was generated less than a threshold amount of time ago, if the trajectory is consistent with a current or previous position of the autonomous vehicle 102, if the trajectory is compatible with the performance of the autonomous vehicle 102, and / or if the trajectory is collision-free or associated with a collision within more than a threshold amount of time. Additionally, the secondary system 108 can determine an invalid state for the trajectory if the trajectory was generated more than a threshold amount of time ago, if the trajectory is inconsistent with a current or previous position of the autonomous vehicle 102 (e.g., achieving the trajectory would force a large (and substantially immediate) change in the vehicle's state), if the trajectory is incompatible with the performance of the autonomous vehicle 102 (e.g., if the trajectory exhibits steering angles and / or accelerations that are not achievable by the hardware of the autonomous vehicle 102), and / or if the trajectory is associated with a collision.

[0032] In an example, the secondary system 108 may also monitor one or more components of the autonomous vehicle 102. For example, the secondary system 108 may monitor the primary system 106, the secondary system 108, and / or any other hardware / software components (e.g., motors, controllers, etc.) of the autonomous vehicle 102 to detect errors. The secondary system 108 may detect errors when there is a hardware and / or software failure of the primary system 106, the secondary system 108, and / or another component, when power to the primary system 106, the secondary system 108, and / or another component is disrupted, when no signals have been received from the primary system 106, the secondary system 108, and / or another component for a period of time, etc.

[0033] The example of FIG. 1 illustrates that the secondary system 108 can determine an illegal trajectory 128 (eg, a trajectory associated with a collision) and / or a component error 130 .

[0034] The secondary system 108 may then determine an output trajectory 122 to provide to the system controller 110 based on errors related to the trajectory conditions and / or components. For example, the secondary system 108 may generally strive to select the primary trajectory 116 as the output trajectory 122 when the primary trajectory 116 is reasonable (e.g., the primary trajectory 116 is associated with the highest level). If the primary trajectory 116 is invalid, the secondary system 108 may select the secondary trajectory 118 (e.g., the secondary trajectory 118 is associated with the next highest level) as the output trajectory 122. If both the primary trajectory 116 and the secondary trajectory 118 are invalid, the secondary system 108 may select a trajectory generated by the secondary system 108 as the output trajectory 122, such as a collision avoidance trajectory that brings the autonomous vehicle 102 to a halt along a modified version of the primary trajectory 116 or the secondary trajectory 118 (e.g., to avoid a potential collision). For example, the secondary system 108 may stop the autonomous vehicle 102 using steering controls associated with the primary trajectory 116 or the secondary trajectory 118 while modifying acceleration parameters associated with the primary trajectory 116 or the secondary trajectory 118. Additionally, if an error is detected by the primary system 106, the secondary system 108, and / or any other hardware / software components of the autonomous vehicle 102, the secondary system 108 may select the secondary trajectory 118 and / or the trajectory generated by the secondary system 108 as the output trajectory 122.

[0035] In some examples, if the secondary system 108 is unable to select or generate a trajectory that avoids a collision, the secondary system 108 (or another component, such as a drive manager) may generate a maximum deceleration trajectory that brings the autonomous vehicle 102 to a stop at a maximum deceleration rate available to the autonomous vehicle 102 (e.g., from among multiple deceleration rates available). This may help mitigate damage from a potential collision.

[0036] In an example, the secondary system 108 can maintain control of the autonomous vehicle 102 based on the output trajectory 122 until a signal is received to release control from the output trajectory 122. For example, the secondary system 108 can transition to a lower level trajectory whenever needed and refrain from transitioning again to a higher level trajectory until a signal is received to release control to a higher level trajectory. To illustrate, if the secondary system 108 selects the secondary trajectory 118 because the primary trajectory 116 is invalid, control of the autonomous vehicle 102 can be maintained along the secondary trajectory 118 (or an even lower level trajectory) until a signal is received from a teleoperation system (e.g., a system associated with an operator) to release control to a higher level trajectory. By doing so, the autonomous vehicle 102 can avoid frequent changes between trajectories.

[0037] Further, in an example, the secondary system 108 can provide a message 132 to the primary system 106 about an error with respect to the trajectory provided by the primary system. For example, if the secondary system 108 determines that a collision is estimated to occur for the primary trajectory 116 far enough ahead (e.g., more than a threshold amount of time away) that the autonomous vehicle 102 does not need to brake immediately, the secondary system 108 can send a message 132 to the primary system 106 to alert the primary system 106. The message 132 can indicate that the primary trajectory 116 is associated with a collision and / or that control of the autonomous vehicle 102 will be taken over by the secondary system 108 if the autonomous vehicle 102 is maintained based on the primary trajectory 116. This can enable the primary system 106 to adjust the primary trajectory 116 before a collision occurs. For illustrative purposes, while depicted in FIG. 1 as being sent when an error is detected, the present disclosure is not intended to be so limited. For example, the secondary system 108 may, in some examples, continuously send output from one or more processes (recognition, localization, collision / validity determination, etc.) to the primary system 106 at one or more frequencies that may depend on the information sent.

[0038] The system controller 110 may control the steering, propulsion, braking, safety, emitter, communication, and / or other systems of the autonomous vehicle 102. The system controller 110 may communicate with and / or control corresponding systems of the drive system and / or other components of the autonomous vehicle 102. In some instances, the system controller 110 translates a selected trajectory into instructions usable by the drive system to cause the autonomous vehicle 102 to traverse the trajectory.

[0039] FIG. 2 depicts example details of the primary system 106 and the secondary system 108 of the autonomous vehicle 102 of FIG.

[0040] In this example, the primary system 106 includes at least a localization component 202 (sometimes referred to as a localizer 202), a perception component 204 (sometimes referred to as a perceiver 202), a prediction component 206 (sometimes referred to as a predictor 206), and a planning component 208 (sometimes referred to as a planner 208). The localization component 202, the perception component 204, the prediction component 206, and / or the planning component 208 can collectively and / or individually include processing pipelines that implement one or more machine-learned models, such as neural networks, to perform various functions. Each processing pipeline can receive data and provide output. In one example, the perception component 204 implements one or more neural networks in the processing pipeline to implement object detection, segmentation, and / or classification in addition to (or as an alternative to) determining a predicted motion of such detected objects. Each stage in the pipeline can provide some aspect of perception (e.g., object detection, classification, bounding box generation, etc.) that can feed into another stage in the pipeline (e.g., providing output from one machine-learned model as input to another machine-learned model). In an example, the primary system 106 implements multiple neural networks trained using thousands or hundreds of thousands of hours of vehicle training data (which may include synthetic or simulated data). Furthermore, the multiple neural networks can be configured to perceive a myriad of complex scenarios and control the autonomous vehicle 102 taking those scenarios into account. In general, in any example, the primary system 106 and the secondary system 108 can rely on similar and / or dissimilar hardware. In an example, the primary system 106 uses parallel computing (e.g., in a GPU) while the secondary system 108 does not.In other examples, parallel computing is used in the primary system 106 and / or the secondary system 108 .

[0041] As described herein, a typical neural network is a biologically inspired algorithm that passes input data through a succession of connected layers to produce an output. Each layer in a neural network may also include another neural network or may include any number of layers (convolutional or not). As may be understood in the context of the present disclosure, a neural network may utilize machine learning, which may refer to a broad class of such algorithms in which an output is generated based on learned parameters.

[0042] Although discussed in the context of neural networks, any type of machine learning may be used consistently with this disclosure. For example, machine learning algorithms include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute value shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative dichotomizer 3 (ID3), chi-squared automated interaction detection (CHAID), decision strain, conditional decision tree), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average unidependent estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, backprocessing, etc.), and so on. The learning algorithms may include, but are not limited to, deep learning algorithms (e.g., deep Boltzmann machines (DBM), deep belief networks (DBN), convolutional neural networks (CNN), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (blending), gradient boosting machines (GBM), gradient boosted regression trees (GBRT), random forests), support vector machines (SVM), supervised learning, unsupervised learning, semi-supervised learning, and others.Additional examples of architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, PointNet, and others.

[0043] The localization component 202 may include functionality for receiving sensor data 114 from one or more sensors 104 (not illustrated in FIG. 2 ) to determine the attitude (e.g., position and / or orientation) of the autonomous vehicle 102 relative to a map of the environment surrounding the autonomous vehicle 102. For example, the localization component 202 may determine and / or receive a map of the environment and may continually determine the location of the autonomous vehicle 102 within the map. The map may include a topological map, an occupancy grid, a point map, a landmark map, a mesh, an attitude constraint graph, and / or any other suitable map. In some instances, the localization component 202 may receive image data, LIDAR data, RADAR data, IMU data, GPS data, wheel encoder data, and / or others utilizing SLAM (simultaneous localization and mapping) or CLAMS (calibration, localization, and mapping simultaneously) to accurately determine the position / attitude of the autonomous vehicle 102. The position of the autonomous vehicle 102 may include attitude, which includes the relative position and / or orientation of the autonomous vehicle 102 with respect to points and / or objects on a map, local coordinates, and / or global coordinates (e.g., GPS coordinates). In additional or alternative examples, the orientation may include an indication of the yaw, roll, and / or pitch of the autonomous vehicle 102 with respect to a reference plane and / or with respect to points and / or objects on a map. The position and / or orientation together may describe "attitude."

[0044] In some instances, the localization component 202 can provide data to various components of the autonomous vehicle 102. For example, the localization component 202 can provide the attitude of the autonomous vehicle 102 to the planning component 208 to generate a trajectory, as discussed below. In other examples, the localization component 202 can provide the attitude of the autonomous vehicle 102 to other components of the autonomous vehicle 102, such as the perception component 204 and / or the prediction component 208.

[0045] The perception component 204 can include functionality for performing object detection, segmentation, and / or classification. In some examples, the perception component 204 can receive sensor data 114 as input and output primary perception data indicative of the presence of an object closest to the autonomous vehicle 102 and / or a classification of the object as an object type (e.g., a semantic label such as a car, pedestrian, cyclist, animal, building, tree, road surface, curve, sidewalk, unknown, etc.). Additionally or alternatively, the perception component 204 can provide processed sensor data indicative of one or more characteristics associated with the detected object and / or associated with the environment in which the object is positioned. In some examples, characteristics associated with an object may include, but are not limited to, x-position (e.g., overall position, relative position, etc.), y-position (e.g., overall position, relative position, etc.), z-position (e.g., overall position, relative position, etc.), orientation, object type (e.g., classification), velocity of the object, width (size) of the object, etc., and / or associated uncertainty. In some examples, the velocity of the object may be determined from multiple detections of the same object in subsequent data, RADAR processing, and / or other techniques. Characteristics associated with an environment may include, but are not limited to, the presence of other objects in the environment, the state of other objects in the environment, the time of day, the day of the week, the season, weather conditions, indications of darkness / lightness, etc.

[0046] In some instances, the perception component 204 may provide the primary perception data and / or the processed sensor data to various components of the autonomous vehicle 102. For example, the perception component 204 may provide the primary perception data and / or the processed sensor data to the planning component 208 and / or the prediction component 206. In other examples, the perception component 204 may provide the primary perception data and / or the processed sensor data to other components of the autonomous vehicle 102, such as the localization component 202. In examples, the primary perception data and / or the processed sensor data may be in the form of a primary object list including a list of objects and / or characteristics for each of the objects.

[0047] The prediction component 206 can associate tracks with detected objects and / or predict object trajectories. The object tracks can include historical object positions, velocities, accelerations, and / or orientations. In some instances, the prediction component 206 can determine whether to associate historical object tracks with objects recently detected by the perception component 204. For example, the prediction component 206 can determine whether a recently detected object has similar features (e.g., similar feature maps, closest embeddings, similar visual features (FAST, BRISK, ORB, etc.)) to a previously detected object, is within a threshold distance of a previous position of the object associated with the historical track, is within a threshold speed of a previous speed of the object associated with the historical track, etc. In some examples, the prediction component 206 can include machine-learned models for predicting object behavior in an environment based on lighting conditions (e.g., blinker detection), object speed / acceleration, map data (e.g., lane information indicative of reasonable driving behavior), and / or learned object behavior. In some examples, the prediction component 206 may predict one or more predicted object trajectories for a specific object detected by the perception component 204 based on, for example, probabilistic determinations or multi-modal distributions of historical, current, and / or predicted object positions, velocities, accelerations, and / or orientations.

[0048] In some instances, the prediction component 206 may provide data regarding the object track, the predicted object trajectory (e.g., one or more trajectories and associated uncertainties), and / or other data to various components of the autonomous vehicle 102. For example, the prediction component 206 may provide such data to the planning component 208. In some examples, the prediction component 206 may generate a heat map associated with the predicted motion of the object and provide such heat map to the planning component 208. In other examples, the prediction component 206 may provide data regarding the object to other components of the autonomous vehicle 102, such as the localization component 202 and / or the perception component 204. Additionally, in some examples, the data regarding the object may be provided to the secondary system 108. In examples, the data regarding the object may be stored in a primary object list.

[0049] In some examples, the perception component 204 and / or the prediction component 206 may implement techniques discussed in U.S. Patent No. 6,399,433, entitled “Detecting Blocking Objects,” filed February 14, 2018, and U.S. Patent No. 6,399,433, entitled “Feature-Based Prediction,” filed April 6, 2018, the entire contents of all of which are incorporated herein by reference.

[0050] The planning component 208 can determine a path and / or trajectory that the autonomous vehicle 102 follows to traverse an environment. For example, the planning component 208 can determine various routes and / or trajectories as well as various levels of detail. For example, the planning component 208 can determine a route for traveling from a first location (e.g., a current location) to a second location (e.g., a target location). In an example, the route may be a sequence of waypoints for traveling between the two locations. As non-limiting examples, the waypoints include streets, intersections, Global Positioning System (GPS) coordinates, etc. Furthermore, the planning component 208 can generate instructions for guiding the autonomous vehicle 102 along at least a portion of the route from the first location to the second location. In at least one example, the planning component 208 can determine how to guide the autonomous vehicle 102 from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instructions can include a trajectory or a portion of a trajectory.

[0051] In examples, the planning component 208 can generate the primary trajectory 116 and / or send the primary trajectory 116 to the secondary system 108. For example, the planning component 208 can generate multiple trajectories substantially simultaneously (e.g., within engineering tolerances) according to a receding horizon technique and / or based at least in part on data received from other components of the primary system 106. In some instances, the planning component 208 can perform a cost-based search (or any other suitable algorithm for identifying a suitable trajectory for controlling the motion of the autonomous vehicle 102) over the multiple trajectories to identify the primary trajectory 116 for transmission to the secondary system 108.

[0052] In an example, the planning component 208 can generate the secondary trajectory 118 and / or send the secondary trajectory 118 to the secondary system 108. For example, the planning component 208 can generate multiple contingent trajectories and select one of the contingent trajectories as the secondary trajectory 118. The secondary trajectory 118 may represent a backup trajectory used when the primary trajectory 116 is invalid or otherwise unacceptable. The secondary trajectory 118 may be different from the primary trajectory 116. In one example, the primary trajectory 116 can cause the autonomous vehicle 102 to accelerate or maintain a particular speed, while the secondary trajectory 118 can cause the autonomous vehicle 102 to decelerate to a stop, such as a gentle stop (as discussed below). In another example, the primary trajectory 116 can cause the autonomous vehicle 102 to change lanes, while the secondary trajectory 118 can cause the autonomous vehicle 102 to stay in the same lane. In yet another example, the primary trajectory 116 may cause the autonomous vehicle 102 to maintain a current speed and steering angle, while the secondary trajectory 118 may cause the autonomous vehicle 102 to pull over to the shoulder. In any example, both the primary trajectory 116 and the secondary trajectory 118 may include a set of vehicle states (position, orientation, speed, etc.) and / or commands (acceleration, steering angle, etc.) to be reached by the autonomous vehicle 102 at a future point along the trajectory and / or at a future time along the trajectory.

[0053] In an example, a gentle stop may include applying the brakes at a particular rate, such as a predetermined rate, a rate less than a first rate (e.g., a maximum braking rate), a rate greater than a second rate (e.g., a minimum braking rate), a rate that is substantially midway among a plurality of available rates (e.g., a third rate among five rates), a minimum rate among a plurality of rates, etc. The rate of deceleration may refer to a unit of measurement over time, such as the number of meters or feet per second per second (m / s^2). In one example, a gentle stop may include decelerating at a rate of 5 or 10 feet (1.524 or 3.048 meters) per second per second until the autonomous vehicle 102 comes to a stop.

[0054] Although in many examples, the localization component 202, the perception component 204, and the prediction component 206 are discussed as providing outputs to each other (or to other components of the primary system 106), in some examples, any of these components may provide outputs to the secondary system 108. As a non-limiting example, the primary system 106 may provide object track information to the secondary system 108.

[0055] As illustrated, the secondary system 108 includes a localizer 210, a perceiver 212 (sometimes referred to as a perception / prediction component 212), a trajectory manager 214, and a monitor 216. In examples, the localizer 210, the perceiver 212, the trajectory manager 214, and / or the monitor 216 can implement models based on the positioning, velocity, acceleration, etc. of the autonomous vehicle 102 and / or objects around the vehicle. In some examples, such models can incorporate probabilistic models. For example, the secondary system 108 can implement a Kalman filter (also referred to as a linear quadratic estimation (LQE)) (e.g., an extended Kalman filter (EKF), an unscented Kalman filter (UKF), etc.), a particle filter, a Bayesian filter, etc. To illustrate, the perceiver 212 can implement a Kalman filter or a particle filter to associate sensor data with prior data for detection, tracking, prediction, etc. In some examples, the secondary system 108 may be configured in a manner that precludes the use of or reduces the number of machine-learned models, in contrast to the primary system 106. In other examples, the secondary system 108 may include one or more machine-learned models that are different (or similar or identical) to those of the primary system 106 (e.g., producing similar outputs based on similar inputs, but having a different internal network architecture). In some examples, the secondary system 108 may use different hardware (e.g., processor and memory) than the primary system 106.

[0056] In examples, the secondary system 108 may be designed to be less computationally intensive and / or to operate at a higher integrity level. For example, the processing pipeline of the secondary system 108 may be simpler by relying on less sensor data, include less complex pipeline components, include fewer pipeline components in general, output less data, and / or exclude and / or limit the use of machine-learned models. In some instances, the secondary system 108 may be a "high integrity" system by achieving tight operating tolerances and / or have the ability to be inspected (verified). In examples, the secondary system 108 may have a higher level of reliability and / or verifiability than the primary system 106. For example, the output of the subcomponents of the secondary system 108 may be certified to operate with perfect accuracy or near-perfect accuracy (e.g., 99.99% of scenarios or higher). In some examples, the secondary system 108 may include Automotive Safety Integrity Level (ASIL) D certification.

[0057] The localizer 210 may process the sensor data 120 from the sensors 104 to determine one or more of a position and / or orientation (together, an attitude) of the autonomous vehicle 102, where the position and / or orientation may be relative to points and / or objects in an environment in which the autonomous vehicle 102 is located. In an example, the orientation may include an indication of the yaw, roll, and / or pitch of the autonomous vehicle 102 relative to a reference surface and / or relative to the points and / or objects. In an example, the localizer 210 may perform less processing (e.g., a higher level of localization) than the localization component 202 of the primary system 106. For example, the localizer 210 may not determine the attitude of the autonomous vehicle 102 relative to a map, but may simply determine the attitude of the autonomous vehicle 102 relative to objects and / or surfaces detected around the autonomous vehicle 102 (e.g., a local position, not a global position) and / or relative to a previous attitude of the autonomous vehicle 102. Such position and / or orientation may be determined using, for example, probabilistic filtering techniques such as a Bayesian filter (Kalman filter, extended Kalman filter, unscented Kalman filter, etc.) using some or all of the sensor data 120.

[0058] The localizer 210 may provide the position and / or orientation of the autonomous vehicle 102 to various components of the secondary system 108, such as the perceiver 212, the trajectory manager 214, the drive manager 226, and others.

[0059] In some examples, the localizer 210 may include a pose validator to verify the pose of the autonomous vehicle 102. For example, the pose validator may check that the pose is consistent with a previous pose of the autonomous vehicle 102. If the pose has not changed (e.g., with respect to lateral motion, vertical motion, orientation, etc.) from the previous pose by more than a threshold amount, the pose may be valid. In examples, the pose validator may provide data about whether the pose is valid to various components of the secondary system 108, such as the perceiver 212, the trajectory manager 214, the drive manager 226, etc.

[0060] The perceiver 212 may include functionality for performing object detection, object segmentation, object classification, track determination, and / or predicting one or more trajectories for each object (e.g., including uncertainty), etc. In some examples, the perceiver 212 may receive as input data the sensor data 120 from one or more of the sensors 104, the attitude of the autonomous vehicle 102 from the localizer 210, data indicative of a direction of motion of the autonomous vehicle 102, data indicative of a speed of the autonomous vehicle 102, data indicative of an acceleration of the autonomous vehicle 102, data indicative of a yaw rate of the autonomous vehicle 102, data indicative of a yaw acceleration, and / or data indicative of a steering angle and / or a steering angle rate of the autonomous vehicle 102. Additionally, in some examples, the perceiver 212 may receive as input data from the primary system 106, such as a primary object list from the perception component 204, the attitude of the autonomous vehicle 102, one or more object tracks determined by the primary system 106, etc. As noted above, the object list from the primary system 106 may include primary perception data, processed sensor data, data regarding the track / trajectory of the object, etc. In such an example, the perceiver 212 may perform data association (e.g., by using probabilistic filters, clustering, closest point analysis, etc.) to associate the sensor data with the track.

[0061] The perceiver 212 can process the input data to determine secondary perception data. The secondary perception data can be indicative of the presence of an object nearest the autonomous vehicle 102, characteristics associated with the object, etc., where the characteristics associated with the object can include the position of the object relative to the autonomous vehicle 102, the orientation of the object relative to the autonomous vehicle 102, the velocity of the object, the acceleration of the object, the extent of the object (size of the object), etc. In many examples, the characteristics included in the secondary perception data do not include a classification (e.g., a semantic label) of the object. Although, in some examples, the secondary perception data may be indicative of a classification.

[0062] The perceiver 212 can also process the input data to determine one or more tracks for the object. As noted above, the track of the object can include historical position, velocity, acceleration, and / or orientation, extent, etc., and / or uncertainties associated therewith. In one example, the perceiver 212 can implement a Kalman filter for data association and / or use features in the data (e.g., unique / salient points of the data, such as unique pixels) to associate new detections with previously detected objects. As another example, the perceiver 212 can determine the object in the environment a first time, determine the object in the environment a second time, determine the overlap of the object between the first and second times, and generate a track for the object. In yet another example, the perceiver 212 can determine the velocity of the object a first time, compare the predicted motion of the object to the data of the object captured a second time, determine the overlap, and generate a track for the object. The perceiver 212 can determine a track for each object detected around the autonomous vehicle 102. In some examples, the perceiver 212 may determine the track of the object independently from the data received from the primary system 106 (e.g., without using the primary object list and / or the posture of the autonomous vehicle 102 received from the primary system 106), while in other examples, the perceiver 212 may use the data received from the primary system 106.

[0063] The perceiver 212 can also process the input data to determine one or more predicted trajectories for the object. For example, based on the current location of the object and the velocity of the object over a period of several seconds, the perceiver 212 can predict the path the object will move along over the next several seconds. In some examples, such predicted paths can include using linear assumptions of motion given the position, orientation, velocity, and / or orientation, such as using straight-line approximations and / or fitting to higher order models. In other examples, such predicted paths can include more complex analyses, such as using Extended Kalman Filter (EKF) propagation, models of object behavior, or other prediction techniques.

[0064] The perceiver 212 may output the secondary perception data and / or data regarding the objects (e.g., tracks, trajectories, etc.) to the trajectory manager 214 and / or any other components of the secondary system 108. In an example, the secondary perception data and / or data regarding the objects is represented in a secondary object list. For example, the object list may indicate, for each object, the characteristics of the object and / or the track / trajectory of the object. In an example, the perceiver 212 may output the secondary perception data and / or data regarding the objects to the primary system 106.

[0065] For ease of illustration, the perceiver 212 (and other components of the primary system 106 and secondary system 108) are illustrated using a single block. However, the perceiver 212 (and / or other components of the primary system 106 and secondary system 108) may be implemented as any number of blocks, with each block located in one or more systems or subsystems.

[0066] The trajectory manager 214 can evaluate the primary system 106 and take over control of the autonomous vehicle 102 if a problem is detected. For example, the trajectory manager 214 can evaluate the primary trajectory 116 and / or the secondary trajectory 118 to determine whether the primary trajectory 116 and / or the secondary trajectory 118 should be used or whether an alternative trajectory should be used. As illustrated, the trajectory manager 214 can include collision detection components 218(1)-(2) (sometimes referred to as a collision detector 218), trajectory validation components 220(1)-(3) (sometimes referred to as a trajectory validator 220), a trajectory selector 222, and a releaser 224.

[0067] Collision detection component 218 may evaluate primary trajectory 116 and secondary trajectory 118 to detect potential collisions. For example, collision detection component 218(1) may determine whether primary trajectory 116 intersects (or comes within a threshold distance of) a predicted trajectory of an object (e.g., as determined by perceiver 212) and whether the object and autonomous vehicle 102 will meet at an intersection at the same time (or time window) and / or whether the probability associated therewith is less than or equal to a threshold. These determinations may be repeated for any number of detected objects. In at least some examples, the determination may be made in order starting with the object closest to autonomous vehicle 102. In some examples, such determinations for all objects may be performed substantially simultaneously. Furthermore, such determinations may additionally or alternatively include information regarding the trajectory of autonomous vehicle 102 as well as one or more uncertainties of the object. The collision detection component 218(2) can determine whether the secondary trajectory 118 intersects (and / or meets or exceeds a threshold certainty of a collision) with a predicted trajectory of the object (e.g., as determined by the perceiver 212) and whether the object and the autonomous vehicle 102 will meet at an intersection at the same time (or time window) and / or within a threshold distance from each other.

[0068] The collision detection component 218 can detect whether any number of objects detected by the autonomous vehicle 102 intersect with the trajectory for the autonomous vehicle 102 at a time now or in the future. In some instances, this can include determining whether the predicted object trajectory and the vehicle trajectory intersect and / or come within a threshold distance (e.g., a nearby location) of each other, determining whether an object along the predicted object trajectory and the autonomous vehicle 102 along the vehicle trajectory will meet at any intersection or nearby location at the same time or time frame and / or within a threshold distance of each other, and / or determining whether a bounding box for the object overlaps with a bounding box for the autonomous vehicle 102 around any intersection or nearby location and / or within a threshold distance from each other. Further, in examples, the same process may be implemented with a safety margin around each bounding box (e.g., a safety bounding box representing a bounding box that is expanded by a certain amount, such as by adjusting the length / width / height of the bounding box). The bounding box can represent the spatial location, orientation, and / or size (e.g., extent) of the object. For example, a bounding box may be a representation of an object having a position, orientation, length, width, and / or height defined by eight corners (e.g., a rectangular solid). In an example, a bounding box may be a minimum volume rectangular solid that encloses an object.

[0069] Collision detection component 218 can provide data indicative of the detection of a collision to trajectory authorization component 220. Additionally, as illustrated, collision detection component 218(1) can pass primary trajectory 116 (denoted as "PT116") to trajectory authorization component 220(1). Collision detection component 218(2) can pass secondary trajectory 118 (denoted as "ST118") to trajectory authorization component 220(2) and trajectory authorization component 220(3).

[0070] Although two collision detection components 218 are illustrated in FIG. 2, any number of collision detection components 218 may be implemented, such as a single collision detection component 218, more than two collision detection components 218 (e.g., for additional primary / secondary orbits), etc.

[0071] In examples, trajectory authorization component 220 can perform one or more authorization actions to determine whether the trajectory is valid. In some examples, trajectory authorization component 220(1) can determine whether primary trajectory 116 is valid by performing one or more authorization actions discussed below with reference to FIGURE 3. Additionally, in some examples, trajectory authorization component 220(2) can determine whether secondary trajectory 118 is valid by performing one or more authorization actions discussed below with reference to FIGURE 4.

[0072] Moreover, in some examples, the trajectory approval component 220(3) can determine whether the secondary trajectory 118 is valid. If the secondary trajectory 118 is invalid, the trajectory approval component 220(3) can generate an alternative trajectory and determine whether the alternative trajectory is valid. The trajectory generated by the trajectory approval component 220(3) can include a collision avoidance trajectory that decelerates the autonomous vehicle 102 to a stop in an attempt to avoid or mitigate a potential collision. For example, the collision avoidance trajectory can include deceleration along the secondary trajectory 118 (or the primary trajectory 116 in some cases). By modifying the acceleration profile along the second trajectory 118 (e.g., decelerating faster, accelerating in some portions, etc.), a similar move can be made to avoid potential other objects while coming to a stop before an expected collision. In some examples, the collision avoidance trajectory can include deviations of the secondary trajectory 118 (e.g., changes in any one or more of the desired vehicle state and / or associated controls) to mitigate any potential collisions detected in the secondary trajectory 118. Trajectory authorization component 220(3) may generate a collision avoidance trajectory based on information about the attitude of autonomous vehicle 102 (e.g., provided by localizer 210), information about the object (e.g., track or trajectory information provided by perceiver 212), information provided by primary system 106, and / or any other information available to secondary system 108. In some examples, trajectory authorization component 220(3) may perform one or more of the authorization operations discussed below with reference to FIG.

[0073] Trajectory authorization component 220 can output data indicative of the validity of the trajectory to trajectory selector 222. For example, trajectory authorization component 220(1) can determine a state of primary trajectory 116 indicating whether primary trajectory 116 is valid or invalid and output the state to trajectory selector 222. Further, trajectory authorization component 220(2) can determine a state of secondary trajectory 118 indicating whether secondary trajectory 118 is valid or invalid and output the state to trajectory selector 222. Moreover, trajectory authorization component 220(3) can determine a state of a collision avoidance trajectory (e.g., modified secondary trajectory 118 or secondary trajectory 118) and output the state to trajectory selector 222.

[0074] 2 illustrates three trajectory authorization components 220, any number of trajectory authorization components 220 may be implemented, such as a single trajectory authorization component 220, more than three trajectory authorization components 220 (e.g., for additional trajectories), etc. Additionally, in some instances, collision detection component 218(2) and / or trajectory authorization components 220(2)-(3) may not process data if one or more of primary trajectory 116 or secondary trajectory 118 are valid. As a non-limiting example, to conserve computational resources, if trajectory authorization component 220(1) determines that primary trajectory 116 is valid (and there are no other errors in either system), no other verification processes may be performed.

[0075] In an example, trajectory manager 214 can provide a message to planning component 208 (or another component of primary system 106) indicating the status of the trajectory (e.g., valid or invalid). For example, one or more of trajectory authorization components 220 can send a message to planning component 208 indicating whether a trajectory is valid or invalid. In an example, the message can indicate one or more of the time until collision, the width of an object associated with the potential collision, the speed of the object, the location of the object, or the point of collision (e.g., the location of the detected collision).

[0076] The trajectory selector 222 can determine and send an output trajectory to the system controller 110. The trajectory selector 222 can generally select the primary trajectory 116, the secondary trajectory 118, the collision avoidance trajectory (or another trajectory generated by the secondary system 108), and / or the maximum deceleration trajectory. Such a selection can be based on one or more conditions for one or more trajectories provided by the trajectory authorization component 220. The primary trajectory 116 can have a higher level than the secondary trajectory 118, which can have a higher level than the collision avoidance trajectory, which can have a higher level than the maximum deceleration trajectory. In many examples, a lower level trajectory can be associated with a higher deceleration rate than a higher level trajectory. For example, the collision avoidance trajectory can be associated with a higher deceleration rate (e.g., more feet per second) than the secondary trajectory 118. The trajectory selector 222 can generally select the highest level trajectory that is reasonable. This can enable the autonomous vehicle 102 to decelerate with the least amount of deceleration to avoid a collision and provide a relatively comfortable ride for the passengers.

[0077] In examples, the maximum deceleration trajectory (also referred to as a hard stop or emergency stop) can include applying the brakes at a particular rate, such as a predetermined rate, a rate greater than a threshold (e.g., a maximum braking rate), a rate that is the greatest rate among multiple available rates, etc. In one example, the maximum deceleration trajectory can include decelerating at a rate of 15 or 20 feet (4.572 or 6.096 meters) per second until the autonomous vehicle 102 comes to a stop.

[0078] In an example, the trajectory selector 222 may select the primary trajectory 116 as the output trajectory when the primary trajectory 116 is valid. If the primary trajectory 116 is invalid, the trajectory selector 222 may select the secondary trajectory 118 as the output trajectory. If both the primary trajectory 116 and the secondary trajectory 118 are invalid, the trajectory selector 222 may select the collision avoidance trajectory as the output trajectory. If the primary trajectory 116, the secondary trajectory 118, and the collision avoidance trajectory are all invalid, the trajectory selector 222 may select the maximum deceleration trajectory. Once the trajectory selector 222 selects a trajectory, the trajectory selector 222 may send a signal to the drive manager 226 (or the system controller 110 of FIG. 1 ) indicating the trajectory to use to control the autonomous vehicle 102.

[0079] In an example, if the output trajectory is associated with deceleration, the trajectory selector 222 can send a signal to another component of the autonomous vehicle 102 to prepare for such deceleration. For example, if a maximum deceleration trajectory is selected due to a likely collision, the trajectory selector 222 can output a signal to alert other components so that they can prepare for the collision (e.g., tension seat belts, prepare airbags for deployment, etc.).

[0080] In examples, the trajectory selector 222 maintains data indicative of a selected trajectory (e.g., output trajectory). For example, the trajectory selector 222 can maintain a state machine that transitions between states each associated with a trajectory. Example states that may be implemented by the trajectory selector 222 are discussed below with reference to FIG. 6. In examples, the trajectory selector 222 may transition to a lower level state and not transition to a higher level state until a signal is received to release control from the lower level state. In some examples, the trajectory selector 222 may maintain control based on the same lower level trajectory (e.g., not consider an updated trajectory of the same level). In other examples, a new lower level trajectory (e.g., secondary trajectory) may be received from the primary system 106, and the new lower level trajectory may be used to maintain the lower state (e.g., accept an updated trajectory of the same level).

[0081] The releaser 224 can send a signal to the trajectory selector 222 to release control from the current state (e.g., release control from the trajectory). In one example, a remote system can provide a signal to the secondary system 108 to release control from the current state, and in response, the releaser 224 can send a signal to the trajectory selector 222 to release control from the current state. The remote system can include an automated system, a remote operation system associated with an operator (e.g., a user), and / or another entity. The operator can help manage the autonomous vehicle 102 when a problem arises, such as an interruption in the movement of the autonomous vehicle 102 (e.g., an unexpected stop). In one illustration, upon transitioning to a state associated with a trajectory other than the primary trajectory 116, the operator can determine whether it is clear for the autonomous vehicle 102 to resume normal operation (or transition to a different state), such as by communicating with a passenger in the autonomous vehicle 102, analyzing data associated with the autonomous vehicle 102, etc. Here, when the operator determines that it is clear for the autonomous vehicle 102 to resume normal operation (or transition to a different state), the operator can cause a signal to be sent to the secondary system 108 to release control from the current state. In another example, a component of the autonomous vehicle 102 can perform processing to determine that it is clear for the autonomous vehicle 102 to resume normal operation (or transition to a different state) and send a signal to the releaser 224 to release control from the current state.

[0082] The monitor 216 may monitor the primary system 106, the secondary system 108, and / or other components of the autonomous vehicle 102 to determine if there is an error. In many examples, the monitor 216 may monitor the primary system 106 more generally than the trajectory manager 214 (e.g., monitor overall vehicle status information instead of just the trajectory). In examples, the monitor 216 may determine that an error has occurred if there is a hardware and / or software failure for the primary system 106, the secondary system 108, and / or another component (e.g., a failure of a sensor 104, an actuator, or any other component of the autonomous vehicle 102), if power to the primary system 106, the secondary system 108, and / or another component is disrupted, if no signals have been received from the primary system 106, the secondary system 108, and / or another component for a period of time (e.g., longer than a period of time that would indicate that the primary system 106 is experiencing a problem), etc. Such a determination may be based on signals from the primary system 106, signals from components of the primary / secondary systems 106 / 108, signals from the drive manager 226 (or actuators (brakes, steering, etc.) that may be in communication with the drive manager 226), and / or signals (e.g., diagnostic information) from a service provider (e.g., a remote device) indicating a hardware or software fault. Additionally or alternatively, such a determination may be based on signals from a battery of the autonomous vehicle 102 or from other components associated with providing power that indicate whether power is being provided to the primary / secondary systems 106 / 108 (or components of the primary / secondary systems 106 / 108). As illustrated in FIG. 2, the monitor 216 may receive a signal from the drive manager 226. Such signals may indicate an error with the drive manager 226 and / or components in communication with the drive manager 226, e.g., actuators, etc.Additionally, the monitor 216 can communicate with the planning component 208 to receive signals indicating errors related to the planning component 208 and / or send information regarding the monitoring (e.g., a state of the autonomous vehicle 102 that indicates that an error is occurring).

[0083] If the monitor 216 determines an error related to the primary system 106, the monitor 216 can send a signal indicating such error to the trajectory manager 214 (e.g., the trajectory selector 222) and / or the drive manager 226. In one example, if the autonomous vehicle 102 is not already being controlled by a trajectory associated with deceleration (e.g., the secondary trajectory 118, the collision avoidance trajectory, or the maximum deceleration trajectory), the trajectory selector 222 can select a trajectory associated with deceleration and send a signal to the system controller 110 to control the autonomous vehicle 102 (e.g., to a stop). In another example, if a signal is sent to the drive manager 226, the drive manager 226 can control the autonomous vehicle 102 to implement the maximum deceleration trajectory or another trajectory.

[0084] The drive manager 226 can receive a trajectory (e.g., from the trajectory selector 222) and / or generate commands / instructions to control the autonomous vehicle 102. For example, the drive manager 226 can control steering, propulsion, braking, safety, emitter, communication, and / or other systems of the autonomous vehicle 102. In some instances, the drive manager 226 translates the selected trajectory into instructions usable by the drive system to cause the autonomous vehicle 102 to traverse the trajectory. In examples, the drive manager 226 is implemented in place of or in cooperation with the system controller 110. Further, in examples, the drive manager 226 may be replaced by the system controller 110. In examples, the drive manager 226 receives attitude information from the localizer 210. Additionally, the drive manager 226 can provide data to the monitor 216, such as actuator diagnostic data indicative of errors related to actuators. In examples, the drive manager 226 can maintain / store one or more previous trajectories and / or the current trajectory being used.

[0085] In some examples, the secondary system 108 or another system performs a ground removal technique on the sensor data 120. This may be done before the sensor data 120 is used by components of the secondary system 108. For example, the secondary system 108 may analyze the LIDAR data, or voxel data associated with the LIDAR data, to determine whether the LIDAR data or voxels represent ground. In some instances, the ground may correspond to a surface drivable by a vehicle. In some examples, the secondary system 108 may determine a dot product between a vector in a height dimension (e.g., a reference direction) of a device holding the LIDAR system expressed in a common coordinate system and a normal vector (e.g., may be determined based on a per-voxel eigenvalue decomposition, etc.). In such examples, a dot product that meets or exceeds a threshold of approximately 15 degrees may indicate, for example, that the voxel does not include ground. Conversely, a dot product that is less than the threshold may indicate that the voxel includes ground. Additionally, in some examples, the secondary system 108 may cluster voxels determined to be locally flat voxels to augment a surface corresponding to ground. In examples, once a ground surface is identified in the sensor data, data associated with the ground surface may be removed from the sensor data and the remaining data may be processed (e.g., object detection, segmentation, classification, etc.). In some examples, the primary system 106, the secondary system 108, and / or another system may perform the ground removal techniques discussed in U.S. Patent Application Publication No. 2013 / 0133994, entitled “Voxel Based Ground Plane Estimation and Object Segmentation,” filed June 14, 2017, the entire contents of which are incorporated herein by reference.

[0086] The components of the primary system 106 and / or the secondary system 108 may represent hardware and / or software components. In a software context, one or more of the components may include instructions executable by one or more processors. In a hardware context, one or more of the components may be implemented as an integrated circuit (e.g., an application specific integrated circuit (ASIC), etc.), a gate array (e.g., a field programmable gate array (FPGA), etc.), etc. In at least some examples, both the hardware and software of the primary system 106 and the secondary system 108 may be different.

[0087] 3, 4, and 5 illustrate example processes 300, 400, and 500 for performing one or more authorization operations. Processes 300, 400, and 500 are illustrated as logical flow graphs, each operation of which represents a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media that perform the recited operations when executed by one or more processors. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular abstract data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the process. Processes 300, 400, and / or 500 may be performed by any device or component, such as the autonomous vehicle 102, a service provider (e.g., a computing device of the service provider), or any other device.

[0088] 3 illustrates an example process 300 for determining whether a trajectory for a vehicle is valid or invalid. In some examples, process 300 may be performed for a primary trajectory. For example, process 300 may be performed by trajectory approval component 220(1) of FIG. 2. In other examples, process 300 may be performed for other trajectories, such as secondary trajectories, collision avoidance trajectories, and / or any other trajectory.

[0089] At operation 302, the system may determine whether a trajectory for the vehicle was generated less than a threshold amount of time ago (e.g., a threshold amount of time from a current time). For example, the system may reference the current time and the time the trajectory was generated (e.g., from data associated with the trajectory and / or from data in a data store). If the system determines that the trajectory was generated less than the threshold amount of time ago (e.g., indicating the trajectory is not stale), process 300 may proceed to operation 304. Alternatively, if the system determines that the trajectory was not generated less than the threshold amount of time ago (e.g., indicating the trajectory is stale), process 300 may proceed to operation 306.

[0090] At operation 304, the system may determine whether the trajectory is consistent with the current or previous attitude of the vehicle. For example, the system may determine whether the trajectory will cause the current attitude and / or one or more previous attitudes to change by more than a threshold amount (e.g., change in heading, position, heading, etc., by more than a threshold amount). In some instances, the trajectory may be compared to a number of previous trajectories, such as the last three trajectories, to determine whether there is a change from each of the previous trajectories by more than a threshold amount. The system may store the trajectory over time, such that one or more previous trajectories may be referenced. If the system determines that the trajectory is consistent with the current or previous attitude of the vehicle, the process 300 may proceed to operation 308. Alternatively, if the system determines that the trajectory is not consistent with the current or previous attitude of the vehicle, the process 300 may proceed to operation 306.

[0091] At operation 308, the system may determine whether the trajectory is compatible with the vehicle's performance. The vehicle's performance may be based on the hardware and / or software components of the autonomous vehicle 102 and / or limitations of those components. In one example, the system may identify the autonomous vehicle 102's performance, such as a maximum steering angle, an amount of time to change from one steering angle to another, a maximum acceleration, etc. If the trajectory requires a change beyond the maximum steering angle, requires steering to change more quickly than the amount of time required to make a steering angle adjustment, and / or requires acceleration above the maximum acceleration, the system may determine that the trajectory is not compatible with the vehicle's performance. If the system determines that the trajectory is compatible with the vehicle's performance, the process 300 may proceed to operation 310. Alternatively, if the system determines that the trajectory is not compatible with the vehicle's performance, the process 300 may proceed to operation 306.

[0092] At operation 310, the system may determine whether the trajectory is collision-free. For example, the system may determine whether a vehicle moving along the vehicle trajectory will collide with an object moving along the object trajectory. If so, the trajectory is associated with a collision. In some examples, the system may determine a probability that the vehicle will collide with the object and determine whether the probability meets or exceeds a probability threshold. If the probability meets or exceeds a probability threshold, the system may determine that the vehicle will collide with the object. In examples, operation 310 may be performed for each object detected around the vehicle (e.g., to check for one or more collisions).

[0093] If the system determines that the trajectory is collision-free, then the process 300 may proceed to operation 312. Alternatively, if the system determines that the trajectory is not collision-free (e.g., associated with a collision), then the process 300 may proceed to operation 314.

[0094] In operation 314, the system can determine whether a collision will occur within more than a threshold amount of time. For example, the system can determine a time to last second braking (TTLSB) before colliding with an object. The time to last second braking may be the last possible time when the vehicle needs to start a maximum deceleration trajectory to avoid colliding with the object. If the time to last second braking is relatively far in the future (e.g., more than a threshold amount of time), there may be time for the vehicle to correct the trajectory and avoid starting a maximum deceleration trajectory. That is, the system does not need to start a maximum deceleration trajectory yet. To illustrate, if the time to last second braking for an object is within 10 milliseconds, the system can determine that a collision will occur within less than a threshold amount of time. In contrast, if the time to last second braking is within 2 seconds, the system can determine that a collision will occur within more than a threshold amount of time. In an example, the system can evaluate potential collisions with each object around the vehicle (e.g., each object in an object list).

[0095] If the system determines that a collision will occur within more than a threshold amount of time (e.g., there is still time before the time to brake at the last moment), process 300 may proceed to operation 316. Alternatively, if the system determines that a collision will occur within the threshold amount of time or within less than the threshold amount of time, process 300 may proceed to operation 306.

[0096] At operation 306, the system may determine that the trajectory is invalid. In an example, the system may update the state of the trajectory to indicate that the trajectory is invalid. In some instances, the state may indicate why the trajectory is invalid (e.g., associated with a collision, time when a collision will occur, inconsistent with the current / previous position of the vehicle, incompatible with the vehicle's performance, etc.).

[0097] In some examples, process 300 may proceed to operation 318 after performing operation 306. At operation 318, the system may send a message indicating that the trajectory is invalid. For example, the system may be a secondary system and may send a message to the primary system to inform the primary system that the trajectory generated by the primary system is invalid. In an example, the message may indicate why the trajectory is invalid.

[0098] At operation 316, the system may send a message indicating that the trajectory is associated with a collision. For example, the system may be a secondary system and may send a message to the primary system to inform the primary system that the trajectory generated by the primary system is associated with a collision down the road. In an example, the message may indicate one or more of a time to collision, an extent of an object associated with the potential collision, a velocity of the object, a location of the object, or a point of collision (e.g., a location of the detected collision).

[0099] At operation 312, the system may determine that the trajectory is valid and / or may store the trajectory. In an example, the system may update the status of the trajectory to indicate that the trajectory is valid (e.g., collision-free, etc.). In an example, the system may be a secondary system and may send a message to the primary system to inform the primary system that the trajectory is valid.

[0100] In one example, at 11:30:40 AM, the system may determine that if the evaluated trajectory is used, the vehicle will collide with the object in 5 seconds (at 11:30:45 AM). The system may also determine that the amount of time required to stop the vehicle (e.g., using a maximum deceleration trajectory) and avoid a collision with the object is 3 seconds (e.g., based on the speed of the vehicle and / or the object). The system may use such information to determine the last possible time when the vehicle needs to begin a maximum deceleration trajectory to avoid a collision (also referred to as time to last moment braking (TTLSB)). In this example, the time to last moment braking is 11:30:42 AM. The system may now determine that a collision will occur in more than a threshold amount of time and send a message indicating as such, noting that the trajectory is valid for now. At 11:30:42 AM (2 seconds into the future), the system may determine whether there is a potential collision for the vehicle (e.g., based on the trajectory or new trajectory, the vehicle is still on a path to collide with the object). If a potential collision is detected, the system can initiate a maximum deceleration trajectory (or another movement) to avoid the collision.

[0101] 4 illustrates an example process 400 for determining whether a trajectory for a vehicle is valid or invalid. In some examples, process 400 may be performed by trajectory authorization component 220(2) of FIG. 2. In other examples, process 400 may be performed by other components and / or for other trajectories.

[0102] At operation 402, the system may determine whether a secondary trajectory for the vehicle was generated less than a threshold amount of time ago. For example, the system may determine whether a secondary trajectory currently being output by the primary system was generated less than a threshold amount of time ago. If the system determines that the secondary trajectory was generated less than the threshold amount of time ago, process 400 may proceed to operation 404. Alternatively, if the system determines that the secondary trajectory was not generated less than the threshold amount of time ago, process 400 may proceed to operation 406.

[0103] In operation 404, the system may select a trajectory for evaluation. In one example, if a primary trajectory is currently being used to control the vehicle, the system may select a secondary trajectory that passed evaluation in operation 402 (e.g., a secondary trajectory that was generated less than a threshold amount of time ago). In another example, if the current trajectory being used has a lower level than the primary trajectory (e.g., a secondary trajectory, a collision avoidance trajectory, or any other lower level trajectory) and a release signal has been received to release control to the primary trajectory, the system may select the secondary trajectory that passed evaluation in operation 402. In yet another example, if a release signal has not been received to release control from the previous secondary trajectory, the system may select the previous secondary trajectory (e.g., ignoring any updated secondary trajectory that has been received).

[0104] Thus, in operation 404, the system may select a previously stored secondary trajectory or a secondary trajectory currently being output from the primary system. To do so, the system may determine the selected state (e.g., primary trajectory state, secondary trajectory state, collision avoidance trajectory state, etc.) and determine whether a release signal has been received to release control to a higher level trajectory. For example, a secondary trajectory currently being output by the primary system may be selected if another secondary trajectory has not been previously obtained or a release signal has been received. Additionally, a secondary trajectory previously output by the primary system may be selected if such a secondary trajectory has been obtained and a release signal has not been received.

[0105] At operation 408, the system may determine whether the selected trajectory is consistent with the current or previous position of the vehicle. If the system determines that the selected trajectory is consistent with the current or previous position of the vehicle, process 400 may proceed to operation 410. Alternatively, if the system determines that the trajectory is not consistent with the current or previous position of the vehicle, process 400 may proceed to operation 406.

[0106] At operation 410, the system may determine whether the selected trajectory is compatible with the vehicle's performance. If the system determines that the selected trajectory is compatible with the vehicle's performance, process 400 may proceed to operation 412. Alternatively, if the system determines that the selected trajectory is not compatible with the vehicle's performance, process 400 may proceed to operation 406.

[0107] At operation 412, the system may determine whether the trajectory is collision-free. If the system determines that the trajectory is collision-free, process 400 may proceed to operation 414. Alternatively, if the system determines that the trajectory is not collision-free (e.g., associated with a collision), process 400 may proceed to operation 416.

[0108] At operation 416, the system may determine whether a collision will occur in more than a threshold amount of time. If the system determines that a collision will occur in more than a threshold amount of time (e.g., there is still time before the time to brake at the last moment), process 400 may proceed to operation 414. In some instances when proceeding along the route to operation 414, the system may send a message indicating that the trajectory is associated with a collision, similar to operation 316 of FIG. 3. Alternatively, if the system determines that a collision will occur within the threshold amount of time or within less than the threshold amount of time, process 400 may proceed to operation 406.

[0109] At operation 406, the system may determine that the selected trajectory is illegal. In examples, the system may update the state of the selected trajectory to indicate that the selected trajectory is illegal. In some instances, the state may indicate why the selected trajectory is illegal.

[0110] In some examples, process 400 may proceed to operation 418 after performing operation 406. At operation 418, the system may send a message indicating that the trajectory is invalid. For example, the system may be a secondary system and may send a message to the primary system to notify the primary system that the trajectory generated by the primary system is invalid. In an example, the message may indicate why the trajectory is invalid.

[0111] At operation 414, the system may determine that the selected trajectory is valid and / or may store the selected trajectory. In examples, the system may update the status of the selected trajectory to indicate that the selected trajectory is valid. In at least some examples, such a determination that the secondary trajectory is valid may also be sent to the primary system. In examples, the system may be a secondary system and may send a message to the primary system to inform the primary system that the trajectory is valid.

[0112] 5 illustrates an example process 500 for determining whether a trajectory for a vehicle is valid or invalid. In some examples, process 500 may be performed by trajectory authorization component 220(3) of FIG. 2. In other examples, process 500 may be performed by other components and / or for other trajectories.

[0113] At operation 502, the system may determine whether a secondary trajectory for the vehicle was generated less than a threshold amount of time ago. For example, the system may determine whether a secondary trajectory currently being output by the primary system was generated less than a threshold amount of time ago. If the system determines that the secondary trajectory was generated less than the threshold amount of time ago, process 500 may proceed to operation 504. Alternatively, if the system determines that the secondary trajectory was not generated less than the threshold amount of time ago, process 500 may proceed to operation 506.

[0114] At operation 504, the system may select a trajectory for evaluation. In one example, if a primary trajectory is currently being used to control the vehicle, the system may select a secondary trajectory that passed evaluation at operation 502 (e.g., a secondary trajectory that was generated less than a threshold amount of time ago). In another example, if the current trajectory being used has a lower level than the primary trajectory (e.g., a secondary trajectory, a collision avoidance trajectory, or any other lower level trajectory) and a release signal has been received to release control to the primary trajectory, the system may select the secondary trajectory that passed evaluation at operation 502. In yet another example, if a release signal has not been received to release control from a previous collision avoidance trajectory, the system may select the previous collision avoidance trajectory (e.g., by not considering any updates and / or by not updating the collision avoidance trajectory).

[0115] Thus, at operation 504, the system may select a previously stored collision avoidance trajectory or a secondary trajectory currently being output from the primary system. To do so, the system may determine the selected state (e.g., primary trajectory state, secondary trajectory state, collision avoidance trajectory state, etc.) and determine whether a release signal has been received to release control to a higher level trajectory. For example, the secondary trajectory currently being output by the primary system may be selected if another secondary trajectory has not been previously obtained, a release signal has been received, or the system otherwise determines that an additional update should be ignored (e.g., if the system may not be able to verify the integrity of the additional secondary trajectory). Furthermore, a previous collision avoidance trajectory may be selected if such a collision avoidance trajectory has been obtained and a release signal has not been received. In some examples, the collision avoidance trajectory (determined / output by process 500, as discussed below) may be a secondary trajectory. In other examples, the collision avoidance trajectory (determined / output by process 500) may be a modified secondary trajectory.

[0116] At operation 508, the system may determine whether the selected trajectory is consistent with the current or previous position of the vehicle. If the system determines that the selected trajectory is consistent with the current or previous position of the vehicle, process 500 may proceed to operation 510. Alternatively, if the system determines that the trajectory is not consistent with the current or previous position of the vehicle, process 500 may proceed to operation 506.

[0117] At operation 510, the system may determine whether the selected trajectory is compatible with the performance of the vehicle. If the system determines that the selected trajectory is compatible with the performance of the vehicle, process 500 may proceed to operation 512. Alternatively, if the system determines that the selected trajectory is not compatible with the performance of the vehicle, process 500 may proceed to operation 506.

[0118] At operation 512, the system may determine whether the trajectory is collision-free. If the system determines that the trajectory is collision-free, process 500 may proceed to operation 514. Alternatively, if the system determines that the trajectory is not collision-free (e.g., associated with a collision), process 500 may proceed to operation 516.

[0119] At operation 516, the system may determine whether a collision will occur in more than a threshold amount of time. If the system determines that a collision will occur in more than a threshold amount of time (e.g., there is still time before the time to brake at the last moment), process 500 may proceed to operation 514. Alternatively, if the system determines that a collision will occur in the threshold amount of time or in less than the threshold amount of time, process 500 may proceed to operation 506.

[0120] 5, in some examples, operation 516 may not be performed. For example, if the trajectory is associated with a collision, process 500 may proceed to operation 506.

[0121] At operation 514, the system may determine that the selected trajectory is valid and / or may store the selected trajectory. In an example, the system may update the status of the selected trajectory to indicate that the selected trajectory is valid. In an example, the system may be a secondary system and may send a message to the primary system to inform the primary system that the trajectory is valid.

[0122] At operation 506, the system may determine whether an iteration limit (sometimes referred to as a limit) has been met. The iteration limit may be used such that if the system cannot determine an acceptable collision avoidance trajectory within a number of iterations, the collision avoidance trajectory is deemed invalid. This may limit the amount of time spent attempting to find an acceptable collision avoidance trajectory. In an example, the iteration limit may be based on the time to braking at the last moment (e.g., set to the number of iterations that can be reached before the time to braking at the last moment). In an example, operation 520 is performed at each iteration.

[0123] If the system determines that the iteration limit is met, process 500 may proceed to operation 518. Alternatively, if the system determines that the iteration limit is not met, process 500 may proceed to operation 520.

[0124] At operation 520, the system may modify the selected trajectory and return to perform operation 508, etc. In some examples, the system may add an amount of deceleration needed to stop the vehicle before colliding with the object. This may be based on the current speed of the vehicle, the predicted trajectory of the object, and / or the current or predicted speed of the object. Alternatively, or in addition, the system may modify the selected trajectory in another manner, such as by changing the steering angle, by adding acceleration, etc. This may also avoid a collision with the object. In an example, modifying the selected trajectory may create a collision avoidance trajectory.

[0125] At operation 518, the system may determine that the selected trajectory is illegal. In examples, the system may update the status of the selected trajectory to indicate that the selected trajectory is illegal. In some instances, the status may indicate why the selected trajectory is illegal.

[0126] In some examples, process 500 may proceed to operation 522 after performing operation 518. At operation 522, the system may send a message indicating that the selected trajectory is invalid. For example, the system may be a secondary system and may send a message to the primary system to notify the primary system that the trajectory is invalid. In an example, the message may indicate why the trajectory is invalid.

[0127] In an example, the trajectory determined to be valid in act 514 may be the secondary trajectory without any deceleration (or other modification) applied in act 520. If the secondary trajectory selected in act 504 does not require modification, for example because the secondary trajectory passed the checks in acts 508, 510, and 512, the secondary trajectory may be determined / output by process 500. In another example, the trajectory determined to be valid in act 514 may be the secondary trajectory with deceleration (or other modification) applied. In an example, either type of trajectory determined / output by performing process 500 (and / or stored for future reference) may be referred to as a collision avoidance trajectory.

[0128] In an example, if the maximum amount of deceleration / correction is applied to the selected trajectory (e.g., by considering the vehicle's performance and performing operation 520 one or more times) and the selected trajectory is still associated with a collision, the system may proceed to override the detected collision and determine at operation 514 that the selected trajectory is reasonable. This may allow the modified trajectory (the collision avoidance trajectory with the maximum amount of deceleration / correction applied) to be used. In an example, this may be more beneficial than indicating that the modified trajectory is unreasonable and initiating a maximum deceleration trajectory (e.g., an emergency stop trajectory), because the maximum deceleration trajectory may fix the vehicle's steering angle, while the collision avoidance trajectory may maintain lateral tracking (e.g., allow the vehicle to continue along the collision avoidance trajectory). In an example, if the collision avoidance trajectory with the maximum amount of deceleration / correction applied is selected to be used (e.g., determined / output from process 500), a signal may be sent to one or more components of the vehicle to tension a seat belt, prepare an airbag for deployment, etc.

[0129] 6 illustrates an example hierarchy 600 of states 602-608 that may be implemented by a secondary system, such as by trajectory selector 222. For example, primary trajectory state 602 may be active when a primary trajectory is selected to control the vehicle, secondary trajectory state 604 may be active when a secondary trajectory is selected, collision avoidance trajectory state 606 may be active when a collision avoidance trajectory is selected, and maximum deceleration trajectory state 608 may be active when a maximum deceleration trajectory (e.g., an emergency stop trajectory) is selected. In FIG. 6, hierarchy 600 is ordered according to levels, with the highest level states at the top (primary trajectory state 602) and the lowest level states at the bottom (maximum deceleration trajectory state 608).

[0130] A secondary system may generally transition to a lower level state when a higher level state is found to be invalid. The secondary system may transition from a current state to any lower state, as illustrated by the descending lines. In at least some examples, the current state may be associated with the highest valid state with no invalid states below (e.g., if the secondary trajectory state 604 is invalid and the collision avoidance trajectory state is invalid, such a system may default to the maximum deceleration trajectory state 608). Further, in examples, to transition up the hierarchy 600, the secondary system may wait to receive a signal to release control from a lower level state. For example, if the secondary system is currently controlling the vehicle based on a collision avoidance trajectory, and thus, if the secondary system is operating in the collision avoidance trajectory state 606, the secondary system may transition back to the primary trajectory state 602 (the vehicle is controlled based on the primary trajectory) when a signal is received to release control back to the primary trajectory state 602.

[0131] 7A-7B illustrate an example process 700 for implementing the techniques discussed herein. Process 700 is illustrated as a logical flow graph, each operation of which represents a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media that perform the recited operations when executed by one or more processors. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular abstract data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the process. Process 700 may be performed by any device or component, such as the autonomous vehicle 102, a service provider (e.g., a computing device of the service provider), or any other device.

[0132] For ease of illustration, the process 700 will be discussed as being performed by a first system 702 (sometimes referred to as a first component) and a second system 704 (sometimes referred to as a second component). For example, the first system 702 may include the primary system 106 of Figures 1 and 2, while the second system 704 may include the secondary system 108 of Figures 1 and 2.

[0133] 7A, the first system 702 may receive first sensor data at operation 706. For example, the first system 702 may receive one or more of LIDAR data, RADAR data, image data, depth data (time-of-flight, structured light, etc.), wheel encoder data, IMU data, etc. from a vehicle sensor.

[0134] At operation 708, the first system 702 may determine a first trajectory for the vehicle and a second trajectory for the vehicle. For example, the first system 702 may analyze the first sensor data to generate the first and second trajectories. In an example, the first system 702 may perform localization, perception, prediction, and / or planning operations to generate the first and second trajectories. In an example, the second trajectory may be a contingency trajectory for the first trajectory.

[0135] In operation 710 , the first system 702 can send the first and second trajectories to the second system 704 .

[0136] At operation 712, the second system 704 may receive second sensor data. For example, the second system 704 may receive one or more of LIDAR data, RADAR data, image data, wheel encoder data, IMU data, depth data (time-of-flight, structured light, etc.), etc. from the vehicle's sensors. In some examples, the second sensor may include a subset of the first sensor data, such as LIDAR data, RADAR data, wheel encoder data, IMU data, GPS data, and / or time-of-flight data. In other examples, the second sensor data may be the same as the first sensor data or may include more data than the first sensor data.

[0137] At operation 714 , the second system 704 may receive the first and second trajectories from the first system 702 .

[0138] At operation 716, the second system 704 may determine objects in the environment and / or determine object trajectories for the objects. For example, the second system 704 may detect objects and / or predict object trajectories for the objects based on the second sensor data.

[0139] At operation 718, the second system 704 may perform one or more first validation actions on the first trajectory. For example, the second system 704 may perform one or more first validation actions on the first trajectory for the vehicle and the object trajectory for the object to determine a first state associated with the first trajectory. In an example, the one or more first validation actions include determining whether the first trajectory was generated less than a threshold amount of time ago, determining whether the first trajectory is consistent with a current or previous attitude of the vehicle, determining whether the first trajectory is compatible with the performance of the vehicle, and / or determining whether a vehicle moving along the first trajectory will collide with an object moving along the object trajectory (e.g., performing a first collision detection). In an example, the first state indicates that the first trajectory is valid, that the first trajectory is invalid, or that the first trajectory will become invalid due to an estimated collision occurring in more than a threshold amount of time. In examples, the second system 704 may evaluate a first trajectory for each object detected in the environment (e.g., determine whether there is a potential collision for each detected object), where the first trajectory may be valid if there are no collisions and invalid if there is at least one collision. In at least some examples, the objects selected for collision detection may be determined in order from closest (i.e., closest potential collision) to furthest.

[0140] At operation 720, the second system 704 may perform one or more second validation actions with respect to the second trajectory. For example, the second system 704 may perform one or more second validation actions on the second trajectory for the vehicle and the object trajectory for the object to determine a second state associated with the second trajectory. In examples, the one or more second validation actions include determining whether the second trajectory was generated less than a threshold amount of time ago, determining whether the second trajectory is consistent with a current or previous attitude of the vehicle, determining whether the second trajectory is compatible with the performance of the vehicle, and / or determining whether a vehicle moving along the second trajectory will collide with an object moving along the object trajectory (e.g., performing a second collision detection). In examples, the second state indicates that the second trajectory is valid, that the second trajectory is invalid, or that the second trajectory becomes invalid due to an estimated collision occurring in more than a threshold amount of time. In an example, the second system 704 may evaluate a second trajectory for each object detected in the environment.

[0141] At operation 722, the second system 704 may generate a third trajectory for the vehicle. For example, the second system 704 may generate the third trajectory based on the second trajectory (or the first trajectory in some cases) and / or independently from the previously generated trajectory. The third trajectory may include deceleration or other corrections along the second trajectory, and / or any other movement (e.g., a collision avoidance trajectory). In an example, the second system 704 may apply deceleration / correction to the second trajectory any number of times based on an iteration limit and / or apply deceleration / correction up to a maximum amount of deceleration / correction available to apply.

[0142] At operation 724, the second system 704 may perform one or more third validation actions with respect to the third trajectory. For example, the second system 704 may perform one or more third validation actions on the third trajectory for the vehicle and the object trajectory for the object to determine a third state associated with the third trajectory. In an example, the one or more third validation actions include determining whether the third trajectory was generated less than a threshold amount of time ago, determining whether the third trajectory is consistent with a current or previous attitude of the vehicle, determining whether the third trajectory is compatible with the performance of the vehicle, and / or determining whether a vehicle moving along the third trajectory will collide with an object moving along the object trajectory (e.g., performing a third collision detection). In an example, the second state indicates that the third trajectory is valid, that the third trajectory is invalid, or that the third trajectory becomes invalid due to an estimated collision occurring in more than a threshold amount of time.

[0143] In examples, any number of the one or more first authorization actions, the one or more second authorization actions, and / or the one or more third authorization actions may be the same. In other examples, any number of the authorization actions may be different.

[0144] At operation 726, the second system 704 may monitor the first system 702, the second system 704, and / or components of the vehicle. Based on the monitoring, the second system 704 of the vehicle may determine an error related to the first system 702, the second system 704, and / or components.

[0145] In operation 728, the second system 704 may determine a fourth trajectory for the vehicle. For example, the second system 704 may select the first trajectory, the second trajectory, the third trajectory, or the fifth trajectory as the fourth trajectory (e.g., a maximum deceleration trajectory) based on the first state, the second state, and / or the third state.

[0146] In an example, the second system 704 can select the first trajectory when the first state indicates that the first trajectory is valid. Further, in an example, the second system 704 can select the second trajectory when (i) the first state indicates that the first trajectory is invalid (e.g., associated with an estimated crash within less than a threshold amount of time) and the second state indicates that the second trajectory is valid, or (ii) an error is detected in the first system 702, the second system 704, and / or a component of the vehicle and the second state indicates that the second trajectory is valid.

[0147] Moreover, in examples, the second system 704 can select a third trajectory (e.g., a collision avoidance trajectory) when (i) the first state indicates that the first trajectory is invalid, the second state indicates that the second trajectory is invalid, and the third state indicates that the third trajectory is valid; (ii) an error is detected in the first system 702, the second system 704, and / or a component of the vehicle and the second state indicates that the second trajectory is invalid and the third state indicates that the third trajectory is valid; or (iii) the first state indicates that the first trajectory is invalid, the second state indicates that the second trajectory is invalid, the third trajectory is associated with a collision, the third trajectory is associated with a maximum amount of deceleration / correction, and an iteration limit is not met in applying the deceleration / correction to the third trajectory. Thus, in some examples, the second system 704 can determine whether the maximum amount of deceleration / correction has been applied to generate the third trajectory and / or whether an iteration limit has been met.

[0148] Further, in an example, the second system 704 may select a maximum deceleration trajectory (e.g., an emergency stop trajectory) if the first trajectory, the second trajectory, or the third trajectory are not selected as discussed above (e.g., the first trajectory is illegal, the second trajectory is illegal, and the third trajectory is illegal). As noted above, the third trajectory may be illegal due to an iteration limit being met in applying deceleration / correction to the third trajectory.

[0149] In FIG. 7B, at operation 730, the second system 704 can send a message to the first system 702 (e.g., if the trajectory is invalid). The message can indicate that the trajectory is invalid and / or that the second system 704 will intervene if control of the vehicle is maintained based on the trajectory. In some examples, a message may not be sent (e.g., if the first trajectory is selected). In at least some examples, such message data can include, for example, time to collision, point of collision, information about the potentially colliding object (e.g., any one or more of object width, object identifier, object position, object velocity), etc. In at least some examples, one or more messages can be sent constantly to the first system 702 regardless of the validity of any trajectory to indicate the validity, one or more outputs from the process (recognition, localization, confirmation, etc.), time to collision, etc.

[0150] Additionally or alternatively, in an example, if a maximum deceleration trajectory or a collision avoidance trajectory (with the maximum amount of deceleration / correction applied) is selected in operation 728, a message or signal may be sent to one or more components of the vehicle to tension seat belts, prepare airbags for deployment, etc.

[0151] At operation 732, the first system 702 may receive a message from the second system 704. In an example, the first system 702 may adjust an trajectory, such as a primary trajectory, based on the message.

[0152] In operation 734, the second system 704 can control the vehicles based on the fourth trajectory. For example, the second system 704 can send a signal to a drive manager, a system controller, etc., instructing the control vehicles based on the fourth trajectory.

[0153] In operation 736, the second system 704 may maintain control of the vehicle based on the fourth trajectory. For example, the second system 704 may maintain the second system 704 with the second trajectory, the third trajectory, and / or the maximum deceleration trajectory obtained, which may include continuing to send a signal to the system controller.

[0154] At operation 738, the second system 704 can receive a signal from the fourth track to release control of the vehicle. In an example, the signal can be received from a remote operation system or another system. The signal can indicate that the vehicle is collision-free or clear to return to normal operation in another situation. For example, if the fourth track brought the vehicle to a halt due to a potential collision, the signal can indicate that it is safe for the vehicle to move again.

[0155] In operation 740, the first system 702 may generate a sixth trajectory. The sixth trajectory may include a primary / main trajectory.

[0156] In operation 742 , the first system 702 may send the sixth trajectory to the second system 704 .

[0157] In operation 744 , the second system 704 may receive the sixth trajectory from the first system 702 .

[0158] In operation 746, the second system 704 returns to operation 716 to evaluate the sixth trajectory and, if applicable, controls the vehicle based on the sixth trajectory.

[0159] 8-10 illustrate example processes 800, 900, and 1000 for implementing the techniques discussed herein. Processes 800, 900, and 1000 are illustrated as logical flow graphs, each operation of which represents a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media that perform the recited operations when executed by one or more processors. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular abstract data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the process. Processes 800, 900, and 1000 may be performed by any device or component, such as the autonomous vehicle 102, a service provider (e.g., a computing device of the service provider), or any other device.

[0160] 8 illustrates an example process 800 that may be performed by various components of a primary system and / or a secondary system. For ease of illustration, process 800 is illustrated in the context of FIG. 1 and / or FIG. 2. However, process 800 may be performed in other environments.

[0161] In an example, the primary system 106 implements a first technique to implement at least one of a first object detection, a first object segmentation, a first object classification, or a first object tracking, while the secondary system 108 implements a second technique to implement at least one of a second object detection, a second object segmentation, a second object classification, or a second object tracking. The first and second techniques may be similar, the same, or different.

[0162] At operation 802, the primary system 106 may receive sensor data from one or more sensors. For example, the primary system 106 may receive sensor data 114 from the sensors 104 of the autonomous vehicle 102.

[0163] At operation 804, the primary system 106 may generate a trajectory for the vehicle. For example, the primary system 106 may generate the primary trajectory 116 and / or the secondary trajectory 118 based at least in part on the sensor data 114.

[0164] At operation 806, the primary system 106 may output the trajectory. For example, the primary system 106 may send the primary trajectory 116 and / or the secondary trajectory 118 to a secondary system 108, such as the trajectory manager 214.

[0165] At 808, the trajectory manager 214 may receive the trajectory. For example, the trajectory manager 214 may receive the primary trajectory 116 and / or the secondary trajectory 108 from the primary system 106.

[0166] At operation 810, localizer 210 may receive at least a portion of the sensor data. For example, localizer 210 may receive at least a portion of sensor data 120 from sensor 104. Localizer 210 may be communicatively coupled to trajectory manager 214, perceiver 212, drive manager 226, and / or any other components.

[0167] At operation 812, the localizer 210 may determine a vehicle attitude. For example, the localizer 210 may determine an attitude of the autonomous vehicle 102 based at least in part on the sensor data 120. As above, such attitude may be either relative to a map (which may be received from the primary system 106) or relative to one or more previous attitudes of the autonomous vehicle 102.

[0168] At operation 814, the localizer 210 may output data indicative of the vehicle's attitude. For example, the localizer 210 may send data indicative of the autonomous vehicle 102's attitude to the perceiver 212, the trajectory manager 214, and / or the drive manager 226.

[0169] At operation 816, the drive manager 226 may receive data indicative of the vehicle's attitude from the localizer 210. The drive manager 226 may be communicatively coupled to the localizer 210, the trajectory manager 214, the monitor 216, the drive system, and / or any other components.

[0170] In operation 818, the trajectory manager 214 may receive data from the localizer 210 indicative of the vehicle's attitude.

[0171] At operation 820, the perceiver 212 may receive data indicative of the vehicle's attitude from the localizer 210. The perceiver 212 may be communicatively coupled to the localizer 210, the trajectory manager 214, and / or any other components.

[0172] At operation 822, the perceiver 212 may receive at least a portion of the sensor data. For example, the perceiver 212 may receive at least a portion of the sensor data 120 from the sensor 104. In at least some examples, the portion of the sensor data received at 822 may be different than the portion of the sensor data received at 810.

[0173] At operation 824, the perceiver 212 may determine objects in the environment, tracks for the objects, and / or object trajectories for the objects. For example, the perceiver 212 may determine the presence of an object 126, characteristics of the object 126, tracks for the object 126, and / or object trajectories for the object 126 based at least in part on the sensor data 120.

[0174] At operation 826, the perceiver 212 may output data indicative of the object characteristics, the track for the object, and / or the object trajectory. For example, the perceiver 212 may send data indicative of the object characteristics (e.g., extent, speed, acceleration, position, orientation, etc.) of the object 126, the track for the object 126, and / or the object trajectory for the object 126 to the trajectory manager 214.

[0175] In operation 828, the trajectory manager 214 may receive data from the perceiver 212 indicative of characteristics of the object, a track for the object, and / or an object trajectory.

[0176] At operation 830, trajectory manager 214 may receive at least a portion of the sensor data. For example, trajectory manager 214 may receive at least a portion of sensor data 120 from sensor 104. In some examples, the portion received at 830 may be different than either portion received at 810 or 812. Trajectory manager 214 may be communicatively coupled to localizer 210, perceiver 212, monitor 216, drive manager 226, primary system 106, and / or any other component or system.

[0177] At operation 832, the monitor 216 may monitor the primary system 106 and / or any other components of the vehicle. For example, the monitor 216 may receive signals from components of the primary system 106 / secondary system 108, signals from the drive manager 226 (or actuators (brakes, steering, etc.)), signals (e.g., diagnostic information) from a service provider (e.g., a remote device) indicating hardware or software faults, signals from a battery or other components of the autonomous vehicle 102 associated with providing power to the primary system 106 / secondary system 108, etc. The monitor 216 may be communicatively connected to the primary system 106, the trajectory manager 214, the drive manager 226, and / or any components of the vehicle, such as the autonomous vehicle 102.

[0178] At operation 834, the monitor 216 may determine that an error has occurred. For example, the monitor 216 may determine that an error has occurred if there is a hardware and / or software failure (e.g., a sensor 104, actuator, etc. failure) for the primary system 106, the secondary system 108, and / or another component, if power to the primary system 106, the secondary system 108, and / or another component is disrupted, if no signals are received from the primary system 106, the secondary system 108, and / or another component for a period of time (e.g., longer than the period of time that would indicate that the primary system 106 is experiencing a problem), etc.

[0179] At operation 836, the monitor 216 may output a signal indicating the error. For example, the monitor 216 may send a signal to the trajectory manager 214 and / or the drive manager 226 indicating that an error has occurred.

[0180] At operation 838 , the trajectory manager 214 may receive a signal from the monitor 216 .

[0181] At operation 840, the drive manager 226 may receive a signal from the monitor 216. In an example, if the error is severe (e.g., a predetermined type of error, an error with the secondary system 108, a combination of errors with components, etc.), the drive manager 226 may control the autonomous vehicle 102 based at least in part on a maximum deceleration trajectory or another trajectory.

[0182] At operation 842, the primary system 106 and the monitor 216 may communicate data. For example, the monitor 216 may receive data regarding an error in the primary system 106. Such data may be used at operation 834 to determine that an error has occurred. In another example, the monitor 216 may send data to the primary system 106 indicating that the monitor 216 has detected an error.

[0183] 9 illustrates an example process 900 that may be performed by various components of a primary system and / or a secondary system. For ease of illustration, process 900 is illustrated in the context of FIG. 1 and / or FIG. 2. However, process 900 may be performed in other environments. In an example, process 900 may be performed after process 800 is performed. In other examples, process 900 may be performed at other times.

[0184] At operation 902, collision detection component 218(1) may perform a first collision detection. For example, collision detection component 218(1) may receive primary trajectory 116 and determine whether primary trajectory 116 is associated with a collision with one or more objects. The determination may be based on data received from localizer 210, perceiver 212, and / or (in some cases) components of primary system 106. Collision detection component 218(1) may be communicatively coupled to primary system 106, trajectory authorization component 220(1), and / or any other components.

[0185] At operation 904, collision detection component 218(1) can output data regarding the first collision detection. For example, collision detection component 218(1) can output data indicating whether a collision is detected, the probability of a collision, etc. to trajectory authorization component 220(1).

[0186] In operation 906, the trajectory authorization component 220(1) may receive data from the collision detection component 218(1).

[0187] At operation 908, collision detection component 218(2) may perform a second collision detection. For example, collision detection component 218(2) may receive secondary trajectory 118 and determine whether secondary trajectory 118 is associated with a collision with one or more objects. The determination may be based on data received from localizer 210, perceiver 212, and / or (in some cases) components of primary system 106. Collision detection component 218(2) may be communicatively coupled to primary system 106, trajectory authorization component 220(2), and / or any other components.

[0188] At operation 910, collision detection component 218(2) can output data regarding the second collision detection. For example, collision detection component 218(2) can output data indicative of whether a collision is detected, the probability of a collision, etc. to trajectory authorization component 220(2) and / or trajectory authorization component 220(3).

[0189] At operation 912, the trajectory authorization component 220(2) may receive data from the collision detection component 218(2).

[0190] At operation 914, the trajectory authorization component 220(3) may receive data from the collision detection component 218(2).

[0191] At operation 916, trajectory authorization component 220(1) may perform one or more first authorization actions. For example, trajectory authorization component 220(1) may perform one or more first authorization actions on primary trajectory 116 to determine a first state associated with primary trajectory 116. The one or more first authorization actions may be based on data from localizer 210, perceiver 212, collision detection component 218(1) (e.g., data received at operation 906), etc. The one or more authorization actions may include determining whether primary trajectory 116 was generated less than a threshold amount from a current time, determining whether primary trajectory 116 is consistent with a current or previous attitude of autonomous vehicle 102, determining whether primary trajectory 116 is compatible with performance of autonomous vehicle 102, and / or determining whether autonomous vehicle 102 moving along primary trajectory 116 will collide with an object moving along an object trajectory. In some examples, one or more authorization actions may be performed in a particular order, such as the order in which they are described above, where an authorization action may not be performed if at any point the primary trajectory 116 is determined to be invalid. For example, if it is determined that the primary trajectory 116 was generated more than a threshold amount of time ago, the primary trajectory 116 may be determined to be invalid and no other authorization action is performed.

[0192] The trajectory authorization component 220(1) may be communicatively coupled to the collision detection component 218(1), the trajectory selector 222, and / or any other components.

[0193] At operation 918, trajectory authorization component 220(1) may output a first status. For example, trajectory authorization component 220(1) may send data to trajectory selector 222 indicating whether primary trajectory 116 is valid or invalid, and (if it is invalid) why it is invalid, etc. In an example, the first status may indicate whether primary trajectory 116 is associated with a collision.

[0194] At operation 920, trajectory authorization component 220(2) may perform one or more second authorization actions. For example, trajectory authorization component 220(2) may perform one or more second authorization actions on secondary orbit 118 to determine a second condition associated with secondary orbit 118. The one or more second authorization actions may be based on data from localizer 210, perceiver 212, collision detection component 218(2) (e.g., data received at operation 912), etc. In an example, the one or more second authorization actions may be similar or the same as the one or more first authorization actions.

[0195] The trajectory authorization component 220(2) may be communicatively coupled to the collision detection component 218(2), the trajectory selector 222, and / or any other components.

[0196] At operation 922, trajectory authorization component 220(2) can output a second status. For example, trajectory authorization component 220(2) can send data to trajectory selector 222 indicating whether secondary trajectory 118 is valid or invalid, and (if it is invalid) why it is invalid, etc. In an example, the second status can indicate whether secondary trajectory 118 is associated with a collision.

[0197] In operation 924, trajectory authorization component 220(3) may perform one or more third authorization actions. For example, trajectory authorization component 220(3) may perform one or more third authorization actions on secondary orbit 118 and / or a tertiary orbit (e.g., a collision avoidance trajectory) generated by trajectory authorization component 220(3) to determine a third condition associated with secondary orbit 118 or the tertiary orbit. In an example, if secondary orbit 118 is invalid, trajectory authorization component 220(3) may generate a tertiary orbit and perform one or more third authorization actions on the tertiary orbit. The tertiary orbit may include a correction to primary orbit 116 or secondary orbit 118. The one or more third authorization actions may be based on data from localizer 210, perceiver 212, collision detection component 218(2) (e.g., data received in operation 914), etc. In examples, the one or more third authorization actions may be similar or the same as the one or more first authorization actions.

[0198] The trajectory authorization component 220(3) may be communicatively coupled to the collision detection component 218(2), the trajectory selector 222, and / or any other components.

[0199] At operation 926, trajectory authorization component 220(3) can output a third state. For example, trajectory authorization component 220(3) can send data to trajectory selector 222 indicating whether secondary trajectory 118 or tertiary trajectory is valid or invalid, and (if it is invalid) why it is invalid, and / or the modified trajectory (tertiary trajectory) itself, etc. In an example, the third state can indicate whether secondary trajectory 118 or tertiary trajectory is associated with a collision.

[0200] At operation 928, trajectory selector 222 may receive one or more conditions for the trajectory. For example, trajectory selector 222 may receive data from trajectory authorization component 220(1) indicating a first condition, data from trajectory authorization component 220(2) indicating a second condition, and / or data from trajectory authorization component 220(3) indicating a third condition.

[0201] Trajectory selector 222 may be communicatively coupled to trajectory authorization component 220(1), trajectory authorization component 220(2), trajectory authorization component 220(3), releaser 224, monitor 216, drive manager 226, and / or any other components.

[0202] At operation 930, the trajectory selector 222 may determine an output trajectory. For example, the determination may be based at least in part on a first state for the primary trajectory 116, a second state for the secondary trajectory 118, a third state for the secondary trajectory 118 or the tertiary trajectory, a signal from the monitor 216, and / or a signal from the releaser 224.

[0203] In an example, the trajectory selector 222 may select the primary trajectory 116 when the primary trajectory 116 is valid. The trajectory selector 222 may select the second trajectory 118 when (i) the primary trajectory 116 is invalid and the second trajectory is valid, or (ii) an error is reported by the monitor 216 and the secondary trajectory 118 is valid. The trajectory selector 222 may select a tertiary trajectory (e.g., a collision avoidance trajectory) when (i) the primary trajectory 116 is invalid, the secondary trajectory 118 is invalid, and the tertiary trajectory is valid, (ii) an error is reported by the monitor 216, the secondary trajectory 118 is invalid, and the tertiary trajectory is valid, or (iii) the primary trajectory 116 is invalid, the secondary trajectory 118 is invalid, the tertiary trajectory is associated with a collision, the tertiary trajectory is associated with a maximum amount of deceleration / correction, and an iteration limit is not met in applying the deceleration / correction to the tertiary trajectory. Additionally, the trajectory selector 222 may select a maximum deceleration trajectory (eg, an emergency stop trajectory) if the primary trajectory 116, the secondary trajectory 118, or the tertiary trajectory is not selected.

[0204] 10 illustrates an example process 1000 that may be performed by various components of a primary system and / or a secondary system. For ease of illustration, process 1000 is illustrated in the context of FIG. 1 and / or FIG. 2. However, process 1000 may be performed in other environments. In an example, process 1000 may be performed after process 900 is performed. In other examples, process 1000 may be performed at other times.

[0205] In operation 1002, the trajectory selector 222 may output an output trajectory. For example, the trajectory selector 222 may output the output trajectory determined in operation 930 of Figure 9. The output trajectory may be sent to the drive manager 226.

[0206] At operation 1004, the drive manager 226 may receive the output trajectory from the trajectory selector 222. The drive manager 226 may be communicatively coupled to the trajectory selector 222, the monitor 216, the localizer 210, and / or any other components.

[0207] At operation 1006, the drive manager 226 may control the vehicle. For example, the drive manager 226 may generate commands / instructions for controlling the autonomous vehicle 102 based at least in part on the output trajectory. The drive manager 226 may control steering, propulsion, braking, safety, emitter, communication, and / or other systems of the autonomous vehicle 102. In an example, the drive manager 226 may generate commands / instructions for controlling the autonomous vehicle 102 based on a previous trajectory for the autonomous vehicle 102, such as a trajectory used to control the autonomous vehicle 102 prior to implementing the output trajectory. Although not illustrated here in FIG. 10, in some examples, a system controller may receive commands provided by the drive manager 226 to actuate one or more control systems (steering, propulsion, braking, etc.).

[0208] In some examples, the drive manager 226 receives the attitude of the autonomous vehicle 102 from the localizer 210. If the attitude is invalid (e.g., an attitude validator of the localizer 210 indicates so), the drive manager 226 can control the autonomous vehicle 102 based on a maximum deceleration trajectory (e.g., an emergency stop trajectory).

[0209] At operation 1008, the trajectory authorization components 220 can output a message regarding an illegal trajectory or collision. For example, if a trajectory is illegal or is associated with a collision (e.g., in the future), any one of the trajectory authorization components 220 can send a message regarding such trajectory to the primary system 106. The message can indicate the time until collision, the width of the object, the velocity of the object, the location of the object, the point of the collision, the reason it is illegal, etc.

[0210] In an operation 1010 , the primary system 106 may receive a message from any one of the orbit authorization components 220 .

[0211] At operation 1012, the trajectory selector 222 may maintain the output trajectory. For example, if the secondary trajectory 118, the tertiary trajectory, or the maximum deceleration trajectory is output to the drive manager 226 due to an error, an estimated collision, or otherwise, the trajectory selector 222 may maintain control of the autonomous vehicle 102 based on the output trajectory (e.g., maintain control at a lower level state). In at least some examples, this corresponds to not considering any updated or newly provided trajectories, whereas in other examples, this corresponds to maintaining the level of the trajectory used for control. As a non-limiting example, the secondary system 108 may accept the updated primary and secondary trajectories and use the updated secondary trajectory even though the updated primary trajectory is valid.

[0212] At operation 1014, the releaser 224 may determine to release control of the vehicle from the output trajectory. In one example, a remote system may provide a signal to the secondary system 108 to release control from the current state, and in response, the releaser 224 may determine to release control from the current state. In another example, a component of the autonomous vehicle 102 may perform processing to determine that it is clear for the autonomous vehicle 102 to resume normal operation (or transition to a different state) and send a signal to the releaser 224 to release control from the current state. The releaser 224 may be communicatively coupled to the trajectory selector 222 and / or any other components.

[0213] At operation 1016, the releaser 224 may output a signal to release control of the vehicle from the output trajectory. For example, based at least in part on the determination at operation 1014, the releaser 224 may send a signal to the trajectory selector 222 to release control from a lower level trajectory / state (e.g., lower than the primary trajectory 116) to a higher level trajectory / state.

[0214] At 1018 , the trajectory selector 222 can receive a signal from the releaser 224 .

[0215] At operation 1020, the trajectory selector 222 may output a trajectory to the drive manager 226. For example, based at least in part on the signal received at operation 1018, the trajectory selector 222 may send the primary trajectory currently being provided by the primary system 106 (or, in some cases, a different trajectory) to the drive manager 226.

[0216] In operation 1022 , the drive manager 226 may receive a trajectory from the trajectory selector 222 .

[0217] At operation 1024, the drive manager 226 may control the vehicle. For example, the drive manager 226 may generate commands / instructions for controlling the autonomous vehicle 102 based at least in part on the trajectory.

[0218] FIG. 11 depicts a block diagram of an example system 1100 for implementing the techniques described herein. In some instances, the system 1100 may include a vehicle 1102 that may correspond to the autonomous vehicle 102 in FIG. 1. In some instances, the vehicle 1102 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire journey without a driver (or passenger) being expected to control the vehicle at any time. However, in other instances, the autonomous vehicle 1102 may be a fully or partially autonomous vehicle having any other level or classification. Moreover, in some instances, the techniques described herein may be usable by non-autonomous vehicles as well.

[0219] The vehicle 1102 may include a first system 1104 (e.g., a first computing device), a second system 1106 (e.g., a second computing device), one or more sensors 1108, one or more emitters 1110, one or more communication connections 1112, one or more drive systems 1114, and / or a direct connection 1116 (e.g., for physically connecting the vehicle 1102 to exchange data and / or to provide power).

[0220] In some instances, the sensors 1108 may include light detection and ranging (LIDAR) sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., Global Positioning System (GPS), compass, etc.), inertial sensors (e.g., Inertial Measurement Units (IMU), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., Red-Green-Blue (RGB), Infrared (IR), luminance, depth, time-of-flight, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. The sensors 1108 may include multiple instances of each of these or other types of sensors. For example, the LIDAR sensors may include individual LIDAR sensors positioned at the corners, front, rear, sides, and / or top of the vehicle 1102. As another example, the cameras may include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle 1102. The sensor 1108 can provide input to the first system 1104 and / or the second system 1106 .

[0221] The vehicle 1102 may also include emitters 1110 for emitting light and / or sound, as described above. The emitters 1110 in this example may include interior audio and visual emitters for communicating with occupants of the vehicle 1102. By way of example and not limitation, the interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The emitters 1110 in this example may also include exterior emitters. By way of example and not limitation, the exterior emitters in this example include lights or other indicators of vehicle action (e.g., indicator lights, signs, light arrays, etc.) to signal direction of movement, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which may include acoustic beam steering technology.

[0222] The vehicle 1102 may also include communication connections 1112 that facilitate communication between the vehicle 1102 and one or more other local or remote computing devices. For example, the communication connections 1112 may facilitate communication with other local computing devices on the vehicle 1102 and / or the drive system 1114. The communication connections 1112 may also additionally or alternatively enable the vehicle 1102 to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). The communication connections 1112 may additionally or alternatively facilitate the vehicle 1102 to communicate with a computing device 1118.

[0223] The communication connection 1112 may include physical and / or logical interfaces for connecting the first system 1104 and / or the second system 1106 to another computing device or network, such as the network 1120. For example, the communication connection 1112 may facilitate Wi-Fi-based communications, such as over frequencies defined by the IEEE 800.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communications protocol that facilitates each computing device interfacing with other computing devices. In some instances, the first system 1104, the second system 1106, and / or the sensor 1108 may send sensor data over the network 1120 to the computing device 1118 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0224] In some instances, the vehicle 1102 may include a drive system 1114. In some instances, the vehicle 1102 may have a single drive system 1114. In some instances, the drive system 1114 may include one or more sensors for detecting conditions surrounding the drive system 1114 and / or the vehicle 1102. By way of example and not limitation, the sensors of the drive system 1114 may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive module, cameras or other image sensors, ultrasonic sensors for acoustically detecting objects in the surroundings of the drive module, LIDAR sensors, RADAR sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive system 1114. In some cases, the sensors on the drive system 1114 may overlap with or supplement corresponding systems of the vehicle 1102 (e.g., sensors 1108).

[0225] The drive system 1114 can include many of the vehicle systems, including a high-voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights for illuminating the exterior surroundings of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high-voltage junctions, high-voltage cables, charging systems, charging ports, etc.). Additionally, the drive system 1114 can include a drive system controller for receiving and processing data from sensors and for controlling operation of various vehicle systems. In some instances, the drive system controller can include one or more processors and a memory communicatively coupled to the one or more processors. The memory can store one or more modules for implementing various functionality of the drive system 1114. In addition, the drive system 1114 can also include one or more communication connections that facilitate communication by the respective drive system with one or more other local or remote computing devices.

[0226] The first system 1104 may include one or more processors 1122 and a memory 1124 communicatively coupled to the one or more processors 1122. The second system 1106 may include one or more processors 1126 and a memory 1128 communicatively coupled to the one or more processors 1126. The computing device 1118 may also include a processor 1130 and / or a memory 1132 communicatively coupled to the processor 1130. The processors 1122, 1126, and / or 1130 may be any suitable processor capable of processing data and executing instructions to perform operations as described herein. By way of example and not limitation, processors 1122, 1126, and / or 1130 may include one or more central processing units (CPUs), graphics processing units (GPUs), integrated circuits (e.g., application specific integrated circuits (ASICs), etc.), gate arrays (e.g., field programmable gate arrays (FPGAs), etc.), and / or any other device or portion of a device that processes electronic data and converts the electronic data into other electronic data that may be stored in registers and / or memory.

[0227] The memories 1124, 1128, and / or 1132 may be examples of non-transitory computer-readable media. The memories 1124, 1128, and / or 1132 may store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods described herein and the functions attributed to the various systems. In various implementations, the memories may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which those shown in the accompanying figures are merely examples relevant to the discussion herein.

[0228] In an example, the memory 1124 of the first system 1104 can store one or more components 1134. For example, the first system 1104 may correspond to the primary system 106 of Figures 1 and 2 and store the localization component 202, the perception component 204, the prediction component 206, and / or the planning component 208. The processor 1122 can execute the one or more components 1134 to cause the first system 1104 to perform various actions discussed herein.

[0229] In an example, the memory 1128 of the second system 1106 can store one or more components 1136. For example, the second system 1106 may correspond to the secondary system 108 of Figures 1 and 2 and store the localizer 210, the perceiver 212, the trajectory manager 214, the monitor 216, and / or the drive manager 226. The processor 1126 can execute the one or more components 1136 to cause the second system 1106 to perform various actions discussed herein.

[0230] 11 as residing in memory 1124 and / or memory 1128, it is contemplated that components 1134 and / or components 1136 may additionally or alternatively be accessible to computing device 1118 (e.g., stored remotely). For example, memory 1132 can store a first system component 1138 that corresponds to at least a portion of component 1134 and / or a second system component 1140 that corresponds to at least a portion of component 1136. Additionally or alternatively, memory 1132 can store one or more maps 1142.

[0231] In at least one example, the first system 1104 and / or the second system 1106 can include one or more system controllers 1144 that can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 1102. In an example, the system controller 1144 can be stored in the memory 1124 and / or the memory 1128. The system controller 1144 can communicate with and / or control corresponding systems of the drive system 1114 and / or other components of the vehicle 1102. In some instances, the system controller 1144 can translate a trajectory generated by the first system 1104, a trajectory generated by the second system 1106, and / or a trajectory selected by the second system 1106 into instructions usable by the drive system 1114 to cause the vehicle 1102 to traverse the trajectory. In some examples, the one or more system controllers 1144 may be located on a hardware component separate from the first system 1104 and / or the second system 1106.

[0232] In some instances, the first system 1104, the second system 1106, the system controller 1144, and / or any components thereof may process the sensor data as described above and may send their respective outputs over the network 1120 to the computing device 1118 (e.g., at a particular frequency, after a predetermined period of time, in near real-time, etc.).

[0233] The first system 1104 and / or the second system 1106 can store one or more maps 1146 that can be used by the vehicle 1102 to navigate within the environment. For purposes of this discussion, a map can be any number of data features modeled in two, three, or N dimensions that can provide information about the environment, such as, but not limited to, topology (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some instances, the map can include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and / or intensities), reflectivity information (e.g., specularity information, retroreflectivity information, BRDF information, BSSRDF information, etc.). In one example, the map can include a three-dimensional mesh of the environment. In some instances, the map may be stored in a tiled format, with each tile of the map representing a discrete portion of the environment, and may be loaded into the working memory as needed. In some instances, the map may include an occupancy grid, a point map, a landmark map, and / or a posture constraint graph. In some instances, the vehicle 1102 may be controlled based at least in part on the map 1146. That is, the map 1146 may be used in conjunction with a localization component of the first system 1104 (and / or a localization component of the second system 1106 in some instances) to determine a position of the vehicle 1102, identify objects in the environment, and / or generate a route and / or trajectory for navigating within the environment.

[0234] In some instances, some or all aspects of the components discussed herein may include any model, algorithm, and / or machine learning algorithm. For example, the components 1134 in memory 1124, the components 1136 in memory 1128, and / or the components in memory 1132 may be implemented as neural networks.

[0235] 11 is illustrated as a distributed system, it should be noted that in alternative examples, components of the vehicle 1102 may be associated with the computing device 1118 and / or components of the computing device 1118 may be associated with the vehicle 1102. That is, the vehicle 1102 may perform one or more of the functions associated with the computing device 1118, and vice versa.

[0236] In some examples, the one or more components 1136 (and / or the one or more components 1134) include an estimation and verification component to determine a direction of motion of the vehicle 1102, a speed of the vehicle 1102, an acceleration of the vehicle 1102, a yaw rate of the vehicle 1102, a yaw acceleration, and / or a steering angle of the vehicle 1102. In examples, such information may be based on signals from the system controller 1144, the drive system 1114, etc. The estimation and verification component may generate data indicative of one or more of such information and provide the data to various components of the second system 1106 and / or the first system 1104.

[0237] Further, in some examples, one or more components 1136 (and / or one or more components 1134) can include a data processing component to filter the sensor data to generate a reduced amount of data. In one example, the data processing component can remove data from the sensor data that is outside of a certain range. This can be based on the speed or acceleration of the vehicle 1102, the track of the object, the speed or acceleration of the object, etc. (e.g., remove a significant amount of data when moving relatively slowly and remove less data when moving relatively quickly). To illustrate, if the vehicle 1102 is moving at 15 miles (24.14 kilometers) per hour and there are no vehicles moving toward the autonomous vehicle 102, the data processing component can remove data that is 200 feet (60.96 meters) away, as this data is not likely to be needed to evaluate the trajectory or identify a potential imminent collision. In another example, the data processing component can remove data from the sensor data based on the type of environment in which the vehicle 1102 is located. For example, if the vehicle 1102 is in an urban area with a large number of vehicles, roads, road intersections, etc., the data processing component may not remove data (or remove a limited amount) from the sensor data. However, if the vehicle 1102 is in a rural area or a sparsely populated area of ​​the urban environment with relatively few vehicles, roads, road intersections, etc., the data processing component may remove more data from the sensor data. In an example, reducing the amount of sensor data may enable the second system 1106, which can process the reduced sensor data, to operate more quickly (e.g., process less data, resulting in reduced computation time).

[0238] Example clauses A. An example vehicle system including a second system communicatively connected to a first system, the second system including one or more processors and a memory, the second system including a perceiver for receiving a portion of the sensor data, determining at least one of an object in an environment, a track for the object, or an object trajectory for the object based at least in part on the portion of the sensor data, and outputting data indicative of at least one of a property of the object, the track for the object, or the object trajectory, and a trajectory manager communicatively connected to the perceiver for receiving a trajectory for the vehicle from the first system generated by the first system based at least in part on the sensor data, receiving data from the perceiver, performing collision detection based at least in part on the data, determining states associated with the trajectory, determining an output trajectory based at least in part on the states associated with the trajectory, and controlling the autonomous vehicle based at least in part on the output trajectory.

[0239] B. The vehicle system of Example A, wherein the portion of the sensor data is a first portion of the sensor data, the data includes the first data, and the second system further includes a localizer communicatively connected to the perceiver and the trajectory manager, wherein the localizer is configured to receive a second portion of the sensor data, determine a vehicle attitude based at least in part on the second portion of the sensor data, and output second data indicative of the autonomous vehicle attitude to at least one of the perceiver or the trajectory manager, and the trajectory manager is configured to receive the second data from the localization component and perform collision detection based at least in part on the second data.

[0240] C. The vehicle system of Examples A or B, wherein the second system further includes a monitor communicatively connected to at least one of the components of the first system or the autonomous vehicle for monitoring at least one of the components of the first system or the autonomous vehicle, determining that an error has occurred based at least in part on the monitoring, and outputting a signal indicating that an error has occurred, and the trajectory manager is configured to receive the signal from the monitor and determine the output trajectory based at least in part on the signal from the monitor.

[0241] D. The vehicle system of any one of Examples A-C, wherein the second system further includes: a monitor communicatively connected to at least one of the components of the first system or the autonomous vehicle for monitoring at least one of the components of the first system or the autonomous vehicle, determining that an error has occurred based at least in part on the monitoring, and outputting a signal indicating that an error has occurred; and a drive manager communicatively connected to the trajectory manager for receiving a signal from the monitor and controlling the vehicle based at least in part on the maximum deceleration trajectory.

[0242] E. The vehicle system of any one of examples A-D, wherein the trajectory includes a first trajectory, a state associated with the first trajectory indicates that the first trajectory is associated with a collision, and the output trajectory includes at least one of a correction of the first trajectory or a second trajectory associated with a maximum deceleration rate.

[0243] F. An example vehicle system including a second system communicatively connected to a first system, the second system including one or more processors, and a memory, the second system including a perceiver for receiving at least a portion of the sensor data and outputting data indicative of at least one of a characteristic of the object, a track for the object, or an object trajectory for the object; and a trajectory manager configured to receive a first trajectory for the vehicle and a second trajectory for the vehicle from the first system, receive the data from the perceiver, perform one or more first authorization operations on the first trajectory to determine a first state associated with the first trajectory, perform one or more second authorization operations on the second trajectory to determine a second state associated with the second trajectory, and output an output trajectory for the vehicle including at least one of the first trajectory, the second trajectory, or a third trajectory generated by the trajectory manager based at least in part on the first state and the second state.

[0244] G. The vehicle system of Example F, further including a first system including one or more additional processors and additional memory, the additional memory including instructions that, when executed by the one or more additional processors, cause the first system to receive sensor data from a sensor, perform at least one of object detection, object segmentation, object classification, or object tracking for the object based at least in part on the sensor data, and generate a first trajectory and a second trajectory based at least in part on the sensor data.

[0245] H. The vehicle system of example F or G, wherein the one or more first authorization actions include at least one of determining whether the first trajectory was generated less than a threshold amount from a current time, determining whether the first trajectory is consistent with a current or previous position of the vehicle, determining whether the first trajectory is compatible with vehicle performance, or determining whether a vehicle moving along the first trajectory is associated with a collision.

[0246] I. A vehicle system as described in any one of Examples F-H, wherein the first state indicates that the first trajectory is associated with a collision, the output trajectory includes at least one of the second trajectory or the third trajectory, and the second system further includes a trajectory selector for maintaining control of the vehicle based at least in part on the output trajectory.

[0247] J. The vehicle system of any one of Examples F-I, wherein the second system further includes a releaser communicatively connected to the trajectory selector for determining to release control of the vehicle from the output trajectory and outputting a signal to the trajectory selector to release control of the vehicle from the output trajectory, the trajectory selector further configured to receive a signal from the releaser and output the first trajectory, the second trajectory, or the fourth trajectory received from the first system to control the vehicle.

[0248] K. The vehicle system of any one of Examples F-J, wherein the second system further includes a monitor configured to monitor at least one of the first systems or components of the vehicle, determine that an error has occurred based at least in part on the monitoring, and output a signal indicating that an error has occurred, and the trajectory manager is configured to receive the signal from the monitor and determine an output trajectory, including at least one of the second trajectory or the third trajectory, based at least in part on the signal.

[0249] L. A vehicle system as described in any one of examples F-K, wherein the first state indicates that the first trajectory is associated with an estimated collision, and the trajectory manager is configured to send a message to the first system indicating one or more of time to collision, object width, object velocity, object position, or point of collision based at least in part on the first state, receive a fourth trajectory from the first system, perform one or more first authorized actions on the fourth trajectory, and output the fourth trajectory to control the vehicle.

[0250] M. The vehicle system of any one of Examples F-L, wherein the first system implements a first technique for performing at least one of a first object detection, a first object segmentation, a first object classification, or a first object tracking based at least in part on the sensor data, and the second system implements a second technique for performing at least one of a second object detection, a second object segmentation, a second object classification, or a second object tracking based at least in part on a portion of the sensor data.

[0251] N. The vehicle system of any one of examples F-M, wherein the second system further includes a drive manager configured to receive an output trajectory from the trajectory manager and control the vehicle based at least in part on the output trajectory.

[0252] O. A vehicle system as described in any one of Examples F-N, wherein the data includes first data, and the second system further includes a localizer for receiving at least a portion of the sensor data, determining an attitude of the vehicle based at least in part on at least a portion of the sensor data, and outputting second data indicative of the attitude of the vehicle to at least one of the perceiver or the trajectory management component.

[0253] P. An example method including: determining, by a perceiver of a second component including one or more processors, perception data including at least one of a characteristic of an object in an environment, a track for the object, or an object trajectory for the object based at least in part on the sensor data; receiving, by a trajectory manager of the second component, a first trajectory for the vehicle from the first component, where the first component generates the first trajectory based at least in part on the sensor data; determining, by the trajectory manager, a state indicating whether the first trajectory is associated with a collision based at least in part on the perception data; and outputting, by the trajectory manager, an output trajectory for the vehicle based at least in part on the state, where the output trajectory includes at least one of the first trajectory or the second trajectory generated by the trajectory manager.

[0254] Q. The method of example P, further comprising the steps of: monitoring at least one of the first component or an additional component of the vehicle by a monitor of the second component; determining, by the monitor, that an error has occurred based at least in part on the monitoring; outputting, by the monitor, a signal indicating that an error has occurred; receiving, by a trajectory manager, a signal from the monitor; and determining, by the trajectory manager, an output trajectory based at least in part on the signal from the monitor.

[0255] R. The method of example P or Q, wherein the state indicates that the first trajectory is associated with a collision and the output trajectory includes a second trajectory, and the method further includes, by the trajectory manager, maintaining control of the vehicle based at least in part on the output trajectory, determining, by a releaser of the second component, to release control of the vehicle from the output trajectory, and outputting, by the releaser, a signal to release control of the vehicle from the output trajectory.

[0256] S. The method of any one of Examples P-R, wherein the state indicates that the first trajectory is associated with a collision, and the output trajectory includes a second trajectory associated with at least one of a deceleration or a correction relative to the first trajectory.

[0257] T. The method of any one of examples P-S, wherein the state indicates that the first trajectory is associated with a presumed collision with the object, and the method further includes the steps of sending, by the trajectory manager, a message to the first component indicating one or more of a time to collision, a width of the object, a velocity of the object, a position of the object, or a point of collision based at least in part on the state; receiving, by the trajectory manager, a third trajectory for the vehicle from the first system based at least in part on the message; determining an additional state indicating whether the third trajectory is associated with a collision or an additional collision; and outputting the third trajectory for controlling the vehicle based at least in part on the additional state.

[0258] Although the example clauses described above are described with respect to particular implementations, it should be understood that in the context of this document, the contents of the example clauses may be implemented via a method, device, system, computer readable medium, and / or another implementation.

[0259] Conclusion One or more examples of the techniques described herein have been described, and various modifications, additions, permutations, and equivalents thereof fall within the scope of the techniques described herein.

[0260] In describing the examples, reference is made to the accompanying drawings, which form a part hereof and which show, by way of illustration, specific examples of the claimed subject matter. It is understood that other examples may be used and that changes or modifications, such as structural changes, may be made. Such examples, changes, or modifications do not necessarily depart from the intended scope of the claimed subject matter. Although steps herein may be presented in a certain order, in some cases the ordering may be changed such that certain inputs are provided at different times or in a different order without changing the functionality of the described systems and methods. The disclosed procedures may also be performed in different orders. Additionally, the various calculations herein need not be performed in the order disclosed, and other examples using alternative orderings of calculations may be readily implemented. In addition to being reordered, calculations may also be decomposed into sub-calculations with the same results.

[0261] Although the present subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features or acts are disclosed as example forms of implementing the claims.

[0262] The components described herein represent instructions that may be stored on any type of computer-readable medium and that may be implemented in software and / or hardware. All of the methods and processes described above may be embodied in and fully automated via software code modules and / or computer-executable instructions executed by one or more computers or processors, hardware, or some combination thereof. Alternatively, some or all of the methods may be embodied in specialized computer hardware.

[0263] In particular, conditional language such as "may," "could," "may," or "might" is understood in context to present that certain examples include certain features, elements, and / or steps, while other examples do not, unless otherwise stated. Thus, such conditional language is not intended to generally imply that certain features, elements, and / or steps are at all required in one or more examples, or that one or more examples necessarily include logic for determining whether certain features, elements, and / or steps are included or will be performed in any particular example, with or without user input or prompting.

[0264] Conjunctive language such as "at least one of X, Y, or Z," unless otherwise stated, will be understood to present that the item, term, etc. may be either X, Y, or Z, or any combination thereof, including multiples of each element. "a" means singular as well as plural, unless expressly described as singular.

[0265] Any routine descriptions, elements, or blocks in the flow diagrams described herein and / or depicted in the accompanying figures should be understood as potentially representing modules, segments, or portions of code that include one or more computer-executable instructions for implementing specific logical functions or elements in the routines. Alternative implementations are included within the scope of the examples described herein, in which elements or functions therein may be deleted or performed substantially synchronously, in reverse order, with additional operations, or in an order that deviates from that shown or discussed, including omitting operations, depending on the functionality involved as understood by those skilled in the art.

[0266] Many variations and modifications may be made to the examples described above, the elements of which will be understood as particularly acceptable examples, and all such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.

Claims

1. A second system communicatively coupled to the first system, the second system comprising one or more processors and a memory, the second system comprising: receiving at least a portion of the sensor data; outputting data indicative of at least one of a property of an object, a track of said object, or an estimated object trajectory of said object, said track including a path traveled by said object; A first subsystem; receiving from the first system a first trajectory for a vehicle and a second trajectory for a vehicle, the second trajectory being a backup trajectory for the first trajectory; receiving the data from the first subsystem; performing one or more first authorization actions related to the first trajectory to determine a first state associated with the first trajectory, the first state indicating that the first trajectory is valid, that the first trajectory is invalid, or that the first trajectory will become invalid; performing one or more second authorization actions related to the second trajectory to determine a second state associated with the second trajectory, the second state indicating that the second trajectory is valid, that the second trajectory is invalid, or that the second trajectory will become invalid; determining, using a trajectory manager included in the second system, a third trajectory based on the first trajectory and the second trajectory, the third trajectory being a backup trajectory for the second trajectory, and applying a motion including deceleration or correction along the second trajectory; output an output trajectory for the vehicle based at least in part on the first condition and the second condition, the output trajectory including at least one of the first trajectory, the second trajectory, or the third trajectory. A second subsystem configured as follows: A vehicle system comprising:

2. The first system comprises: one or more additional processors; When executed by the one or more additional processors, the first system comprises: receiving the sensor data from a sensor; performing at least one of object detection, object segmentation, object classification, or object tracking on the object based at least in part on the sensor data; generating the first trajectory and the second trajectory based at least in part on the sensor data; Additional memory containing instructions and The vehicle system according to claim 1 , further comprising:

3. The one or more first authorization actions include: determining whether the first trajectory was generated less than a threshold amount of time ago; determining whether the first trajectory controls the vehicle in a current configuration; Determining whether the first trajectory is capable of controlling the vehicle due to the vehicle's capabilities; or determining whether the vehicle moving along the first trajectory is associated with a probable collision with the object; 3. The vehicle system according to claim 1, further comprising at least one of:

4. the first state indicates that the first trajectory is associated with a probable collision with the object, and the output trajectory includes at least one of the second trajectory or the third trajectory, and the second system: A third subsystem that maintains control of the vehicle based at least in part on the output trajectory.

4. The vehicle system according to claim 1, further comprising:

5. The second system comprises: and a fourth subsystem communicatively connected to the track selector, the fourth subsystem comprising: determining to release control of the vehicle from the output trajectory; outputting a signal to a third subsystem to release control of the vehicle from the output trajectory; The third subsystem includes: receiving the signal from the fourth subsystem; outputting the first trajectory, the second trajectory, or a fourth trajectory received from the first system to control the vehicle, the fourth trajectory being a maximum deceleration trajectory; 10. The vehicle system of claim 1, further configured to:

6. The second system comprises: monitoring at least one of the first systems or components of the vehicle; determining that an error has occurred based at least in part on said monitoring; Output a signal indicating that the error has occurred and a third subsystem configured as follows: The second subsystem includes: receiving the signal from the third subsystem; determining the output trajectory based at least in part on the signal, the output trajectory including the second trajectory or the third trajectory; 4. A vehicle system according to claim 1, wherein the vehicle system is a vehicle control system.

7. the first state indicates that the first trajectory is associated with a presumed collision with the object; The trajectory manager: sending a message to the first system indicating one or more of a time to an estimated collision with the object, a size of the object, a velocity of the object, a position of the object, or a point in the environment of an estimated collision with the object based at least in part on the first condition; receiving a fourth trajectory from the first system; performing the one or more first authorizing actions with respect to the fourth trajectory; outputting the fourth trajectory for controlling the vehicle; 7. The vehicle system according to claim 1, wherein the vehicle system is configured to:

8. the first system implements a first technique for at least one of a first object detection, a first segmentation of an object, a first object classification, or a first tracking of an object based at least in part on the sensor data; The second system implements a second technique for at least one of a second object detection, a second segmentation of an object, a second object classification, or a second tracking of an object based at least in part on the portion of the sensor data.

8. A vehicle system according to claim 1, wherein the vehicle system is a vehicle control system.

9. The data includes first data, and the second system comprises: receiving at least the portion of the sensor data; determining a posture of the vehicle based in part on at least the portion of the sensor data; outputting second data indicative of the attitude of the vehicle to at least one of the first subsystem or the second subsystem; or a third subsystem configured as follows: receiving the output trajectory from the trajectory manager; Controlling the vehicle based at least in part on the output trajectory. The fourth subsystem is configured as follows:

9. The vehicle system of claim 1, further comprising at least one of:

10. 1. A method performed by a vehicle system, comprising: determining, by a first subsystem of a second system including one or more processors, perception data comprising at least one of a characteristic of an object in an environment, a track of the object, or an estimated object trajectory of the object based at least in part on the sensor data, wherein the track comprises a path traveled by the object; receiving, by a second subsystem of the second system, a first trajectory for the vehicle from a first system, the first system generating the first trajectory based at least in part on the sensor data; determining, by the second subsystem, based at least in part on the perception data, whether the first trajectory is in a state indicative of being associated with a probable collision with the object; outputting, by the second subsystem, an output trajectory for the vehicle based at least in part on the conditions, the output trajectory including at least one of the first trajectory or a second trajectory generated by the second subsystem, the second trajectory applying a movement along the first trajectory, the movement including a deceleration or correction along the first trajectory; sending, by the second subsystem, a message to the first system indicating one or more of a time to an estimated collision with the object, a size of the object, a velocity of the object, a position of the object, or a point in the environment of an estimated collision with the object based at least in part on the state; receiving, by the second subsystem, from the first system based at least in part on the message, a third trajectory for the vehicle, the third trajectory being a backup trajectory for the first trajectory; 23. A method comprising:

11. monitoring, by a third subsystem of the second system, at least one of the first system or an additional component of the vehicle; determining, by the third subsystem, that an error has occurred based at least in part on the monitoring; and outputting, by the third subsystem, a signal indicating that the error has occurred; receiving, by the second subsystem, the signal from the third subsystem; determining, by the second subsystem, the output trajectory based at least in part on the signal from the third subsystem; The method of claim 10 further comprising:

12. the state indicates that the first trajectory is associated with a probable collision with the object, and the output trajectory includes the second trajectory; maintaining, by the second subsystem, control of the vehicle based at least in part on the output trajectory; determining, by a third subsystem of the second system, to release control of the vehicle from the output trajectory; outputting, by the third subsystem, a signal to release control of the vehicle from the output trajectory; The method of claim 10 further comprising:

13. 13. The method of any one of claims 10 to 12, wherein the state indicates that the first trajectory is associated with an estimated collision with the object, and the output trajectory includes the second trajectory, the second trajectory being associated with at least one of a deceleration or a correction relative to the first trajectory.

14. the state indicates that the first trajectory is associated with a probable collision with the object; determining whether the third trajectory is in an additional state indicative of being associated with a probable collision with the object; outputting the third trajectory for controlling the vehicle based at least in part on the additional state; 14. The method according to any one of claims 10 to 13, further comprising:

15. One or more computer readable media storing instructions that, when executed by one or more processors, cause a computing device to perform the method of any one of claims 10-14.

Images