Method and system for controlling an autonomous machine to avoid potential future collisions

Abstract

A safety evaluation of a trajectory of an autonomous machine can be performed based at least on determining whether the autonomous machine is able to occupy a trajectory point in the space-time using one or more safety procedures while still being able to avoid future potential collisions with one or more objects in the environment. In doing so, a collision evaluation of the trajectory point can be performed based at least on a comparison of times of arrival of points in the space-time corresponding to the autonomous machine executing one or more safety procedures from the point to one or more corresponding locations in the environment of the one or more objects. A trajectory can be sampled and collision evaluations performed at various points throughout the trajectory. Based on results of one or more evaluations, the trajectory can be scored, eliminated from consideration, or otherwise considered for control of the autonomous machine.

Description

[0001] Cross Reference to Related Applications

[0002] This application is related to U.S. Non-Provisional Application No. 16 / 265,780, filed February 1, 2019, U.S. Non-Provisional Application No. 16 / 269,921, filed February 7, 2019, and U.S. Non-Provisional Application No. 16 / 877,127, filed May 18, 2020, the entireties of which are incorporated herein by reference. BACKGROUND

[0003] Designing a system to drive a vehicle autonomously and safely without supervision is very difficult. An autonomous vehicle should at least be able to function as well as a careful driver that utilizes a perception and action system with amazing ability to identify and respond to moving and static obstacles in complex environments to avoid collisions with other objects or structures while operating the vehicle. Behavior planning for an autonomous vehicle can involve using a driving perception system to determine a trajectory for the vehicle to travel, which is a challenging but critical task, especially for complex urban scenarios. For example, to give an autonomous vehicle an understanding of its surroundings to make safe and effective behavior decisions, the autonomous vehicle must process various environmental inputs along with current vehicle trajectory and route information to compute a continued or future trajectory for the vehicle.

[0004] Conventional systems for behavior planning often rely on inflexible decisions that do not account for changing variables in the environment. For example, a traditional approach can choose a route with the shortest travel time and then pick a lane plan that follows that route. However, this rigid planning does not allow the planning system to optimize adjustments to the trajectory based on new or future environmental conditions. For example, by determining the behavior of the vehicle separately from other determinations, such as avoiding collisions, adjustments to the current plan can be passive, rather than active. Thus, in the example of avoiding collisions, the output from the behavior planner can be used to determine control decisions, and when a potential collision is determined, an avoid collision function can generate updated control decisions. As a result, the trajectory of the vehicle is not determined using avoiding collisions as a factor, but is instead overridden by the avoid collision function of the vehicle. Thus, the behavior of the vehicle can not be optimal because they do not take into account dynamic inputs, such as avoiding collisions. SUMMARY

[0005] Embodiments of the present disclosure relate to using predicted times of arrival and safety procedures in motion planning of trajectories for autonomous vehicles. More specifically, the present disclosure relates to techniques and methods for evaluating planned trajectories of an autonomous machine based on whether the autonomous machine can occupy a trajectory point in space-time while still being able to avoid potential future collisions for controlling the autonomous machine.

[0006] In contrast to conventional systems, the disclosed methods can evaluate the safety of trajectory proposals for an autonomous machine based at least on determining whether the autonomous machine can occupy a point of the proposed trajectory in space-time while still being able to avoid a potential collision with one or more objects in the environment in the future while performing one or more safety procedures. To do so, a collision for a point of the trajectory can be evaluated based at least on a comparison between the point in space-time corresponding to the autonomous machine implementing the safety procedure from that point (e.g., a set of claimed sets) and the arrival times for the one or more objects to arrive at respective locations in the environment. Trajectories can be sampled and evaluated for collisions at various points in the trajectory. Based on one or more evaluation results, a trajectory can be scored (e.g., for ranking against other potential trajectories), eliminated from consideration, or otherwise considered for controlling the autonomous machine. BRIEF DESCRIPTION OF DRAWINGS

[0007] The present systems and methods for using arrival times and safety procedures in motion planning for trajectories of autonomous vehicles are described in detail below with reference to the accompanying drawings, wherein:

[0008] Figure 1 is an illustration including an example trajectory evaluation system in accordance with some embodiments of the present disclosure;

[0009] Figure 2 includes a chart for describing an example of a motion curve in accordance with some embodiments of the present disclosure;

[0010] Figure 3A is an illustration including an example object trajectory in accordance with some embodiments of the present disclosure;

[0011] Figure 3B is an illustration of a portion of a contour plot representing arrival times in accordance with some embodiments of the present disclosure;

[0012] Figure 3C is an illustration of a contour plot representing arrival times in accordance with some embodiments of the present disclosure;

[0013] Figures 4A-4C depicts an example of a two-dimensional projection of a safety procedure for a vehicle in accordance with some embodiments of the present disclosure;

[0014] Figures 5A-5C depicts an example of a space-time diagram for a vehicle performing a safety procedure in accordance with some embodiments of the present disclosure;

[0015] Figure 5D depicts an example of a three-dimensional projection of safety procedures for multiple vehicles in space-time in accordance with some embodiments of the present disclosure;

[0016] Figure 6is an illustration of an example including a trajectory, safety programs that can be invoked during the trajectory, and points that can be used to evaluate the trajectory according to some embodiments of the present disclosure;

[0017] Figure 7 is an illustration of an example describing a lateral path and a speed profile according to some embodiments of the present disclosure;

[0018] Figure 8 is an illustration of an example behavior planning architecture for an autonomous vehicle according to some embodiments of the present disclosure;

[0019] Figure 9 is a flowchart illustrating a method for controlling a vehicle based on evaluation points corresponding to safety programs that can be deployed in a trajectory according to some embodiments of the present disclosure;

[0020] Figure 10 is a flowchart illustrating a method for controlling a vehicle based on modeling a set of vehicles corresponding to safety programs that can be implemented in a trajectory according to some embodiments of the present disclosure;

[0021] Figure 11 is a flowchart illustrating a method for controlling a vehicle based on determining that a vehicle will occupy a location before the time at which a location will be occupied by an object when implementing safety programs during a trajectory according to some embodiments of the present disclosure;

[0022] Figure 12 is an example of a suitable operating environment according to some embodiments of the present disclosure;

[0023] Figure 13A is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure;

[0024] Figure 13B is an example of a camera position and field of view of an example autonomous vehicle according to some embodiments of the present disclosure; Figure 13A

[0025] is a block diagram of an example system architecture of an example autonomous vehicle according to some embodiments of the present disclosure; Figure 13C Figure 13A is a system diagram of communication between a cloud-based server and an example autonomous vehicle according to some embodiments of the present disclosure;

[0026] Figure 13D Figure 13A is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0027] Figure 14 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0028] Figure 15 ​​is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0029] Systems and methods related to using time of arrival and safety procedures in motion planning of trajectories of autonomous vehicles are disclosed. More specifically, the present disclosure relates to techniques and methods for evaluating trajectories for controlling an autonomous machine based on whether the autonomous machine can occupy a trajectory point in time-space while still being able to avoid a potential future collision.

[0030] The present disclosure can be described in relation to an example autonomous vehicle 140 (also referred to herein as "vehicle 140" or "autonomous vehicle 140"), examples of which are described herein in relation to Figures 13A-13D The present disclosure can be described in relation to an example autonomous vehicle 140 (also referred to herein as "vehicle 140" or "autonomous vehicle 140"), examples of which are described herein in relation to

[0031] The disclosed methods can evaluate the safety of a trajectory of an autonomous machine based at least on determining whether the autonomous machine can occupy a point of the trajectory in time-space while still being able to avoid a potential future collision with one or more objects in the environment by using one or more safety procedures. If the autonomous machine performs one or more safety procedures from a sample point, for one or more sample points in time-space within the trajectory, the system can determine one or more points in time-space that the autonomous machine will occupy. The system can also determine a potential time of arrival of one or more objects to a location of the one or more points in time-space and compare the time of arrival to a time of the one or more points in time-space that the autonomous machine will occupy. The objects can include pedestrians, animals, bicycles, cars, and / or static obstacles with assumed small accelerations to account for uncertainty in perception. In some embodiments, the objects can be limited to one or more of these types.

[0032] The comparison can be used to identify whether a conflict exists between when the autonomous machine will occupy one or more locations of the trajectory and when one or more objects will occupy one or more proximate locations in the environment. For example, if the comparison indicates that the autonomous machine will occupy the one or more locations before the corresponding time of arrival to implement a safety procedure from a location in the trajectory, then no conflict can be determined (e.g., the vehicle can stop before a collision). However, if the comparison indicates that the autonomous machine will occupy the one or more locations after the corresponding time of arrival, then a conflict can be determined (e.g., the vehicle cannot stop before a collision).

[0033] In at least one embodiment, the time of arrival of one or more objects can be computed based at least on modeling a trajectory of one or more objects into the environment (e.g., from a starting point of the trajectory). For example, each trajectory can model a scenario in which an object travels directly from a starting location of the object toward a given location. Object parameters such as pose, acceleration / deceleration, and velocity can be used to generate the trajectory. In one or more embodiments, a worst-case scenario can be modeled in which each object travels as fast as possible directly toward a given location (e.g., according to the capabilities of the object in the model). This provides a conservative approach in which if an autonomous machine is able to avoid a collision in a worst-case scenario or a highly restricted scenario, then the autonomous machine should be able to avoid a collision in a less restricted scenario. In the case of multiple objects being modeled, the minimum or earliest time of arrival at a given location can be used for comparison to a point corresponding to the autonomous machine.

[0034] In one or more embodiments, the set of machines claimed corresponds to one or more points in space-time that are predicted to be occupied by the autonomous machine when one or more safety procedures are executed at a sampling point. The set of machines claimed for the autonomous machine can include an occupancy trajectory of the autonomous machine (e.g., at least one estimate of each point in space that the machine will occupy when following the trajectory) when the autonomous machine applies a safety procedure. A safety procedure can refer to a procedure for controlling the autonomous machine that defines one or more trajectories (e.g., but not limited to, turning, swerving, driving into a side lane or shoulder, etc.) or actions (e.g., but not limited to, accelerating, braking, decelerating, switching at least partial control to a backup component, returning control to a driver or other operator, etc.) of the autonomous machine in space-time that, if executed at a corresponding time and location, will reduce the likelihood of or avoid a collision and / or its effects. The safety procedure can be triggered based at least in part on a determination by the autonomous machine that a collision is likely and / or imminent, and / or a failure of one or more components of the autonomous machine is likely, imminent, and / or has occurred.

[0035] For example, at different time steps, the trajectories can be sampled and conflicts at various points throughout the trajectories can be evaluated. Multiple points can be evaluated for samples corresponding to the starting point of the trajectory to avoid gaps in coverage. However, for subsequent time steps, only the points at the tip of the safety procedure (e.g., the location that satisfies the actor state goal) can be evaluated. In some embodiments, sampling only the tips can be sufficient because these tips can correspond to the most restricted points of the safety procedure. Thus, checking for conflicts with respect to the safety procedure can be computationally efficient because the number of comparisons can be in a linear relationship with the number of safety procedures evaluated.

[0036] Based on the results of the one or more evaluations, the trajectories can be scored (e.g., for ranking against other potential trajectories), eliminated from consideration, or otherwise considered for controlling the autonomous machine. For example, in evaluating a selected trajectory for controlling the autonomous machine, the conflicts can be used as hard constraints or soft constraints. The constraints can be based on various potential factors, such as a distance and / or elapsed time to a conflict detected along the trajectory (e.g., to a start of a sample or to a point compared to a time of arrival) (e.g., a conflict of a maximum distance from a start of the trajectory and / or an elapsed time).

[0037] Reference is now made to Figure 1 , Figure 1 A diagram is shown that includes an example trajectory evaluation system 100, in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, and groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted depending on the context. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. The various functions described herein as being performed by one or more entities can be carried out by hardware, firmware, and / or software. For instance, some functions can be carried out by a processor executing instructions stored in memory.

[0038] The trajectory evaluation system 100 can receive sensor data 102 representative of an environment 126 and use the sensor data 102 to calculate at least one time of arrival of an object to at least one location in the environment 126. The at least one time of arrival can be used to analyze vehicle trajectory information 124 representative of at least one proposed trajectory or path 144 of a vehicle 140. In one or more embodiments, the at least one time of arrival can be captured in an image or other form of data that is processed to analyze the proposed trajectory or path 144. The image can include, by way of example and not limitation, a 2D grid, a matrix, a single channel rasterized image, and the like.

[0039] The trajectory evaluation system 100 can include, for example, a communication manager 104, an object analyzer 106, an object modeler 108, a safety procedure determiner 110, a time of arrival determiner 112, a trajectory analyzer 114, and a set of claims determiner 116.

[0040] As an overview, the communication manager 104 can be configured to manage communications received by the trajectory evaluation system 100 (e.g., including sensor data 102 and / or vehicle trajectory information 124) and / or communications provided by the trajectory evaluation system 100 (e.g., including data corresponding to results of analyzing vehicle trajectory information 124). The object analyzer 106 can be configured to analyze objects in an environment using sensor data 102. The object modeler 108 can be configured to model objects (e.g., trajectories of objects in the environment 126) based at least on analysis performed by the object analyzer 106. The time of arrival determiner 112 can be configured to analyze trajectories of objects, e.g., trajectories modeled by the object modeler 108 to compute times of arrival of objects to locations within the environment.

[0041] The trajectory analyzer 114 can be configured to evaluate one or more trajectories 144 of the vehicle 140 based at least on the times of arrival. To do so, the trajectory analyzer 114 can use the safety procedure determiner 110 and the set of claims determiner 116. The safety procedure determiner 110 can determine safety procedures of the vehicle 140. The set of claims determiner 116 can determine at least a portion of one or more sets of claims of the vehicle 140 if the vehicle 140 performs a safety procedure at one or more points within the trajectory 144. The trajectory analyzer 114 can compare one or more times of arrival to the portion of the one or more sets of claims. Based on results of the comparison, the trajectory 144 can be scored (e.g., for ranking relative to other potential trajectories), eliminated from consideration, or otherwise considered for control of the vehicle 140.

[0042] As described herein, the communication manager 104 can be configured to manage communications received by the trajectory evaluation system 100 (e.g., including sensor data 102 and / or vehicle trajectory information 124) and / or communications provided by the trajectory evaluation system 100 (e.g., including data corresponding to results of analyzing vehicle trajectory information 124).

[0043] When used as a network communication receiver and / or provider for communication, the communication manager 104 may include a network interface that can communicate over one or more networks using one or more wireless antennas 1326 and / or modems. For example, the network interface may be capable of communication via Long Term Evolution (LTE), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications Service (UMTS), Global System for Mobile Communications (GSM), CDMA2000, etc. The network interface can also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth Low Energy (LE), Z-Wave, ZigBee, etc., and / or low-power wide area networks (LPWANs) such as LoRaWAN, SigFox, etc. However, the communication manager 104 may not need to include a network interface, for example, where the trajectory evaluation system 100 is implemented entirely on an autonomous vehicle (e.g., vehicle 140). In some examples, the one or more communications described herein can be communicated via... Figure 14 The bus 1402 connects the components of the computing device 1400.

[0044] Sensor data 102 can be generated using any combination of sensors, for example Figure 12 Sensor 1238. For example, sensor data 102 may include image data representing an image, image data representing video (e.g., a snapshot of a video), and / or data representing a sensor (e.g., LIDAR sensor 1364, RADAR sensor 1360, etc.). Figure 13B The sensor data 102 represents the field of view of the various cameras (such as cameras on the vehicle 140). In the illustrated example, sensor data 102 includes image data representing the field of view of the various cameras on the vehicle 140, which may include one or more images. The images may depict areas of environment 126, which may include any number of objects, examples of which include objects 128A and 128B. Objects may include any combination of vehicles, people (e.g., pedestrians), motorcycles, bicycles, trees, animals, buildings, signs or other structures, objects or obstacles that the vehicle 140 may be able to collide with in environment 126.

[0045] The sensor data 102 can be provided to the object analyzer 106 and used by the object analyzer 106 to analyze objects in the environment 126. For example, the object analyzer 106 can detect one or more of the objects 128A or 128B, identify the one or more objects, and / or determine parameters of the one or more objects using the sensor data 102. To do so, the object analyzer 106 can employ one or more machine learning models. For example, but not limited to, the machine learning models can include any type of machine learning model, such as machine learning models that use linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutional, recurrent, perceptron, long / short-term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid machines, etc.), and / or other types of machine learning models.

[0046] Examples of parameters of objects that the object analyzer 106 can determine using the sensor data 102 include one or more of a position (e.g., in the environment 126, such as position coordinates), a pose, a current and / or observed velocity, a maximum velocity, a predicted velocity, at least one dimension (e.g., a physical dimension, such as a length, a width, a footprint, a height, etc.), a current and / or observed acceleration or deceleration, a maximum acceleration or deceleration, a predicted acceleration or deceleration, a mass, a reaction time, and / or other parameters, such as those described herein, for example, but not limited to. The one or more parameters can represent observed characteristics of the object (e.g., position / location), and the one or more parameters can represent inferred characteristics of the object (e.g., maximum acceleration).

[0047] In some examples, the parameters of the objects can be outputs of machine learning models, such as a convolutional neural network that receives at least some of the sensor data 102 as input. In further examples, the object analyzer 106 can use at least one or more machine learning models to classify one or more objects captured by the sensor data 102. Examples of classifications include stationary, moving, vehicle, car, truck, pedestrian, bicycle, motorcycle, etc.

[0048] Object analyzer 106 can use classification to determine one or more parameters. For example, classification can be provided as input to a machine learning model used to determine one or more parameters. As another example, one or more classifications and / or other object information (e.g., other parameters) can be applied to a lookup table or otherwise used to look up, determine, and / or compute one or more parameters. For example, classification can include vehicle models or types (e.g., cars, trucks, motorcycles, SUVs) that can be used to define one or more parameters, such as a predetermined shape and / or size, braking capability, handling capability, acceleration capability, maximum speed, maximum acceleration, etc. In some examples, a machine learning model such as a convolutional neural network can be trained to simultaneously output the object's classification and one or more parameters of the object.

[0049] As an example, object analyzer 106 can use a machine learning model (e.g., a neural network) to implement object awareness, which can be specifically configured (e.g., trained) to identify certain objects and / or object features. One or more trained machine learning models used by object analyzer 106 (e.g., trained and deployed for use by trajectory evaluation system 100) can determine the presence and / or location of objects (e.g., X and Y coordinates), the pose of the objects, etc. The size of the obstacle (e.g., width and length) and / or the classification of the object. Furthermore, a trained machine learning model (e.g., a neural network) can be used to determine the object's maximum acceleration (A). MAX+ ) and maximum deceleration (A MAX -).

[0050] As a further example, the reaction time (T) of an object can be determined using a trained machine learning model (e.g., a neural network) that identifies or outputs the object type (e.g., object classification) and / or specifies the corresponding reaction time. REACT An object's reaction time can represent the future time before its motion begins to be affected by another object, such as vehicle 140, present in environment 126, or the time before its motion begins to be affected by another object, such as vehicle 140, present in environment 126. This can correspond to the object's perception, noticing, and / or determining that it may collide with other objects at a certain location in the environment. For example, a pedestrian staring at his or her smartphone may have a long reaction time T. REACT Similarly, cyclists or motorcyclists looking from a distance can be considered to have a longer reaction time T than, for example, a more attentive driver. REACT In contrast, autonomous vehicles (such as self-driving taxis) may have a shorter reaction time T. REACTBecause autonomous vehicles can be less prone to distraction and can assume to be monitoring objects on its path and react to objects faster than human drivers.

[0051] As an example, the object analyzer can detect one or more of the objects 128A and 128B, identify one or more of the objects 128A and 128B, and / or determine parameters and / or classifications of one or more of the objects 128A and 128B using the sensor data 102 and / or other information received from one or more of the sensors 138 and / or the client device 1232 in the operating environment 130. For example, the communication manager 104 can receive respective data over the network 1234. Additionally, the object analyzer 106 can detect, identify, and / or determine parameters and / or classifications of one or more objects that cannot be directly represented by the sensor data 102, but can be inferred, predicted, or otherwise determined by the object analyzer 106 to be present or likely present (e.g., based at least on the sensor data 102). This can be used to account for dead zones or blind spots in the perception of the trajectory assessment system 100. Figure 12

[0052] The object modeler 108 can be configured to model one or more objects in the environment 126, such as the objects 128A and / or 128B, based at least on the analysis performed by the object analyzer 106 (e.g., based at least on one or more parameters associated with the object). Additionally or alternatively, the object modeler 108 can filter one or more objects out of the modeling (e.g., trajectory modeling). For example, the object modeler 108 can filter out an object based at least on a classification of the object (e.g., as stationary) and / or based at least on a type of the object. Additionally or alternatively, the object modeler 108 can filter out one or more objects based at least on one or more parameters of the object, such as any parameters determined by the object analyzer 106 (e.g., by filtering out an object based at least on a location of the object exceeding a threshold distance from the vehicle 140). Such filtering can reduce processing power for determining time of arrival, and can further be used to improve trajectory assessment in lane driving scenarios where a vehicle is in a neighboring lane or parallel to a vehicle of interest with respect to oncoming traffic.

[0053] ​The object modeler 108 can use any suitable method to model objects in the environment 126, for example by representing each object as, for example, a radial distance function (and / or other function(s) or functions) or a list of known locations of the object in the environment 126 (e.g. represented as a plane). For example, the representation of an object can be defined using at least one dimensional parameter (e.g. width, length), a position parameter (e.g. X, Y in the environment 126) and / or an attitude of the object determined using the object analyser 106. Optionally, a safety buffer can be included in the representation of an object (e.g. by extending its dimensions). The model of an object can also include one or more motion vectors of the object, for example a current and / or instantaneous motion vector (e.g. an observed motion vector from sensor data 102). For example, each object can have a motion vector and an acceleration vector defined using at least one parameter determined for the object using the object analyser 106 (e.g. using the attitude, acceleration / deceleration and velocity of the object). The object modeler 108 can further model an object using a maximum acceleration and / or deceleration parameter, a maximum velocity parameter and / or a mass parameter of the object.

[0054] The object modeler 108 can use the representation or model of an object to model one or more trajectories of the object. To do so, the object modeler 108 can apply one or more motion curves to the representation of the object to define the velocity and acceleration of the object at a particular time and along the trajectory of the object in the environment 126, examples of which are described with reference to Figure 2 .

[0055] Reference is now made to Figure 2 , Figure 2 includes a graph 200 for describing examples of motion curves that can be used to define a modelled trajectory of an object. The graph 200 includes a plot 210 that can correspond to a motion curve that defines the motion of an object over time according to a trajectory. The object modeler 108 can represent the motion curve, for example, using one or more functions (e.g. continuous functions) and / or rules for defining the plot 210 of the object.

[0056] The motion curve that defines the plot 210 can include a plurality of phases, for example the object following a trajectory that includes a maximum acceleration A MAX+ a phase of the trajectory that points to a location in the environment 126 (e.g. even if the object is currently moving away from the object), optionally subject to a maximum velocity V MAX or velocity cap V CAP a phase of the trajectory that includes a maximum deceleration A MAX that can be selected to be the same as A MAX+The motion curve can be a phase of deceleration (same as above), until the object comes to a complete stop, and / or a phase in which the object remains stationary. The motion curve can include parameters used by the object modeler 108 to set the time at which the object begins to decelerate in the motion curve, such as reaction time T. REACT (For example, as described in this article). Observed acceleration A OBS It can be used to replace the maximum acceleration A MAX+ Or maximum deceleration A MAX- It can also be adjusted by a safety factor to account for the possibility that the observed acceleration is inaccurate, and / or the possibility that the acceleration may increase.

[0057] Corresponding to Figure 2 The motion curve can be used to represent the worst-case scenario for vehicle safety, calculated as if the vehicle 140 were located at one or more time points. The reaction time T in these scenarios... REACT This can represent the time taken for an operator (e.g., a driver) of an object (e.g., another vehicle) to notice the vehicle at that location and attempt to avoid a collision. However, object modeler 108 can use other motion curves to model one or more trajectories of any object from a variety of objects, which may include any number of stages. Furthermore, one or more motion curves can be more complex than the described motion curves, including at least based on object modeler 108 analyzing and applying variables representing different characteristics of the state of the environment 126 at a given time to change the conditions or rules used on or during the entry stage of the object's motion.

[0058] The trajectory modeled by object modeler 108 can have a start time T representing the start time of the trajectory. S and the end time T representing the end time of the trajectory E . Figure 2 It shows a starting velocity V S and initial acceleration A S Start time T S A specific example. The object modeler 108 can determine the initial velocity V. S and initial acceleration A S Set to the current, observed, and / or instantaneous velocity and acceleration parameters provided by the object analyzer 106, or the initial velocity V. S and / or initial acceleration A S These values ​​can be derived. In at least one embodiment, the initial acceleration A can be... S Set to maximum acceleration A MAX .

[0059] Figure 2 The example in the figure represents the start time T of the trajectory. S The initial velocity V of the objectS and initial acceleration A S A positive value relative to vehicle 140 means the object is moving towards vehicle 140 at an increasing rate. However, if the object initially moved at start time T... S If the object moves 140 degrees away from the vehicle, its initial velocity V is... S and / or initial acceleration A S It can be negative relative to the vehicle's speed of 140. In other examples, it could be the initial speed V. s Or initial acceleration A S It can be zero. This may depend on the object's start time T. S The stage of the motion curve during that period.

[0060] Figure 2 It also shows the terminal velocity V E and the final acceleration A E End time T E A specific example. As shown in the figure, the final velocity V... E and the final acceleration A E The velocity V can be zero, indicating that the object has come to a complete stop at the end of the trajectory. However, in other examples, the ending velocity V... E and / or termination acceleration A E The end of the trajectory can be positive or negative. For example, the start time T S This can correspond to any position along drawing 210, depending on which stage of the motion curve the object is in at the start of the trajectory, and similarly, the end time T. E This can correspond to the start time T. S Then, at any point along drawing 210, depending on the stage of the object in the motion curve at the end of the trajectory (e.g., the object may not have completely stopped yet).

[0061] Furthermore, when object modeler 108 uses the same motion curve to model the trajectories of multiple objects, different objects can be at different stages of the motion curve simultaneously. For example, an object might have a start time T in its modeled trajectory. S It decelerates to a stop and may remain stopped for a period of time until the end time T. E The second object may be in its modeled trajectory at start time T S Accelerate toward the vehicle at 140, and may finish at time T. E At maximum speed V MAXMovement. Furthermore, in at least one embodiment, the motion curve may exclude the phase in which the object begins to decelerate. For example, the motion curve may be used to model the time it takes for the object to reach a point in space without deceleration (e.g., modeling the earliest possible arrival time of the object to that point), and thus the object is modeled as reaching a maximum velocity V. MAX The area 220 can last until the end time T. E At this point, the point is reached (unless the end time is T). E Upon reaching maximum speed V MAX (Previously appeared).

[0062] Figure 2 As shown in some examples, the object modeler 108 can use the object's maximum speed. VMAX The maximum acceleration A of the object MAX+ Or the maximum deceleration A of the object MAX- One or more motion curves are applied. These values ​​can represent limitations on the object's ability to move, and any of these values ​​can correspond to parameters determined for the object using object analyzer 106, as described herein. For example, curve plot 210 includes region 220 in which the object is modeled to reach maximum velocity V during the modeling trajectory. MAX However, in other trajectories, the object may not reach its maximum velocity V during the trajectory. MAX Maximum acceleration A MAX + or maximum deceleration A MAX One or more of these factors may be used, or the object modeler 108 may not use one or more of these factors when applying motion curves.

[0063] Figure 2 The upper speed limit V is also shown. CAP This can be illustrated in the trajectory modeling. Velocity upper limit V CAP It can be greater than the maximum speed V MAX And it can correspond to a speed greater than the maximum speed V. MAX The observation speed of the object. For example, the upper limit of speed V. CAP This can correspond to a parameter determined by object analyzer 106 based on sensor data 102, which indicates that the object is moving at a speed greater than the maximum speed V. MAX Traveling at a faster speed or being able to travel at a speed greater than the maximum speed V MAX Traveling at a faster speed. For example, the object modeler 108 can initially use the maximum speed VMAX to apply the object's motion curve to model the object's trajectory. This is done after determining that the object is traveling or has been traveling faster than the maximum speed VMAX. MAX Subsequently, the object modeler 108 can alternatively use the speed cap V. CAPThis is used to update the trajectory of the model or different trajectories of the modeled object. Similar methods can be used for maximum acceleration or deceleration.

[0064] Now for reference Figure 3A , Figure 3A This is an illustration including example trajectory 302 according to some embodiments of the present disclosure. Trajectory 302 may correspond to... Figure 2 The described modeling trajectory. The object modeler 108 can use the object's starting position X. S To model the trajectory 302 of object Y1 (e.g., object 128A or 128B), the starting position X S Corresponding to object Y1 at start time T S The object modeler 108 can also model the trajectory 302 using the end position X1 of object Y1, which corresponds to the position or location of object Y1 at the end time T. E Location or position within environment 126.

[0065] The starting position X of object Y1 S This can correspond to the position parameters determined using object analyzer 106. The position parameters can be determined by object analyzer 106 in real-time or near real-time, for example, the starting position representing the basic current location of object Y1 in environment 126. However, the starting position X... S In some examples, it could be the predicted current or future location of an object in the environment, such as the location where object Y1 is occluded and not perceived by sensor 138, where object Y1 does not necessarily exist in the environment but is modeled to address blind spots or dead zones in the perception of trajectory evaluation system 100, or other scenarios.

[0066] exist Figure 3A In this context, object modeler 108 may have already modeled trajectory 302 as directly toward the end position X1 of object Y1 (e.g., a location in environment 126 where arrival time is being determined), as shown in the figure. Object modeler 108 can use this method, in conjunction with the aforementioned motion curves, to model the worst-case scenario where the operator of object Y1 attempts to reach the end position X1 as quickly and directly as possible. Object modeler 108 can use a similar approach to determine the arrival time of each location in environment 126 and each modeled object in the environment, or one or more trajectories may follow alternative paths and / or use different motion curves. In some examples, object modeler 108 may determine to model trajectories using default paths and / or motion curves (e.g., the motion paths and direct paths corresponding to plot 210), and in other examples, object modeler 108 may determine to model trajectories using different paths and / or motion curves (e.g., when the planned trajectory of the object is known, such as after it has been provided by the object).

[0067] As described herein, arrival time determiner 112 can be configured to analyze trajectories modeled by object modeler 108 to calculate one or more arrival times for one or more objects to arrive at one or more locations in environment 126. For example, arrival time determiner 112 can use the modeled trajectories to determine when an object will occupy one or more locations in environment 126, and the object's velocity and acceleration at those locations. This combination of location and time can form one or more points in spacetime and can be determined using a variety of possible criteria. For example, a point can represent the earliest possible arrival time of an object at that point (e.g., system estimate or prediction). In one or more embodiments, the arrival time to a point can represent the safe arrival time of vehicle 140 at that point (e.g., the most recent safe arrival time), where vehicle 140 arriving at the point at least before the safe arrival time will not result in a collision with an object. However, arrival times can be evaluated under other criteria, such as the impact of a collision being less than a threshold impact, which can be calculated at least based on the modeling quality and velocity of the object, the probability of a collision occurring being less than a threshold, or vehicle 140 being able to come to a complete stop before the object's arrival time to the point.

[0068] In some examples, to determine at least one safe arrival time for the ending position X1, the object modeler 108 models the trajectory 302 to the ending position X1. This is based at least on the calculation that object Y1 will arrive at the trajectory 302 at time T. E At (or in other examples at the end time T) E (Previously) completely stopped, the arrival time determiner 112 can determine the end time T. E and / or the end time T in trajectory 302 E Any time prior to this is the safe arrival time for the vehicle to reach its destination X1. For example, given a location P in the environment, since it can be determined that the destination X1 has an end time T. E At the end time T E At this point, object Y1 may stop at position P in the worst case of the object's trajectory, so the end time T E This can be considered the safe arrival time to position P. Therefore, it can be assumed that any earlier time is also a safe arrival time to position P. Therefore, the start time T of trajectory 302... S and end time T E This can constitute a safe time interval for the vehicle relative to object Y1 and position P. In other examples, object Y1 may not be at the end time T of trajectory 302. Emay have an end position X1' instead of stopping completely. In this case, the safe arrival time for position P and object Y1 can be determined to be zero because object Y1 has the potential to put itself in a state that has already committed to pass position P.

[0069] Another case where the safe arrival time can be set to zero can be when object Y1 commits to position P from the start and the parabolic acceleration in the opposite direction passes position P. This can happen, for example, if position P is located at the start position X S of trajectory 302. This can happen, for example, when vehicle 140 follows object Y1, which is a lead vehicle moving away from vehicle 140 at a fast speed (examples of which are shown in FIGS. 15A-15C). In that case, vehicle 140 can inevitably pass some positions in front of it (and typically pass them). Here, it is clear that the parabola passes position P for a time interval (e.g., an unsafe time interval). Figure 3A

[0070] The following is a non-limiting example of calculating the safe arrival time for object Y1 at an obstacle point Y (e.g., start position X S in the plane) and a target point X (e.g., position P in the plane). Given that object Y1 at obstacle point Y has a velocity vector V (e.g., start velocity V S in the plane), a scalar maximum velocity v max (e.g., maximum velocity V MAX ), a scalar maximum acceleration a max (e.g., maximum acceleration A MAX ), and a reaction time τ react (e.g., reaction time T REACT ), the scalar distance d can be calculated as

[0071] d = |X - Y|

[0072] and the scalar (signed) velocity v can be calculated as

[0073]

[0074] In some examples, this can be generalized to any function that distributes the maximum start velocity to each direction, the dot product gives a concrete suggestion for such a function. The safe arrival time t lsa (e.g., the latest safe arrival time) for object Y1 at obstacle point Y can be calculated by the following equation:

[0075] v cap = max(v, v max )

[0076] ​

[0077] For each location in the plane of evaluation, all safe arrival times t lsa from all objects (e.g., points) can be computed in parallel. If the input is a radial distance function, then for all points on rays further than the radial distance function, the time can be set to zero. Arrival time determiner 112 can use this approach to produce a visualization (e.g., an image). Moreover, in some examples, any safe arrival time t lsa of any object can be computed at any suitable time, e.g., to test and / or generate paths and / or trajectories (e.g., this can not involve generating a visualization, as desired).

[0078] In any examples, object modeler 108 and arrival time determiner 112 can consider additional factors to determine arrival times of objects. For example, a slope to location P can be estimated using sensor data 102 received and processed by object analyzer 106. An uphill slope to a target point can be used as a factor to reduce acceleration (or increase deceleration) to the target point. Similarly, a road surface coefficient μ can be estimated using, e.g., information received and processed by a perception block of vehicle 140 (e.g., similar to object perception that can be performed by object analyzer 106). For example, a trained neural network can detect the presence of ice, snow, or oil road and return a road surface coefficient μ that can be used as a factor to reduce a maximum deceleration to a location P.

[0079] Figure 3A End locations X2, X3, and X4 of other trajectories that object modeler 108 can model are also shown, e.g., using motion curves corresponding to graph 210 or different motion curves and / or direct paths (or different types of paths) to each of end locations X2, X3, and X4. End locations X2, X3, X4 are provided as examples of locations in a trajectory that collectively have at least one arrival time. This can be in Figure 3AThe diagram shows that the ending positions X1, X2, X3, and X4 are plotted in the spatiotemporal domain, forming a point outline 306. Outline 306 represents the shared arrival time between positions or locations within environment 126. Outline 306 may additionally or alternatively represent a shared time interval between positions (e.g., outline 306 represents the end time of an interval). In at least one embodiment, trajectory analyzer 114 can use one or more points of outline 306 to determine whether one or more points in the spatiotemporal domain of trajectory 144 are safe for vehicle 140 to occupy. For example, trajectory analyzer 114 can use one or more points of outline 306 to detect or determine potential intersections with one or more points in the spatiotemporal domain that vehicle 140 will occupy if vehicle 140 implements one or more safety procedures at one or more points in the spatiotemporal domain of trajectory 144. Using this method, trajectory analyzer 114 can determine that a given time and location of trajectory 144 would be unsafe because a collision could occur under the modeling conditions.

[0080] Although reference Figure 3A The described arrival time determiner 112 analyzes the position and time relative to a single object Y1. When calculating one or more arrival times and / or time intervals, the arrival time determiner 112 can consider any number of objects, such as any of the various objects modeled using sensor data 102. For example, Figure 3B and Figure 3C The diagram illustrates contour lines representing the arrival times of multiple objects in the environment.

[0081] exist Figure 3C In the context, object Y1 can correspond to Figure 1 Object 128A, object Y2 can correspond to Figure 1 Object 128B, and object Y3 may correspond to an object not represented in sensor data 102 but modeled using object modeler 108. For any of positions P1, P2, P3, P4, P5, P6, and P7, arrival time determiner 112 may use the arrival time and / or time interval of the earliest appearing object in the trajectory of the modeled object and the specific location. For example, for position P1, arrival time determiner 112 may use the arrival time of object Y1 because it occurs before the arrival times of objects Y2 and Y3 for position P1.

[0082] Object modeler 108 for location P l The arrival times and / or time intervals of P2 and P3 can be determined by... Figure 3CThe diagram shows the points where contour lines 308 are formed, drawn in the spatiotemporal domain. Contour lines 308 represent corresponding locations along the environment 126 (e.g., similar to contour 306) but taking into account the arrival times T1 (and / or time intervals) of multiple objects. Similarly, the object modeler 108 can use the arrival times and / or time intervals of locations P4, P5, P6, and P7. Figure 3C The text indicates that the points where contour lines 310 were drawn in the spatiotemporal domain are formed. Contour lines 310 represent the arrival time T2 (and / or time interval) of the positions along the environment 126.

[0083] Figure 3B It shows including Figure 3C The additional contour lines 308 and 310 of contour map 300, as well as other contour lines in environment 126 for different arrival times and / or time intervals and locations. Figure 3B Region 330 in the middle can correspond to Figure 3C Region 330 in the middle. Although Figure 3B The contour lines correspond to multiple objects; in other examples, any one of these contour lines may correspond to a single object. Furthermore, there may be an infinite number of contour lines, each corresponding to a different arrival time and / or time interval (if any). In any example, the arrival time determiner 112 may calculate data representing one or more of the contour lines, portions of the contour lines, and / or corresponding locations and arrival times (e.g., points in spacetime).

[0084] Furthermore, in some examples, the data may include image data representing an image that captures any of the aforementioned information. For example, the arrival time determiner 112 may generate a visualization of the image based at least on the arrival time. The data values ​​of the image used for visualization (e.g., pixel values, vector graphics values, analog values, digital values, etc.) can be computed in parallel, for example, by using parallel processing that can occur on one or more graphics processing units (GPUs), such as... Figure 14 One or more graphics processing units (GPUs) 1508 Figure 13C GPU 1308 and / or Figure 13CGPU 1320. In at least one embodiment, the data values ​​of the image (e.g., pixel values, vector graphics values, analog values, digital values, etc.) may represent at least a portion of a safe time interval. For example, the arrival time determiner 112 may use one or more pixels or points of visualization to represent the arrival time, for example by setting brightness values, color values, and / or other values ​​associated with or used to generate pixels or points. Each point or pixel of visualization generated by the arrival time determiner 112 may represent the arrival time and location in environment 126. The location of a point or pixel in visualization may correspond to a location in environment 126, and the color or other attributes of a point or pixel in visualization may correspond to the arrival time (e.g., safe arrival time) at that location.

[0085] One or more trajectories analyzed by trajectory analyzer 114 may have been generated by another system or component (e.g., a behavior planner for vehicle 140), which may not have considered, for example, arrival time and / or time intervals. In at least one embodiment, one or more trajectories may have already used... Figure 8 The behavior planning architecture 800 is generated. The trajectory can be represented using any suitable method, including using spatiotemporal trajectory points and / or spline curves or other functions. For example, vehicle trajectory information 124 may include trajectory points 142 along trajectory 144 (e.g., represented by 2D or 3D coordinates in world space and / or 2D coordinates in image space). In some examples, only a single trajectory point 142 (e.g., the next trajectory point of vehicle 140 in a discrete trajectory step sequence) is available (e.g., output by a machine learning model). In other examples, more than one trajectory point 142 may be available. In at least one embodiment, one or more trajectory points 142 can be calculated and / or extrapolated from one or more other trajectory points 142. In some examples, one or more trajectory points 142 can be calculated using the radius or reciprocal radius of trajectory 144 (e.g., where trajectory 144 includes turns, lane changes, lane splits, lane edges, and / or curves).

[0086] As described herein, trajectory analyzer 114 can be configured to evaluate one or more trajectories 144 based at least on arrival times determined by arrival time determiner 112. For example, trajectory analyzer 114 can calculate one or more points in spacetime corresponding to one or more objects in environment 126 (e.g., object 128A and / or object 128B). Trajectory analyzer 114 can use one or more point objects in spacetime corresponding to one or more objects to evaluate trajectory 144 with respect to one or more points in spacetime within trajectory 144. For example, trajectory analyzer 114 can calculate one or more points in spacetime corresponding to a vehicle 140 implementing one or more safety procedures (e.g., points that vehicle 140 would occupy if vehicle 140 implements one or more safety procedures) from one or more points within trajectory 144. Trajectory analyzer 114 can then compare these one or more points with one or more points corresponding to one or more objects to evaluate trajectory 144. Based on the comparison results, trajectory 144 can be scored, eliminated from consideration, or otherwise considered for the control of vehicle 140.

[0087] Figure 1 Examples indicating at least one or more points in space and time corresponding to object 128A (e.g., a pedestrian). In particular, Figure 1 The arrival times T of C0, C1, C2, and C3 of object 128A are shown. A The contour lines. One or more points calculated by the arrival time determiner 112 and / or used by the trajectory analyzer 114 to evaluate the trajectory 144 may include one or more points on any of those contour lines. As an example and not a limitation, Figure 1 The contour lines form concentric circles around object 128A, and the arrival time increases with distance from the starting position of object 128A. As described in this paper, when considering multiple objects, the contour lines / points can be calculated as the minimum arrival time of the objects.

[0088] Figure 1 It also indicates that trajectory analyzer 114 can evaluate one or more points within trajectory 144 with respect to points associated with arrival times. Specifically, Figure 1 The location of vehicle 140 is indicated for trajectory times TT of T1, T2, and T3, where, by way of example and not limitation, T0 = 0, T1 = 0.5, and T2 = 1.0. The location and trajectory time may include one or more points that can be evaluated relative to one or more points associated with the arrival time. For example, trajectory analyzer 114 may sample trajectory 144 over one or more trajectory times for comparison relative to points associated with the arrival time. In various embodiments, trajectory analyzer 114 may be able to perform analysis over any number of trajectory times T. TAny number of samples of the trajectory 144 can be used. In one or more embodiments, the trajectory analyzer 114 can evaluate samples at different time steps, which can be spaced apart by 0.5 second intervals, by way of example and without limitation. In general, any number of time steps can be used for samples having any time interval or amount of time between time steps. Further, the time interval can be different from the sampling to the next one of the trajectory times T T The same or different.

[0089] Figure 1 Further indicated are examples of one or more points in space-time corresponding to vehicles 140 implementing one or more safety procedures, which can be computed from one or more points within the trajectory 144. Figure 1 represent points of the vehicle 140 in space-time occupied by the vehicle 140 implementing safety procedures 140A, 140B, and 140C at trajectory times TT of TO, T1, and T2 within the trajectory 144, respectively. To evaluate in samples of the trajectory 144, the trajectory analyzer 114 can compare one or more points of a safety procedure corresponding to a sample to one or more points of the time of arrival.

[0090] As described herein, the trajectory analyzer 114 can employ the safety procedure determiner 110 to determine a safety procedure for a vehicle 140 for a given sample. The safety procedure determiner 110 can determine a safety procedure for a sample at a trajectory time TT based at least on a state of the vehicle 140 at a trajectory time TT T T The state of the vehicle 140 can be determined using the object analyzer 106, can include at least some information the same as or similar to information determined for other objects described herein, and can be obtained using similar or different methods. In one or more embodiments, the object analyzer 106 can determine the state of the vehicle 140 using any combination of sensors such as the GNSS sensor 1358, the IMU sensor 1366, the speed sensor 1344, the steering sensor 1340, etc. of the vehicle 140. For example, the state information can correspond to one or more sensor readings.

[0091] Examples of the state information include a position, a velocity, a direction (e.g., a heading), a speed, an acceleration (e.g., a scalar, a rotation, etc.), a pose (e.g., an orientation), and / or other information. The state can encode or represent a position of the vehicle 140 in two-dimensional space (e.g., (x, y) coordinates), a unit direction of the vehicle 140, and / or a scalar speed of the vehicle 140 at a point in time. In some examples, the state can encode or represent additional or alternative information such as a rotational speed (e.g., a yaw) and / or a scalar acceleration in any direction. For example, the state x A ​which can be parameterized as an m-dimensional state vector, denoted in equation (1) as follows:

[0092]

[0093] As an example, the state x A is a five-dimensional vector (e.g., m = 5), the state vector can be denoted in equation (2) as follows:

[0094] x A = [y T d T v] T (2)

[0095] where y is the position of the vehicle 140 in two-dimensional space, d is a unit direction vector, and v is a scalar velocity.

[0096] When the state of the vehicle 140 is considered as a function of time, the vector can represent a state trajectory X A of the vehicle 140 (e.g., the state trajectory X A may represent or encode every state x A of the vehicle 140 at every point in time over a time period).

[0097] The safety program determiner 110 can use the state of the vehicle 140 (and / or other objects in the environment 126) to determine a control model for the vehicle 140. For example, the control model can be denoted in equation (3) as follows:

[0098]

[0099] Thus, the control model for the vehicle 140 can represent the derivative of the state x A of the vehicle 140 with respect to time t. The explicit differential equation can include control parameters c, which can model user inputs such as steering, braking, and acceleration. For example, in some examples, the control model for the vehicle 140 can be denoted according to the following equation (4):

[0100]

[0101] where v is a scalar velocity, d is a unit direction vector, a is a scalar acceleration quantity, b is a scalar steering parameter, d ⊥ is a perpendicular to d, generated by flipping the coordinates of d and taking the negation of the first coordinate. In the example of equation (4), the control parameters can be the scalar acceleration quantity a and the scalar steering parameter b.

[0102] Once the control model is determined, a control policy can be determined (e.g., by the safety program determiner 110). For example, the control parameters can be a function of the world state x w(or perception of the world state based on sensor data generated by sensors of the vehicle 140) and time t. Thus, the control policy can be smooth and bounded a function of the joint state space of the world and time, where m is the dimension of the state space of the vehicle 140. For example, the control policy can be represented in Equation (5) as follows:

[0103]

[0104] Once the control policy is determined, a safety program can be determined for the vehicle 140 (e.g., by the safety program determiner 110). For example, it can be assumed that the vehicle 140 has a safety program S A that is derived from any starting state x A of the vehicle 140. In embodiments, the starting state x A of a sample of the trajectory 144 can be or correspond to the state at the sample’s trajectory time T T . The safety program can represent a trajectory of the vehicle 140 as the agent transitions from the state x A to an agent state goal (e.g., a final location where the agent can stop). The agent state goal can refer to one or more end conditions of the safety program that involve the state of the agent (e.g., vehicle) implementing the safety program.

[0105] In the example of Figure 1 each sample’s safety program includes an agent state goal where the vehicle 140 stops. For example, for the trajectory time T T of TO, the agent state goal can be satisfied at TO + T stop，0 , where T stop，0 can represent the amount of time it takes for the vehicle 140 to reach the agent state goal from the corresponding state at TO. Similarly, for the trajectory time T T of T1, the agent state goal can be satisfied at T1 + T stop，1 , where T stop，1 can represent the amount of time it takes for the vehicle 140 to reach the agent state goal from the corresponding state at T1, and for the trajectory time T T of T2, the agent state goal can be satisfied at T2 + T stop，2 , where T stop，2 can represent the amount of time it takes for the vehicle 140 to reach the agent state goal from the corresponding state at T2. While the agent state goal includes a complete stop, other agent state goals can be used and different agent state goals can be used for different samples. Moreover, in some embodiments, more than one agent state goal can be used for the same sample or trajectory time T T .

[0106] In some examples, the actor state goal can be determined by analyzing sensor data received from one or more sensors (e.g., one or more sensors of the vehicle 140) to determine a position, orientation, and velocity of an object (or other actor) in the environment 126. Control parameters (e.g., for steering, braking, acceleration, etc.) as described herein can then be determined for the vehicle 140, and a set of functions for directing the vehicle 140 to the actor state goal can be determined.

[0107] The safety program can cause the trajectory to vary smoothly from its initial state (e.g., because the safety program can be a continuous deceleration to a stop). In some examples, the safety program S A may be represented in Equation (6) as follows:

[0108]

[0109] where W represents properties of the world (or environment). Depending on the embodiment, the safety program of the vehicle 140 can or can not depend on fixed properties of the world. For example, the safety program can not rely on fixed properties of the world, such as road shape or a map. In such examples, the safety program can include a frozen direction vector (e.g., by setting the scalar steering parameter b to zero), and a full stop by decelerating through a series of acceleration values [a min , a'] (where a min is a minimum amount of acceleration or a negative of a maximum amount of braking, and a' is a negative value greater than a min ). This type of safety program S A may be represented by the following Equation (7):

[0110]

[0111] In any examples, the safety program can include braking until a full stop is reached and / or until the actor is traveling below a threshold speed. At high speeds, but not limited to, the safety program can include aligning with the current lane (or aligning with the direction of the road, such as when the vehicle 140 is in the middle of a lane change), and then stopping completely (and thus can depend on fixed properties of the world, such as lane markings). For example, but not limited to, at low speeds, the safety program can include the vehicle 140 turning itself to the side of the road as it decelerates to a stop (and thus can depend on fixed properties of the world). For example, one or more neural networks (e.g., convolutional neural networks) can be used to identify the side of the road and / or help maneuver the vehicle 140 to the side of the road. As another example, the HD map 1322 and / or another map type can be used. In such examples, the HD map 1322 can be received over the network 1390 and / or can be embedded in the vehicle 140.

[0112] In yet another example, safety procedures can be modified to provide a certain level of comfort (e.g., maximum comfort) for passengers in vehicle 140 (e.g., minimal deceleration or change of direction) while still ensuring that a collision is avoided or where forces and / or actor speeds are below a threshold. In such examples, a route, trajectory, and / or control sequence can be determined for vehicle 140 as a safety procedure to maximize comfort and / or minimize forces exerted on passengers while ensuring that other objects (e.g., vehicles, entities, structures, etc.) are avoided and / or other criteria are met. In some examples, such as when a collision is unavoidable or the probability of a collision exceeds a threshold risk level, safety procedures can be modified or configured to minimize the risk of injury to passengers and other entities in vehicle 140 in the event of a collision.

[0113] Example reference for security procedures Figures 4A-4C Explanation was provided. For example, regarding... Figure 4A Safety procedure 402 may include actor 404 (or a shape representing actor 404, such as a subscribed rectangle or polygon) coming to a complete stop while maintaining a low or zero rate of lateral change. For example, in an unstructured environment, or when the fixed properties of the world are ignored, safety procedure 402 may include moving straight forward and / or continuing along the current steering circle (which may or may not include a rate of lateral change) until actor 404 (e.g., vehicle 140) comes to a complete stop. For example, if the actor is currently turning right at a steering angle, safety procedure 402 may include continuing at the steering angle until a complete stop is reached. If the actor is currently moving straight, safety procedure 402 may include continuing straight until a complete stop is reached (e.g., as...). Figure 4A (As shown).

[0114] In any example, the safety procedures of vehicle 140 may include safety margins (e.g., in addition to, or as an alternative to, the safety margins described herein with respect to the dimensions of vehicle 140). For example, with respect to safety procedure 402, the safety margin of the safety procedure may increase as time increases spatiotemporally from the time associated with the actor's current state. For example, with respect to safety procedure 402, the width W1 of the set of protections claimed by safety procedure 402 may be smaller than the width W2 of the set of protections claimed by safety procedure 402. In such an example, because width W1 may correspond to an earlier time, the error tolerance may be smaller compared to the time associated with width W2. Therefore, the safety margin may increase over spatiotemporal time to address this error.

[0115] As another example, safety procedure 406 may include, during a lane change from first lane 408A to second lane 408B, actor 404 (or representing the shape of actor 404) aligning itself with the road (e.g., aborting the lane change and aligning with the road direction, such as parallel to lane markings 410A, 410B, and / or 410C) and coming to a complete stop. In such an example, the safety procedure may take into account fixed properties of the world (e.g., lane markings, road direction, etc.). Safety procedure 406 can be determined to minimize the lateral rate of change (e.g., relative to the road shape) while still aborting the lane change and realigning actor 404 with the road.

[0116] As a further example, and regarding Figure 4C Safety procedure 412 may include actor 404 (or a shape representing actor 404) that follows the road shape to adapt to curves in the road and comes to a complete stop. For example, if actor 404 is already following the road shape, and therefore takes into account the fixed properties of the world, safety procedure 412 may include continuing to follow the road shape (e.g., as defined by lane markings 410D and 410E). Similar to safety procedure 406, safety procedure 412 may be determined to minimize the rate of lateral change while continuing to follow the road shape.

[0117] Once the safety procedures for vehicle 140 are determined, the protection set determiner 116 can determine the protection set of vehicle 140 in environment 126. The protection set of an actor can include information about when the actor is in state x. A Start applying its security program S A The occupation trajectory of the actor (e.g., each point in space-time occupied by the actor when following the trajectory).

[0118] To determine the set of claims, the claim set determiner 116 can determine the area and / or volume occupied by the actor in space, given its state. For example, it can be assumed that the actor moves around and occupies n-dimensional real space. In some examples, for simplicity, a two-dimensional (2D) space modeled as a top-down view of the real world can be used. In other examples, a three-dimensional (3D) space can be used. In any example, to determine the claimed set, vehicle 140 can first determine the actor's occupancy set, which represents the set of points in the space occupied by the actor according to its state. An actor's occupancy set o A The following can be determined in equation (8):

[0119]

[0120] If a point in the space is in the actor's occupancy set, then the actor can be determined to occupy the point.

[0121] To determine each point in the occupancy set, a size of the actor (e.g., an actual size of the actor) or a representative size (e.g., a shape that surrounds and / or includes the actor) can be determined. In some examples, the size or representative size can include an optional safety margin. With respect to the vehicle 140, the size of the vehicle 140 can be known (e.g., based on calibration information, vehicle information, vehicle make and model, and / or other parameters). In some examples, to determine the size of the actor, a shape (e.g., a predefined or subscribed shape, such as a square, a polygon, a bounding box, a cuboid, a circle, an oval, an ellipse, etc.) can be fit around the actor (e.g., to include at least the actor) and the size of the actor can be determined as the size of the predefined shape (e.g., including a safety margin in some examples, as described herein). For example, the shape can be a 2D shape (e.g., a rectangle or a circle) to serve as a bounding box that at least partially encloses the actor. In other examples, the shape can be a 3D shape (e.g., a cuboid) to serve as a bounding cuboid that at least partially encloses the actor. In any example, the claim set determiner 116 can use the size of the vehicle to determine points (e.g., (x, y) coordinates) in the space that the actor occupies as part of the occupancy set o A .

[0122] In examples of the Figure 1 , a shape 132 is shown that can be used to determine one or more points in the spatiotemporal claim set of the vehicle 140. The shape 132 can include a minimum or bounding rectangle that at least surrounds the body of the vehicle 140. In other examples, a hexagon or other shape can be used.

[0123] In some examples, the size of the actor can be determined, as well as a representative shape that corresponds to the size of the actor, such that the size and / or shape completely includes the actor in at least two dimensions (e.g., lateral and longitudinal). By completely containing the actor (with an additional safety margin in examples), the occupancy set, the occupancy trajectory, and thus the claim set are more likely to more accurately represent the actual points in the space that the actor will occupy when performing a safety procedure.

[0124] Once the occupancy set is determined, the claim set determiner 116 can determine an occupancy trajectory o A for each actor. The occupancy trajectory can include a set of points in the spatiotemporal space that the actor will occupy as a function of its trajectory over time. For example, the occupancy trajectory o A may be determined as follows, in Equation (9), as follows:

[0125]

[0126] When the safety program S of an actor is applied A , the occupancy trajectory can include a protected set C A . Any point determined to be within the protected set C of an actor can be a point in space-time that the actor can need to maintain the integrity of its safety program. The protected set C A may be determined as follows in equation (10) below:

[0127]

[0128] where equation (10) can represent an occupancy trajectory of an actor that results if the actor applies its safety program starting from state x A . In some examples, the protected set can represent a combination or aggregation of each occupancy trajectory that results from applying the safety program of the actor with different parameters. For example, the safety program can apply a maximum braking curve, a minimum braking curve, a braking curve between the maximum and minimum values, a maximum steering curve, a minimum steering curve, a steering curve between the maximum and minimum values, and / or the like. In such examples, the protected set can include the occupancy trajectories of any number of different applications of the safety program (e.g., for each different application) that are combined or aggregated.

[0129] As a first example, the protected set can represent a safety program with a first braking curve for stopping completely faster than a second braking curve, and can represent a safety program with a second braking curve for stopping completely slower while still avoiding a collision (e.g., than the first braking curve). In such examples, a threshold or bound can be set and / or determined to define the first braking curve (e.g., a defined maximum or upper bound braking curve) and the second braking curve (e.g., a defined minimum or lower bound braking curve). In such examples, the protected set can represent each of the points in space-time that the actor occupies through application of the safety program of the first braking curve, the second braking curve, and points in space-time that the actor occupies that fall between the first and second braking curves (e.g., as shown in FIG. 6 and described with respect to Figure 5C Figure 5C According to particular embodiments, the threshold or bound used to generate the protected set can be different.

[0130] ​The threshold or limit can be defined without limitation based on a percentage (e.g., a percentage of braking intensity) and / or a time (e.g., a time to come to a complete stop, e.g., based on a current rate or speed). For example, a first braking curve can include a braking intensity of 95%, while a second braking curve can include a braking intensity of 85%. As another example, if the vehicle 140 is traveling at sixty miles per hour (mph), a first braking curve can include stopping within three seconds (e.g., decelerating at an average of twenty miles per hour per second), while a second braking curve can include stopping within four seconds (e.g., decelerating at an average of fifteen miles per hour per second). To determine the time, a speed factor can be used in some examples. In such examples, a first braking curve can include the vehicle 140 being traveling at x A ten miles per hour per second, and a second braking curve can include the vehicle 140 being traveling at x A fifteen miles per hour per second, based on a current state x

[0131] As another example, the set of claims can represent a safety procedure with a first steering curve (e.g., a defined maximum steering curve) for reaching a lateral position (e.g., a roadside, aligned with a road, etc.) faster than a second steering curve, and can represent a safety procedure with a second steering curve for reaching the lateral position slower than the first steering curve. In such examples, a threshold or limit can be set and / or determined to define the first steering curve (e.g., a defined maximum or upper limit steering curve) and the second braking curve (e.g., a defined minimum or lower limit steering curve). In such examples, the set of claims can represent a safety procedure by applying the first steering curve, the second steering curve, and each of the points in space-time occupied by the actor that fall between the first and second steering curves. According to embodiments, the threshold or limit used to generate the set of claims can be different.

[0132] Similar to the threshold or limit for the braking curve, the steering curve can also be based on a percentage (e.g., a percentage of steering angle intensity) and / or a time (e.g., an amount of time to reach a lateral position).

[0133] As a simplified example of the set of claims, and with reference to Figure 5A shown, the vehicle 140 can be traveling towards the static object 502 (with the direction of travel in the Figure 5A(Represented in one dimension in the spatiotemporal diagram). Vehicle 140 may have an occupied trajectory 504 when not performing safety procedures, which could lead to a collision between vehicle 140 and static object 502 (e.g., vehicle 140 continues along the same path at the same speed without performing safety procedures until it collides with the object). Static object 502 is fixed along the spatial dimension of the spatiotemporal diagram because it does not move. In this example, a boundary polygon can be used to represent the dimensions of vehicle 140 (e.g., the boundary polygon extends from the front to the rear of vehicle 140 between boundary lines 516A and 516B).

[0134] To explain Figure 5A In such cases, safety procedures can be implemented, such as braking vehicle 140 until it comes to a complete stop before colliding with the stationary object 502. For example, as... Figure 5B As shown, the trajectory generator can generate a trajectory 506 corresponding to a safety procedure, represented by a corresponding set of protections determined by the protection set determiner 116, and the vehicle 140 can implement the safety procedure at time T1 such that the vehicle 140 stops just before colliding with the static object 502. The trajectory 506 (e.g., the set of protections representing the trajectory of the safety procedure) can be continuously projected into space until the trajectory 506 is determined to intersect the static object 502 almost completely (e.g., at time T1), then the vehicle 140 can implement the safety procedure by actuating the brakes with an intensity corresponding to the safety procedure (e.g., the intensity that will stop the vehicle 140 before colliding with the static object 502, with a certain error margin, depending on the embodiment). In some examples, the trajectory 506 may correspond to a second braking curve (e.g., a defined minimum or lower braking limit curve).

[0135] As another example, and regarding Figure 5CThe generated trajectories 506 and 508 are represented by corresponding sets of claims determined by the sets of claims determiner 116, which correspond to safety procedures implemented using two different braking profiles (e.g., the sets of claims can include each of the points in time-space occupied by the vehicle 140 if the vehicle 140 were to navigate the first trajectory 506, the second trajectory 508, and / or any trajectory in between). For example, the trajectory 506 can include a first braking profile (e.g., a defined minimum or lower limit braking profile) and the trajectory 508 can include a second braking profile (e.g., a defined maximum or upper limit braking profile). In this example, the vehicle 140 can implement a safety procedure before the trajectory 506 collides with the static object 502 (e.g., at time T1). Thus, the trajectories 506 and / or 508 (represented by the corresponding sets of claims) can be projected into time-space until it is determined that the trajectory 506 (or in some examples 508) nearly intersects the static object 502, then the vehicle 140 can implement the safety procedure by actuating the brakes at an intensity corresponding to the braking profile selected for the safety procedure.

[0136] A property of the set of claims can be that the set of claims does not grow over time as the safety procedure is applied. Thus, for a safety procedure, a minimum (or safe) braking profile and a maximum braking profile can be defined such that the set of claims does not increase (although the set of claims can decrease). Thus, at each time step or phase of implementing the safety procedure, the set of claims can not increase. Thus, when the safety procedure is first implemented, and at the end of the safety procedure (e.g., when stopped completely), the set of claims should remain the same.

[0137] As another example, and with respect to Figure 5D, the vehicle 140, the first object 540A (e.g., a vehicle in this example), and the second object 540B (e.g., a vehicle in this example) in the environment 126. In this example, the trajectories can occupy a spatio-temporal three-dimensional space (e.g., volume) within the environment 126. Thus, the trajectories can include longitudinal distance (e.g., braking or stopping distance), lateral variation (e.g., turning variation), and / or vertical space occupied by the actors (e.g., the vehicle 140, the first object 540A, and the second object 540B) if they were to implement their respective safety procedures (e.g., from the ground plane to the top of the bounding polygon or other shape representing the set of occupancies of the actor). Thus, the trajectories can be analyzed as solid volumes of length that increase with speed (e.g., the faster an actor moves, the longer the trajectory and the spatio-temporal correspondences contained in the set of occupancies), and the actors can be characterized as driving around with these volumes (e.g., protruding from them) attached to them while performing collision analysis on these volumes (e.g., trajectories) instead of on their actual shapes. As a result, by guaranteeing no collisions in the spatio-temporal volumes, a guarantee of no collisions in the actual space can be induced. This can be beneficial because avoiding collisions between actual physical objects in the actual space requires foresight because actors have inertia and it can be too late once physical overlap occurs. However, in spatio-temporal space, once an intersection, overlap, or near-intersection or overlap between spatio-temporal volumes is determined, the volumes or trajectories can be considered frozen because of the simultaneous presence of lateral and longitudinal dimensions, and the shared geometry between the vehicle 140 and the objects can not allow for a collision to occur (or at least can not cause a likelihood of a collision to occur because the actions of other actors are not within the control of the vehicle 140).

[0138] In such examples, the vehicle 140 can generate a vehicle occupancy trajectory 520 (in some examples, applied to a series of curves) representing the safety procedure of the vehicle 140, an object occupancy trajectory 522 representing the safety procedure of the first object 540A, and an object occupancy trajectory 524 representing the safety procedure of the second object 540B. In Figure 5D In the illustration of FIG. 5, there is no overlap or intersection, or near overlap or intersection, between any of the trajectories 520, 522, and 524. Thus, at the point in time shown in Figure 5D In the illustration of FIG. 5, there is no overlap or intersection, or near overlap or intersection, between any of the trajectories 520, 522, and 524. Thus, at the point in time shown in

[0139] When implementing safety procedures, the points in space-time occupied by the projection of the trajectory 520 can comprise a required protection set for the vehicle 140. Similar to what is described herein with respect to Figure 4C The vehicle-occupied trajectory 520 can comprise the first braking curve, the second braking curve, a braking curve for another threshold, and / or combinations thereof. Similarly, the points in space-time occupied by the projection of the object-occupied trajectory 522 can comprise a required protection set for the first object 540A, and the points in space-time occupied by the projection of the object-occupied trajectory 524 can comprise a required protection set for the second object 540B.

[0140] In some examples, the latency, discretization, and / or reaction time can be at least some of the practical limitations that can be modeled. For example, the vehicle 140 can be subject to limitations of perception, or more accurately, limitations of perception and action, in the sense that when an actor takes an action, it is inevitably based on an incomplete current perception (e.g., with a time delay). Thus, when an actor takes an action, it can be based on a perception of the world at some earlier point in time. For example, an actor (e.g., a human actor, a manually driven vehicle, or a pedestrian) can have some reaction time before noticing that a potential collision can occur (e.g., based on being distracted due to looking at a phone or reaching for something, etc.). In such examples, the vehicle 140 can account for this reaction time. In other examples, an actor such as a vehicle can include a delay or lag between receiving a command and when the actuation actually occurs. The delay or lag can be known (e.g., after identifying the type of vehicle) or can be perceived (e.g., using one or more neural networks). In such examples, the vehicle 140 can account for such a delay or lag. In any examples, the shape (e.g., length, width, height, etc.) of the trajectory for a required protection set for an actor (e.g., the vehicle 140 and / or an object) can be adjusted (e.g., lengthened, widened, etc.) to account for the delay, lag, or reaction time.

[0141] In some examples, it can be assumed that the amount of delay is At. To account for At, in some examples, a form of worst-case forward prediction can be used, such that the forward set of an actor A in a time interval At A (x A , At) is all states that the actor A can reach after being in state x A after a time interval At. The forward set of the set of actors Θ in a time interval At can be the union of the forward sets of all actors in Θ, as shown in the following equation (11):

[0142] Φ(Θ, At) = U A∈Θ Φ A (x A, At) (11)

[0143] Actors can generally have a better ability to predict their own state compared to other actors. In particular, in the control system of vehicle 140, the actual command sequence sent previously can be known, providing the ability to predict where the actor itself will be when the actuation command currently under consideration (e.g., being passed to an actuation component of vehicle 140) is actually issued. For practical purposes, this can allow the forwarding set to include only one point, effectively resulting in deterministic forwarding, and further resulting in a single actor state. Generally, the forwarding mechanism can be non-deterministic forwarding, potentially resulting in a set of states. While non-deterministic forwarding of an actor itself can be used in some examples, and can require that the control policy be safe for all possible states of the actor, in other examples, deterministic forwarding of the actor itself can be assumed in order to reduce complexity.

[0144] The result can be a control policy for the forwarding actor, implicitly assuming that state parameterization is updated through prediction based on all actuation commands in the queue up to the actuation command currently under consideration. With these assumptions, a control command can be applied to the actor state under consideration, and the only latency can be information about other actors (e.g., objects other than vehicle 140).

[0145] Despite the latency limit between perception and action, the forwarding control policy can be safe for where the perceived set of actors moves at the current time. This can in turn be a direct result of the definition of the worst-case assumption and safe control policy. Since all constraints that can exist are assumed to exist (e.g., other actors can arrive from anywhere in the environment while performing control of vehicle 140), vehicle 140 can be compliant with all relevant constraints.

[0146] Further, the vehicle 140 can combine the delayed perception with the visibility perception and can use this information to avoid entering an unreasonable state. For example, consider the set Φ(V, Δt)∪(Φ(Λ, Δt)∩Ψ), where V, Λ, Ψ are the sets of visible actors, non-visible actors, and reasonable actors, respectively. First, visibility can be considered to provide a complete set of actors (visible and non-visible, as described herein) that can be desirably considered in the world at a certain point in time. Then, by forwarding this set of actors, the delay can be considered on this complete world representation. Finally, unreasonable actors can be excluded from the forwarded set of non-visible actors. In some examples, unreasonable actors can be excluded prior to forwarding; however, this would not allow for consideration of unreasonable actors that can be brought into a reasonable state during the forwarding process. Further, although unreasonable non-visible actors can be excluded, in some examples unreasonable visible actors can not be excluded as removing an actually perceived actor can not lead to an accurate world state.

[0147] As described herein, to evaluate a sample of the trajectory 144, the trajectory analyzer 114 can compare one or more points of the safety program corresponding to the sample with one or more points of the arrival time. For example, to evaluate a sample of the trajectory time T T corresponding to T0, the trajectory analyzer 114 can compare one or more points of the set of required protection determined for the safety program 140A with one or more arrival times determined for the object 128A (and / or other objects). In one or more embodiments, the arrival times compared with the points from the set of required protection can be arrival times at locations of the points from the set of required protection.

[0148] In the case where the arrival time is less than (or equal to) the time of the point from the set of required protection, the trajectory analyzer 114 can determine a conflict between the trajectory 144 of the sample and the arrival time. For example, the trajectory analyzer 114 can determine that the vehicle state of the trajectory sample would be unsafe based at least on determining that the arrival time is less than the time of the point of the set of required protection. This can indicate, for example, that if the object 128A arrives at the one or more locations at the arrival time and the vehicle 140 is in the state of the sample of the trajectory 144, the vehicle 140 would not be able to avoid a collision with the object 128A (and / or other objects captured by the arrival time).

[0149] In the event that the time of arrival is greater than the time of the point from the set of points of protection, the trajectory analyzer 114 can determine that there is no conflict between the trajectory 144 and the time of arrival of the sample. For example, the trajectory analyzer 114 can determine that the vehicle state of the trajectory sample is safe based at least on determining that the time of arrival is greater than the time of the point from the set of points of protection. This can indicate, for example, that the vehicle 140 will be able to avoid a collision with the object 128A (and / or other objects captured by the time of arrival) if the object 128A arrives at a location at the time of arrival and the vehicle 140 is in the state of the sample of the trajectory 144.

[0150] For a given trajectory sample, the trajectory analyzer 114 can compare any number of points of the set of points of protection of the sample to any number of times of arrival of locations in the environment 126. Since the vehicle 140 occupies many points in space at a given time, the trajectory analyzer 114 can compare multiple points of the set of points of protection to times of arrival to determine whether a conflict exists at a given time. As an example and not by way of limitation, these points can lie on a shape 132 in Figure 1 points P 1a , P 1b , and P 1c may be available to evaluate the sample of the trajectory 144 at a given time of the set of points of protection. In one or more embodiments, only points on the shape boundary can be used, as it can be assumed that if the boundary is conflict free, then the interior will also be conflict free.

[0151] Also in one or more embodiments, only points on one side of the shape can be used for evaluation at a given time. For example, a side of the vehicle 140 corresponding to the direction of travel of the vehicle 140. Thus, the trajectory analyzer 114 can determine that the vehicle 140 is backing up and use points from the back of the vehicle 140 instead of points from the front of the vehicle 140. Further, in some examples, one or more points can correspond to a plane perpendicular to the direction of travel. For example, if the vehicle 140 is making a right turn, one or more points can correspond to the front and right side of the vehicle 140 (close to the front to ensure at least the front edge of the vehicle is tested). Further, while these points lie on line segments in Figure 1 in the shape 132, these points can fall on the edges of a hexagon, semi-circle, or other shape.

[0152] In general, the trajectory analyzer 114 evaluates samples at multiple different times within the set of points of protection. In one or more embodiments, the trajectory analyzer 114 can be configured to avoid or eliminate gaps between adjacent sets of points of protection of samples being evaluated for conflicts with times of arrival. For example, in Figure 1In some embodiments, the region occupied by the vehicle 140 being evaluated is contiguous (e.g., in temporal order of the corresponding samples) such that there are no gaps. In one or more examples, gaps can be reduced or avoided by setting the time between time steps such that the actor state goal of each safety program is satisfied at a time greater than or equal to the next time step of the sample. For example, the time step can be calculated from the current speed of the vehicle 140, from the speed at the point in the trajectory 144 being sampled, and / or statically or dynamically configured such that the condition is likely or will be satisfied. In embodiments, if a gap is identified, interpolation (e.g., linear) between the sampled points can be used to account for any gaps that can be between regions (e.g., the set of claims). For example, points in the spatiotemporal of the set of claims being evaluated can be interpolated from one or more other points (e.g., to the end or boundary of the gap) and evaluated to narrow the gap between adjacent sets of claims.

[0153] Reference is now made to Figure 6 , Figure 6 is a diagram including an example of a trajectory, safety programs that can be invoked during the trajectory, and points that can be used to evaluate the trajectory, in accordance with some embodiments of the present disclosure. In particular, Figure 6 shows a trajectory 144 and one or more points P0to P N (where N is an integer) can be evaluated for a sample corresponding to T0, where the safety programs 140A, 140B, and 140C are individually labeled. For a sample at trajectory time T T = T0, e.g., at the beginning of the trajectory 144 and / or the current location of the vehicle 140, the trajectory 144 can evaluate one or more points in the spatiotemporal for multiple times through the safety program 140A. For simplicity, one or more points per time can be discussed with reference to a single point. However, the evaluation can include multiple points (e.g., according to the shape 132 described herein). As shown, the points P0and P o,1 to P 0,M (where M is an integer) can be evaluated for a sample corresponding to T0. In one or more embodiments, the trajectory analyzer 114 can determine a conflict for a sample based at least on determining a conflict for at least one point evaluated for the sample. For example, a conflict can be determined when any of the points conflict with the corresponding time of arrival.

[0154] In the example of Figure 1 , multiple points can be evaluated for the sample corresponding to T0to avoid gaps in coverage. However, for subsequent time steps, only the point at the tip of the safety program (e.g., the location that satisfies the actor state goal) can be evaluated. In Figure 6In some embodiments, the points at the tip of the safety program are represented by points P0to PN. Sampling only the tip can be sufficient because the tip can correspond to the most stringently limited point in the safety program (e.g., the latest safety arrival time). For example, if a collision can be avoided at the end of the sampled safety program, it can be assumed that a collision can be avoided at any earlier time in the safety program. Thus, checking for conflicts with respect to the safety program can be computationally efficient because the number of comparisons can be linear with respect to the number of safety programs evaluated. Moreover, each of these comparisons can be computed efficiently in parallel on a GPU. While multiple points are evaluated for the sample corresponding to T0, in other examples, a single point (e.g., point P0) can be sampled and / or evaluated points can leave an acceptable gap in the coverage. For example, a gap in the coverage can be acceptable because the vehicle 140 can have already evaluated a point for one or more trajectories of a previous frame that cover the gap.

[0155] As described herein, the trajectory analyzer 114 can analyze a plurality of trajectories, including the trajectory 144, in order to select a trajectory therefrom for controlling the vehicle 140. As an example and not by way of limitation, one or more trajectories can be represented using one or more lateral paths and one or more speed profiles. In particular, a trajectory can include a lateral path in combination with a speed profile for the vehicle along the lateral path. Figure 7 is an illustration for describing examples of lateral paths and speed profiles in accordance with some embodiments of the present disclosure. A set of trajectories can be determined based at least on a nominal trajectory or lane and / or lane edges from a world model of the vehicle 140 and based at least on one or more selected actions of the vehicle 140, as described herein with respect to Figure 8 Further described. As shown in FIG. 7A, Figure 7 As shown, the nominal trajectory can correspond to a right lane in which the vehicle 140 can be located. As can be seen, a series of cones 724 present an obstacle to be avoided. The vehicle 140 can generate 11 different lateral variation selections or lateral paths related to the nominal trajectory. The selections include path 702, path 704, path 706, path 708, path 710, path 712, path 714, path 716, path 718, path 720, and path 722. Any combination of paths and speed profiles can be used to form a trajectory.

[0156] The plurality of trajectories evaluated by the trajectory analyzer 114 for selecting a trajectory can correspond to different combinations of paths and speed profiles. In one or more embodiments, the comparisons between the sample and arrival times can be with respect to the lateral path Np, the speed profile N s and the time step N tin linear relationships, each of which can be computed in parallel on the GPU. Since the initial sample for each trajectory can be the same for each velocity profile sharing the same lateral path (e.g., at trajectory time T T = To) for each trajectory, the trajectory analyzer 114 can evaluate the sample for each lateral path once and apply the result to each corresponding trajectory. Thus, the complexity of the evaluation can be O(N p * N s * N t ).

[0157] In some embodiments, the trajectory analyzer 114 can stop analyzing a trajectory once a conflict is detected and / or based on the location at which the conflict is detected. In other embodiments, the trajectory analyzer 114 can always analyze each sample of a trajectory.

[0158] As described herein, based on the results of the comparison of the trajectory samples, a trajectory can be scored (e.g., for ranking against other potential trajectories), eliminated from consideration, or otherwise considered for control of the vehicle 140. The trajectory analyzer 114 can use various possible factors or criteria to determine a score for a trajectory and / or whether to exclude a trajectory from consideration. In cases where a trajectory is eliminated from consideration based on satisfying one or more criteria, the one or more criteria can be considered a hard constraint on the trajectory. In cases where a trajectory is reduced or lowered in score based on satisfying one or more criteria (assuming a lower score means a less likely trajectory to select), the one or more criteria can be considered a soft constraint on the trajectory.

[0159] In one or more embodiments, a constraint on a trajectory can be based at least on a distance along the trajectory to a detected conflict and / or a time of passage (e.g., to the start of the sample or to a point compared to the time of arrival) (e.g., a conflict that is a maximum distance and / or a time of passage from the start of the trajectory). For example, assuming there is a conflict detected for a trajectory, the farther along the trajectory the conflict is and / or the longer the time of passage to the conflict, the higher the score (or weight used to calculate the score) can be. Thus, if the trajectory analyzer 114 determines a conflict for trajectory time T T = T2 of the trajectory 144, the score for the trajectory 144 can be higher than if the trajectory analyzer 114 determines a conflict for trajectory time T T = T1 of the trajectory 144. In cases where multiple conflicts are detected, the score can be based at least on the conflict with the shortest distance, the least time of passage, and / or the number of detected conflicts.

[0160] In at least one embodiment, one or more criteria can be used as a hard constraint or a soft constraint depending on whether one or more conditions are satisfied. For example, a conflict of a sample can result in exclusion of a trajectory from consideration for control of vehicle 140 based at least on a distance to the conflict and / or an elapsed time (e.g., a detected conflict closest to a start of a trajectory) being below a threshold. A conflict can be used as a soft constraint (e.g., by penalizing a score of a trajectory) for selecting a trajectory based at least on a distance to the conflict and / or an elapsed time (e.g., a detected conflict closest to a start of a trajectory) being greater than a threshold. For example, when a conflict is detected by trajectory analyzer 114, trajectory analyzer 114 can compare a distance to the conflict and / or an elapsed time to one or more thresholds to determine whether to use the conflict as a soft constraint or a hard constraint and / or to determine how much or whether a score of a trajectory should be reduced by the conflict. Using this approach, conflicts can be allowed, or the later a conflict occurs in a trajectory the more it is allowed. Thus, a trajectory can still be selected for control of vehicle 140, and replanning a trajectory or selecting a different trajectory before reaching the conflict sample or time step can ensure that a control objective (e.g., avoiding a collision) is satisfied.

[0161] As described herein, arrival times evaluated by trajectory analyzer 114 for a world state can be evaluated for multiple trajectories, can be pre-computed and stored in an image or texture for an entire scene perceived by vehicle 140 or a subset of scenes that vehicle 140 can be in during a duration of a planning horizon. Values corresponding to arrival times for an image or texture can be computed using a distance transform or in parallel on a GPU. Conflicts for an image or texture can then be evaluated for points corresponding to trajectories, e.g., using texture reads and inequality checks. In other examples, arrival times can be computed on the fly or as needed during evaluation of a trajectory and / or can not be stored in an image or texture.

[0162] Reference is now made to Figure 8 , Figure 8 is a diagram of an example behavior planning architecture for an autonomous vehicle in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted or consolidated. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. The various functions described herein as being performed by an entity can be performed by hardware, firmware and / or software. For instance, various functions can be performed by a processor executing instructions stored in memory.

[0163] The behavior planning architecture 800 can use different planning components, each with different planning scopes. In one embodiment, the different planning components include a route planner 810, a lane planner 820, and a behavior planner 840. Each component can produce a result and / or a set of results and pass it to the next component in the planning pipeline. The route planner 810 can pass a set of routes to the lane planner 820, and the lane planner 820 can pass a lane plan to the behavior planner 840. The behavior planner 840 can develop and evaluate multiple trajectories, and can send a selected or preferred trajectory from the behavior planner to a motion controller 880 that is responsible for implementing the planned trajectory.

[0164] In embodiments, the route planner 810 can use geographic information that covers a geographic region (e.g., a town, a metropolitan area, a state, a country, a region, etc.), but with a relatively low amount of detail. The planning scope of the route planner can be the entire region between a start point (e.g., a current location) and a destination point (e.g., a final location).

[0165] The route planner 810 can be architected to be modularly separable from other planning modules. The architectural design can be effective to support long distance routes. The route planner 810 can hand off a resulting route plan, along with some basic state information, to the lane planner 820. The route plan can include one or more GPS trajectories that have been determined to be potential approximate routes for reaching a waypoint or a parking zone from a current location. The GPS points in the route plan can also have expected time rewards (or, if desired, time rewards can be expressed as expected times to a destination). The GPS trajectories of the route plan can give the lane planner 820 an approximate guidance, rather than a precise adherence to a particular lane or road. The route planner 810 can provide multiple routes or route variations that fall within a threshold travel time of each other, and in embodiments, each route can be associated with an estimated travel time. In one embodiment, each route segment is assigned a travel time that gives the lane planner 820 and other components information that can be used to calculate the cost of missing an exit and being forced to enter an alternative route. In some embodiments, the travel time can also be calculated by other components, such as the lane planner 820, which can use distance and speed limits to calculate the travel time, but can also refine the travel time using real-time traffic data, construction data, weather data, and / or the like.

[0166] The lane planner 820 can generate lane plans based on one or more routes received from the route planner 810. The lane planner 820 can use a second geographic map that includes more detail than found in the map used by the route planner. The second geographic map can cover a smaller geographic area than the map used by the route planner. For example, the smaller geographic area can be less than 50 square miles. The lane plans can take the form of annotated lane graphs that can have a planning horizon of several miles. The lane graphs can represent lanes available to the autonomous vehicle on a route (e.g., a road). The annotations on the lane graphs can indicate time scores for various locations on the lane graph. For example, time scores can be provided on the lane graph every 10 meters, 20 meters, or some other interval. The time scores can be described as continuous time rewards for potential future locations. Larger or better rewards can be associated with faster paths toward the destination of the route, or in other embodiments, lower scores can indicate better or faster paths. The scale of the rewards can be in time units, such as seconds.

[0167] The time scores convey an overall value of getting the autonomous vehicle to a particular point on the lane graph. The behavior planner 840 can then consider these scores when evaluating various trajectories. The output of the lane planner 820 can be represented as an expected time reward encoded on a large lane graph associated with the local lane graphs of the lanes. This encoding can provide more flexibility than just a direct command or suggestion of which lane to enter, as it allows the behavior planner 840 to weigh the value of being in a particular lane against the difficulty of the maneuver to get there. The large lane graph can correspond to a lightweight representation of lane positions and expected times to transition between the lane positions. The large lane graph can be derived from map data that is also linked to the local lane graphs of the lanes.

[0168] The behavior planner 840 can consider one or more actions and proceed with one or more iterations of hypothesis generation and evaluation of more detailed implementations of those actions. The more detailed implementations can be represented as precise motion plans (trajectories of poses for the next few seconds) and can be evaluated by the trajectory analyzer 114. The behavior planner 840 can include an action generator 842, a longitudinal pre- limiter 850, a hypothesis generator 860, and an action plan selector 870.

[0169] The behavior planner 840 can have a smaller planning horizon than the route planner 810 or the lane planner 820. In general, the behavior planner 840 can plan the vehicle’s movements for the next few seconds. The distance covered can vary depending on the vehicle’s speed of travel. For example, the behavior planner 840 can have a planning horizon of 50 m, 100 m, 200 m, 300 m, etc. The behavior planner 840 can use a map that shows a high level of detail over a small area. For example, the small area can be smaller than the area in the map used by the lane planner 820 or the route planner 810. The higher amount of detail can include objects (e.g., other vehicles, pedestrians) detected by sensors associated with the autonomous vehicle. In an aspect, at least some of these objects can not be in the information available to the lane planner 820. The selected actual trajectory can control the motion of the autonomous vehicle for only a short period of time (e.g., one second). New trajectories can be constantly evaluated and implemented to adapt to changing conditions in the autonomous vehicle’s environment (e.g., moving vehicles, moving pedestrians).

[0170] The computational flow of each action evaluated by the behavior planner 840 can be initiated by an action generator 842. The action generator 842 can initially select one or more desired actions for the autonomous vehicle to take. The actions can be based on the annotated lane graph provided by the lane planner 820. The selected one or more actions can be based on achieving the best outcome indicated by the time score included in the annotated lane graph. Possible actions include, but are not limited to, lane following, lane change speed adaptation, passing, lane change push, stop shoulder, parked loiter, parked dilemma, summon, and limp. As with previously discussed components, the action generator 842 can select or score multiple actions. In some embodiments, an action can be represented as a nominal trajectory and / or each action can be associated with multiple nominal trajectories. At the end of the process after the action generator 842, each action can be associated with a nominal trajectory and / or a lane edge and lane from the map.

[0171] The longitudinal pre-restrictor 850 can receive nominal trajectories corresponding to the actions generated by the action generator 842. Each of these nominal trajectories can be evaluated by the longitudinal pre-restrictor 850. The longitudinal pre-restrictor 850 can consider basic longitudinal constraints on a per-lane (or per-nominal trajectory) basis. Different components can consider different constraints, such as those related to curve speed adaptation, speed regulation, distance keeping, safe braking, and / or yielding. The output of the longitudinal pre-restrictor 850 can include acceleration constraints that restrict the speed associated with each nominal trajectory / lane as input. Thus, the longitudinal pre-restrictor 850 can augment the lane / trajectory with acceleration, speed, or distance restrictions.

[0172] The combined output from the various components of the longitudinal pre- limiter 850 can be a plurality of longitudinally limited nominal trajectories. The trajectories generated by the motion generator 842 can be combined with a plurality of different acceleration plans to generate a large number of trajectories along the trajectories. The plurality of longitudinally limited nominal trajectories includes trajectories that satisfy the various constraints computed by the longitudinal pre-limiter 850. The plurality of longitudinally limited nominal trajectories can be passed to the hypothesis generator 860 for further refinement.

[0173] Motion planning can continue through evaluation by the hypothesis generator 860 and the action plan selector 870. The action plan selector 870 can use the trajectory analyzer 114 to evaluate the quality of one or more given trajectories (e.g., the autonomous vehicle’s future complete planned pose), the hypothesis generator 860 can provide promising or plausible trajectories because the space of all trajectories is too large to search.

[0174] According to embodiments, high variability can be used for hypothesis generation. In some embodiments, the hypothesis generator 860 includes a path sector generator that starts from the longitudinally limited nominal trajectories and generates lateral variations around them while maintaining their curvatures. This is designed for following a lane while nudging around static and dynamic obstacles. A longitudinal variation within the longitudinal limit can be added to each lateral selection, resulting in a 2D trajectory grid. This approach can be used for standard lane following situations, as well as actions that can be built upon such as, for example and without limitation, parking meanders, parking impasses, summoning, and limping.

[0175] The hypothesis generator 860 can also use other less structured approaches such as dynamic programming to search for free-form paths or employ a linear quadratic regulator (LQR) / obstacle-aware MPC control that takes an initial path (e.g., nominal trajectory) and iteratively adjusts it to move away from obstacles. This can be done on static scenarios that ignore motion, trusting that the motion planner will see and avoid motion-induced situations, or directly use the set of claims in the search.

[0176] The trajectory analyzer 114 can evaluate each of the hypothesis trajectories to identify the highest quality or best trajectory. Quality can include a number of considerations, but can be quantified by a score as described herein. Different qualities can be weighted differently when calculating the score. Ideal driving can be traced back to various goals. Goals can include, but are not limited to, maximizing comfort and making progress (e.g., reaching a destination with minimal resource expenditure (e.g., time, money, fuel, and wear and tear)). Other goals can be maximizing collision safety (e.g., obstacle / object avoidance), following a lane (e.g., path) when other conditions are the same, and operating while adhering to applicable rules and conventions (e.g., waiting for a yield condition).

[0177] The score can be normalized to time. The continuous time reward of the potential future position can be used directly for the score and serve as a starting point, while other components contributing to the score can add time as a penalty. For example, a predicted collision or conflict of a trajectory can add 100 hours, which can represent a substantial adjustment or offset (e.g., penalty) and result in the trajectory not being selected. Smaller but still undesirable conditions can result in smaller adjustments. The adjustments can be scaled or normalized according to the route distance or estimated travel time. In this way, adjustments due to undesirable features of the route can add more time on longer routes and less time on shorter routes. Normalization can be done by calculating the adjustment as a percentage of the time to reach the destination. For example, certain conditions on the route can result in a 2% adjustment to the projected time remaining to reach the destination.

[0178] The trajectory analyzer 114 can also be configured to favor keeping close to the nominal lane or between the edges of the considered lane. These preferences can be used in calculating the score. When lane following, it can be strongly preferred not to extend beyond the edge of the lane line, even if the lane edge is not a physical barrier. However, it can be more desirable to extend beyond the edge of the lane line if it is the only viable method to avoid a collision. This is different from making a planned controlled lane change, however. In particular, because in a planned controlled lane change there is time to activate a turn signal and give others time to notice the signal. To this end, the trajectory analyzer 114 can consider trajectories differently in the context of lane following and lane changing, even if they can be the same trajectory. The closer to the center of the lane, the more the score of the trajectory is increased. The score is decreased when the trajectory crosses a lane boundary. The amount of score reduction can depend on the situation at the lane intersection. For example, crossing a dashed line can be less penalized than crossing a solid line or double solid line. Crossing oncoming traffic can be a very large penalty.

[0179] The trajectory analyzer 114 can analyze trajectories for obstacles or object competitors. The analysis can include attention to intersection involving the claimed set. Instead of considering whether there is a collision between physical bodies now or in the future, motion planning according to disclosed embodiments can consider whether there is an intersection involving one or more claimed sets. This can provide a way to handle how to upgrade static motion planning to an environment that does not allow competitors to move at a rate that approaches instantaneous stop. As described herein, in one or more embodiments, the claimed set can refer to a shape in space-time that an actor will trace when attempting to safely stop and align laterally with the road. The actor can virtually "claim" this set to remain collision-safe. Even if the bodies do not intersect, it can be difficult to allow for an intersection involving one or more claimed sets while maintaining some form of controlled safety. Instead, if the claimed set is respected, the actor can stay within it. If they do so, the claimed set plays out in space-time and can not expand. This allows for the upgrade of a static motion planning problem to one with motion. The claimed set can stop instantaneously, but a moving actor cannot. Another benefit of this approach is that it does not need to rely on training data or predictions in near or actual collision situations, which are rare or difficult to obtain. It also ensures collision safety, not just escape from without physical collision.

[0180] The trajectory analyzer 114 can select a trajectory based at least on a rating or score of the trajectories. In at least one embodiment, the trajectory with the highest score is selected. Longitudinal adjustments and / or lateral adjustments can be used to slightly smooth the selected trajectory. Various methods can be used to evaluate each trajectory and aggregate to form a score for selection. For example, different methods (e.g., such as the methods described with respect to Figure 1 Even in cases where a score is used, any of the various methods can be used to evaluate and enforce hard constraints on trajectories. In one or more embodiments, one trajectory can be selected from multiple trajectories (e.g., based on a score or another method), and then one or more methods (e.g., such as the methods described with respect to Figure 1

[0181] ​An iterative approach can be used to identify the optimal trajectory. In the first iteration, a plurality of hypothetical trajectories can be evaluated, with the trajectory having the highest score serving as a seed to generate additional hypothetical trajectories for evaluation. In subsequent iterations, an additional plurality of hypothetical trajectories can be generated by slightly varying various parameters of the seed trajectory. For example, the seed trajectory can be used as input to a path sector generator instead of the nominal orbit. The path sector generator can then generate an additional plurality of trajectories that are small lateral variations (smaller lateral bumps than those used in the first iteration on the nominal orbit) and small longitudinal variations. The additional plurality of hypothetical trajectories can then be evaluated to determine whether one of the trajectories has a higher score than the seed trajectory. The hypothetical trajectory with the highest score can be selected for implementation.

[0182] A motion controller (MC) 180 can take the selected trajectory and use a more refined vehicle model to select instantaneous lateral and longitudinal acceleration controls 895. This can involve computing a future trajectory based on the control sequence and the vehicle model, and iteratively adjusting the control sequence using a nonlinear optimizer to optimize a cost function. The cost function can be designed to find a tradeoff between smoothness and following the requested trajectory (although since there can be no obstacle perception for this stage, it can be set to follow the trajectory faithfully and the smoothness of the trajectory can be guaranteed by the motion planning stage).

[0183] Reference is now made to Figure 9 , Figure 9 is a flow diagram illustrating a method 900 for controlling a vehicle based on evaluating points corresponding to safety procedures deployable in a trajectory, according to some embodiments of the present disclosure. Each block of the method 900, and other methods described herein, comprise a computing process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The methods can also be embodied as computer-usable instructions stored on computer storage media. The methods can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. The methods described herein can additionally or alternatively be executed by any one of the systems, or any combination of systems, including but not limited to those described herein and not limited to particular examples.

[0184] The method 900 includes determining one or more first future points corresponding to one or more objects, at block B902. For example, the time of arrival determiner 112 can determine one or more future points (e.g., times of arrival) in spacetime corresponding to one or more objects (e.g., the object 128A) in the environment 126.

[0185] The method 900 includes, at block B904, computing one or more second future points, the second future points corresponding to deployment of one or more safety procedures in the first trajectory. For example, if the vehicle 140 implements one or more safety procedures (e.g., safety procedures 140A, 140B, or 140C) from one or more points in the spatiotemporal (e.g., corresponding to a time step or sample) in the trajectory 144, the trajectory analyzer 114 can compute one or more future points in the spatiotemporal corresponding to the vehicle 140 in the environment 126.

[0186] The method 900 includes, at block B906, controlling the autonomous machine using the first trajectory or the second trajectory based at least on the one or more first future points and the one or more second future points. For example, the trajectory analyzer 114 can use the trajectory 144 or a different trajectory to control the vehicle 140 based at least on a comparison between the one or more second future points in the spatiotemporal and the one or more first future points in the spatiotemporal. This can include determining whether the points conflict and scoring or otherwise evaluating the trajectory 144. This can also include selecting the trajectory 144 or other trajectory for controlling the vehicle 140 based on the comparison.

[0187] Reference is now made to Figure 10 , Figure 10 is a flowchart illustrating a method 1000 for controlling a vehicle based on modeling future sets of claims of protection corresponding to a vehicle implementing safety procedures in a trajectory, in accordance with some embodiments of the present disclosure.

[0188] The method 1000 includes, at block B1002, modeling one or more future sets of claims of protection of an autonomous machine implementing one or more safety procedures in a first trajectory. For example, if the vehicle 140 implements safety procedures 140A, 140B, and / or 140C in the trajectory 144, the trajectory analyzer 114 can model one or more future sets of claims of protection of the vehicle 140 in the environment 126 using the set of claims determiner 116.

[0189] The method 1000 includes, at block B1004, computing one or more arrival times of one or more objects. For example, the arrival time determiner 112 can determine one or more arrival times of one or more objects (e.g., object 128A) to one or more locations in the environment 126 (e.g., one or more locations of the tip of the future set of claims of protection).

[0190] Method 1000, in box B1006, includes controlling an autonomous machine using a first or second trajectory based on at least one or more sets of future claims and one or more arrival times. For example, trajectory analyzer 114 may control vehicle 140 using trajectory 144 or different trajectories based at least on a comparison between one or more times of one or more sets of future claims and one or more arrival times. This may include determining whether a conflict exists and scoring or otherwise evaluating trajectory 144. It may also include selecting trajectory 144 or other trajectories for controlling vehicle 140 based on the comparison.

[0191] Now for reference Figure 11 , Figure 11 This is a flowchart according to some embodiments of the present disclosure, illustrating a method 1100 for controlling a vehicle based on determining that the vehicle will occupy a location before the time when the location will be occupied by an object when implementing safety procedures during a trajectory.

[0192] Method 1100 in box B1102 includes: calculating one or more times that one or more objects will occupy at one or more locations in the environment if the one or more objects follow one or more trajectories. For example, arrival time determiner 112 can calculate one or more times that one or more objects (e.g., object 128A) will occupy at one or more locations in environment 126 (e.g., one or more locations of the tip of the protected set) if the one or more objects follow one or more trajectories.

[0193] Method 1100, in box B1104, includes controlling the autonomous machine using a first trajectory or a second trajectory based at least on determining that the autonomous machine will occupy one or more locations one or more times prior. For example, trajectory analyzer 114 may control the vehicle 140 using trajectory 144 or a different trajectory based at least on modeling one or more safety procedures implemented by the vehicle 140 during trajectory 144 to determine that the vehicle 140 will occupy one or more locations one or more times prior. This may include scoring or otherwise evaluating trajectory 144 based on the determination. It may also include selecting trajectory 144 or another trajectory for controlling the vehicle 140 based on that determination.

[0194] Example operating environment

[0195] According to some embodiments of this disclosure, the trajectory evaluation system 100 can... Figure 12 This is implemented in the example operating environment 1230. Among other components not shown, the operating environment 1230 includes a client device 1232, a network 1234, a server device 1236, a sensor 1238, and / or data storage 1246. It should be understood that... Figure 12The operating environment 1230 shown in FIG. 13 is an example of a suitable operating environment. Figure 12 Each of the components shown in FIG. 13 can be implemented by any type of computing device, such as the example computing devices 1400 described in connection with Figure 14 one or more of the computing devices 1400 described. The components can communicate with each other over a network 1234, which can be wired, wireless, or both. The network 1234 can include multiple networks or networks of networks, but is shown in a simple form to avoid obscuring aspects of the disclosure. By way of example, the network 1234 can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet), and / or one or more private networks. In the event that the network 1234 includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (among other components) can provide wireless connectivity. In any example, at least one network 1234 can correspond to the network 1390 of FIG. 13, described further below. Figure 13D

[0196] It should be appreciated that any number of client devices 1232, server devices 1236, sensors 1238, and data stores 1246 can be employed within the operating environment 1230 within the scope of the present disclosure. Each can be configured as a single device or multiple devices cooperating in a distributed environment.

[0197] The client devices 1232 can include at least some of the components, features, and functionality of the example computing devices 1400 described herein with respect to FIG. 14. By way of example and not limitation, the client devices 1232 can be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a robot, a drone, a global positioning system (GPS) or device, a video player, a handheld communication device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, any combination of these described devices, or any other suitable device. In any example, at least one client device 1232 can be part of a vehicle, such as the vehicle 140 described in greater detail herein with respect to FIG. 15. Figure 14 Figures 13A-13D

[0198] The client devices 1232 can include one or more processors, and one or more computer-readable media. The computer-readable media can include computer-readable instructions executable by the one or more processors. When executed by the one or more processors, the instructions can cause the one or more processors to perform any combination and / or portion of the methods described herein and / or implement the example computing devices 1400 described herein. Figure 1 ​​​any portion of the functionality of the trajectory assessment system 100.

[0199] The server device 1236 can also include one or more processors, and one or more computer-readable media. The computer-readable media include computer-readable instructions executable by the one or more processors. When executed by the one or more processors, the instructions can cause the one or more processors to perform any combination and / or portion of the methods described herein and / or implement Figure 1 any portion of the functionality of the trajectory assessment system 100. In any example, the at least one server device 1236 can correspond to the server 1378, described in greater detail herein. Figure 13D

[0200] The data store 1246 can include one or more computer-readable media. The computer-readable media can include computer-readable instructions executable by the one or more processors. When executed by the one or more processors, the instructions can cause the one or more processors to perform any combination and / or portion of the methods described herein and / or implement Figure 1 any portion of the functionality of the trajectory assessment system 100. The data store 1246 (or computer data store) is depicted as a single component, but can be embodied as one or more data stores (e.g., databases) and can be at least partially in the cloud. The one or more data stores 1246 can correspond to the one or more data repositories Figure 13C .

[0201] Although described external to the server device 1236 and the client device 1232, the data store 1246 can be at least partially embodied on any combination of the server device 1236 and / or the client device 1232 (e.g., as the memory 1404 Figure 14 )) of the server device 1236 and / or the client device 1232. For example, some information can be stored on the client device 1232, and other and / or duplicate information can be stored externally (e.g., on the server device 1236). Thus, it should be appreciated that information in the data store 1246 can be distributed in any suitable way across one or more data repositories for storage (which can be hosted externally). For example, the data store 1246 can include at least some of the one or more computer-readable media of the server device 1236 and / or at least some of the one or more computer-readable media of the client device 1232.

[0202] The sensors 1238 include at least one sensor capable of generating sensor data representative of at least some aspect of an environment. For example, the sensors 1238 can generate Figure 1 ​The sensor data 102 can be generated by, for example and without limitation, a global navigation satellite system (GNSS) sensor 1358 (e.g., a global positioning system sensor), a RADAR sensor 1360, an ultrasonic sensor 1362, a LIDAR sensor 1364, an inertial measurement unit (IMU) sensor 1366 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), a microphone 1396, a stereo camera 1368, a wide-angle camera 1370 (e.g., a fisheye camera), an infrared camera 1372, a surround camera 1374 (e.g., a 360-degree camera), a long and / or medium range camera 1398, a speed sensor 1344 (e.g., to measure the speed of the vehicle 140), a vibration sensor 1342, a steering sensor 1340, a brake sensor (e.g., as part of a brake sensor system 1346), and / or other sensor types.

[0203] Referring to Figures 13A-13C The sensor data 102 can be generated by, for example and without limitation, a global navigation satellite system (GNSS) sensor 1358 (e.g., a global positioning system sensor), a RADAR sensor 1360, an ultrasonic sensor 1362, a LIDAR sensor 1364, an inertial measurement unit (IMU) sensor 1366 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), a microphone 1396, a stereo camera 1368, a wide-angle camera 1370 (e.g., a fisheye camera), an infrared camera 1372, a surround camera 1374 (e.g., a 360-degree camera), a long and / or medium range camera 1398, a speed sensor 1344 (e.g., to measure the speed of the vehicle 140), a vibration sensor 1342, a steering sensor 1340, a brake sensor (e.g., as part of a brake sensor system 1346), and / or other sensor types.

[0204] In some examples, the sensor data 102 can be generated by forward and / or side-facing cameras, such as the wide-angle camera 1370, the surround camera 1374, the stereo camera 1368, and / or the long or medium range camera 1398. In some examples, more than one camera or other sensor can be used to merge multiple fields of view (e.g., the fields of view of the long range camera 1398, the forward stereo camera 1368, and / or the forward wide-angle camera 1370). Figure 13B In some examples, the sensor data 102 can be generated by forward and / or side-facing cameras, such as the wide-angle camera 1370, the surround camera 1374, the stereo camera 1368, and / or the long or medium range camera 1398. In some examples, more than one camera or other sensor can be used to merge multiple fields of view (e.g., the fields of view of the long range camera 1398, the forward stereo camera 1368, and / or the forward wide-angle camera 1370).

[0205] Figure 1Components of the trajectory assessment system 100 generally can be implemented using any combination of client device 1232 and server device 1236. As such, it should be appreciated that the trajectory assessment system 100 can be provided by multiple devices in a distributed environment collectively providing the functionality described herein, or can be embodied on a single device (e.g., the vehicle 140). Thus, while some examples for describing the trajectory assessment system 100 can refer to particular devices and / or configurations, it is contemplated that these examples can be more generally applicable to potential combinations of the devices and configurations described above. For example, in some embodiments, at least some of the sensors 1238 used to generate one or more portions of the sensor data 102 can be distributed among multiple vehicles and / or objects in the environment, and / or at least one of the sensors 1238 can be included in the vehicle 140.

[0206] Example autonomous vehicle

[0207] Figure 13A A diagram of an example autonomous vehicle 1300 in accordance with some embodiments of the present disclosure. The autonomous vehicle 1300 (alternatively referred to herein as “vehicle 1300”) can include, but is not limited to, a passenger vehicle such as a car, truck, bus, ambulance, shuttle, electric or motorized bicycle, motorcycle, fire truck, police car, ambulance, boat, engineering vehicle, underwater vessel, drone, vehicle coupled to a trailer, and / or other type of vehicle (e.g., unmanned and / or capable of accommodating one or more passengers). Autonomous vehicles are often described in terms of levels of automation as defined by a division of the United States Department of Transportation, the National Highway Traffic Safety Administration (NHTSA), and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806 published June 15, 2018, Standard No. J3016-201609 published September 30, 2016, and previous and future versions of this standard). The vehicle 1300 can be capable of implementing functionality consistent with one or more of Levels 3-5 of autonomous driving. For example, depending on the embodiment, the vehicle 1300 can be capable of implementing conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0208] Vehicle 1300 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 1300 may include a propulsion system 1350, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 1350 may be connected to the drivetrain of vehicle 1300, which may include a transmission, to enable propulsion of vehicle 1300. Propulsion system 1350 may be controlled in response to receiving a signal from throttle / accelerator 1352.

[0209] A steering system 1354, which may include a steering wheel, can be used to steer the vehicle 1300 (e.g., along a desired path or route) when the propulsion system 1350 is operating (e.g., when the vehicle is in motion). The steering system 1354 may receive signals from the steering actuator 1356. For fully automatic (level 5) functionality, the steering wheel may be optional.

[0210] The brake sensor system 1346 can be used to operate the vehicle brakes in response to receiving signals from the brake actuator 1348 and / or the brake sensor.

[0211] It can include one or more System-on-Chip (SoC) 1304 ( Figure 13C One or more controllers 1336, including and / or one or more GPUs, may provide signals (e.g., signals representing commands) to one or more components and / or systems of vehicle 1300. For example, one or more controllers may send signals to operate vehicle brakes via one or more brake actuators 1348, to operate steering system 1354 via one or more steering actuators 1356, and to operate propulsion system 1350 via one or more throttles / accelerators 1352. One or more controllers 1336 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 1300. One or more controllers 1336 may include a first controller 1336 for autonomous driving functions, a second controller 1336 for functional safety functions, a third controller 1336 for artificial intelligence functions (e.g., computer vision), a fourth controller 1336 for infotainment functions, a fifth controller 1336 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1336 can handle two or more of the functions described above, and two or more controllers 1336 can handle a single function, and / or any combination thereof.

[0212] One or more controllers 1336 can provide signals for controlling one or more components and / or systems of the vehicle 1300 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data can be received from, for example and without limitation, global navigation satellite system sensors 1358 (e.g., global positioning system sensors), RADAR sensors 1360, ultrasonic sensors 1362, LIDAR sensors 1364, inertial measurement unit (IMU) sensors 1366 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1396, stereo cameras 1368, wide-angle cameras 1370 (e.g., fisheye cameras), infrared cameras 1372, surround cameras 1374 (e.g., 360 degree cameras), long and / or medium range cameras 1398, speed sensors 1344 (e.g., to measure the speed of the vehicle 1300), vibration sensors 1342, steering sensors 1340, brake sensors (e.g., as part of a brake sensor system 1346), and / or other sensor types.

[0213] One or more of the controllers 1336 can receive inputs (e.g., represented by input data) from the instrument cluster 1332 of the vehicle 1300 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1334, audible annunciators, speakers, and / or via other components of the vehicle 1300. These outputs can include information such as vehicle speed, velocity, time, map data (e.g., the HD map 1322 of the controller 1336), location data (e.g., the location of the vehicle 1300, e.g., on a map), direction, locations of other vehicles (e.g., an occupancy grid), information about objects and object states as perceived by the controller 1336, and so on. For example, the HMI display 1334 can display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., change lanes now, exit 34B in two miles, etc.). Figure 13C

[0214] The vehicle 1300 further includes a network interface 1324 that can communicate over one or more networks using one or more wireless antennas 1326 and / or modems. For example, the network interface 1324 can be capable of communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The one or more wireless antennas 1326 can also enable communication between objects (e.g., vehicles, mobile devices, etc.) in an implementation environment such as one or more local area networks (e.g., Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc.) and / or one or more low power wide area networks (LPWANs) (e.g., LoRaWAN, SigFox, etc.). ​

[0215] Figure 13B FIG. 1 illustrates an example autonomous vehicle 1300 according to some embodiments of the present disclosure. Figure 13A FIG. 2 illustrates example camera positions and fields of view of the example autonomous vehicle 1300 of FIG. 1. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras can be included, and / or the cameras can be located at different positions on the vehicle 1300.

[0216] Camera types for the cameras can include, but are not limited to, digital cameras that can be suitable for use with components and / or systems of the vehicle 1300. The cameras can operate at Automotive Safety Integrity Level (ASIL) B and / or at another ASIL. The camera types can have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The cameras can be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, a color filter array can include a red- white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras with a

[0217] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-functional mono camera can be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more (e.g., all) of the cameras can simultaneously record and provide image data (e.g., video).

[0218] One or more of the cameras can be installed in mounting assemblies, such as custom designed (3-D printed) assemblies, in order to cut off stray light and reflections from within the car (e.g., reflections from the dashboard reflected in the windshield mirror) that can interfere with the image data capture capabilities of the cameras. With regard to wing mirror mounting assemblies, the wing mirror assemblies can be custom 3-D printed such that the camera mounting plates match the shape of the wing mirrors. In some examples, one or more cameras can be integrated into the wing mirrors. For side view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0219] Cameras with fields of view that include portions of the environment in front of the vehicle 1300 (e.g., front-facing cameras) can be used for surround view to help identify the forward path and obstacles, and to assist in providing information critical to generating an occupancy grid and / or determining a preferred vehicle path with the aid of one or more controllers 1336 and / or control SoCs. Front-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Front-facing cameras can also be used for ADAS functions and systems, including lane departure warning ("LDW"), adaptive cruise control ("ACC"), and / or other functions such as traffic sign recognition.

[0220] A wide variety of cameras can be used in a front-facing configuration, including, for example, monocular camera platforms including CMO S (complementary metal-oxide semiconductor) color imagers. Another example can be a wide-angle camera 1370, which can be used to perceive objects (e.g., pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although Figure 13B Although only one wide-angle camera is illustrated in FIG. 13, any number of wide-angle cameras 1370 can be present on the vehicle 1300. In addition, long-range cameras 1398 (e.g., pairs of long-view stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not been trained. Long-range cameras 1398 can also be used for object detection and classification and basic object tracking.

[0221] One or more stereo cameras 1368 can also be included in a front-facing configuration. Stereo cameras 1368 can include an integrated control unit that includes a scalable processing unit that can provide a multi-core microprocessor with integrated CAN or Ethernet interfaces and programmable logic (FPGA) on a single chip. Such a unit can be used to generate a 3-D map of the vehicle's environment, including distance estimates for all points in the image. Alternative stereo cameras 1368 can include compact stereo vision sensors that can include two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1368 can be used in addition to or instead of those described herein.

[0222] Cameras with fields of view that include portions of the environment on the sides of the vehicle 1300 (e.g., side-view cameras) can be used for surround view to provide information used to create and update an occupancy grid and to generate side impact collision warnings. For example, surround cameras 1374 (e.g., as illustrated in FIG. 13) can be used to provide a 360-degree view of the vehicle's environment. The surround cameras 1374 can be used to provide a 360-degree view of the vehicle's environment, which can be used to create and update an occupancy grid and to generate side impact collision warnings. Figure 13BFour surround cameras 1374) can be placed on the vehicle 1300. The surround cameras 1374 can include wide-view cameras 1370, fisheye cameras, 360-degree cameras, and / or the like. In one example, four fisheye cameras can be placed on the front, back, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1374 (e.g., left, right, and back) and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround view camera.

[0223] Cameras with a field of view that includes an environmental portion behind the vehicle 1300 (e.g., rearview cameras) can be used to assist with parking, surround view, rear collision warnings, and creating and updating an occupancy grid. A wide variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras (e.g., long and / or mid-range cameras 1398, stereo cameras 1368, infrared cameras 1372, etc.) as described herein.

[0224] Figure 13C FIG. 1 illustrates an example autonomous vehicle 1300 for use in accordance with some embodiments of the present disclosure. Figure 13A FIG. 1 illustrates an example autonomous vehicle 1300 for use in accordance with some embodiments of the present disclosure.

[0225] Figure 13C Each of the components, features, and systems of the vehicle 1300 in FIG. 1 are illustrated as being connected via a bus 1302. The bus 1302 can include a controller area network (CAN) data interface (alternatively referred to herein as a "CAN bus"). The CAN can be a network within the vehicle 1300 that is used to assist in controlling various features and functions of the vehicle 1300, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, and the like. The CAN bus can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus can be ASIL B compliant.

[0226] Although bus 1302 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet can be used in addition to or instead of a CAN bus. Further, although bus 1302 is represented with a single line, this is not intended to be limiting. For example, there can be any number of buses 1302, which can include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses that use different protocols. In some examples, two or more buses 1302 can be used to perform different functions, and / or can be used for redundancy. For example, a first bus 1302 can be used for collision avoidance functions, and a second bus 1302 can be used for drive control. In any example, each bus 1302 can communicate with any component of vehicle 1300, and two or more buses 1302 can communicate with the same components. In some examples, each SoC 1304, each controller 1336, and / or each computer within the vehicle can have access to the same input data (e.g., input from sensors of vehicle 1300), and can be connected to a common bus, such as a CAN bus.

[0227] Vehicle 1300 can include one or more controllers 1336, such as those described herein with respect to Figure 13A controllers. Controllers 1336 can be used for a wide variety of functions. Controllers 1336 can be coupled to any other different components and systems of vehicle 1300, and can be used for control of vehicle 1300, artificial intelligence of vehicle 1300, infotainment for vehicle 1300, and / or the like.

[0228] Vehicle 1300 can include one or more system on chips (SoCs) 1304. SoCs 1304 can include CPUs 1306, GPUs 1308, processors 1310, caches 1312, accelerators 1314, data stores 1316, and / or other components and features not illustrated. In a wide variety of platforms and systems, SoCs 1304 can be used to control vehicle 1300. For example, one or more SoCs 1304 can be used in a system (e.g., a system of vehicle 1300) in conjunction with HD map 1322, which can obtain map refreshes and / or updates from one or more servers (e.g., one or more servers 1378) via network interface 1324. Figure 13D

[0229] ​CPU 1306 can include a CPU cluster or CPU complex (alternatively referred to herein as a“CCPLEX”). CPU 1306 can include multiple cores and / or L2 caches. For example, in some embodiments, CPU 1306 can include eight cores in a coherent multi-processor configuration. In some embodiments, CPU 1306 can include four dual-core clusters with each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). CPU 1306 (e.g., the CCPLEX) can be configured to support simultaneous cluster operation such that any combination of clusters of CPU 1306 can be active at any given time.

[0230] CPU 1306 can implement power management capabilities including one or more of the following features: individual hardware blocks can be automatically clock-gated when idle to save dynamic power; each core clock can be gated when the core is not actively executing instructions due to execution of WFI / WFE instructions; each core can be independently power-gated; each core cluster can be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster can be independently power-gated when all cores are power-gated. CPU 1306 can further implement enhanced algorithms for managing power states in which the allowed power states and desired wake-up times are specified and the hardware / microcode determines the optimal power state for the cores, clusters, and CCPLEX to enter. The processing cores can support a simplified power state entry sequence in software, with the work being offloaded to microcode.

[0231] GPU 1308 can include an integrated GPU (alternatively referred to herein as an“iGPU”). GPU 1308 can be programmable and efficient for parallel workloads. In some examples, GPU 1308 can use an enhanced tensor instruction set. GPU 1308 can include one or more streaming microprocessors, where each streaming microprocessor can include an Ll cache (e.g., an Ll cache having at least 96 KB of storage) and two or more of the streaming microprocessors can share an L2 cache (e.g., an L2 cache having 512 KB of storage). In some embodiments, GPU 1308 can include at least eight streaming microprocessors. GPU 1308 can use a compute application programming interface (API). In addition, GPU 1308 can use one or more parallel computing platforms and / or programming models (e.g., NVIDIA’s CUDA).

[0232] In the case of automotive and embedded uses, the GPU 1308 can be power-optimized for best performance. For example, the GPU 1308 can be fabricated on a fin field-effect transistor (FinFET) for lower power consumption. However, this is not intended to be limiting, and the GPU 1308 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor Cores for deep learning matrix arithmetic, an L0 instruction cache, a thread warp scheduler, a dispatch unit, and / or a 64 KB register file. Further, the streaming microprocessor can include independent parallel integer and floating point data paths to exploit the mix of computation and address computation for efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for finer-grain synchronization and cooperation between parallel threads. The streaming microprocessor can include a combined LI data cache and shared memory unit to improve performance while simplifying programming.

[0233] The GPU 1308 can include a high bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem that provides approximately 900 GB / s of peak memory bandwidth in some examples. In some examples, in addition to or alternatively from HBM memory, synchronous graphics random access memory (SGRAM) can be used, such as fifth generation graphics double data rate synchronous random access memory (GDDR5).

[0234] The GPU 1308 can include a unified memory technology that includes an access counter to allow memory pages to be migrated more precisely to the processors that access them most frequently, improving efficiency of memory ranges shared between processors. In some examples, address translation services (ATS) support can be used to allow the GPU 1308 to access CPU 1306 page tables directly. In such examples, when the GPU 1308 memory management unit (MMU) experiences a miss, an address translation request can be transmitted to the CPU 1306. In response, the CPU 1306 can look up the virtual-to-physical mapping for the address in its page tables and transmit the translation back to the GPU 1308. In this way, the unified memory technology can allow a single unified virtual address space for the memory of both the CPU 1306 and the GPU 1308, simplifying GPU 1308 programming and porting applications to the GPU 1308.

[0235] In addition, GPU 1308 can include an access counter that can track how often GPU 1308 accesses memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that most frequently accesses those pages.

[0236] SoC 1304 can include any number of caches 1312, including those described herein. For example, caches 1312 can include an L3 cache that is available to both CPU 1306 and GPU 1308 (e.g., connected to both CPU 1306 and GPU 1308). Caches 1312 can include a write-back cache that can track the state of a line, for example, by using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache can include 4MB or more, although smaller cache sizes can also be used.

[0237] SoC 1304 can include one or more arithmetic logic units (ALUs) that can be used to perform processing with respect to any of a variety of tasks or operations of vehicle 1300, such as processing a DNN. In addition, SoC 1304 can include a floating point unit (FPU) or other mathematical coprocessor or digital coprocessor type for performing mathematical operations within the system. For example, SoC 104 can include one or more FPUs integrated as execution units within CPU 1306 and / or GPU 1308.

[0238] SoC 1304 can include one or more accelerators 1314 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 1304 can include a hardware acceleration cluster that can include optimized hardware accelerators and / or a large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used to supplement GPU 1308 and offload some of the tasks of GPU 1308 (e.g., freeing up more cycles of GPU 1308 for performing other tasks). As one example, accelerators 1314 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to accelerate easily. As used herein, the term “CNN” can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0239] The accelerators 1314 (e.g., hardware acceleration cluster) can include a deep learning accelerator (DLA). The DLA can include one or more tensor processing units (TPUs) that can be configured to provide an additional 100 billion operations per second for deep learning applications and inferencing. The TPU can be an accelerator that is configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and is optimized for performing image processing functions. The DLA can be further optimized for a specific set of neural network types and floating point operations and inferencing. The design of the DLA can provide higher performance per mm than general purpose GPUs and far exceeds the performance of CPUs. The TPU can perform several functions including single instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, for example, and post-processor functions.

[0240] The DLA can perform neural networks, especially CNNs, on processed or unprocessed data for any of a wide variety of functions, such as and not by way of limitation: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and identification and detection using data from microphones; CNNs for facial recognition and vehicle owner identification using data from camera sensors; and / or CNNs for safety and / or safety related events.

[0241] The DLA can perform any of the functions of the GPU 1308, and by using an inferencing accelerator, the designer can target the DLA or the GPU 1308 for any function, for example. For example, the designer can focus the processing and floating point operations of the CNNs on the DLA and leave other functions to the GPU 1308 and / or other accelerators 1314.

[0242] The accelerators 1314 (e.g., hardware acceleration cluster) can include a programmable vision accelerator (PVA), which can be alternatively referred to herein as a computer vision accelerator. The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA can include any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors, for example, and not by way of limitation.

[0243] The RISC cores can interact with image sensors (e.g., image sensors of any of the cameras described herein), image signal processors, and / or the like. Each of these RISC cores can include any number of memories. Depending on the embodiment, the RISC cores can use any of several protocols. In some examples, the RISC cores can execute a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include instruction caches and / or tightly coupled RAM.

[0244] The DMA can enable components of the PVA to access system memory independently of the CPU 1306. The DMA can support any number of features to provide optimization to the PVA, including but not limited to supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0245] The vector processor can be a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystems can operate as the main processing engines of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core can include a digital signal processor such as, for example, a single instruction multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0246] Each of the vector processors can include an instruction cache and can be coupled to a dedicated memory. As a result, in some examples, each of the vector processors can be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms on the same image simultaneously, or even different algorithms on a sequence of images or portions of an image. Any number of PVAs can be included in the hardware acceleration cluster, and any number of vector processors can be included in each of the PVAs, among other things. Furthermore, the PVAs can include additional error-correcting code (ECC) memory to enhance overall system security.

[0247] The accelerator 1314 (e.g., hardware acceleration cluster) can include an on-chip computer vision network and SRAM to provide high bandwidth, low latency SRAM for the accelerator 1314. In some examples, the on-chip memory can include at least 4 MB of SRAM composed of, for example and without limitation, eight field-programmable memory blocks, which can be accessed by both the PVA and the DLA. Each pair of memory blocks can include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone can include an on-chip computer vision network that interconnects the PVA and the DLA to the memory, for example using an APB.

[0248] The on-chip computer vision network can include an interface that determines that both the PVA and the DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst communications for continuous data transmission. This type of interface can comply with ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0249] In some examples, the SoC 1304 can include a real-time ray tracing hardware accelerator, such as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine locations and extents of objects (e.g., within a world model) in order to generate real-time visualizations simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison to LIDAR data for purposes of localization and / or other functionality, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) can be used to perform one or more ray tracing related operations.

[0250] The accelerator 1314 (e.g., a hardware accelerator cluster) has a wide range of uses for autonomous driving. The PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-dense or dense regular computations, and even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms because they are effective at object detection and integer math operations.

[0251] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, a semi-global matching based algorithm can be used, although this is not intended to be limiting. Many applications for level 3-5 autonomous driving require instant motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on input from two monocular cameras.

[0252] In some examples, the PVA can be used to perform dense optical flow. Raw RADAR data is processed according to a process (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, such as by processing raw time-of-flight data to provide processed time-of-flight data.

[0253] The DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence metric for each object detection. Such a confidence value can be interpreted as a probability, or as providing a relative "weight" for each detection compared to other detections. The confidence value enables the system to make further decisions about which detections should be considered true positive detections rather than false positive detections. For example, the system can set a threshold for confidence, and only consider detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform an emergency brake, which is obviously undesirable. Thus, only the most confident detections should be considered a trigger for AEB. The DLA can run a neural network for regression of a confidence value. The neural network can take as its input at least some subset of parameters, such as a bounding box dimension, a ground plane estimate obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 1366 outputs related to vehicle 1300 orientation, distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 1364 or RADAR sensor 1360), etc.

[0254] SoC 1304 can include one or more data stores 1316 (e.g., memory). Data stores 1316 can be on-chip memory of SoC 1304, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, data stores 1316 can be large enough in capacity to store multiple instances of a neural network. Data stores 1312 can include L2 or L3 cache 1312. References to data stores 1316 can include references to memory associated with PVA, DLA, and / or other accelerators 1314 as described herein.

[0255] SoC 1304 can include one or more processors 1310 (e.g., embedded processors). The processors 1310 can include a boot and power management processor, which can be a specialized processor and subsystem for handling boot power and management functions, as well as security implementation. The boot and power management processor can be part of the SoC 1304 boot sequence and can provide runtime power management services. The boot power and management processor can provide clock and voltage programming, auxiliary system low power state transitions, SoC 1304 thermal and temperature sensor management, and / or SoC 1304 power state management. Each temperature sensor can be implemented as a ring oscillator, whose output frequency is proportional to temperature, and the SoC 1304 can use the ring oscillator to detect the temperature of the CPU 1306, GPU 1308, and / or accelerator 1314. If it is determined that the temperature exceeds a threshold, the boot and power management processor can enter a temperature fault routine and place the SoC 1304 in a lower power state and / or place the vehicle 1300 in a driver safe park mode (e.g., safely park the vehicle 1300).

[0256] The processors 1310 can further include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces, as well as a range of extensive and flexible audio I / O interfaces. In some examples, the audio processing engine is a specialized processor core with a digital signal processor with dedicated RAM.

[0257] The processors 1310 can further include an always-on processor engine, which can provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine can include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0258] The processors 1310 can further include a security cluster engine, which includes a specialized processor subsystem that handles security management for automotive applications. The security cluster engine can include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In a secure mode, the two or more cores can operate in a lockstep mode and act as a single core with comparison logic that detects any differences between their operations.

[0259] The processors 1310 can further include a real-time camera engine, which can include a specialized processor subsystem for handling real-time camera management.

[0260] The processor 1310 can further include a high dynamic range signal processor, which can include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0261] The processor 1310 can include a video image compositor, which can be a processing block (e.g., implemented on a microprocessor), that implements video post-processing functions needed by the video playback application to produce the final image for the player window. The video image compositor can perform lens distortion correction on the wide-angle camera 1370, the surround camera 1374, and / or on the cab-in monitor camera sensors. The cab-in monitor camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to recognize cab-in events and respond accordingly. The cab-in system can perform lip reading to activate mobile phone services and place a call, dictate an email, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode, and are disabled otherwise.

[0262] The video image compositor can include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, where motion is present in the video, the noise reduction appropriately weights the spatial information, reducing the weight of information provided by neighboring frames. Where the image or portions of the image do not include motion, the temporal noise reduction performed by the video image compositor can use information from previous images to reduce noise in the current image.

[0263] The video image compositor can also be configured to perform stereo correction on input stereo lens frames. The video image compositor can further be used for user interface composition when the operating system desktop is in use and the GPU 1308 does not need to continuously render new surfaces. Even when the GPU 1308 is powered on and active, doing 3D rendering, the video image compositor can be used to offload the GPU 1308 to improve performance and responsiveness.

[0264] The SoC 1304 can further include a Mobile Industry Processor Interface (MIPI) camera serial interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The SoC 1304 can further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a particular role.

[0265] The SoC 1304 can further include a wide range of peripheral device interfaces to enable communication with peripherals, audio codecs, power management, and / or other devices. The SoC 1304 can be used to process data from cameras (connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LiDAR sensor 1364, RADAR sensor 1360, etc. that can be connected over Ethernet), data from the bus 1302 (e.g., speed of the vehicle 1300, steering wheel position, etc.), data from GNSS sensor 1358 (connected over Ethernet or CAN bus). The SoC 1304 can further include dedicated high-performance mass storage controllers, which can include their own DMA engines, and which can be used to free up the CPU 1306 from routine data management tasks.

[0266] The SoC 1304 can be an end-to-end platform with a flexible architecture that spans automation levels 3-5, providing an integrated functional safety architecture for a platform that leverages and efficiently uses computer vision and ADAS technology to achieve diversity and redundancy, along with deep learning tools. The SoC 1304 can be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the accelerator 1314, when combined with the CPU 1306, GPU 1308, and data storage 1316, can provide a fast and efficient platform for level 3-5 autonomous vehicles.

[0267] The technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to, for example, execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for on-board ADAS applications and for practical level 3-5 autonomous vehicles.

[0268] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined together to achieve level 3-5 autonomous driving functionality. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1320) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA can further include a neural network that is able to recognize, interpret, and provide a semantic understanding of the sign, and pass that semantic understanding to a path planning module running on the CPU complex.

[0269] As another example, multiple neural networks can be run simultaneously as required for level 3, 4, or 5 driving. For example, a warning sign consisting of the words "Caution: flashing lights indicate icy conditions" along with electric lights can be interpreted by several neural networks independently or collectively. The sign itself can be recognized by a first deployed neural network (e.g., a trained neural network) as a traffic sign, the text "flashing lights indicate icy conditions" can be interpreted by a second deployed neural network that informs the vehicle's path planning software (preferably executing on the CPU complex) that icy conditions exist when flashing lights are detected. The flashing lights can be recognized by operating a third deployed neural network over multiple frames that informs the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can be run simultaneously, for example, within the DLA and / or on the GPU 1308.

[0270] In some examples, a CNN for face recognition and owner recognition can use data from the camera sensors to recognize the presence of an authorized driver and / or owner of the vehicle 1300. A processing engine always on the sensor can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a safe mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 1304 provides security against theft and / or carjacking.

[0271] In another example, a CNN for emergency vehicle detection and recognition can use data from the microphones 1396 to detect and recognize emergency vehicle sirens. In contrast to conventional systems that detect sirens using a general classifier and manually extract features, the SoC 1304 uses a CNN to classify ambient and urban sounds as well as to classify visual data. In a preferred embodiment, a CNN running on the DLA is trained to recognize the relative closing speed of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle is operating as recognized by the GNSS sensor 1358. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to recognize sirens that are only North American. Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine, slow the vehicle down, pull over to the side of the road, stop the vehicle, and / or idle the vehicle until the emergency vehicle passes, with the assistance of the ultrasonic sensors 1362.

[0272] The vehicle can include a CPU 1318 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 1304 via a high-speed interconnect (e.g., PCIe). The CPU 1318 can include, for example, an X86 processor. The CPU 1318 can be used to perform any of a wide variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 1304, and / or monitoring the status and health of the controller 1336 and / or infotainment SoC 1330.

[0273] The vehicle 1300 can include a GPU 1320 (e.g., a discrete GPU or dGPU) that can be coupled to the SoC 1304 via a high-speed interconnect (e.g., NVIDIA’s NVLINK). The GPU 1320 can provide additional artificial intelligence functionality, for example, by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based on input (e.g., sensor data) from sensors of the vehicle 1300.

[0274] The vehicle 1300 can further include a network interface 1324 that can include one or more wireless antennas 1326 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 1324 can be used to enable wireless connections over the Internet with a cloud (e.g., with a server 1378 and / or other network devices), with other vehicles, and / or with computing devices (e.g., a client device of a passenger). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and through the Internet). The direct link can be provided using a car-to-car communication link. The car-to-car communication link can provide the vehicle 1300 with information about vehicles that are approaching the vehicle 1300 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 1300). This functionality can be part of a cooperative adaptive cruise control functionality of the vehicle 1300.

[0275] The network interface 1324 can include a SoC that provides modulation and demodulation functionality and enables the controller 1336 to communicate over a wireless network. The network interface 1324 can include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. The frequency conversion can be performed through well-known processes and / or can be performed using a super-heterodyne process. In some examples, the radio frequency front end functionality can be provided by a separate chip. The network interface can include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0276] The vehicle 1300 can further include a data store 1328, which can include off-chip (e.g., off-SoC 1304) storage. The data store 1328 can include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disks, and / or other components and / or devices that can store data for at least one bit.

[0277] The vehicle 1300 can further include a GNSS sensor 1358. The GNSS sensor 1358 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist in mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1358 can be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0278] The vehicle 1300 can further include a RADAR sensor 1360. The RADAR sensor 1360 can be used by the vehicle 1300 for long-range vehicle detection, even in darkness and / or adverse weather conditions. The RADAR functional safety level can be ASIL B. The RADAR sensor 1360 can use the CAN and / or the bus 1302 (e.g., to transmit data generated by the RADAR sensor 1360) for control as well as access to object tracking data, in some examples, to Ethernet for access to raw data. A wide variety of RADAR sensor types can be used. For example and without limitation, the RADAR sensor 1360 can be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0279] The RADAR sensor 1360 can include different configurations, such as long-range with narrow field of view, short-range with wide field of view, short-range side coverage, and so on. In some examples, long-range RADAR can be used for adaptive cruise control functionality. Long-range RADAR systems can provide a wide field of view (e.g., 250 m range) implemented through two or more independent scans. The RADAR sensor 1360 can help distinguish between static and moving objects, and can be used by the ADAS system for emergency brake assist and forward collision warning. The long-range RADAR sensor can include a single-station multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas, as well as high-speed CAN and FlexRay interfaces. In examples with six antennas, the central four antennas can create focused beam patterns designed to record the surroundings of the vehicle 1300 at higher speed with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 1300.

[0280] As one example, a mid-range RADAR system can include a range of up to 1360 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1350 degrees (rear). A short-range RADAR system can include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the rear and the blind spot next to the vehicle.

[0281] A short-range RADAR system can be used in an ADAS system for blind spot detection and / or lane change assist.

[0282] The vehicle 1300 can further include ultrasonic sensors 1362. The ultrasonic sensors 1362, which can be placed on the front, rear, and / or sides of the vehicle 1300, can be used for parking assist and / or to create and update an occupancy grid. A wide variety of ultrasonic sensors 1362 can be used, and different ultrasonic sensors 1362 can be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensors 1362 can operate at an ASIL B functional safety level.

[0283] The vehicle 1300 can include LIDAR sensors 1364. The LIDAR sensors 1364 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensors 1364 can be at an ASIL B functional safety level. In some examples, the vehicle 1300 can include multiple LIDAR sensors 1364 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0284] In some examples, the LIDAR sensors 1364 can be capable of providing a list of objects and their distances for a 360-degree field of view. A commercially available LIDAR sensor 1364 can have, for example, an advertised range of approximately 500 m, a precision of 2 cm - 3 cm, and support for a 500 Mbps Ethernet connection. In some examples, one or more flush-mounted LIDAR sensors 1364 can be used. In such examples, the LIDAR sensors 1364 can be implemented as small devices that can be embedded into the front, rear, sides, and / or corners of the vehicle 1300. In such examples, the LIDAR sensors 1364 can provide a field of view of up to 120 degrees horizontal and 35 degrees vertical, with a range of 200 m, even for low reflectivity objects. Front-mounted LIDAR sensors 1364 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0285] In some examples, LIDAR technology such as 3D Flash LIDAR can also be used. 3D Flash LIDAR uses a flash of laser light as a source of emission to illuminate the vehicle’s surroundings up to about 200 m. The flash LIDAR unit includes a receptor that records the laser pulse transmission time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR can allow for the generation of highly accurate and distortion-free images of the surroundings with each laser flash. In some examples, four flash LIDAR sensors can be deployed, one on each side of the vehicle 1300. Available 3D flash LIDAR systems include solid-state 3D staring array LIDAR cameras (e.g., non-scanning LIDAR devices) that have no moving parts other than a fan. The flash LIDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture the reflected laser light in the form of 3D range point clouds and co-registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 1364 can be less susceptible to motion blur, vibration, and / or jostling.

[0286] The vehicle can further include an IMU sensor 1366. In some examples, the IMU sensor 1366 can be located at the center of the rear axle of the vehicle 1300. The IMU sensor 1366 can include, for example and without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, the IMU sensor 1366 can include an accelerometer and a gyroscope, for example in a six-axis application, while in a nine-axis application the IMU sensor 1366 can include an accelerometer, a gyroscope, and a magnetometer.

[0287] In some embodiments, the IMU sensor 1366 can be implemented as a micro-electro-mechanical systems (MEMS) based high-performance GPS-aided inertial navigation system (GPS / INS) that combines MEMS inertial sensors, high-sensitivity GPS receivers, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 1366 can enable the vehicle 1300 to estimate heading without input from a magnetic sensor by directly observing the change in velocity from GPS to the IMU sensor 1366 and correlating it. In some examples, the IMU sensor 1366 and the GNSS sensor 1358 can be combined into a single integrated unit.

[0288] The vehicle can include a microphone 1396 placed in and / or around the vehicle 1300. The microphone 1396 can be used for emergency vehicle detection and identification, among other things.

[0289] The vehicle can further include any number of camera types, including stereo cameras 1368, wide-view cameras 1370, infrared cameras 1372, surround-view cameras 1374, long and / or mid-range cameras 1398, and / or other camera types. These cameras can be used to capture image data around the entire periphery of the vehicle 1300. The types of cameras used depend on the embodiment and requirements of the vehicle 1300, and any combination of camera types can be used to provide the necessary coverage around the vehicle 1300. Further, the number of cameras can vary depending on the embodiment. For example, the vehicle can include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As one example and not by way of limitation, the cameras can support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with respect to Figure 13A and Figure 13B are described in more detail.

[0290] The vehicle 1300 can further include vibration sensors 1342. The vibration sensors 1342 can measure vibrations of components of the vehicle, such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 1342 are used, differences between the vibrations can be used to determine the friction or slip of the road surface (e.g., when there is a difference in vibration between a power driven axle and a free spinning axle).

[0291] The vehicle 1300 can include an ADAS system 1338. In some examples, the ADAS system 1338 can include a SoC. The ADAS system 1338 can include adaptive / automatic / autonomous cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functionality.

[0292] The ACC system can use RADAR sensors 1360, LIDAR sensors 1364, and / or cameras. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of the vehicle 1300 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance keeping and, if necessary, suggests a lane change for the vehicle 1300. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0293] CACC uses information from other vehicles, which can be received from other vehicles indirectly via a wireless link or through a network connection (e.g., through the Internet) via the network interface 1324 and / or the wireless antenna 1326. Direct links can be provided by a vehicle-to-vehicle (V2V) communication link, while indirect links can be an infrastructure-to-vehicle (I2V) communication link. In general, the V2V communication concept provides information about the immediately preceding vehicles (e.g., vehicles immediately ahead of and in the same lane as the vehicle 1300), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of I2V and V2V information sources. Given information about vehicles ahead of the vehicle 1300, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0294] FCW systems are designed to alert the driver to a hazard so that the driver can take corrective action. FCW systems use a front-facing camera and / or RADAR sensor 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components. FCW systems can provide warnings in the form of, for example, sound, visual warnings, vibrations, and / or quick brake pulses.

[0295] AEB systems detect an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. AEB systems can use a front-facing camera and / or RADAR sensor 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When an AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the effects of a predicted collision. AEB systems can include technologies such as dynamic brake support and / or crash imminent braking.

[0296] LDW systems provide visual, audible, and / or tactile warnings such as steering wheel or seat vibrations to alert the driver when the vehicle 1300 is crossing lane markers. The LDW system is not activated when the driver indicates an intentional lane departure by activating a turn signal. LDW systems can use a front-side facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components.

[0297] An LKA system is a variation of the LDW system. If the vehicle 1300 begins to leave the lane, the LKA system provides a steering input or brake to correct the vehicle 1300.

[0298] A BSW system detects and warns the driver of vehicles in the car's blind spot. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use rear side-facing cameras and / or RADAR sensors 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0299] A RCTW system can provide visual, audible, and / or tactile notifications when objects are detected outside the range of the rear-facing camera while the vehicle 1300 is backing up. Some RCTW systems include AEB to ensure that vehicle brakes are applied to avoid a collision. The RCTW system can use one or more rear-facing RADAR sensors 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0300] Conventional ADAS systems can be prone to false positive results, which can annoy and distract the driver, but typically are not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether the safety condition is truly present and act accordingly. However, in an autonomous vehicle 1300, in the case of conflicting results, the vehicle 1300 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 1336 or the second controller 1336). For example, in some embodiments, the ADAS system 1338 can be a secondary and / or auxiliary computer for providing perception information to a backup computer plausibility module. The backup computer plausibility monitor can run redundant diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 1338 can be provided to a supervisory MCU. If the outputs from the primary and secondary computers conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0301] In some examples, the host computer can be configured to provide a confidence score to the supervisory MCU indicating the host computer's confidence in the selected result. If the confidence score exceeds a threshold, then the supervisory MCU can follow the host computer's direction, regardless of whether the secondary computer provides conflicting or inconsistent results. In the event that the confidence score does not satisfy the threshold and in the event that the host computer and the secondary computer indicate different results (e.g., a conflict), the supervisory MCU can arbitrate between the computers to determine the appropriate result.

[0302] The supervisory MCU can be configured to run a neural network that is trained and configured to determine conditions under which the secondary computer provides false alarms based on the output from the host computer and the secondary computer. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying metal objects that are not in fact dangerous, such as drain grates or manhole covers that trigger false alarms. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is in fact the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU can include at least one of a DLA or a GPU suitable for running a neural network with associated memory. In preferred embodiments, the supervisory MCU can include and / or be included as a component of the SoC 1304.

[0303] In other examples, the ADAS system 1338 can include a secondary computer that performs ADAS functions using traditional computer vision rules. In this way, the secondary computer can use classic computer vision rules (if-then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially with respect to faults caused by software (or software-hardware interface) functions. For example, if there is a software bug or error in the software running on the host computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervisory MCU can be more confident that the overall result is correct and that the bug in the software or hardware on the host computer did not cause a substantial error.

[0304] In some examples, the output of the ADAS system 1338 can be fed to a perception block of the host computer and / or a dynamic driving task block of the host computer. For example, if the ADAS system 1338 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information in identifying the object. In other examples, the secondary computer can have its own neural network that is trained and thus reduces the risk of false positives as described herein.

[0305] The vehicle 1300 can further include an infotainment SoC 1330 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a SoC, the infotainment system can not be a SoC and can include two or more discrete components. The infotainment SoC 1330 can include a combination of hardware and software that can be used to provide audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., a navigation system, a park assist, a radio data system, vehicle-related information such as a fuel level, a total distance covered, a brake fuel level, an oil level, a door open / close, air filter information, etc.) to the vehicle 1300. For example, the infotainment SoC 1330 can include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an in-car computer, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, a heads-up display (HUD), an HMI display 1334, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 1330 can further be used to provide information (e.g., visual and / or audible) to a user of the vehicle, such as information from the ADAS system 1338, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0306] The infotainment SoC 1330 can include GPU functionality. The infotainment SoC 1330 can communicate with other devices, systems, and / or components of the vehicle 1300 over the bus 1302 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1330 can be coupled to a supervisory MCU such that, in the event of a failure of the host controller 1336 (e.g., a primary and / or backup computer of the vehicle 1300), the GPU of the infotainment system can perform some autonomous driving functions. In such examples, the infotainment SoC 1330 can place the vehicle 1300 in a driver safe park mode as described herein.

[0307] The vehicle 1300 can further include an instrument cluster 1332 (e.g., a digital dashboard, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 1332 can include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 1332 can include a set of instruments, such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seat belt warning light, parking brake warning light, engine malfunction light, supplemental restraint system (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information can be displayed and / or shared between the infotainment SoC 1330 and the instrument cluster 1332. In other words, the instrument cluster 1332 can be included as part of the infotainment SoC 1330, or vice versa.

[0308] Figure 13D FIG. 13 illustrates a system diagram of communication between a cloud-based server and an example autonomous vehicle 1300 in accordance with some embodiments of the present disclosure. Figure 13A FIG. 13 illustrates a system diagram of communication between a cloud-based server and an example autonomous vehicle 1300 in accordance with some embodiments of the present disclosure. The system 1376 can include servers 1378, a network 1390, and vehicles including the vehicle 1300. The servers 1378 can include a plurality of GPUs 1384(A)-1384(H) (collectively referred to herein as GPUs 1384), PCIe switches 1382(A)-1382(H) (collectively referred to herein as PCIe switches 1382), and / or CPUs 1380(A)-1380(B) (collectively referred to herein as CPUs 1380). The GPUs 1384, CPUs 1380, and PCIe switches can be interconnected with high-speed interconnects such as, for example and without limitation, NVLink interfaces 1388 developed by NVIDIA and / or PCIe connections 1386. In some examples, the GPUs 1384 are connected via NVLink and / or NVSwitch SoC connections, and the GPUs 1384 and PCIe switches 1382 are connected via PCIe interconnects. Although eight GPUs 1384, two CPUs 1380, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 1378 can include any number of GPUs 1384, CPUs 1380, and / or PCIe switches. For example, each of the servers 1378 can include eight, sixteen, thirty-two, and / or more GPUs 1384.

[0309] The server 1378 can receive image data from vehicles over the network 1390 and representing images showing unexpected or changing road conditions such as a road work that recently started. The server 1378 can transmit neural networks 1392, updated neural networks 1392, and / or map information 1394, including information about traffic and road conditions, to vehicles over the network 1390. Updates to the map information 1394 can include updates to the HD map 1322, e.g., information about construction sites, potholes, curves, floods, or other obstacles. In some examples, the neural networks 1392, updated neural networks 1392, and / or map information 1394 can have been produced from experience representing and / or based on training performed at a data center (e.g., using the server 1378 and / or other servers) from new training and / or from data received by any number of vehicles in the environment.

[0310] The server 1378 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated by vehicles and / or can be generated in simulations (e.g., using game engines). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other pre-processing, while in other examples, the training data is not labeled and / or pre-processed (e.g., in cases where the neural network does not require supervised learning). The training can be performed according to any class or more classes of machine learning techniques, including but not limited to the following classes: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including spare dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning models are trained, the machine learning models can be used by vehicles (e.g., transmitted to vehicles over the network 1390), and / or the machine learning models can be used by the server 1378 to remotely monitor vehicles.

[0311] In some examples, the server 1378 can receive data from vehicles and apply the data to the latest real-time neural networks for real-time intelligent inference. The server 1378 can include deep learning supercomputers and / or specialized AI computers powered by GPUs 1384, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the server 1378 can include deep learning infrastructure of data centers powered by CPUs only.

[0312] The deep learning infrastructure of the server 1378 can be capable of fast real-time inference, and can use this capability to assess and validate the health of the processors, software, and / or associated hardware in the vehicle 1300. For example, the deep learning infrastructure can receive periodic updates from the vehicle 500, such as a sequence of images and / or objects located in the sequence of images that the vehicle 1300 has located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify the objects and compare them to the objects identified by the vehicle 1300, and if the results do not match and the infrastructure concludes that the AI in the vehicle 1300 is malfunctioning, the server 1378 can transmit a signal to the vehicle 1300 instructing the fail-safe computer of the vehicle 1300 to take control, notify the passengers, and complete a safe parking operation.

[0313] For inference, the server 1378 can include GPUs 1384 and one or more programmable inference accelerators (such as NVIDIA’s TensorRT 3). The combination of GPU-powered servers and inference-accelerated servers can make real-time responses possible. In other examples, such as where performance is less important, CPU-, FPGA-, and other processor-powered servers can be used for inference.

[0314] Example Computing Device

[0315] Figure 14 A block diagram of an example computing device 1400 suitable for implementing some embodiments of the present disclosure is shown. The computing device 1400 can include an interconnect system 1402 that directly or indirectly couples the following devices: memory 1404, one or more central processing units (CPUs) 1406, one or more graphics processing units (GPUs) 1408, a communication interface 1410, input / output (I / O) ports 1412, I / O components 1414, a power supply 1416, one or more presentation components 1418 (e.g., a display), and one or more logic units 1420. In at least one embodiment, the computing device 1400 can include one or more virtual machines (VMs), and / or any component thereof can include a virtual component (e.g., a virtual hardware component). For a non-limiting example, the one or more GPUs 1408 can include one or more vGPUs, the one or more CPUs 1406 can include one or more vCPUs, and / or the one or more logic units 1420 can include one or more virtual logic units. Thus, the computing device 1400 can include discrete components (e.g., a full GPU dedicated to the computing device 1400), virtual components (e.g., a portion of a GPU dedicated to the computing device 1400), or a combination thereof.

[0316] WhileFigure 14 The various boxes of the computing device 1400 are shown connected via an interconnection system 1402 with lines, but this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 1418 such as a display device can be considered an I / O component 1414 (e.g., if the display is a touchscreen). As another example, the CPU 1406 and / or GPU 1408 can include memory (e.g., the memory 1404 can represent a storage device in addition to the memory of the GPU 1408, CPU 1406, and / or other components). In other words, Figure 14 The computing device of FIG. 1400 is merely illustrative. No distinction is made between a “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of the computing device of FIG. 1400. Figure 14 The computing device of FIG. 1400 is merely illustrative. No distinction is made between a “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of the computing device of FIG. 1400.

[0317] The interconnection system 1402 can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1402 can include one or more link or bus types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 1406 can be directly connected to the memory 1404. Also, the CPU 1406 can be directly connected to the GPU 1408. Where there are direct or point-to-point connections between components, the interconnection system 1402 can include a PCIe link to perform the connection. In these examples, a PCI bus need not be included in the computing device 1400.

[0318] The memory 1404 can include any of a wide variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 1400. Computer-readable media can include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0319] Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, memory 1404 can store computer readable instructions (e.g., which represent program and / or program elements, such as an operating system). Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computing device 1400. Computer storage media, as used herein, does not include signals per se.

[0320] Computer storage media can include computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

[0321] The CPUs 1406 can be configured to execute at least some of the computer readable instructions in order to control one or more components of the computing device 1400 to perform one or more of the methods and / or processes described herein. Each of the CPUs 1406 can include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling a large number of software threads simultaneously. The CPUs 1406 can include any type of processors and can include different types of processors depending on the type of computing device 1400 being implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1400, the processors can be Advanced RISC Machines (ARM) processors implemented using reduced instruction set computing (RISC) or x86 processors implemented using complex instruction set computing (CISC). The computing device 1400 can include one or more CPUs 1406 in addition to one or more microprocessors or supplemental co-processors such as math co-processors.

[0322] In addition or alternatively to CPU 1406, GPU 1408 can be configured to execute at least some computer-readable instructions to control one or more components of computing device 1400 to perform one or more methods and / or processes described herein. One or more GPUs 1408 can be integrated GPUs (e.g., with one or more CPUs 1406) and / or one or more GPUs 1408 can be discrete GPUs. In embodiments, one or more GPUs 1408 can be co-processors to one or more CPUs 1406. Computing device 1400 can use GPU 1408 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPU 1408 can be used for general-purpose computing on GPUs (GPGPU). GPU 1408 can include hundreds or thousands of cores capable of processing hundreds or thousands of software threads concurrently. GPU 1408 can generate pixel data for an output image in response to rendering commands (e.g., rendering commands from CPU 1406 received via a host interface). GPU 1408 can include graphics memory, such as display memory, for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory can be included as part of memory 1404. GPU 1408 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or through a switch (e.g., using NVSwitch). When combined together, each GPU 1408 can generate different portions of pixel data or GPGPU data for an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory or can share memory with other GPUs.

[0323] In addition or alternatively to CPU 1406 and / or GPU 1408, logic unit 1420 can be configured to execute at least some computer-readable instructions to control one or more components of computing device 1400 to perform one or more methods and / or processes described herein. In embodiments, CPU 1406, GPU 1408, and / or logic unit 1420 can execute any combination of methods, processes, and / or portions thereof, discretely or jointly. One or more logic units 1420 can be part of and / or integrated in one or more CPUs 1406 and / or one or more GPUs 1408, and / or one or more logic units 1420 can be discrete components or otherwise external to CPU 1406 and / or GPU 1408. In embodiments, one or more logic units 1420 can be processors of one or more CPUs 1406 and / or one or more GPUs 1408.

[0324] Examples of logic units 1420 include one or more processing cores and / or components thereof, such as tensor cores (TCs), tensor processing units (TPUs), pixel vision cores (PVCs), vision processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multi-processors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application specific integrated circuits (ASICs), floating point units (FPUs), input / output (I / O) elements, peripheral component interconnects (PCI) or peripheral component interconnect express (PCIe) elements, and the like.

[0325] Communication interface 1410 can include one or more receivers, transmitters, and / or transceivers that enable computing device 1400 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communication. Communication interface 1410 can include components and functionality enabling communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, and the like), wired networks (e.g., communication over Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, and the like), and / or the Internet.

[0326] I / O ports 1412 can enable the computing device 1400 to be logically coupled to other devices including the I / O components 1414, the presentation components 1418, and / or other components, some of which can be built into (e.g., integrated with) the computing device 1400. Illustrative I / O components 1414 include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 1414 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs can be transmitted to an appropriate network element for further processing. A NUI can implement any combination of speech recognition, handwriting recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device 1400. The computing device 1400 can include depth cameras, infrared cameras, RGB cameras, touch screens, and combinations of these, such as a stereoscopic camera system to provide a depth map.

[0327] A power supply 1416 can include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 1416 can supply power to the computing device 1400 to enable the components of the computing device 1400 to operate.

[0328] The presentation components 1418 can include a display (e.g., a monitor, a touch screen, a television, a heads-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation components 1418 can receive data from other components (e.g., the GPU 1408, the CPU 1406, etc.) and output that data (e.g., as a

[0329] Example data center

[0330] Figure 15 An example data center 1500 is shown, which can be used in at least one embodiment of the present disclosure. The data center 1500 can include a data center infrastructure layer 1510, a framework layer 1520, a software layer 1530, and an application layer 1540.

[0331] As Figure 15As shown, the data center infrastructure layer 1510 can include a resource orchestrator 1512, grouped computing resources 1514, and node computing resources (“node C.R.s”) 1516(1)-1516(N), where “N” represents any whole, positive integer. In at least one embodiment, the node C.R.s 1516(1)-1516(N) can include, but are not limited to, any number of central processing units (“CPUs” or “processors”) including accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules, cooling modules, and the like. In some embodiments, one or more of the node C.R.s 1516(1)-1516(N) can correspond to a server having one or more of the above-described computing resources. Moreover, in some embodiments, the node C.R.s 1516(1)-1516(N) can include one or more virtual components, such as vGPUs, vCPUs, and the like, and / or one or more of the node C.R.s 1516(1)-1516(N) can correspond to a virtual machine (VM).

[0332] In at least one embodiment, the grouped computing resources 1514 can include separate groupings of node C.R.s 1516 housed within one or more racks (not shown), or housed within a number of racks (also not shown) within various geographic locations. The separate groupings of node C.R.s 1516 within the grouped computing resources 1514 can include groupings of computing, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s 1516 including CPUs, GPUs, and / or other processors can be grouped within one or more racks to provide computing resources to support one or more workloads. The one or more racks can also include any quantity of power modules, cooling modules, and / or network switches in any combination.

[0333] The resource orchestrator 1522 can configure or otherwise control the one or more node C.R.s 1516(1)-1516(N) and / or the grouped computing resources 1514. In at least one embodiment, the resource orchestrator 1522 can include a software design infrastructure (“SDI”) management entity for the data center 1500. The resource orchestrator can include hardware, software, or some combination thereof.

[0334] In at least one embodiment, as Figure 15As shown, the framework layer 1520 can include a job scheduler 1532, a configuration manager 1534, a resource manager 1536, and a distributed file system 1538. The framework layer 1520 can include a framework that supports the software 1532 of the software layer 1530 and / or one or more applications 1542 of the application layer 1540. The software 1532 or the applications 1542 can include web-based service software or applications, respectively, such as the service software or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1520 can be, but is not limited to, a free and open-source software web application framework, such as Apache Spark TM (hereinafter “Spark”) that can utilize the distributed file system 1538 for large-scale data processing (e.g., “big data”). In at least one embodiment, the job scheduler 1532 can include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 1500. In at least one embodiment, the configuration manager 1534 can be capable of configuring different layers, such as the software layer 1530 and the framework layer 1520 including Spark and the distributed file system 1538 for supporting large-scale data processing. The resource manager 1536 can manage clustered or grouped computing resources mapped to or allocated for supporting the distributed file system 1538 and the job scheduler 1532. In at least one embodiment, the clustered or grouped computing resources can include the grouped computing resources 1514 at the data center infrastructure layer 1510. The resource manager 1536 can coordinate with the resource orchestrator 1512 to manage these mapped or allocated computing resources.

[0335] In at least one embodiment, the software 1532 included in the software layer 1530 can include software used by at least portions of the node C.R.s 1516(1)-1516(N), the grouped computing resources 1514, and / or the distributed file system 1538 of the framework layer 1520. One or more types of software can include, but are not limited to, Internet web page search software, e-mail virus scanning software, database software, and streaming video content software.

[0336] In at least one embodiment, one or more application programs 1542 included in application layer 1540 can include one or more types of application programs used by at least portions of node C.R.s 1516(1)-1516(N), grouped computing resources 1514, and / or distributed file system 1538 of framework layer 1520. One or more types of application programs can include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications including training or inferencing software, machine learning framework software (e.g., PyTorch, TenSorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0337] In at least one embodiment, any of configuration manager 1534, resource manager 1536, and resource orchestrator 1512 can implement any number and type of self-modifying actions based on any number and type of data acquired in any technically feasible fashion. Self-modifying actions can relieve data center operators of data center 1500 from making possibly poor configuration decisions and can avoid underutilized and / or poorly performing portions of a data center.

[0338] Data center 1500 can include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information in accordance with one or more embodiments described herein. For example, a machine learning model can be trained in accordance with a neural network architecture computing weight parameters by using software and computing resources described above with respect to data center 1500. In at least one embodiment, using weight parameters computed through one or more training techniques, a trained machine learning model corresponding to one or more neural networks can be used to infer or predict information using resources described above with respect to data center 1500, such as but not limited to those described herein.

[0339] In at least one embodiment, data center 1500 can use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or inference using resources described above. Moreover, one or more software and / or hardware resources described above can be configured as a service to allow users to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0340] Example network environment

[0341] Network environments suitable for implementing embodiments of the present disclosure can include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) can be implemented on one or more instances of computing devices 1400, which can include similar components, features, and / or functionality of computing devices 1400. Further, where backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices can be included as part of data center 1500, an example of which is described herein with respect to FIG. 2, in more detail. Figure 14 Figure 15

[0342] Components of the network environment can communicate with each other by way of the network, which can be wired, wireless, or both. The network can include multiple networks, or network(s). For example, the network can include one or more wide area networks (WAN)s, one or more local area networks (LAN)s, one or more public networks (e.g., the Internet and / or the public switched telephone network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunication network, components such as base stations, communication towers, or even access points (among other components) can provide wireless connectivity.

[0343] Compatible network environments can include one or more peer-to-peer network environments (in which case servers can not be included in the network environment), as well as one or more client-server network environments (in which case one or more servers can be included in the network environment). In a peer-to-peer network environment, functionality described herein with respect to servers can be implemented on any number of client devices.

[0344] In at least one embodiment, the network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework for supporting one or more applications of a software layer and / or an application layer. The software or applications can include network-based service software or applications, respectively. In embodiments, the one or more client devices can use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (API)s). The framework layer can be, but is not limited to, a type of free and open-source software web application framework, such as HADOOP®, which can use the distributed file system for large-scale data processing (e.g., “big data”). ​​

[0345] The cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these various functions can be distributed across multiple locations from a central or core server (e.g., across one or more data centers in a state, region, country, globally, etc.). The core server can designate at least a portion of the functions to an edge server if the connection to the user (e.g., client device) is relatively close to the edge server. The cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0346] The client device can include at least some components, features, and functionality of the example computing device 1400 described herein with respect to Figure 14 As examples and not by way of limitation, the client device can embody a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a watercraft, an aircraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, an in-vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these described devices, or any other suitable device.

[0347] The present disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal digital assistant or other handheld devices. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, etc. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0348] As used herein, "and / or" shall mean either or both depending on the context. For example, "A and / or B" shall mean A alone, B alone, or both A and B. Further, "at least one of A or B" shall mean A alone, B alone, or both A and B.

[0349] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms "step" and / or "block" might be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

Claims

1. A method for controlling an autonomous machine to avoid potential future collisions, comprising: determining one or more first future points in space-time corresponding to one or more objects in an environment; computing one or more second future points in space-time corresponding to execution of one or more safety procedures by the autonomous machine along a first trajectory; comparing the one or more first future points in space-time and the one or more second future points in space-time; sampling the first trajectory at a particular time step and evaluating collisions at different points throughout the first trajectory, wherein a sample at each sampling point of the first trajectory corresponds to an invoked safety procedure; and determining whether to use the first trajectory in performing one or more operations using the autonomous machine based at least on the comparison and the evaluation, wherein sampling the first trajectory at a particular time step and evaluating collisions at different points throughout the first trajectory comprises: evaluating, for a safety procedure invoked by a sample at a first sampling point of the first trajectory, a plurality of points between a start point and an end point of the safety procedure; evaluating, for a safety procedure invoked by a sample after the first sampling point of the first trajectory, only the end point of the safety procedure; and determining a collision for a sample based on evaluating at least one point for the sample.

2. The method of claim 1, wherein the one or more second future points in space-time are computed based at least on evaluating a time and a location at which the autonomous machine will stop according to execution of the one or more safety procedures.

3. The method of claim 1, wherein the one or more first future points in space-time are determined based at least on evaluating one or more predicted trajectories computed for the one or more objects to one or more locations of the one or more first future points.

4. The method of claim 1, wherein determining the one or more first future points in space-time comprises: evaluating a direct path for an object of the one or more objects from a first location at a start time of the first trajectory to a second location of the one or more first future points in space-time.

5. The method of claim 1, wherein using a second trajectory to control the autonomous machine is based at least on determining, based on the comparison, that the autonomous machine will reach a first location of the one or more first future points in space-time at a first time at which the one or more objects will occupy the first location upon execution of a first set of one or more safety procedures.

6. The method of claim 1, further comprising determining, based at least on the comparison, a potential collision between a time at which the autonomous machine will occupy a first location on the first trajectory and a time at which the one or more objects will occupy a second location of the one or more first future points in space-time, wherein the first trajectory is dis-qualified from consideration in controlling the autonomous machine based at least on a distance along the first trajectory to the first location being less than a threshold value.

7. The method of claim 1, further comprising: determine, based at least on the comparison, a potential conflict between a time at which the autonomous machine will occupy a first location on the first trajectory and a time at which the one or more objects will occupy a second location in space-time at the one or more first future points; and assign a score to the first trajectory corresponding to a distance along the first trajectory to the first location, wherein one or more operations are performed using the autonomous machine to use the first trajectory based at least on the score assigned to the first trajectory.

8. The method of claim 1, wherein at a beginning of the first trajectory, the one or more objects occupy one or more first locations in the environment that are different from one or more second locations in space-time at the one or more first future points.

9. The method of claim 1, wherein the one or more objects comprise one or more of a pedestrian, an animal, or a stationary obstacle.

10. A processor comprising: one or more processing units to model one or more future sets of claimed locations in space-time for an autonomous machine in an environment, the one or more future sets of claimed locations corresponding to one or more safety procedures to be performed along a first trajectory, determine one or more arrival times for one or more objects to one or more future locations in the environment, compare the one or more future sets of claimed locations to the one or more arrival times; sample the first trajectory at a particular time step and evaluate a conflict at different points throughout the first trajectory, wherein a sample at each sampled point of the first trajectory corresponds to an invoked safety procedure; and determine, based at least on the comparison and the evaluation, whether to use the first trajectory in performing one or more operations using the autonomous machine, wherein sampling the first trajectory at a particular time step and evaluating a conflict at different points throughout the first trajectory comprises: for a sample at a first sampled point of the first trajectory corresponding to an invoked safety procedure, evaluate a plurality of points from a start point to an end point of the safety procedure; for a sample after the first sampled point of the first trajectory corresponding to an invoked safety procedure, evaluate only the end point of the safety procedure; and determine a conflict for a sample based on determining a conflict for at least one point evaluated for the sample.

11. The processor of claim 10, wherein the one or more processing units are further to evaluate a second trajectory comprising a same lateral path as the first trajectory based at least on a comparison of the first trajectory and the second trajectory.

12. The processor of claim 10, wherein the comparison is between the one or more future locations and at least a leftmost vertex and a rightmost vertex representing a shape of the autonomous machine in the one or more future sets of claimed locations.

13. The processor of claim 10, wherein the comparison is between the one or more times of arrival and one or more portions of the one or more future sets of claim corresponding to a side of the autonomous machine, the side based on a direction of travel of the autonomous machine.

14. The processor of claim 10, wherein the one or more times of arrival are based at least on modeling an earliest time at which the one or more objects are likely to occupy the one or more locations.

15. The processor of claim 10, wherein the one or more future sets of claim are modeled from one or more first velocities of the autonomous machine at times within the first trajectory, and the one or more times of arrival are determined from one or more second velocities of the one or more objects at a start time of the first trajectory.

16. A system for controlling an autonomous machine to avoid potential future collisions, comprising: one or more processors; and one or more memory devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a method comprising: computing one or more times at which one or more objects will occupy one or more locations in an environment when following one or more trajectories; modeling a position in space-time of an autonomous machine corresponding to one or more safety procedures performed along a first trajectory; determining whether the autonomous machine will occupy the one or more locations prior to the one or more times based on the modeling and the computing; sampling the first trajectory at particular time steps and evaluating conflicts at different points throughout the first trajectory, wherein a sample at each sampling point of the first trajectory corresponds to an invoked safety procedure; and determining whether to use the first trajectory in performing one or more operations using the autonomous machine based at least on the determination of whether the autonomous machine will occupy the one or more locations prior to the one or more times and the evaluation, wherein sampling the first trajectory at particular time steps and evaluating conflicts at different points throughout the first trajectory comprises: for an invoked safety procedure corresponding to a sample at a first sampling point of the first trajectory, evaluating a plurality of points from a start point to an end point of the safety procedure; for an invoked safety procedure corresponding to a sample after the first sampling point of the first trajectory, evaluating only the end point of the safety procedure; and determining a conflict for a sample based on determining a conflict for at least one point evaluated for the sample. using the modeling to compute a time and a place at which the autonomous machine will stop when implementing the one or more safety procedures during the first trajectory.

17. The system of claim 16, wherein determining that the autonomous machine will occupy the one or more locations prior to the one or more times comprises: ​ 18. The system of claim 16, wherein the one or more times at which the one or more objects will occupy the one or more locations are determined based at least on modeling the one or more trajectories of the one or more objects to the one or more locations.

19. The system of claim 16, wherein the one or more trajectories comprise a direct path of an object of the one or more objects from a first location at a start of the one or more trajectories to a second location of the one or more locations.

20. The system of claim 16, wherein the system is included in at least one of: a control system of a fully autonomous or semi-autonomous machine; a perception system of a fully autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using edge devices; a system implemented using robots; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Images