Use arrival times and safety procedures in motion planning trajectories for autonomous vehicles.

Abstract

To provide use of predicted arrival time and one or more safety procedures in a motion planning of trajectory for an autonomous type vehicle.SOLUTION: A trajectory for an autonomous machine is evaluated for safety based at least on determining whether the autonomous machine is capable of occupying points of the trajectory in space-time while still being able to avoid a potential future collision with one or more objects in an environment through use of one or more safety procedures. To do so, a point of the trajectory is evaluated for conflict based at least on a comparison between points in space-time that correspond to the autonomous machine executing the safety procedures from the points and arrival times of the one or more objects to corresponding positions in the environment. A trajectory is sampled and evaluated for conflicts at various points throughout the whole trajectory. Based on results of one or more evaluations, the trajectory is scored, eliminated from consideration, or otherwise considered for control of the autonomous machine.SELECTED DRAWING: Figure 1

Description

[Technical Field] 【0001】 This application relates to U.S. Patent Application No. 16 / 265,780, filed on 1 February 2019, U.S. Patent Application No. 16 / 269,921, filed on 7 February 2019, and U.S. Patent Application No. 16 / 877,127, filed on 18 May 2020, the entirety of each of those applications is incorporated herein by reference. [Background technology] 【0002】 Designing a system to drive a vehicle autonomously and safely without supervision is an enormous challenge. While operating, an autonomous vehicle should at least have the ability to act as a functional equivalent of a careful driver—a driver relying on a perception and action system with a remarkable ability to identify and react to moving and static obstacles in a complex environment—in order to avoid collisions with other objects or structures. Behavior planning for autonomous vehicles may involve using a driving perception system to determine the trajectory the vehicle should take, which is a challenging but crucial task, especially in complex urban scenarios. For example, in order for an autonomous vehicle to understand its surroundings in order to make safe and effective behavioral decisions, it must process various environmental inputs, along with current vehicle trajectory and route information, to calculate continuous or future trajectories for the vehicle. 【0003】 Conventional systems for behavioral planning often rely on inflexible decisions that do not take into account constantly changing and fluctuating elements in the environment. For example, a conventional method might select a route with the shortest travel time and then choose a lane plan to follow along that route. However, this rigid planning does not allow the planning system to optimally adjust the trajectory based on new or future environmental conditions. For example, by determining the behavior for the vehicle separately from other decisions—such as collision avoidance—adjustments to the current plan may be passive rather than proactive. As such, in the example of collision avoidance, a control decision may be determined using the output from the behavioral planner, and if it determines that a collision is possible, the collision avoidance function may generate an updated control decision. As a result, the vehicle's trajectory is not determined using collision avoidance as a factor, but rather overridden by the vehicle's collision avoidance function. Thus, the vehicle's behavior may be suboptimal because the behavior does not take dynamic inputs—such as collision avoidance—into consideration. [Prior art documents] [Patent Documents] 【0004】 [Patent Document 1] U.S. Patent Application No. 16 / 101,232 [Overview of the project] [Means for solving the problem] 【0005】 Embodiments of this disclosure relate to the use of predicted arrival times and safety procedures in motion planning of trajectories for autonomous vehicles. More specifically, this disclosure relates to techniques and approaches for evaluating a planned trajectory of an autonomous machine for use in controlling the autonomous machine, based on whether the autonomous machine has the ability to occupy a point in the trajectory in spacetime while still being able to avoid potential future collisions. 【0006】 In contrast to conventional systems, the disclosure approach may evaluate a trajectory proposal to an autonomous machine for safety, at least on the basis of determining whether, when performing one or more safety procedures, the autonomous machine should still be able to avoid potential future collisions with one or more objects in the environment while simultaneously having the ability to occupy a proposed trajectory point in spacetime. To do so, the trajectory points may be evaluated for competition at least on the basis of a comparison between points in spacetime (e.g., a claimed set) corresponding to the autonomous machine performing safety procedures from the points and arrival times of one or more objects to the corresponding positions in the environment. The trajectory may be sampled and evaluated for competition at various points throughout the trajectory. Based on the results of one or more evaluations, the trajectory may be scored (e.g., for ranking against other potential trajectories), excluded from consideration, or otherwise considered for control of the autonomous machine. 【0007】 The system and method for using arrival times and safety procedures in motion planning of tracks for autonomous vehicles are described in detail below with reference to the attached drawings. [Brief explanation of the drawing] 【0008】 [Figure 1] This is an illustration including an exemplary trajectory evaluation system according to some embodiments of the present disclosure. [Figure 2] This chart is used to illustrate examples of exercise profiles according to some embodiments of this disclosure. [Figure 3A] Illustrations including exemplary object trajectories according to some embodiments of the present disclosure. [Figure 3B] This is a partial diagram of an contour map representing arrival times, according to some embodiments of the present disclosure. [Figure 3C] This figure shows an contour map representing arrival times according to some embodiments of the present disclosure. [Figure 4A]This figure illustrates an example of a two-dimensional projection of a safety procedure for a vehicle, according to some embodiments of the present disclosure. [Figure 4B] This figure illustrates an example of a two-dimensional projection of a safety procedure for a vehicle, according to some embodiments of the present disclosure. [Figure 4C] This figure illustrates an example of a two-dimensional projection of a safety procedure for a vehicle, according to some embodiments of the present disclosure. [Figure 5A] This figure illustrates an example of a spatiotemporal plot of a vehicle performing a safety procedure, according to some embodiments of the present disclosure. [Figure 5B] This figure illustrates an example of a spatiotemporal plot of a vehicle performing a safety procedure, according to some embodiments of the present disclosure. [Figure 5C] This figure illustrates an example of a spatiotemporal plot of a vehicle performing a safety procedure, according to some embodiments of the present disclosure. [Figure 5D] This figure illustrates an example of a three-dimensional projection of safety procedures for multiple vehicles in spacetime, according to some embodiments of the present disclosure. [Figure 6] This figure includes examples of orbits, safety procedures that may be activated in the orbit, and points that may be used to evaluate the orbit, according to some embodiments of the present disclosure. [Figure 7] These are illustrations used to illustrate examples of lateral paths and speed profiles according to some embodiments of the present disclosure. [Figure 8] Illustrations of exemplary behavioral planning architectures for autonomous vehicles, according to some embodiments of the present disclosure. [Figure 9] This flowchart illustrates a method for controlling a vehicle based on evaluating points corresponding to safety procedures that may be deployed on the track, according to some embodiments of the present disclosure. [Figure 10] This flowchart illustrates a method for controlling a vehicle based on modeling a future set of vehicle requirements corresponding to safety procedures that may be implemented on the track, according to some embodiments of the present disclosure. [Figure 11]This flowchart illustrates a method for controlling a vehicle based on determining that the vehicle should occupy a position before the time at which the position should be occupied by an object, when performing safety procedures in a track, according to some embodiments of the present disclosure. [Figure 12] Illustrations of exemplary operating environments according to some embodiments of the present disclosure. [Figure 13A] Illustrations of exemplary autonomous vehicles according to some embodiments of the present disclosure. [Figure 13B] Figure 13A shows examples of camera positions and fields of view of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 13C] Figure 13A is a block diagram of an exemplary system architecture of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 13D] This is a system diagram of communication between a cloud-based server and the exemplary autonomous vehicle shown in Figure 13A, according to some embodiments of the present disclosure. [Figure 14] This is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 15] This is an exemplary data center block diagram suitable for use in implementing some embodiments of the present disclosure. [Modes for carrying out the invention] 【0009】 A system and method relating to the use of arrival times and safety procedures in motion planning of trajectories for autonomous vehicles is disclosed. More specifically, the disclosure relates to techniques and approaches for evaluating trajectories for use in controlling an autonomous machine, based on whether the autonomous machine has the ability to occupy a point in the trajectory in spacetime while still being able to avoid potential future collisions. 【0010】 This disclosure may describe an exemplary autonomous vehicle 140 (referred to herein as "vehicle 140" or "autonomous vehicle 140" as otherwise), which is described in more detail herein with respect to Figures 13A–13D. While this disclosure primarily provides examples using autonomous vehicles, other types of objects or machines may be controlled using the approaches of the disclosure. 【0011】 The disclosed approach may evaluate a trajectory for an autonomous machine for safety purposes, at least on the basis of determining whether the autonomous machine should have the ability to occupy a point in its trajectory in spacetime while still being able to avoid potential future collisions with one or more objects in the environment through the use of one or more safety procedures. For a sample point in spacetime in the trajectory, the system may determine one or more points in spacetime that the autonomous machine should occupy if it were to perform one or more safety procedures from the sample point. The system may also determine the potential arrival times of one or more objects to the location of one or more points in spacetime and compare the time of the one or more points in spacetime that the autonomous machine should occupy with the arrival time. Objects may include pedestrians, animals, motorcycles, cars, and / or static obstacles with small accelerations assumed to account for perceptual uncertainty. In some embodiments, objects may be limited to one or more of these classes or types. 【0012】 The comparison may be used to identify whether there is a conflict between when an autonomous machine is supposed to occupy a position in its orbit and when one or more objects are supposed to occupy a position in the environment very close by. For example, if the comparison indicates that the autonomous machine should occupy one or more positions before the corresponding arrival time, given that safety procedures are performed from a position in the orbit, then no conflict is determined (e.g., the vehicle can stop before the collision). However, if the comparison indicates that the autonomous machine should occupy one or more positions after the corresponding arrival time, then a conflict may be determined (e.g., the vehicle cannot stop before the collision). 【0013】 In at least one embodiment, the arrival time of one or more objects can be calculated based at least on modeling the trajectories of one or more objects to one or more locations in the environment (e.g., from the start of the trajectory). For example, each trajectory may model a scenario in which an object travels directly from its starting position toward a given position. Object parameters such as attitude, acceleration / deceleration, and velocity may be used to generate the trajectory. In one or more embodiments, a worst-case scenario can be modeled in which each object travels directly toward a given position as quickly as possible (e.g., according to the capabilities of the objects in the model). This provides a conservative approach in which, if the autonomous machine has the ability to avoid collisions in the worst-case or very limited scenario, then the autonomous machine should also have the ability to avoid collisions in less limited scenarios. When multiple objects are modeled, the minimum or earliest arrival time to a given position may be used for comparison with the point corresponding to the autonomous machine. 【0014】 In one or more embodiments, the set of machine requirements corresponds to one or more locations in spacetime that the autonomous machine is expected to occupy when one or more safety procedures are performed at a sample site. The set of requirements for the autonomous machine may include the trajectory of the autonomous machine when the autonomous machine applies the safety procedures (for example, at least an estimate of each location in space that the machine should occupy as it moves along the trajectory). The safety procedures may refer to procedures for controlling the autonomous machine that define one or more trajectories (without limitation, such as changing direction, turning around, moving to a side lane or shoulder, etc.) or actions (without limitation, such as accelerating, braking, slowing down, switching at least partial control to an alternative component, returning control to a driver or other operator) of the autonomous machine in spacetime that, if performed at the corresponding time and location, should reduce the likelihood of a collision and / or its effects, or avoid a collision and / or its effects. Safety procedures may be triggered at least in part on the autonomous machine's determination that a collision is likely and / or imminent, and / or that a failure of one or more components of the autonomous machine is likely, imminent, and / or has occurred. 【0015】 The trajectory can be sampled and evaluated for competition at various points along the entire trajectory, for example, at different time steps. To avoid gaps in coverage, multiple points may be evaluated for the sample corresponding to the start of the trajectory. However, for subsequent time steps, only the point at the tip of the safety procedure (for example, if the actor's state objective is satisfied) may be evaluated. Sampling only the tip may be sufficient in some embodiments, since the tip may correspond to the most limited points of the safety procedure. Therefore, checking for competition with respect to safety procedures may be computationally efficient, as the number of comparisons may be linear with the number of safety procedures being evaluated. 【0016】 Based on the results of one or more evaluations, a trajectory may be scored (for example, for ranking against other potential trajectories), excluded from consideration, or otherwise considered for the control of an autonomous machine. For example, competition may be used as a hard or soft constraint when evaluating trajectories for selection during autonomous machine control. The constraint may be based on various potential factors, such as the distance and / or elapsed time to competition detected along the trajectory (e.g., competition at the maximum distance and / or elapsed time from the start of the trajectory) (e.g., to the start of the sample, or to a point equivalent to the arrival time). 【0017】 Referring here to Figure 1, Figure 1 is an illustration including an exemplary trajectory evaluation system 100 according to several embodiments of the present disclosure. It should be understood that this and other configurations described herein are presented merely as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, and groupings of functions) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely for clarity. Furthermore, many of the elements described herein are functional entities that may be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. Various functions described herein as being performed by one or more entities may be performed by hardware, firmware, and / or software. For example, some functions may be performed by a processor executing instructions stored in memory. 【0018】 The trajectory evaluation system 100 receives sensor data 102 representing the environment 126 and may use the sensor data 102 to calculate at least one arrival time of an object to at least one location within the environment 126. The at least one arrival time may be used to analyze vehicle trajectory information 124 representing at least one proposed trajectory or route 144 of the vehicle 140. In one or more embodiments, the at least one arrival time may be captured in an image or other form of data processed for analysis of the proposed trajectory or route 144. The image may include, but is not limited to, a 2D grid, a matrix, a one-channel rasterized image, etc., as examples. 【0019】 The trajectory evaluation system 100 may include, for example, a communications manager 104, an object analyzer 106, an object modeler 108, a safety procedure determiner 110, an arrival time determiner 112, a trajectory analyzer 114, and a request set determiner 116. 【0020】 Overall, the communication manager 104 may be configured to manage communications received by the trajectory evaluation system 100 (including, for example, sensor data 102 and / or vehicle trajectory information 124) and / or provided by the trajectory evaluation system 100 (including, for example, data corresponding to the analysis results of the vehicle trajectory information 124). The object analyzer 106 may be configured to analyze objects in the environment using the sensor data 102. The object modeler 108 may be configured to model objects, such as the trajectory of an object in the environment 126, based at least on the analysis performed by the object analyzer 106. The arrival time determiner 112 may be configured to analyze the trajectory of an object, such as the trajectory modeled by the object modeler 108, in order to calculate the arrival time of the object to a position in the environment. 【0021】 The trajectory analyzer 114 may be configured to evaluate one or more trajectories 144 for the vehicle 140 based at least on arrival times. To do so, the trajectory analyzer 114 may use a safety procedure determiner 110 and a requirements set determiner 116. The safety procedure determiner 110 may determine safety procedures for the vehicle 140. The requirements set determiner 116 may determine at least a portion of the requirements set for the vehicle 140, assuming the vehicle 140 performs safety procedures at one or more points on the trajectory 144. The trajectory analyzer 114 may compare one or more arrival times with a portion of the requirements set. Based on the comparison results, the trajectory 144 may be scored (for example, for ranking against other potential trajectories), excluded from consideration, or otherwise considered for control of the vehicle 140. 【0022】 As described herein, the communications manager 104 may be configured to manage communications received by the trajectory evaluation system 100 (including, for example, sensor data 102 and / or vehicle trajectory information 124) and / or provided by the trajectory evaluation system 100 (including, for example, data corresponding to the analysis results of the vehicle trajectory information 124). 【0023】 When communication is received and / or provided as network communication, the communication manager 104 may have a network interface capable of communicating on one or more networks using one or more wireless antennas 1326 and / or modems. For example, the network interface may have the capability to communicate using Long-Term Evolution (LTE), Wideband Code Division Multiple Access (WCDMA®), General-Purpose Mobile Communications Services (UMTS), Global System for Mobile Communications (GSM), CDMA2000, etc. The network interface may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth Low Energy (LE), Z-Wave, ZigBee, and / or low-power wide area networks (LPWANs) such as Long-Range Wide Area Networks (LoRaWAN) and SigFox. However, the communication manager 104 does not need to include a network interface, for example, when the trajectory evaluation system 100 is fully implemented in an autonomous vehicle (e.g., vehicle 140). In some examples, one or more of the communications described herein may be between components of the computing device 1400 on the bus 1402 in Figure 14. 【0024】 Sensor data 102 can be generated using any combination of sensors, such as sensor 1238 in Figure 12. For example, sensor data 102 may include image data representing images, image data representing video (e.g., video snapshots), and / or sensor data representing the field of view of a sensor (e.g., LIDAR sensor 1364, RADAR sensor 1360, camera in Figure 13B). In the illustrated example, sensor data 102 may include image data representing the field of view of each camera of the vehicle 140, which may include one or more images. The images may depict an area of ​​the environment 126, where the area may include any number of objects, in this example, objects 128A and 128B. The objects may include vehicles, people (e.g., pedestrians), motorcycles, cyclists, trees, animals, buildings, signs, or any combination of other structures, objects, or obstacles in the environment 126 that the vehicle 140 may be able to collide with. 【0025】 Sensor data 102 is provided to an object analyzer 106, which may use it to analyze objects in the environment 126. For example, the object analyzer 106 may detect one or more objects 128A or 128B, identify one or more objects, and / or determine one or more parameters of an object using the sensor data 102. To do so, the object analyzer 106 may use one or more machine learning models. For example, and not limited to, machine learning models may include any type of machine learning model, and / or other types of machine learning models, such as machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbors (KNNs), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, and neural networks (e.g., autoencoders, convolutions, recurrent, perceptrons, long / short-term memory / LSTMs, Hopfield, Boltzmann, deep belief, deconvolution, adversarial generative, liquid state machines, etc.). 【0026】 Examples of object parameters that the object analyzer 106 can determine using sensor data 102 include location (e.g., location coordinates within the environment 126), orientation, current and / or observed velocity, maximum velocity, predicted velocity, at least one dimension (e.g., physical dimensions such as length, width, footprint, height), current and / or observed acceleration or deceleration, maximum acceleration or deceleration, predicted acceleration or deceleration, mass, reaction time, and / or one or more other parameters, including but not limited to those described herein. One or more parameters may represent an observed characteristic of the object (e.g., location / position), and one or more parameters may represent an inferred characteristic of the object (e.g., maximum acceleration). 【0027】 In some examples, the parameters of an object may be the output of a machine learning model, such as a convolutional neural network, which takes at least some of the sensor data 102 as input. In further examples, the object analyzer 106 may use at least one machine learning model to classify one or more of the objects captured by the sensor data 102. Examples of classification include stationary objects, moving objects, vehicles, cars, trucks, pedestrians, cyclists, motorcycles, etc. 【0028】 The object analyzer 106 may use classification to determine one or more parameters. For example, classification may be provided as input to a machine learning model used to determine one or more parameters. In another example, one or more classifications and / or other object information (e.g., other parameters) may be applied to a search table or otherwise used for the search, determination, and / or calculation of one or more parameters. In one example, classification may include a vehicle model or type (e.g., sedan, truck, motorcycle, SUV), and the vehicle model or type may have one or more predetermined shapes and / or dimensions, braking ability, handling ability, acceleration ability, maximum speed, maximum acceleration, etc., which may be used to define one or more parameters. In some examples, a machine learning model, such as a convolutional neural network, may be trained to simultaneously output object classifications and one or more of the object's parameters. 【0029】 As an example, the object analyzer 106 may perform object perception using a machine learning model (e.g., a neural network) that can be specifically configured (e.g., trained) to recognize a particular object and / or the features of an object. One or more trained machine learning models used by the object analyzer 106 (e.g., trained and deployed for use by the trajectory evaluation system 100) may determine the presence and / or location (e.g., X and Y coordinates) of an object, the orientation (φ) of an object, the dimensions of an obstacle (e.g., width and length), and / or the classification of an object. Furthermore, the trained machine learning model (e.g., a neural network) may determine the maximum acceleration (A) of an object. MAX+ ) and maximum deceleration (A MAX- ) can be used to determine 【0030】 As a further example, consider the reaction time of an object (T REACTThe reaction time of an object can be determined using a trained machine learning model (e.g., a neural network) that identifies or outputs the type of object (e.g., object classification) and / or assigns a corresponding reaction time. The reaction time of an object may represent the future time at which the object's motion begins to be affected by the presence of another object, such as a vehicle 140, in the environment 126, or the time until the object's motion begins to be affected. This may correspond to the object perceiving, noticing, and / or determining that a collision with another object may occur at its position in the environment. For example, a pedestrian staring intently at their smartphone may have a long reaction time T. REACT Similarly, a cyclist or motorcyclist looking far away from their position may have a longer reaction time T than, for example, a more attentive driver. REACT It can be considered that it has a shorter reaction time T. REACT It may have. 【0031】 For example, an object analyzer may detect one or more of objects 128A and 128B, identify one or more of objects 128A and 128B, and / or use one or more of objects 128A and 128B, sensor 138, and / or sensor data 102 and / or other information received from client device 1232 in the operating environment 130 to determine parameters and / or classifications of one or more of objects 128A and 128B (for example, on vehicle 140 and / or server 1378). For example, a communication manager 104 may receive the corresponding data on network 1234 in Figure 12. Furthermore, the object analyzer 106 may detect, identify, and / or determine parameters and / or classifications of one or more objects that cannot be directly represented by the sensor data 102, but which can be (for example, based at least on the sensor data 102) inferred, predicted, or otherwise determined to be present or potentially present by the object analyzer 106. This can be used to take into account blind spots or blind spots perceived by the trajectory evaluation system 100. 【0032】 The object modeler 108 may be configured to model one or more objects, such as objects 128A and / or 128B in the environment 126, based at least on an analysis performed by the object analyzer 106 (for example, based on at least one or more parameters associated with the object). In addition or otherwise, the object modeler 108 may filter one or more objects from the model (e.g., a trajectory model). For example, the object modeler 108 may filter objects based at least on the classification of the object (e.g., stationary) and / or on the type of the object. In addition or otherwise, the object modeler 108 may filter one or more objects based at least on one or more parameters of the object, such as any of the parameters determined by the object analyzer 106 (for example, by filtering the object based at least on the location of the filtered object being beyond a threshold distance from the vehicle 140). This filtering process can reduce the processing power used to determine arrival times and may be further used to improve trajectory evaluation in lane-driving scenarios, either by vehicles driving parallel to or oncoming traffic of interest adjacent to the lane. 【0033】 The object modeler 108 may use any suitable approach to model objects in the environment 126, for example, by representing each object as a function of radial distance (and / or one or more other functions), or as a list of known locations of objects in the environment 126 (represented as a plane). The representation of an object may be defined using, for example, at least one-dimensional parameters (e.g., width, length), location parameters (e.g., X, Y in the environment 126), and / or the object's orientation determined using the object analyzer 106. Optionally, a safety buffer may be included in the object's representation (e.g., by scaling the object's dimensions). The model of an object may further include one or more motion vectors of the object, such as current and / or instantaneous motion vectors (e.g., observed motion vectors derived from sensor data 102). For example, each object may have motion and acceleration vectors defined using at least one parameter determined for the object using the object analyzer 106 (e.g., using the object's orientation, acceleration / deceleration, and velocity). The object modeler 108 can further model the object using the object's maximum acceleration and / or deceleration parameters, maximum velocity parameter, and / or mass parameter. 【0034】 The object modeler 108 can model one or more trajectories of an object using an object representation or model. To do so, the object modeler 108 can apply one or more motion profiles to an object representation to define the velocity and acceleration of the object at specific times and locations along the object's trajectory within the environment 126, an example of which is illustrated with Figure 2. 【0035】 Referring now to FIG. 2, FIG. 2 includes a chart 200 used to illustrate an example of a motion profile that can be used to define a modeled trajectory of an object. The chart 200 includes a plot 210 corresponding to a motion profile that defines the motion of the object over time according to the trajectory. The object model 108 can represent the motion profile using, for example, one or more functions (e.g., continuous functions) and / or rules used to define the plot 210 for the object. 【0036】 The motion profile that defines the plot 210 can optionally be constrained by a maximum speed V MAX or a speed cap V CAP and a maximum acceleration A MAX+ such that the object progresses along a trajectory directed towards a position within the environment 126 (even if the object is currently moving away from the object), a stage where the object decelerates at a maximum deceleration A MAX+ (which can be the same as A MAX- ) until the object comes to a complete stop, and / or a stage where the object remains stationary. The motion profile can include parameters used by the object model 108 to set the time at which the object begins to decelerate in the motion profile, such as a reaction time T REACT . The observed acceleration A OBS can be used instead of the maximum acceleration A MAX+ or the maximum deceleration A MAX- and can be adjusted by a safety factor to allow for the possibility that the observed acceleration is inaccurate and / or to allow for the possibility that the acceleration can increase. 【0037】 The motion profile corresponding to FIG. 2 can be used to represent a worst-case scenario for the safety of the vehicle, calculated as if the vehicle 140 were at a position at one or more points in time. The reaction time T REACTThis could represent the time it takes for an operator of an object (e.g., another vehicle) (e.g., a driver) to become aware of the vehicle at this location and attempt to avoid a collision at this location. However, other motion profiles may be used by the object modeler 108 to model the trajectories of one or more of various objects, and these motion profiles may include any number of stages. Furthermore, one or more motion profiles may be more complex than the described motion profile, including conditions or rules for entering a stage, or used in stages that change the motion of an object, at least based on the object modeler 108 analyzing and applying variables that represent different characteristics of the state of the environment 126 at a given time. 【0038】 The trajectory modeled by object modeler 108 has a start time T that represents the start time of the trajectory. S , and the end time T representing the end time of the trajectory. E It may have. Figure 2 shows the starting speed V S and starting acceleration A S Start time T S A specific example is shown. The object modeler 108 takes the current, observed and / or instantaneous velocity and acceleration parameters provided by the object analyzer 106 and the starting velocity V S and starting acceleration A S Can it be set to start speed V? S and / or starting acceleration A S This can be derived from these values. In at least one embodiment, the starting acceleration A S is the maximum acceleration A MAX It can be set to [this value]. 【0039】 The example in Figure 2 shows the orbit start time T. S In this case, the initial velocity V of the object S and starting acceleration A S This indicates that the value is positive relative to vehicle 140, meaning that the object is moving toward vehicle 140 at an increasing rate. However, if the object initially... S When moving away from vehicle 140, the object's initial velocity V Sand / or starting acceleration A S This can be negative for vehicle 140. In other examples, the starting speed V S Or starting acceleration A S This can be zero. S The presence of an object inside can depend on the stage of its motion profile. 【0040】 Figure 2 shows the termination speed V. E and termination acceleration A E End time T E Specific examples are also shown. Termination speed V E and termination acceleration A E As shown in the diagram, the final velocity V can be zero, indicating that the object has come to a complete stop at the end of its trajectory. However, in other examples, the final velocity V E and / or termination acceleration A E This can be positive or negative at the end of the trajectory. For example, start time T S This can correspond to any position along plot 210 depending on which stage the object is in within the motion profile at the start of the trajectory, and similarly, at the end time T E The start time T is determined by the stage in the motion profile of the object at the end of its trajectory (for example, the object may not have reached a complete stop), depending on the stage the object is at. S This can correspond to any position along plot 210 after that. 【0041】 Furthermore, if the object modeler 108 models the trajectories of multiple objects using the same motion profile, different objects may be at different stages of the motion profile simultaneously. For example, one object may be at a start time T in its modeled trajectory. S It may be slowing down until it comes to a complete stop, and the end time T E It is possible that it remained stationary for some time until then, while the second object, in its modeled trajectory, started at time T. S It is possible that the vehicle is accelerating towards vehicle 140, and the end time T E At maximum speed V MAXIt is possible that this is progressing. Also, in at least one embodiment, the motion profile may not include the stage in which the object begins to decelerate. For example, the motion profile may be used to model the time in which the object can reach a point in space without decelerating (for example, to model the earliest possible time of arrival of the object to that point in space), and thus the time in which the object can reach a point at maximum velocity V MAX The region 220, which is modeled to reach (maximum speed V MAX End time T before reaching E (Unless it arrives) End time T when reaching this point E It could continue until then. 【0042】 Figure 2 shows the maximum velocity V of an object in several examples. MAX The maximum acceleration A of an object MAX+ , or the maximum deceleration A of an object MAX- This shows that a motion profile can be applied by the object modeler 108 using one or more of the following values. These values ​​may represent the limits of the object's motion capabilities as described herein, and any of these values ​​may correspond to parameters determined for the object using the object analyzer 106. As an example, plot 210 shows that the object reaches a maximum velocity V in the modeled trajectory. MAX The region 220 is modeled to reach [a certain value]. However, in other orbits, the object reaches a maximum velocity V within the orbit. MAX , maximum acceleration A MAX+ , or maximum deceleration A MAX- Either one or more of these factors cannot be reached, or the object modeler 108 cannot use one or more of these factors when applying a motion profile. 【0043】 Figure 2 shows the velocity cap V that can be considered in the modeled trajectory. CAP Also shows: Speed ​​cap V CAP The maximum speed V MAX It can become larger, maximum speed V MAX It can correspond to larger observed velocities of objects. For example, a velocity cap V CAP This is when the object reaches its maximum velocity V MAXIs it moving faster, or is it at maximum speed V? MAX The object analyzer 106 may determine from sensor data 102 that it has the ability to move faster. For example, the object modeler 108 first applies the object's motion profile to model the object's trajectory, which corresponds to the maximum velocity V MAX It can be used. The object has a maximum velocity V MAX After determining that it is moving faster or has moved faster, object modeler 108 instead sets a velocity cap V CAP The modeled trajectory can be updated using this method, or different trajectories of the object can be modeled. A similar approach can be used for maximum acceleration or deceleration. 【0044】 Referring here to Figure 3A, Figure 3A is an illustration including an exemplary trajectory 302 according to several embodiments of the present disclosure. Trajectory 302 may correspond to the modeled trajectory described with respect to Figure 2. The object modeler 108 starts at time T S The starting position X of an object corresponding to the location or position of object Y1 (for example, object 128A or 128B) in environment 126. S The trajectory 302 of object Y1 can be modeled using the object modeler 108. The object modeler 108 can also model the end time T. E The trajectory 302 can be modeled using the end position X1 of object Y1, which corresponds to the location or position of object Y1 within the environment 126. 【0045】 Starting position X of object Y1 S This may correspond to location parameters determined using the object analyzer 106. Location parameters can be determined by the object analyzer 106, for example, in real-time or near real-time, such that the starting position substantially represents the current position of object Y1 within the environment 126. However, the starting position X SThis could be the predicted current or future location of the object in the environment, such as when, in some examples, it becomes unclear to perceive object Y1 with sensor 138, when object Y1 is not necessarily present in the environment and is modeled to take into account blind spots or blind spots perceived by the trajectory evaluation system 100, or in other scenarios. 【0046】 In Figure 3A, the object modeler 108 may model the trajectory 302 so that it proceeds directly toward the endpoint X1 of object Y1 (for example, a position in environment 126 whose arrival time is being determined), as shown in the figure. This approach may be used by the object modeler 108 in conjunction with the motion profiles described above to model a worst-case scenario in which the operator of object Y1 attempts to reach the endpoint X1 as quickly and directly as possible. The object modeler 108 may use a similar approach when determining the arrival time of each position in environment 126 and each modeled object in the environment, or one or more trajectories may proceed along other paths and / or different motion profiles may be used. In some instances, the object modeler 108 may decide to model the trajectory (e.g., the motion path corresponding to plot 210, and the direct path) using a default path and / or motion profile, while in other instances, the object modeler 108 may decide to model the trajectory (e.g., after the object has provided it, if the object's planned trajectory is known) using a different path and / or motion profile. 【0047】 As described herein, the arrival time determination unit 112 may be configured to analyze the trajectory modeled by the object modeler 108 in order to calculate the arrival times of one or more objects to one or more locations within the environment 126. For example, the arrival time determination unit 112 may use the modeled trajectory to determine when an object occupies one or more locations within the environment 126, as well as the velocity and acceleration of the object at these locations. Such combinations of location and time may form one or more points in spacetime, which may be determined using a variety of possible measures. For example, a point may represent the earliest possible arrival time of an object to that point (as estimated or predicted by the system). In one or more embodiments, the arrival time to a point may represent the safe arrival time of a vehicle 140 to that point (e.g., the latest safe arrival time), where a vehicle 140 arriving at this point at least before the safe arrival time should not collide with an object. However, the arrival time can be evaluated on other measures, such as the fact that the resulting collision should be less affected by a threshold that can be calculated at least based on the modeled mass and velocity of the object, that the collision is less than the threshold probability of occurrence, or that vehicle 140 has the ability to come to a complete stop before the time of the object's arrival at the point. 【0048】 In some examples, to determine at least one safe arrival time to the destination position X1, the object modeler 108 models the trajectory 302 to the destination position X1. E Based at least on the calculation that a complete stop occurs at (or earlier in other instances), the arrival time determination unit 112 determines the end time T E , and / or end time T in track 302 E Any time earlier than the end time T can be determined to be the safe arrival time for the vehicle to the end position X1. For example, considering the position P in the environment, if under the worst-case scenario for the object's trajectory, the object Y1 can stop at position P, then the end position X1 can be determined to be the end time T. E Since it can be determined that it has the end time T Ecan be regarded as the safe arrival time at position P. Therefore, any earlier time can also be assumed to be the safe arrival time to position P. As such, the start time T of the trajectory 302 S and the end time T E can constitute the safe time interval of the vehicle for the object Y1 and position P. In other examples, the object Y1 does not reach a complete stop before the end time T of the trajectory 302 E Instead, it may have an end position X1'. In this case, since the object Y1 can make itself in a state where it is already promised to pass position P, it can be determined that the safe arrival time of position P and the object Y1 is zero. 【0049】 Another case where the safe arrival time can be set to zero is when the object Y1 is promised to be at position P from the beginning and the parabola of the reverse acceleration passes through position P. As an example, this can occur if position P is at the start position X of the trajectory 302 S This can happen, for example, when the vehicle 140 is chasing the object Y1 which is the leading vehicle moving rapidly away from the vehicle 140 (this example is shown in FIG. 3A). In this case, the vehicle 140 will necessarily pass through some previous positions (and typically pass through them). Here, it may be clear that the parabola passes through position P during a time interval (e.g., an unsafe time interval). 【0050】 A non-limiting example of the calculation of the safe arrival time for the object Y1 at the obstacle point Y (e.g., the start position X in the plane S ) and the target point X (e.g., position P in the plane) follows. The object Y1 at the obstacle point Y has a velocity vector V (e.g., the start velocity V in the plane S ), a scalar maximum velocity v max (e.g., the maximum velocity V MAX ), a scalar maximum acceleration a max (e.g., the maximum acceleration A MAX ), and a reaction time τ react (e.g., the reaction time T REACTConsidering having (), the scalar distance d can be calculated as d = |X - Y| and the scalar (signed) velocity v can be calculated as 【Number】 and can be calculated as 【0051】 In some examples, this can be generalized to any function that distributes the maximum starting velocity in each direction, and the dot product makes a specific proposal for such a function. The safe arrival time t of the object Y1 at the obstacle location Y lsa (e.g., the latest safe arrival time) can be calculated by the following formula. 【Number】 【0052】 For each position in the plane to be evaluated, all the minimum values (e.g., locations) of the safe arrival time t from all objects can be calculated in parallel. When the input is a function of the radial distance, the time can be set to zero for all points on the radiation farther away from the function of the radial distance. This approach can be used by the arrival time determiner 112 to generate a visualization (e.g., an image). Further, in some examples, any safe arrival time t of any object lsa can be calculated at any appropriate time for purposes such as inspecting and / or generating a path and / or trajectory (e.g., without necessarily generating a visualization if required). lsa can be calculated at any appropriate time for purposes such as inspecting and / or generating a path and / or trajectory (e.g., without necessarily generating a visualization if required). 【0053】 In any given example, the object modeler 108 and the arrival time determiner 112 may consider additional factors to determine the arrival time of an object. For example, the incline to position P may be estimated using sensor data 102 received and processed by the object analyzer 106. The incline to the target point may be used as a factor to reduce acceleration (or increase deceleration) to the target point. Similarly, the road surface coefficient μ may be estimated using information received and processed by a perception block of the vehicle 140 (similar to object perception which may be performed by the object analyzer 106). For example, a trained neural network may detect the presence of an ice, snow, or oily road and return a road surface coefficient μ, which may be used as a factor to reduce maximum deceleration to position P. 【0054】 Figure 3A also shows the endpoints X2, X3, and X4 of other trajectories that the object modeler 108 can model, such as by using a motion profile corresponding to or different from plot 210, and / or by using a direct path (or a different type of path) to each of the endpoints X2, X3, and X4. Endpoints X2, X3, and X4 are provided as examples of positions in trajectories having at least one common arrival time. This may be represented in Figure 3A by endpoints X1, X2, X3, and X4 plotted within a spatiotemporal domain that forms points of contour lines 306 representing arrival times shared between positions or locations in the environment 126. Contour lines 306 may, in addition or otherwise, represent a shared time interval of positions (for example, contour lines 306 may represent the end time of an interval). In at least one embodiment, one or more points on contour lines 306 may be used by the track analyzer 114 to determine whether one or more points in the spacetime of track 144 are safe for vehicle 140 to occupy. For example, the track analyzer 114 may use the points on contour lines 306 to detect or determine potential intersections with one or more points in the spacetime of track 144 that vehicle 140 would occupy if vehicle 140 were to perform one or more safety procedures at one or more points in the spacetime of track 144. Using this approach, the track analyzer 114 may determine, given a time, that the position of track 144 is unsafe because a collision could occur under the modeled conditions. 【0055】 Figure 3A illustrates how the arrival time determination unit 112 analyzes the position and time for a single object Y1 when calculating one or more arrival times and / or time intervals. However, the arrival time determination unit 112 can consider any number of objects, such as any of the various objects modeled using sensor data 102. For example, Figures 3B and 3C show contour lines representing arrival times considering multiple objects in the environment. 【0056】 In Figure 3C, object Y1 may correspond to object 128A in Figure 1, object Y2 may correspond to object 128B in Figure 1, and object Y3 may correspond to an object modeled using object modeler 108 rather than being represented by sensor data 102. For any of positions P1, P2, P3, P4, P5, P6, and P7, the arrival time determiner 112 may use the arrival time and / or time interval of the earliest occurring object among the modeled objects and trajectories at a particular position. For example, for position P1, the arrival time determiner 112 may use the arrival time of object Y1 if the arrival time of object Y1 occurs before the arrival times of objects Y2 and Y3 for position P1. 【0057】 The arrival times and / or time intervals used by the object modeler 108 for positions P1, P2, and P3 represent the arrival times T1 (and / or time intervals) along the corresponding positions in the environment 126 (similar to contour lines 306, for example), but can be shown in Figure 3C as plotted within a spatiotemporal domain that forms the points of contour lines 308 considering multiple objects. Similarly, the arrival times and / or time intervals used by the object modeler 108 for positions P4, P5, P6, and P7 can be shown in Figure 3C as plotted within a spatiotemporal domain that forms the points of contour lines 310 representing the arrival times T2 (and / or time intervals) along the positions in the environment 126. 【0058】 Figure 3B shows an contour map 300, which includes additional contours 308 and 310 from Figure 3C, as well as other contours for different arrival times and / or time intervals and locations within the environment 126. Region 330 in Figure 3B may correspond to region 330 in Figure 3C. While the contours in Figure 3B correspond to multiple objects, in other instances, any of these contours may correspond to a single object. Furthermore, an infinite number of contours may exist, each corresponding to a different arrival time and / or time interval (if any). In any instance, the arrival time determiner 112 may compute data representing one or more contours, portions of contours, and / or corresponding locations and arrival times (e.g., points in spacetime). 【0059】 Furthermore, in some examples, the data may include image data representing an image that captures any of the information to be withheld. For example, the arrival time determination unit 112 may generate an image as a visualization based at least on the arrival time. The data values ​​of the image used as a visualization (e.g., pixel values, vector graphics values, analog values, digital values, etc.) may be computed in parallel by using parallel processing that may occur on one or more image processing units (GPUs), such as one or more image processing units (GPUs) 1508 in Figure 14, GPU 1308 in Figure 13C, and / or GPU 1320 in Figure 13C. In at least one embodiment, the data values ​​of the image (e.g., pixel values, vector graphics values, analog values, digital values, etc.) may represent at least a portion of a safe time interval. For example, the arrival time determination unit 112 may represent the arrival time by using one or more pixels or points of the visualization to set, for example, brightness values, color values, and / or other values ​​associated with or used to generate the pixels or points. Each point or pixel in the visualization generated by the arrival time determination unit 112 may represent an arrival time and location within the environment 126. The location of a point or pixel in the visualization may correspond to a location within the environment 126, and the color or other properties of a point or pixel in the visualization may correspond to an arrival time to that location (e.g., safe arrival time). 【0060】 One or more trajectories analyzed by the trajectory analyzer 114 may be generated by another system or component (e.g., a behavior planner for vehicle 140), which, in practice, may not take into account arrival times and / or time intervals. In at least one embodiment, one or more trajectories may be generated using the behavior planning architecture 800 of Figure 8. Trajectories may be represented using trajectory points in spacetime and / or any suitable approach including splines or other functions. For example, vehicle trajectory information 124 may include trajectory points 142 along trajectory 144 (e.g., represented by 2D or 3D coordinates in world space and / or 2D coordinates in image space). In some examples, only a single trajectory point 142 (e.g., the next trajectory point of vehicle 140 in a discretized trajectory step sequence) may be available (e.g., output by a machine learning model). In other examples, two or more trajectory points 142 may be available. In at least one embodiment, one or more track points 142 can be calculated and / or extrapolated from one or more other track points 142. In some examples, the radius or inverse radius of track 144 can be used to calculate one or more track points 142 (for example, if turns, lane changes, lane splits, lane merges, and / or curves are included in track 144). 【0061】 As described herein, the trajectory analyzer 114 may be configured to evaluate one or more trajectories 144 based at least on the arrival times determined by the arrival time determiner 112. For example, the trajectory analyzer 114 may calculate one or more points in spacetime corresponding to one or more objects in the environment 126 (e.g., object 128A and / or object 128B). The trajectory analyzer 114 may use one or more points in spacetime corresponding to one or more objects to evaluate the trajectory 144 with respect to another point in spacetime within the trajectory 144. For example, the trajectory analyzer 114 may calculate one or more points in spacetime corresponding to a vehicle 140 performing one or more safety procedures (e.g., points that vehicle 140 would occupy if vehicle 140 were to perform one or more safety procedures) from one or more points in the trajectory 144. The track analyzer 114 may then evaluate the track 144 by comparing one or more of these points with one or more points corresponding to one or more objects. Based on the comparison results, the track 144 may be scored, excluded from consideration, or otherwise considered for the control of the vehicle 140. 【0062】 Figure 1 illustrates an example of one or more points in spacetime corresponding to at least one object 128A (e.g., a pedestrian). Specifically, Figure 1 shows the arrival times T of object 128A at points C0, C1, C2, and C3. A Contour lines are shown for the object. One or more points calculated by the arrival time determination unit 112 and / or used by the trajectory analyzer 114 to evaluate the trajectory 144 may include one or more points on any of these contour lines. As an example, but not limited to, the contour lines in Figure 1 form concentric circles around object 128A with arrival times that increase with distance from the starting position of object 128A. When multiple objects are considered as described herein, contour lines / points may be calculated as the minimum arrival times of the objects. 【0063】 Figure 1 also shows examples of one or more points in the orbit 144 that the orbital analyzer 114 can evaluate for points associated with arrival time. Specifically, Figure 1 shows orbital times T1, T2, and T3. T The location of vehicle 140 is indicated for T0=0, T1=0.5, and T2=1.0, where, as an example but not limited to, T0=0, T1=0.5, and T2=1.0. The location and trajectory time may include one or more points that can be evaluated against one or more points associated with the arrival time. For example, the trajectory analyzer 114 may sample the trajectory 144 at one or more trajectory times for comparison with points associated with the arrival time. In various embodiments, the trajectory analyzer 114 may sample any number of trajectory times T T Any number of samples of orbital 144 in can be used. In one or more embodiments, the orbital analyzer 114 may evaluate samples for different time steps, the time steps may be spaced at intervals of 0.5 seconds, as an example, but not limited to. In general, any number of time steps may be used for samples depending on the time interval or the time between time steps. Furthermore, the time interval may be the time of one orbital time T to be sampled next. T It may be the same as or different from it. 【0064】 Figure 1 further illustrates examples of one or more points in spacetime corresponding to a vehicle 140 performing one or more safety procedures, which can be calculated from one or more points within track 144. Specifically, Figure 1 shows the track times T0, T1, and T2 within track 144. T This indicates the location of vehicle 140 in spacetime that would be occupied by vehicle 140 performing safety procedures 140A, 140B, and 140C, respectively. To evaluate a sample of trajectory 144, the trajectory analyzer 114 may compare one or more locations corresponding to the safety procedures of the sample with one or more locations of arrival time. 【0065】 As described herein, the trajectory analyzer 114 may use the safety procedure determiner 110 to determine a safety procedure for the vehicle 140 for a given sample. The safety procedure determiner 110 determines the trajectory time T T Based at least on the state of vehicle 140 in, track time T T Safety procedures for samples in the vehicle may be determined. The state of the vehicle 140 may be determined using the object analyzer 106, which may include at least some information that is the same as or similar to the information determined for other objects described herein, and which may be derived using similar or different approaches. In one or more embodiments, the object analyzer 106 may determine the state of the vehicle 140 using any combination of sensors, such as the GNSS sensor 1358, IMU sensor 1366, speed sensor 1344, and steering sensor 1340 of the vehicle 140. For example, the state information may correspond to readings from one or more sensors. 【0066】 Examples of state information include location, speed, direction (e.g., direction of movement), velocity, acceleration (e.g., scalar, rotation, etc.), attitude (e.g., orientation), and / or other information. The state may encode or represent the position of vehicle 140 in two-dimensional space (e.g., (x,y) coordinates), the unit direction of vehicle 140, and / or the scalar velocity of vehicle 140 at a given time. In some examples, the state may encode or represent additional or alternative information, such as rotational velocity (e.g., yawing), and / or scalar acceleration in any direction. For example, the state of vehicle 140 (x A ) can be parameterized as an m-dimensional state vector, which is expressed in equation (1) as follows: 【number】 【0067】 As an example, state (x A When ) is a 5-dimensional vector (for example, m=5), the state vector can be expressed in equation (2) as follows: x A =[y T dT v] T (2) Here, y is the position of vehicle 140 in two-dimensional space, d is the unit direction vector, and v is the scalar velocity. 【0068】 When the state of vehicle 140 is considered a function of time, the vector is the state trajectory X of vehicle 140. A It can represent (for example, state orbit X) A This refers to each state x of vehicle 140 at each time within the period. A (It can represent or encode) 【0069】 The safety procedure determiner 110 may use the state of the vehicle 140 (and / or other objects in the environment 126) to determine a control model for the vehicle 140. For example, the control model may be expressed in equation (3) as follows: 【number】 【0070】 As such, the control model for vehicle 140 is the state x of vehicle 140 over time t. A The derivative of can be expressed. The explicit differential equation may include control parameters c that can model user inputs such as steering, braking, and acceleration. For example, in some examples, a control model for vehicle 140 can be expressed according to the following equality (4): 【number】 Here, v is the scalar velocity, d is the unit direction vector, a is the scalar acceleration, b is the scalar steering parameter, and d ⊥ This is perpendicular to d and is generated by inverting the coordinates of d and canceling out the first coordinate. In the example of equation (4), the control parameters may be a scalar acceleration a and a scalar steering parameter b. 【0071】 Once the control model is determined, the control policy can be determined (for example, by the safety procedure decisioner 110). For example, the control parameters are the world state x w (or perception of the world state based on sensor data generated by the sensors of the vehicle 140) and may be a function of time t. As such, the control policy is smooth and boundary formed 【number】 It can be a function of the coupled state space of the world and time (where m is the dimension of the state space of vehicle 140). For example, the control policy can be expressed in equation (5) as follows: 【number】 【0072】 Once the control policy is determined, a safety procedure for the vehicle 140 may be determined (for example, by the safety procedure determiner 110). For example, the vehicle 140 follows safety procedure S A It can be assumed that it has. The safety procedure is any starting state x of the vehicle 140 A It may have associated orbitals derived from. In the example, the starting state x of the sample of orbital 144 A This is the sample's orbital time T T The state is either in this state or can correspond to this. The safety procedure is that the actor's transition leads to state x of the actor's state goal (e.g., the final location where the actor can stop). A Since it forms a trajectory, it can represent the track of vehicle 140. The actor state objective may refer to one or more termination conditions of a safety procedure, which involve the state of the actor (e.g., vehicle) performing the safety procedure. 【0073】 In the example in Figure 1, the safety procedure for each sample includes the actor state target when vehicle 140 comes to a stop. For example, trajectory time T0 T Regarding this, the actor's state target is T0+T stop,0 This can be satisfied in T stop,0Therefore, this can represent the time it takes for vehicle 140 to reach the actor's state target from the corresponding state at T0. Similarly, the trajectory time T1 T Regarding this, the actor's state goal is T1+T stop,1 This can be satisfied in, where T stop,1 This can represent the time it takes for vehicle 140 to reach the actor's state target from the corresponding state at T1, and the trajectory time T at T2. T Regarding this, the actor's state goal is T2+T stop,2 This can be satisfied in, where T stop,2 This can represent the time it takes for the vehicle 140 to reach the actor's state target from the corresponding state at T2. The actor's state target includes reaching a complete stop, but other actor state targets may be used, and different actor state targets may be used for different samples. Furthermore, two or more actor state targets may be used for the same sample or trajectory time T in some embodiments. T It can be used for this purpose. 【0074】 In some examples, the actor's state target may be determined by analyzing sensor data received from one or more sensors (e.g., of vehicle 140) to determine the location, orientation, and velocity of an object (or other actor) in the environment 126. Control parameters (e.g., steering, braking, acceleration, etc.), as described herein, may then be determined for vehicle 140, and a set of functions for guiding vehicle 140 to the actor's state target may be determined. 【0075】 A safety procedure can produce a trajectory that changes smoothly depending on its starting state (for example, since a safety procedure can be a continuous deceleration towards a stop). In some examples, safety procedure S A This can be expressed in equation (6) as follows: 【number】 Here, W represents a property of the world (or environment). The safety procedure for vehicle 140 may or may not depend on fixed properties of the world, depending on the embodiment. For example, the safety procedure may not depend on fixed properties of the world, such as road geometry or map. In such an embodiment, the safety procedure may freeze the direction vector (for example, by setting the scalar steering parameter b to zero) and control the acceleration value range [a] until a complete stop. min This may include reaching a complete stop by reducing the speed by a'] (where a min a' is the negative number of the minimum acceleration amount or the maximum braking amount, and a' is a min (A larger negative value). Safety procedure S A This type can be expressed by the following equation (7). 【number】 【0076】 In any example, the safety procedure may include applying the brakes until a complete stop is reached and / or until the actor is moving below a threshold speed. At high speeds, and not limited thereto, the safety procedure may include aligning with the current lane (or with the direction of the road, such as when vehicle 140 is changing lanes) and then bringing it to a complete stop (and thus depending on fixed properties of the world, such as lane markings). In an example, and not limited thereto, at low speeds, the safety procedure may include steering vehicle 140 itself to the edge of the road while decelerating to a stop (and thus depending on fixed properties of the world). For example, one or more neural networks (e.g., convolutional neural networks) may be used to identify the edge of the road and / or to assist in moving vehicle 140 to the edge of the road. In another example, HD map 1322 and / or other map types may be used. In such an example, the HD map 1322 may be received by the network 1390 and / or embedded in the vehicle 140. 【0077】 In yet another example, safety procedures may be modified to provide a specific level of reassurance (e.g., maximum reassurance) to the passengers of vehicle 140, while still ensuring that a collision is avoided or, if a collision is possible, that forces and / or actor velocities remain below a threshold (e.g., minimum deceleration or change of direction). In such an example, a course, trajectory, and / or control sequence may be determined for vehicle 140 as a safety procedure to maximize reassurance and / or minimize forces exerted on passengers, while still ensuring that collisions with other objects (e.g., vehicles, entities, structures, etc.) are avoided and / or that other measures are met. In some examples, such as when a collision is unavoidable or the probability of a collision exceeds a threshold risk level, safety procedures may be modified or configured to minimize the risk of harm to passengers and other entities in vehicle 140 in the event of a collision. 【0078】 Examples of safety procedures are shown for Figures 4A to 4C. For example, for Figure 4A, safety procedure 402 may include actor 404 (or a shape representing actor 404, such as a signed rectangle or polygon) reaching a complete stop while maintaining a low or zero lateral rate of change. For example, in an unstructured environment or when ignoring the fixed properties of the world, safety procedure 402 may include actor 404 (e.g., vehicle 140) driving straight ahead and / or continuing along the current steering circle (which may or may not include the lateral rate of change) until reaching a complete stop. For example, if actor is currently steering at a rightward steering angle, safety procedure 402 may include continuing at this steering angle until reaching a complete stop. If actor is currently steering straight, safety procedure 402 may include continuing straight until reaching a complete stop (as shown, for example, in Figure 4A). 【0079】 In any given example, a safety procedure for vehicle 140 may include a safety margin (for example, in addition to or otherwise, a safety margin specified herein for the size of vehicle 140). For example, for safety procedure 402, the safety margin for the safety procedure may increase as time increases in spacetime from the time associated with the actor's current state. For example, for safety procedure 402, the width W1 of the request set for safety procedure 402 may be smaller than the width W2 of the request set for safety procedure 402. In such an example, since width W1 may correspond to an earlier time, the margin for error may be smaller compared to the time associated with width W2. As a result, the safety margin may increase in spacetime to account for this error. 【0080】 As another example, safety procedure 406 may include actor 404 (or the shape representing actor 404) aligning itself with the road (e.g., abandoning the lane change and aligning itself with the direction of the road, such as being parallel to the lane markings 410A, 410B, and / or 410C) and coming to a complete stop while changing lanes from the first lane 408A to the second lane 408B. In such an example, the safety procedure may take into account fixed properties of the world (e.g., lane markings, road direction, etc.). Safety procedure 406 may still be determined to minimize the rate of lateral change (e.g., relative to the road shape) while abandoning the lane change and realigning actor 404 with the road. 【0081】 As a further example, and with respect to Figure 4C, safety procedure 412 may include actor 404 (or the shape representing actor 404) moving along the road shape to adapt to the curve in the road and coming to a complete stop. For example, if actor 404 is already moving along the road shape and thus taking into account the fixed properties of the world, safety procedure 412 may include continuing to move along the road shape (for example, as defined by lane markings 410D and 410E). Similar to safety procedure 406, safety procedure 412 may be determined to minimize the rate of lateral change while continuing to move along the road shape. 【0082】 Once safety procedures are determined for vehicle 140, the request set determiner 116 may determine the request set for vehicle 140 in environment 126. The request set for actor is determined by the actor's state x A Actor safety procedure S that starts from A When applying this, it may include the actor's occupied trajectory (for example, each of the points in spacetime occupied by the actor as it travels along the trajectory). 【0083】 To determine the request set, the request set determiner 116 may determine the area and / or volume of space occupied by the actor, taking into account the actor's state. For example, if the actor is in n-dimensional real space R n It is conceivable that the actor will move around and occupy the space. In some examples, for simplification, a two-dimensional (2D) space may be used to model a top-down view of the real world. In other examples, a three-dimensional (3D) space may be used. In any example, to determine the request set, vehicle 140 may first determine the occupied set of the actor, the occupied set representing the set of points in space that the actor occupies depending on its state. The occupied set for the actor A This can be determined as follows in equation (8) below. 【number】 If a point in space is within a set occupied by an actor, the actor may decide to occupy that point. 【0084】 To determine each point within the occupied set, the size of the actor (e.g., the actual size of the actor) or a representative size (e.g., the shape around the actor and / or the shape including the actor) may be determined. In some examples, the size or representative size may include an optional safety margin. For vehicle 140, the size of vehicle 140 may be known (e.g., based on calibration information, vehicle information, vehicle type and model, and / or other parameters). In some examples, to determine the size of the actor, a shape (e.g., a predetermined or signed shape such as a square, polygon, bounding box, cube, circle, ellipse, or oblong) may be fitted around the actor (e.g., to at least include the actor), and the size of the actor may be determined to be the size of a predetermined shape (e.g., including a safety margin, as described herein in some examples). For example, the shape may be a 2D shape (e.g., a rectangle or circle) that functions as a bounding box at least partially surrounding the actor. In other examples, the shape may be a 3D shape (e.g., a cube) that functions as a boundary cube that at least partially encloses the actor. In any example, the size of the vehicle is the size of the occupied set (o A It can be used by the request set determiner 116 to determine a point in space (e.g., (x, y) coordinates) occupied by an actor as part of the set. 【0085】 In the example shown in Figure 1, a shape 132 is provided that may be used to determine one or more points in spacetime of a set of requirements for vehicle 140. Shape 132 may include a minimum or boundary rectangle around at least the body of vehicle 140. In other examples, a hexagon or other shape may be used. 【0086】 In some examples, the size of an actor, and therefore the representative shape corresponding to the actor's size, may be determined such that the size and / or shape completely encloses the actor in at least two dimensions (e.g., horizontally and vertically). By completely enclosing the actor (along with additional safety margins in the examples), the occupied set, occupied trajectory, and therefore the requested set are more likely to represent the actual location in space that the actor should occupy when performing safety procedures. 【0087】 Once the occupied sets are determined, the request set determiner 116 determines the occupied trajectory (O) for each actor. A ) can be determined. An occupied orbit may include a set of points in spacetime that an actor will occupy over time according to its orbit. For example, an occupied orbit (O A ) can be determined as follows in the following equation (9): 【number】 【0088】 Occupied orbits are subject to the actor's safety procedures (S A When applying the request set (C A ) may include. Any location determined to be within the actor's request set is a location in spacetime where the actor may need to maintain the integrity of its safety procedures. Request set (C A ) can be determined in the following equation (10) as follows: 【number】 Here, equation (10) is that the actor is in state x AThe requirements set may represent the occupied trajectories of an actor that arise when applying an actor's safety procedure starting from a certain point. In some examples, the requirements set may represent each combination or aggregation of occupied trajectories that arise from applying an actor's safety procedure with different parameters. For example, a safety procedure may be applied with a maximum braking profile, a minimum braking profile, a braking profile between the maximum and minimum, a maximum steering profile, a minimum steering profile, a steering profile between the maximum and minimum, and / or in other ways. In such examples, the requirements set may include occupied trajectories for any number of different applications of the combined or aggregated safety procedure (e.g., for each different application). 【0089】 As a first example, the set of requirements may represent a safety procedure with a first braking profile that leads to a complete stop more quickly than the second braking profile, and a safety procedure with a second braking profile that leads to a complete stop more slowly (for example, more slowly than the first braking profile) while still avoiding a collision. In such an example, thresholds or boundaries may be set and / or determined to define the first braking profile (e.g., a defined maximum or upper limit braking profile) and the second braking profile (e.g., a defined minimum or lower limit braking profile). In such an example, the set of requirements may represent the spatiotemporal locations occupied by the actor by the safety procedure applied with the first braking profile and the second braking profile, and the spatiotemporal locations occupied by the actor located between the first and second braking profiles (e.g., shown and described in Figure 5C). Depending on the particular embodiment, the thresholds or boundaries may differ to generate the set of requirements. 【0090】 Thresholds or boundaries may be defined not as limitations, but as percentages (e.g., brake intensity percentage) and / or as time (e.g., time to a complete stop based on current speed or velocity). For example, a first braking profile may include 95% brake intensity, and a second braking profile may include 85% brake intensity. As another example, if vehicle 140 is traveling at 97 kilometers per hour (km mph) (60 miles per hour (mph)), the first braking profile may include stopping within 3 seconds (e.g., slowing down by an average of 32 km mph (20 mph)), while the second braking profile may include stopping within 4 seconds (e.g., slowing down by an average of 24 km mph (15 mph)). A speed factor may be used in some examples to determine the time. In such examples, the first braking profile is defined by the current state x of vehicle 140. A Based on this, the second braking profile may include 1 second for every 16 km mph (10 mph) the vehicle 140 travels, and the current state of the vehicle 140 x A Based on this, it may include 1 second for every 24 km mph (15 mph) that the vehicle 140 travels. Continuing with the previous example where the vehicle 140 is traveling at 97 km mph (60 mph), the first braking profile may include coming to a stop within 3 seconds, and the second braking profile may include coming to a stop within 4 seconds. 【0091】 As another example, the requirement set may represent a safety procedure with a first steering profile (e.g., a defined maximum steering profile) for reaching a lateral location (e.g., the edge of the road, or alignment with the road) more quickly than a second steering profile, and a safety procedure with a second steering profile for reaching a lateral location more slowly than a first steering profile. In such an example, thresholds or boundaries may be set and / or determined to define a first steering profile (e.g., a defined maximum or upper limit steering profile) and a second braking profile (e.g., a defined minimum or lower limit steering profile). In such an example, the requirement set may represent spatiotemporal locations occupied by an actor by the safety procedures applied with the first and second steering profiles, and spatiotemporal locations occupied by an actor located between the first and second steering profiles. Depending on the embodiment, the thresholds or boundaries for generating the requirement set may differ. 【0092】 Similar to the threshold or boundary of the braking profile, the steering profile can also be based on percentages (e.g., percentage of steering angle intensity) and / or time (e.g., time to reach a lateral position). 【0093】 As a simplified example of the requirements set and with respect to Figure 5A, vehicle 140 may be driving toward a stationary object 502 (where the direction of driving is represented in one dimension in the spatiotemporal plot of Figure 5A). Vehicle 140 may have an occupied trajectory 504 which will result in a collision with the stationary object 502 if no safety procedure is performed (for example, vehicle 140 continues to drive along the same path at the same speed until it collides with the object, rather than performing a safety procedure). The stationary object 502 is fixed along the spatial dimension in the spatiotemporal plot because it does not move. In this example, a boundary polygon may be used to represent the size of vehicle 140 (for example, a boundary polygon extending from the front to the back of vehicle 140 between boundary lines 516A and 516B). 【0094】 To consider the situation in Figure 5A, safety procedures may be implemented, such as applying the brakes to bring the vehicle 140 to a complete stop before it collides with the stationary object 502. For example, in Figures 5A and 5B, the trajectory generator may generate a trajectory 506 represented by a corresponding set of requirements, such as that determined by the requirements set determiner 116, which corresponds to the safety procedure, and the vehicle 140 may implement the safety procedure at time T1 so that the vehicle 140 stops just before it collides with the stationary object 502. The trajectory 506 (for example, representing the set of requirements for the trajectory of the safety procedure) may be projected permanently into space until the trajectory 506 is determined to intersect the stationary object 502 (for example, at time T1), and the vehicle 140 may implement the safety procedure by applying the brakes with an intensity corresponding to the safety procedure (for example, an intensity that would cause the vehicle 140 to stop before it collides with the stationary object 502, with some margin for error, depending on the embodiment). In some examples, track 506 may correspond to a second braking profile (e.g., a defined minimum or lower limit braking profile). 【0095】 As another example, and with respect to Figure 5C, generated trajectories 506 and 508 are represented by corresponding request sets, as determined by the request set determiner 116, corresponding to safety procedures performed using two different braking profiles (for example, the request sets may include each of the points in spacetime occupied by the vehicle 140 if the vehicle 140 navigated both the first trajectory 506 and the second trajectory 508, and / or the trajectory between them). For example, trajectory 506 may include a first braking profile (for example, a defined minimum or lower limit braking profile), and trajectory 508 may include a second braking profile (for example, a defined maximum or upper limit braking profile). In this example, the vehicle 140 may perform the safety procedure immediately before trajectory 506 collides with the stationary object 502 (for example, at time T1). Therefore, trajectories 506 and / or 508 (as represented by the corresponding set of requirements) can be projected into spacetime until trajectory 506 (or, in some instances, 508) is determined so as to intersect the stationary object 502, and then the vehicle 140 can perform the safety procedure by acting the brakes with an intensity corresponding to the selected braking profile for the safety procedure. 【0096】 A property of the requirements set is that when a safety procedure is applied, the requirements set may not increase over time. As such, the minimum (or safety) braking profile and maximum braking profile for a safety procedure may be defined such that the requirements set does not increase (although it may decrease). As such, the requirements set cannot increase at each time step or stage in the implementation of the safety procedure. Consequently, it is preferable that the requirements set does not change when the safety procedure is first implemented and at the end of the safety procedure (e.g., at a complete stop). 【0097】 As another example, and relating to Figure 5D, within the environment 126, there are a vehicle 140, a first object 540A (e.g., the vehicle in this example), and a second object 540B (e.g., the vehicle in this example). In this example, the trajectory may occupy three-dimensional space (e.g., volume) in spacetime within the environment 126. As such, the trajectory may include longitudinal distance (e.g., braking or stopping distance), lateral change (e.g., steering change), and / or vertical space (e.g., from the ground plane to the top of a boundary polygon or other shape representing the set occupied by the actor) occupied by the actor, assuming the actor performs its respective safety procedures. As such, a trajectory can be analyzed as an actual volume with increasing length depending on velocity (for example, as an actor moves faster, the trajectory and corresponding points in spacetime included in the occupied set become longer), and the actor can be characterized as driving around in proportion to these volumes accompanying the actor (e.g., protruding from the actor), while performing collision analyses on the volume (e.g., the trajectory) rather than on these actual shapes. As a result, by ensuring there are no collisions in the spacetime volume, a guarantee of no collisions in real space can be induced. This can be beneficial because, since actors have inertia, avoiding collisions between actual physical objects in real space requires insight, and if physical overlap occurs, this may already be too late. However, in spacetime, if an intersection, overlap, or quasi-intersection or overlap between volumes in spacetime is determined, the volume or trajectory may be considered frozen, and since there are lateral and longitudinal dimensions and no shared shape is permitted between the vehicle 140 and the object, the vehicle 140 may avoid collisions (or at least, the actions of other actors are outside the control of the vehicle 140 and therefore cannot contribute to the possibility of a collision). 【0098】 In such examples, vehicle 140 may generate vehicle-occupied track 520 representing safety procedures for vehicle 140 (applied to the extent of the profile in some examples), object-occupied track 522 representing safety procedures for the first object 540A, and object-occupied track 524 representing safety procedures for the second object 540B. In the illustration of Figure 5D, there is no overlap or intersection, or partial overlap or intersection, between any of the tracks 520, 522, and 524. As such, at the time shown in Figure 5D, neither vehicle 140, the first object 540A, nor the second object 540B can perform these safety procedures. However, if one of tracks 520, 522, or 524 overlaps or nearly overlaps with another of tracks 520, 522, or 524, the actors involved in the overlap or near-overlap should be expected to perform these safety procedures (for example, vehicle 140, if involved, should perform the safety procedures and should expect the other actors to perform each of these safety procedures to avoid a collision). 【0099】 The locations in spacetime occupied by the projection of the trajectory 520 may include a set of requirements for the vehicle 140 when performing safety procedures. As described herein with respect to Figure 4C, the vehicle-occupied trajectory 520 may include a first braking profile, a second braking profile, a braking profile of another threshold, and / or a combination thereof. Similarly, the locations in spacetime occupied by the projection of the object-occupied trajectory 522 may include a set of requirements for the first object 540A, and the locations in spacetime occupied by the projection of the object-occupied trajectory 524 may include a set of requirements for the second object 540B. 【0100】 In some instances, latency, discretization, and / or reaction time may be at least a small practical limit that can be modeled. For example, vehicle 140 may address limitations of perception, or more precisely, perception and action, in the sense that when an actor takes action, the actor necessarily bases their actions on perceptions that are not entirely present (e.g., with a time delay). As a result, when an actor takes action, the actor may base their actions on perceptions of the world at some initial point in time. For example, an actor (e.g., a human actor, a manually driven vehicle, or a pedestrian) may have some reaction time (e.g., based on a lack of attention, such as looking at a phone or reaching for something) before realizing that a potential collision may occur. In such instances, vehicle 140 may take this reaction time into account. In other instances, an actor such as a vehicle may have latency or lag between when a command is received and when the action actually occurs. Latency or lag may be known (for example, after identifying the vehicle type) or perceived (for example, using one or more neural networks). In such an example, vehicle 140 may take this latency or lag into account. In any example, the shape of the trajectory of the request set for an actor (for example, vehicle 140 and / or object) (e.g., length, width, height, etc.) may be adjusted to take latency, lag, or reaction time into account (e.g., be extended, widened, etc.). 【0101】 In some examples, the amount of latency can be assumed to be Δt. To account for Δt, in some examples, the transferred set of actor A up to the time interval Δt (Φ A (x A ,Δt)) may, in some cases, be in state x A Subsequently, a worst-case transfer prediction can be used, such that it is the set of all states that can be reached in the time interval Δt. The transferred set Θ of the collection of actors up to the time interval Δt can be the union of the transferred sets of all actors in Θ, as expressed by equation (11) below. Φ(Θ,Δt)=∪ A∈Θ Φ A (x A ,Δt) (11) 【0102】 An actor may typically possess the ability to predict its own state better than other actors. Specifically, in the control system of vehicle 140, a previously sent actual command sequence may be known, providing the ability to predict where the actor itself will be when the now intended actuation command (e.g., delivered to the actuation components of vehicle 140) is actually issued. For practical purposes, this can effectively result in a deterministic transfer, allowing the transferred set to include only one point that further results in a single actor state. Generally, the transfer mechanism can be a non-deterministic transfer, resulting in a set of states. In some instances, a non-deterministic transfer of the actor itself may be used, requiring all possible states, while in other instances, a deterministic transfer of the actor itself may be assumed to reduce complexity. 【0103】 The result could be a control policy for the transferred actor, implicitly assuming that the state parameterization is updated with predictions based on all actuation commands in the queue up to the currently intended actuation command. Depending on these assumptions, the control commands may be applied to possible actor states, with only delays being information about other actors (e.g., objects other than vehicle 140). 【0104】 The transmitted control policy can be safe in the present time with respect to the location where the actor's perceived collection has moved, despite the latency limits between perception and action. This, too, may be a direct consequence of the worst-case assumption and a definition of a safe control policy. Since it is assumed that all possible constraints exist (for example, from locations that other actors can reach in the environment at the time the control of vehicle 140 is performed), vehicle 140 can therefore be subject to all relevant constraints. 【0105】 Furthermore, vehicle 140 may combine latency recognition with visibility recognition and use this information to avoid entering an irrational state. For example, consider a set (Φ(V,Δt)∪(Φ(Λ,Δt)∩Ψ)) where V, Λ, and Ψ are sets of visible, invisible, and moderate actors, respectively. Firstly, visibility may be considered to provide a complete collection representing all actors in the world (visible and invisible, as described herein) that it may be desirable to consider at a single point in time. Next, latency may be considered for this complete world representation by transferring the set of actors. Finally, irrational actors may be excluded from the transferred set of invisible actors. In some examples, irrational actors may be excluded before transfer, but this should not make it possible to consider irrational actors that would make the state reasonable during transfer. Furthermore, while irrational invisible actors can be excluded, irrational visible actors cannot be excluded, as in some instances, removing actually perceived actors would not result in an accurate world state. 【0106】 As described herein, to evaluate a sample of orbit 144, the orbital analyzer 114 may compare one or more points corresponding to the safety procedure of the sample with one or more points of arrival time. For example, orbital time T0 T To evaluate the corresponding sample, the trajectory analyzer 114 may compare one or more points from the requested set determined for safety procedure 140A with one or more arrival times determined for object 128A (and / or other objects). In one or more embodiments, the arrival time compared to the points from the requested set may be the arrival time of the location of the points from the requested set. 【0107】 If the arrival time is less than (or equal to) the time from the requested set to the location, the trajectory analyzer 114 may determine a conflict between the arrival time of the trajectory 144 and the arrival time of the sample. For example, the trajectory analyzer 114 may determine, at least based on the determination that the arrival time is less than the time from the requested set to the location, that the vehicle state of the sample on the trajectory should be unsafe. This may indicate, for example, that if object 128A arrives at the location at the arrival time, and if vehicle 140 is in the state of the sample on the trajectory 144, then vehicle 140 should be unable to avoid a collision with object 128A (and / or other objects captured by the arrival time). 【0108】 If the arrival time is greater than the time from the requested set to the location, the trajectory analyzer 114 may determine that there is no conflict between the arrival time of trajectory 144 and the arrival time of the sample. For example, the trajectory analyzer 114 may determine, at least based on the determination that the arrival time is greater than the time from the requested set to the location, that the vehicle state of the sample on the trajectory should be safe. This may indicate, for example, that if object 128A arrives at the location at the arrival time, and if vehicle 140 is in the state of the sample on trajectory 144, then vehicle 140 should be able to avoid a collision with object 128A (and / or other objects captured by the arrival time). 【0109】 For a given sample of trajectories, the trajectory analyzer 114 can compare any number of points in the sample's request set with any number of arrival times of locations within the environment 126. Since the vehicle 140 occupies many points in space at a given time, the trajectory analyzer 114 can compare the arrival times of multiple points in the request set with their arrival times to determine if there is any competition at a given time. As an example rather than an limitation, the points may be on a shape used by the request set determiner 116 to determine the request set. For example, point P on shape 132 in Figure 1. 1a , P 1b , and P 1cThis can be used to evaluate a sample of trajectories 144 of the requested set at a given time. In one or more embodiments, if there is no competition at the boundary, it can be assumed that there should also be no competition inside, so only points on the boundary of the shape can be used. 【0110】 Furthermore, in some or more embodiments, only a point on one side of the shape may be used for evaluation at a given time. For example, the side of vehicle 140 corresponding to the direction of movement of vehicle 140. Thus, the trajectory analyzer 114 may determine that vehicle 140 is being driven in reverse and use a point from the rear of vehicle 140 rather than the front. Moreover, in some examples, one or more points may correspond to a plane perpendicular to the direction of movement. For example, if vehicle 140 turns to the right, one or more points may correspond to the front right side of vehicle 140 (closer to the front to ensure that at least the leading edge of the vehicle is inspected). Furthermore, although the points lie on the line segment in Figure 1, the points may be on both sides of a hexagon, semicircle, or other shape. 【0111】 In general, the trajectory analyzer 114 may evaluate samples at several different times within a set of claims. In one or more embodiments, the trajectory analyzer 114 may be configured to avoid or eliminate gaps between adjacent claim sets of samples being evaluated for competition with arrival time. For example, in Figure 1, the areas occupied by the vehicle 140 being evaluated are in contact (e.g., in the temporal order of the corresponding samples) so that there are no gaps. In one or more embodiments, gaps may be reduced or avoided by setting the time between time steps such that the state objective of the actor in each safety procedure is satisfied at a time greater than or equal to the next time step being sampled. For example, the time step may be calculated from the current speed of the vehicle 140, i.e., from the speed at a point in the trajectory 144 being sampled, and / or may be configured statically or dynamically so that this condition may or will be satisfied. In embodiments, if a gap is identified, interpolation (e.g., linear) between sampled points may be used to account for any possible gaps between areas (e.g., set of claims). For example, the spatiotemporal location of a set of requirements used for evaluation may be interpolated from one or more other locations (e.g., to the end or boundary of a gap) to evaluate the gap between adjacent sets of requirements. 【0112】 Referring now to Figure 6, Figure 6 is an illustration including examples of trajectories, safety procedures that may be activated in the trajectory, and points that may be used to evaluate the trajectory, according to some embodiments of the present disclosure. Specifically, Figure 6 shows safety procedures 140A, 140B, and 140C individually labeled, from time T0 to T N Orbit 144 is shown along with the safety procedure that can be activated by (where N is an integer) time T. TFor a sample at =T0, for example, at the start of the track 144 and / or the position of the current vehicle 140, the track 144 may evaluate one or more points in spacetime for multiple times throughout the safety procedure 140A. For simplification, one or more points for each time can be discussed with reference to a single point. However, the evaluation may include multiple points (for example, by the shape 132 described herein). As shown in the figure, points P0 and P 0,1 From P 0,M Up to (where M is an integer), the sample corresponding to T0 can be evaluated. In one or more embodiments, the trajectory analyzer 114 may determine sample conflicts based at least on determining a conflict at least one of the points being evaluated for the sample. For example, a conflict may be determined by determining when any of the points conflict with the corresponding arrival time. 【0113】 In the example in Figure 1, multiple points may be evaluated for the sample corresponding to T0 in order to avoid gaps in coverage. However, for subsequent time steps, only the point at the tip of the safety procedure (for example, if the actor's state objective is satisfied) may be evaluated. In Figure 6, the point at the tip of the safety procedure is from point P0 to P NAs shown, sampling only the tip may be sufficient because the tip can correspond to the most limited point in the safety procedure (for example, relative to the most recent safe arrival time). For example, if a collision can be avoided at the end of the safety procedure for a sample, it can be assumed that it can be avoided at any earlier time in the safety procedure. Therefore, checking for conflicts with respect to safety procedures may be computationally efficient because the number of comparisons may be linear with the number of safety procedures being evaluated. Furthermore, each of these comparisons can be computed efficiently in parallel on a GPU. Multiple points may be evaluated for a sample corresponding to T0, but in other instances, a single point (e.g., point P0) may be sampled similarly, and / or the evaluated points may leave gaps in acceptable coverage. For example, gaps in coverage may be acceptable because vehicle 140 may have evaluated points for one or more trajectories of frames before covering the gap. 【0114】 As described herein, the track analyzer 114 may analyze a plurality of tracks, including track 144, to select a track from there for use in controlling the vehicle 140. Instances, but not limited to them, one or more tracks may be represented using one or more lateral paths and one or more speed profiles. Specifically, a track may comprise a lateral path combined with a vehicle speed profile across the lateral path. Figure 7 is a diagram used to illustrate instances of lateral paths and speed profiles according to some embodiments of the present disclosure. The set of tracks may be determined based at least on nominal rails or lanes and / or roadway outer lines from a world model of the vehicle 140, as further described herein with respect to Figure 8, and at least on one or more selection actions for the vehicle 140. In Figure 7, the nominal rails can correspond to the right lane where the vehicle 140 may be located. As illustrated, a series of cones 724 indicate obstacles to be avoided. Vehicle 140 can generate 11 different laterally changing options, or lateral paths, associated with a nominal rail. The options include paths 702, 704, 706, 708, 710, 712, 714, 716, 718, 720, and 722. Any combination of paths and speed profiles can be used to form a track. 【0115】 Multiple trajectories evaluated by the trajectory analyzer 114 for trajectory selection can correspond to various combinations of course and speed profile. In one or more embodiments, the comparison between the sample and arrival time is performed using multiple trajectories, with respect to the lateral course Np and speed profile N. s , and time step N t These can be linear and can each be computed in parallel on a GPU. The initial sample of each orbit is (for example, the orbital time T of each orbit) TWhen the speed profiles can be identical for each speed profile sharing the same lateral path (at T0), the trajectory analyzer 114 can evaluate this sample once for each lateral path and apply the results to the respective corresponding trajectories. As such, the complexity of the evaluation is O(N). p *N s *N t ) is possible. 【0116】 In some embodiments, the orbital analyzer 114 may terminate the orbital analysis when a conflict is detected and / or based on the location where this conflict is detected. In other embodiments, the orbital analyzer 114 may always analyze a sample of each orbital. 【0117】 As described herein, based on the results of comparisons of a sample of tracks, a track may be scored (for example, for ranking against other potential tracks), excluded from consideration, or otherwise considered for the control of the vehicle 140. The track analyzer 114 may use a variety of possible factors or measures to determine the score of a track and / or whether a track should be excluded from consideration. If a track is excluded from consideration based on the satisfaction of one or more measures, one or more measures may be considered a hard constraint on the track. If a track's score is reduced or lowered based on the satisfaction of one or more measures (assuming a lower score means the track is less likely to be selected), one or more measures may be considered a soft constraint on the track. 【0118】 In one or more embodiments, constraints on the trajectory may be based at least on the distance and / or elapsed time to detected competition along the trajectory (e.g., to the start of the sample, or to a point considered equivalent to the arrival time) (e.g., competition at the maximum distance and / or elapsed time from the start of the trajectory). For example, assuming there is detected competition for a trajectory, a higher score (or weight used to calculate the score) indicates that the competition is further along the trajectory and / or the elapsed time to the competition is greater. Thus, the trajectory time T of trajectory 144 TIf the orbital analyzer 114 determines that there is competition for =T2, the score of orbit 144 is the orbital time T of orbit 144. T The score may be higher than when the trajectory analyzer 114 determines the competition for T1. If multiple competitions are detected, the score may be based on at least the competition with the shortest distance, the minimum elapsed time, and / or the amount of competition detected. 【0119】 In at least one embodiment, depending on whether one or more conditions are met, one or more measures may be used as hard or soft constraints. For example, due to competition for a sample, a trajectory may be excluded from consideration for controlling the vehicle 140, at least on the basis that the distance and / or elapsed time to the competition (e.g., the detected competition closest to the start of the trajectory) is below a threshold. Due to competition, at least on the basis that the distance and / or elapsed time to the competition (e.g., the detected competition closest to the start of the trajectory) is greater than a threshold, the competition may be used as a soft constraint for trajectory selection (e.g., by deducting points from the trajectory score). For example, when the trajectory analyzer 114 detects a competition, the trajectory analyzer 114 may compare the distance and / or elapsed time to the competition with one or more thresholds to determine whether the competition should be used as a soft or hard constraint, and / or how much the competition should, or should, reduce the trajectory score. Using this approach, the later the competition occurs in the trajectory, the more acceptable or acceptable it may become. Therefore, the trajectory can still be selected for the control of the vehicle 140, and replanning the trajectory or selecting a different trajectory before reaching a competing sample or time step can ensure that the control objective is satisfied (e.g., collision is avoided). 【0120】 As described herein, arrival times evaluated by the trajectory analyzer 114 for the state of the world, which can be evaluated for multiple trajectories, may be pre-calculated and stored in an image or texture for the entire scene perceived by the vehicle 140, or a subset of the scene for which the vehicle 140 may, in some cases, be during the period of the planned horizon. Values ​​corresponding to arrival times in the image or texture may be calculated using distance transformations or in parallel on the GPU. Points corresponding to the trajectories may then be evaluated for conflicts with the image or texture, for example, using texture reads and inequality checks. In other examples, arrival times may be calculated on the fly or as needed during trajectory evaluation and / or stored in an image or texture. 【0121】 Referring here to Figure 8, which is an illustration of an exemplary behavior planning architecture for an autonomous vehicle according to several embodiments of the present disclosure. It should be understood that this and other configurations described herein are presented only as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components, or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. 【0122】 The behavior planning architecture 800 may use various planning components, each having a different planning horizon. In one embodiment, the various planning components include a route planner 810, a lane planner 820, and a behavior planner 840. Each component may produce results and / or a set of results, which may be transmitted to the next component in the planning pipeline. The route planner 810 may transmit a route set to the lane planner 820, and the lane planner 820 may transmit the lane plan to the behavior planner 840. The behavior planner 840 may develop and evaluate multiple trajectories, and the selected or preferred trajectory may be sent from the behavior planner to a motion controller 880, which is responsible for implementing the planned trajectory. 【0123】 Route Planner 810 may use geographical information covering geographical areas (e.g., towns, metropolitan areas, states, countries, regions, etc.) in its embodiments, but with relatively little detail. The planned horizon for Route Planner may be the entire area between the starting point (e.g., current location) and the destination point (e.g., final location). 【0124】 The route planner 810 may be designed to be modularly separable from other planning modules. The structural design may efficiently support long-distance routes. The route planner 810 may pass the resulting route plan and some basic state information to the lane planner 820. The route plan may include one or more GPS traces that have been determined to be potential approximate routes to be used to get from the current location to a waypoint or parking area. The GPS points in the route plan may also have an expected time reward for them (or, if preferred, the time reward may be expressed as the expected time remaining to the destination). The GPS traces in the route plan may provide the lane planner 820 with approximate guidance rather than exact assertions about a particular lane or road. The route planner 810 may provide multiple routes or route variations that are included in each other's threshold travel times, and in embodiments, each route may be associated with an estimated travel time. In one embodiment, each route segment is assigned a travel time, which can be used to provide the lane planner 820 and other components with information that can be used to calculate the cost of losing the exit and being forced to take an alternative route. In some embodiments, the travel time can also be calculated by other components such as the lane planner 820, and the travel time can be calculated using distance and speed limits, but can also be refined with real-time traffic data, construction data, weather data, and / or similar. 【0125】 The lane planner 820 can generate a lane plan based on one or more routes received from the route planner 810. The lane planner 820 may use a second geographic map that contains more detail than what is found in the map used by the route planner. The second geographic map may cover a smaller geographic area than the map used by the route planner. For example, a smaller geographic area may be less than 129.5 square kilometers (50 square miles). The lane plan may take the form of an annotated lane graph, which may have a plan horizon of several miles. The lane graph may represent the lanes available to an autonomous vehicle on a route (e.g., a road). Annotations on the lane graph may indicate time scores at various locations on the lane graph. For example, time scores may be provided at every 10 meters, every 20 meters, or at several other intervals on the lane graph. The time scores may be described as a continuous time reward for potential future locations. A larger or better reward may be associated with a faster route to the route destination, or in other embodiments, a lower score may indicate a better or faster route. The reward scale can be in units of time, such as seconds. 【0126】 The time score conveys the overall value of getting an autonomous vehicle to a specific point on the lane graph. The behavior planner 840 can then consider these scores when evaluating various trajectories. The output of the lane planner 820 may be represented as an expected time reward encoded on a larger lane graph associated with the local lane graph of the lane. This encoding can provide more flexibility than just the correct order or recommendation of which lane to be in, as it allows the behavior planner 840 to gauge the value of being in a particular lane relative to the difficulty of maneuvering to get there. The larger lane graph may correspond to a lightweight representation of lane positions and the expected time to transition between them. The larger lane graph may be derived from map data, which is also linked to the local lane graph of the lane. 【0127】 The behavior planner 840 may consider one or more actions and proceed through one or more iterations of hypothesis generation and evaluation of more detailed implementation forms of these actions. The more detailed implementation forms may be represented as an accurate motion plan (a time-series attitude trajectory over several seconds into the future) and may be evaluated by the trajectory analyzer 114. The behavior planner 840 may include an action generator 842, a length pre-limiter 850, a hypothesis generator 860, and an action plan selector 870. 【0128】 The behavior planner 840 may have a smaller planned horizon than the route planner 810 or the lane planner 820. Generally, the behavior planner 840 may plan the vehicle's movement for the next few seconds. The distance covered may vary depending on the speed at which the vehicle is traveling. For example, the behavior planner 840 may have a planned horizon of 50m, 100m, 200m, 300m, or similar. The behavior planner 840 may use a map that shows a high level of detail covering a small area. For example, the small area may be smaller than the area in the map used by the lane planner 820 or the route planner 810. More detail may include objects detected by sensors associated with the autonomous vehicle (e.g., other vehicles, pedestrians). In one embodiment, at least some of these objects may not be in the information available to the lane planner 820. The selected actual trajectory may only affect the autonomous vehicle's motion for a short period, such as one second. The new trajectory can be continuously evaluated and implemented to adapt to changing conditions in the autonomous vehicle environment (e.g., moving vehicles, moving pedestrians). 【0129】 The calculation flow for each action evaluated by the behavior planner 840 may be initiated by the action generator 842. The action generator 842 may first select one or more desirable actions for the autonomous vehicle to take. The actions may be based on an annotated lane graph provided by the lane planner 820. The selected one or more actions may be based on achieving the best outcome indicated by the time scores included in the annotated lane graph. Possible actions include, but are not limited to, lane following, lane change speed adjustment, overtaking, lane change push, shoulder stop, parking patrol, final stage of parking, summoning, and lane limping. As with the components discussed earlier, the action generator 842 may select or score multiple actions. In some embodiments, actions may be represented as nominal rails, and / or each action may be associated with multiple nominal rails. At the end of the processing carried out by the action generator 842, each action may be associated with nominal rails and / or roadway outer lines and lanes from the map. 【0130】 The length pre-limiter 850 may receive nominal rails corresponding to actions generated by the action generator 842. Each of these nominal rails may be evaluated by the length pre-limiter 850. The length pre-limiter 850 may consider basic length constraints for each lane (or each nominal rail). Different constraints may be considered by different components such as those relating to curve speed conforming, speed restrictions, distance holding, safe braking, and / or yielding. The output of the length pre-limiter 850 may include acceleration constraints that limit the speed associated with each of the nominal rails / lanes that arise as inputs. As such, the length pre-limiter 850 may expand lanes / rails by acceleration limits, speed constraints, or distance constraints. 【0131】 The combined output from the various components of the length pre-limiter 850 can be a nominal rail limited to multiple lengths. The rail generated by the action generator 842 can be combined with multiple different acceleration plans to generate many tracks along the rail. The nominal rail limited to multiple lengths includes rails that satisfy various constraints calculated by the length pre-limiter 850. The nominal rail limited to multiple lengths can be passed to the hypothesis generator 860 for further refinement. 【0132】 Motion planning can proceed through evaluation by the hypothesis generator 860 and the action plan selector 870. The action plan selector 870 may use the trajectory analyzer 114 to evaluate the quality of one or more given trajectories (e.g., a fully planned future attitude for an autonomous vehicle), and the hypothesis generator 860 may provide promising or reasonable trajectories because the space for all trajectories is too large to search. 【0133】 According to the embodiments, a high degree of variability can be used for hypothesis generation. In some embodiments, the hypothesis generator 860 includes a path-fan generator that starts with a nominal rail of limited length and generates laterally varying options around it while maintaining its curvature. This is tailored to the intention of moving along a lane while avoiding static and dynamic obstacles. Variations in length within the length limit may be added to each lateral option, creating a 2D grid of tracks. This approach can be used for standard lane-following cases, as well as for actions such as parking patrols, the final stage of parking, summoning, and lane changes, which can be constructed (as examples, and not limited to). 【0134】 The hypothesis generator 860 may also search for free-form paths or linear quadrant regulator (LQR) / obstacle-aware MPC controls, taking an initial path (e.g., a nominal rail) and iteratively adjusting it to stay away from obstacles, using other less structured approaches such as dynamic programming. This can be done for static scenes that ignore motion, which the motion planner then trusts will see and avoid dangerous motion-inducing situations, or by directly using a set of requests during the search. 【0135】 Each hypothetical trajectory can be evaluated by the trajectory analyzer 114 to identify the highest quality or optimal trajectory. Quality can involve many considerations, but can be quantified by a score as described herein. When calculating the score, different quality can be weighted accordingly. Ideal driving can be traced back to various objectives. Objectives may include, without limitation, maximizing safety and progress (e.g., reaching the destination with minimal consumption of resources such as time, money, fuel, and wear). Other objectives may include maximizing collision safety (e.g., obstacle / object avoidance), staying within the lane (e.g., course), and operating in accordance with applicable rules and conventions (e.g., waiting conditions for yielding). 【0136】 The score can be normalized by time. Consecutive time rewards for potential future locations can be used directly in the score and as starting points, while other components contributing to the score may add time as penalties. For example, a predicted collision or competition with a trajectory may add 100 hours, representing a substantial adjustment or offset (e.g., a penalty), preventing the trajectory from being selected. Fewer but still undesirable conditions may result in smaller adjustments. Adjustments can be scaled or normalized according to the route distance or estimated time taken. In this way, adjustments resulting from undesirable features of a route may add more time to longer routes and less time to shorter routes. Normalization can occur by calculating the adjustment as a percentage of the time to the destination. For example, several conditions on the route may result in an adjustment of 2% of the estimated time remaining to reach the destination. 【0137】 The trajectory analyzer 114 may also be configured to prefer staying near the nominal rail or between the edges of the lane under consideration. These choices can be used to calculate the score. When following a lane, it may be very preferable not to extend beyond the edge of the lane line, even if the outer edge of the roadway line is not a physical obstacle. However, this may be preferable if extending beyond the edge of the lane line is the only reasonable way to avoid a collision. This, however, is different from making a planned and controlled lane change, specifically because in a planned and controlled lane change there is time to initiate an indicator signal and give others time to notice the signal. This allows the trajectory analyzer 114 to consider the trajectories in the context of lane following and lane changes, respectively, even if the trajectories would otherwise be the same trajectory. The score of a trajectory can be increased the more it follows more strictly along the center of the lane. The score can be decreased when the trajectory crosses a lane boundary. The amount of the score to decrease may depend on the context of the lane being crossed. For example, crossing an intermittent line may incur a smaller penalty than crossing a solid line or a double solid line. Crossing into oncoming traffic can result in a very large penalty. 【0138】 The trajectory analyzer 114 can analyze the trajectory of an obstacle or object relative to a contender. The analysis may include anticipating intersections with a set of requirements. Instead of considering whether there is a collision between physical bodies, motion planning according to embodiments of the disclosure may now or in the future consider whether there is an intersection with one or more sets of requirements. This may provide a way of handling how to upgrade static motion planning to environments with contenders moving at speeds that do not allow for near instantaneous stopping. As described herein, in one or more embodiments, a set of requirements may refer to a shape in spacetime that an actor will trace while attempting to stop safely and align alongside the road. The actor may virtually "request" this set in order to maintain collision safety. Even if the bodies do not intersect, an intersection with one or more sets of requirements should be less tolerable while maintaining some form of controlled safety. Conversely, if a set of requirements is respected, the actor can remain within it. If a set of requirements is respected, the set of requirements is unfolded in spacetime and cannot expand. This allows for upgrading static motion planning problems to those involving motion. A set of requirements can be stopped instantaneously, but a moving actor cannot. Another benefit of this approach is that it eliminates the need to rely on training data or predictions in rare or unlikely, near-miss or actual collision situations. This also ensures collision safety beyond simply avoiding a physical collision. 【0139】 The orbital analyzer 114 may select orbitals based at least on orbital rating or scoring. In at least one embodiment, the orbital with the highest score is selected. Length and / or lateral conditions may be used for minor smoothing of the selected orbital. Various approaches may be used to evaluate each orbital and aggregated to form a score used for selection. For example, different approaches (e.g., the approaches described for Figure 1) may have different weights used to form the aggregated score. Any of the various approaches may be used to evaluate and enforce hard constraints on the orbital, even when scoring is used. In one or more embodiments, an orbital may be selected from multiple orbitals (e.g., based on scoring or another approach) and then evaluated for implementation using one or more approaches (e.g., the approaches described for Figure 1). 【0140】 An iterative approach may be used to identify the optimal trajectory. In the first iteration, several hypothetical trajectories may be evaluated, and the trajectory with the highest score acts as a seed for generating additional hypothetical trajectories for evaluation. In subsequent iterations, additional hypothetical trajectories may be generated by slightly changing various parameters of the seed trajectory. For example, the seed trajectory may be used by a path-fan generator as input in place of the nominal rail. The path-fan generator may then generate additional trajectories with small lateral variations (smaller lateral bumps than those used in the first iteration on the nominal rail) and small length variations. The additional hypothetical trajectories may then be evaluated to determine whether one of these trajectories has a higher score than the seed trajectory. The hypothetical trajectory with the best score may be selected for implementation. 【0141】 The motion controller (MC) 180 may select instantaneous lateral and longitudinal acceleration control 895 using a more sophisticated vehicle model, taking the chosen trajectory. This may involve calculating the future trajectory based on the control sequence and vehicle model, and optimizing the cost function by iteratively adjusting the control sequence with a nonlinear optimizer. The cost function may be designed to find a trade-off between smoothness and traveling along the requested trajectory (however, since this stage cannot involve obstacle detection, the cost function may be set to travel precisely along the trajectory, and the smoothness of the trajectory may be guaranteed by the motion planning stage). 【0142】 Referring here to Figure 9, which is a flowchart illustrating a method 900 for controlling a vehicle based on evaluating points corresponding to safety procedures that may be deployed on a track, according to some embodiments of the present disclosure. Each block of method 900, and other methods described herein, includes computational processing that may be carried out using any combination of hardware, firmware, and / or software. For example, various functions may be executed by a processor executing instructions stored in memory. The methods may also be embodied as computer-usable instructions stored on a computer storage medium. The methods may be provided, to name a few, as a standalone application, a service or a hosted service (in combination with another hosted service, either standalone or otherwise), or as a plug-in to another product. The methods described herein may be carried out by any one system, or any combination of systems, including but not limited to those described herein, and are not limited to specific examples. 【0143】 Method 900 includes determining one or more first future locations corresponding to one or more objects in block B902. For example, the arrival time determination unit 112 may determine one or more future locations in spacetime (e.g., arrival time) corresponding to one or more objects (e.g., object 128A) in the environment 126. 【0144】 Method 900 includes calculating one or more second future locations corresponding to one or more safety procedures deployed within the first trajectory in block B904. For example, the trajectory analyzer 114 may calculate one or more future locations in spacetime corresponding to vehicle 140 in environment 126, assuming that vehicle 140 performs one or more safety procedures (e.g., safety procedures 140A, 140B, or 140C) from one or more locations in spacetime within trajectory 144 (e.g., corresponding to a time step or sample). 【0145】 Method 900 includes controlling an autonomous machine using a first or second trajectory based on at least one or more first future locations and one or more second future locations in block B906. For example, a trajectory analyzer 114 may control a vehicle 140 using a trajectory 144 or a different trajectory based at least on a comparison between one or more second future locations in spacetime and one or more first future locations in spacetime. This may include determining whether the locations are in conflict and scoring or otherwise evaluating the trajectory 144. This may also include selecting a trajectory 144 or another trajectory for use when controlling the vehicle 140 based on the comparison. 【0146】 Referring now to Figure 10, Figure 10 is a flowchart illustrating a method 1000 for controlling a vehicle based on modeling a future set of requirements for the vehicle corresponding to safety procedures that may be carried out in the track, according to some embodiments of the present disclosure. 【0147】 Method 1000 includes modeling one or more future sets of requirements for one or more autonomous machines performing one or more safety procedures within a first track in block B1002. For example, the track analyzer 114 may use the requirements set determiner 116 to model one or more future sets of requirements for vehicle 140 in environment 126, assuming that vehicle 140 performs safety procedures 140A, 140B, and / or 140C within track 144. 【0148】 Method 1000 includes calculating one or more arrival times for one or more objects in block B1004. For example, the arrival time determiner 112 may determine one or more arrival times for one or more objects (e.g., object 128A) to one or more locations in the environment 126 (e.g., the front of a future set of requests). 【0149】 Method 1000 includes controlling an autonomous machine using a first or second trajectory in block B1006 based at least on one or more sets of future requests and one or more arrival times. For example, a trajectory analyzer 114 may control a vehicle 140 using trajectory 144 or a different trajectory based at least on a comparison between one or more times in one or more sets of future requests and one or more arrival times. This may include determining whether there is a conflict and scoring or otherwise evaluating trajectory 144. This may also include selecting trajectory 144 or another trajectory for use when controlling the vehicle 140 based on the comparison. 【0150】 Referring now to Figure 11, Figure 11 is a flowchart illustrating a method 1100 for controlling a vehicle based on determining that the vehicle should occupy a position before the time at which the position should be occupied by an object, when performing safety procedures in a track, according to some embodiments of the present disclosure. 【0151】 Method 1100 includes, in block B1102, calculating one or more times for one or more objects to occupy one or more locations in the environment, assuming one or more objects travel along one or more trajectories. For example, arrival time determiner 112 may calculate one or more times for one or more objects (e.g., object 128A) to occupy one or more locations in the environment 126 (e.g., the leading edge of the request set), assuming one or more objects travel along one or more trajectories. 【0152】 Method 1100 includes controlling an autonomous machine using a first or second trajectory, at least based on determining that the autonomous machine should occupy one or more positions one or more times in advance. For example, the trajectory analyzer 114 may control the vehicle 140 using trajectory 144 or a different trajectory, at least based on determining that the vehicle 140 should occupy one or more positions one or more times in advance, at least based on modeling that the vehicle 140 performs one or more safety procedures on trajectory 144. This may include scoring or otherwise evaluating trajectory 144 based on the determination. This may also include selecting trajectory 144 or another trajectory for use when controlling the vehicle 140 based on the determination. 【0153】 Exemplary operating environment The trajectory evaluation system 100 may be implemented within the exemplary operating environment 1230 shown in Figure 12, according to some embodiments of this disclosure. Among numerous components not shown, the operating environment 1230 includes a client device 1232, a network 1234, a server device 1236, a sensor 1238, and / or a data store 1246. It should be understood that the operating environment 1230 shown in Figure 12 is just one example of one suitable operating environment. Each of the components shown in Figure 12 may be implemented via any type of computing device, such as one or more of the computing devices 1400 described in relation to Figure 14. These components may communicate with each other via the network 1234, which may be wired, wireless, or both. The network 1234 may include multiple networks, or one of multiple networks, but is shown in a simplified form so as not to obscure the aspects of this disclosure. For example, network 1234 may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet, and / or one or more private networks. If network 1234 includes a wireless telecommunications network, components such as base stations, communication towers, or possibly access points (and other components) may provide wireless connectivity. In any example, at least one network 1234 may correspond to network 1390 in Figure 13D, which is further described below. 【0154】 It should be understood that any number of client devices 1232, server devices 1236, sensors 1238, and data stores 1246 may be employed within the operating environment 1230 within the scope of this disclosure. Each may be configured as a single device or as multiple devices working together in a distributed environment. 【0155】 Client device 1232 may include at least some of the components, features, and functions of the exemplary computing device 1400 described herein with respect to Figure 14. In example, but not limited to, client device 1232 may be embodied as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, robot, drone, global positioning system (GPS) or device, video player, handheld communication device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronic device, workstation, any combination of these described devices, or any other suitable device. In any example, at least one client device 1232 may be part of a vehicle, such as vehicle 140 in Figures 14A to 14D, which are described in more detail herein. 【0156】 The client device 1232 may include one or more processors and one or more computer-readable media. The computer-readable media may include computer-readable instructions that can be executed by one or more processors. When executed by one or more processors, the instructions can cause one or more processors to perform any combination and / or part of the methods described herein, and / or any part of the functions of the trajectory evaluation system 100 of Figure 1. 【0157】 The server device 1236 may also include one or more processors and one or more computer-readable media. The computer-readable media includes computer-readable instructions that can be executed by one or more processors. When executed by one or more processors, the instructions can cause one or more processors to perform any combination and / or part of the methods described herein, and / or perform any part of the functions of the trajectory evaluation system 100 of Figure 1. In any example, at least one server device 1236 may correspond to the server 1478 of Figure 14D, which is described in more detail herein. 【0158】 The data store 1246 may include one or more computer-readable media. The computer-readable media may include computer-readable instructions that can be executed by one or more processors. When executed by one or more processors, the instructions can cause one or more processors to perform any combination and / or part of the methods described herein, and / or any part of the functions of the trajectory evaluation system 100 in Figure 1. Although the data store 1246 (or computer data storage) is described as a single component, it may be embodied as one or more data stores (e.g., databases) and may be at least partially in the cloud. One or more of the data stores 1246 may correspond to one or more of the data stores in Figure 14C. 【0159】 Although depicted externally to server device 1236 and client device 1232, data store 1246 can be at least partially embodied in any combination of server device 1236 and / or client device 1232 (for example, as memory 1404 (Figure 14)). For example, some information may be stored in client device 1232, and other and / or duplicate information may be stored externally (for example, in server device 1236). Therefore, it should be understood that the information in data store 1246 can be distributed in any suitable manner across one or more data stores for storage (which may be hosted externally). For example, data store 1246 may include at least some of the computer-readable media of one or more server devices 1236 and / or at least some of the computer-readable media of one or more client devices 1232. 【0160】 Sensor 1238 comprises at least one sensor capable of generating sensor data representing at least some aspects of the environment. For example, sensor 1238 may generate sensor data 102 in Figure 1. Sensor 1238 may include global navigation satellite system (GNSS) sensors (e.g., Global Positioning System (GPS) sensors), RADAR sensors, ultrasonic sensors, LIDAR sensors, inertial measuring unit (IMU) sensors (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones, stereo cameras, wide-view cameras (e.g., fisheye cameras), infrared cameras, surround cameras (e.g., 360-degree cameras), long-range and / or medium-range cameras, speed sensors (e.g., for measuring the speed of vehicle 140), vibration sensors, steering sensors, brake sensors (e.g., as part of a brake sensor system), and / or any combination of other sensor types. 【0161】 Regarding Figures 14A to 14C, the sensor data 102 is, as an example and not limited to, a Global Navigation Satellite System (GNSS) sensor 1468 (e.g., Global Positioning System sensor), a Radar sensor 1260, an ultrasonic sensor 1462, a LiDAR sensor 1464, an Inertial Measurement Unit (IMU) sensor 1466 (e.g., an accelerometer, gyroscope, magnetic compass, magnetometer, etc.), a microphone 1496, a stereo camera 1468, and a Wi-Fi sensor. This can be generated by a de-view camera 1470 (e.g., a fisheye camera), an infrared camera 1472, a surround camera 1474 (e.g., a 360-degree camera), a long-range and / or medium-range camera 1498, a speed sensor 1444 (e.g., for measuring the speed of a vehicle 140), a vibration sensor 1442, a steering sensor 1440, a brake sensor (e.g., as part of a brake sensor system 1446), and / or other sensor types. 【0162】 In some examples, sensor data 102 may be generated by forward-facing and / or side-facing cameras, such as a wide-view camera 1470, a surround camera 1474, a stereo camera 1468, and / or a long-range or medium-range camera 1498. In some examples, two or more cameras or other sensors may be used to incorporate multiple fields of view (for example, the fields of view of the long-range camera 1498, the forward-facing stereo camera 1468, and / or the forward-facing wide-view camera 1470 in Figure 14B). 【0163】 The components of Figure 1 can generally be implemented using any combination of client device 1232 and server device 1236. Therefore, it should be understood that the trajectory evaluation system 100 may be provided through multiple devices configured in a distributed environment that collectively provide the functions described herein, or it may be embodied on a single device (e.g., vehicle 140). Thus, while some examples used to illustrate the trajectory evaluation system 100 may refer to specific devices and / or configurations, it is assumed that these examples may more generally be applicable to potential combinations of devices and configurations described above. For example, in some embodiments, at least some of the sensors 1238 may be used to generate one or more portions of sensor data 102 and may be distributed among multiple vehicles and / or objects in the environment, and / or at least one of the sensors 1238 may be included in vehicle 140. 【0164】 Exemplary autonomous vehicle Figure 13A is an illustration of an exemplary autonomous vehicle 1300 according to some embodiments of the present disclosure. The autonomous vehicle 1300 (as otherwise referred to herein as "Vehicle 1300") may include, but is not limited to, passenger vehicles, such as cars, trucks, buses, first responder vehicles, shuttles, electric or motorized bicycles, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, submarines, drones, vehicles attached to trailers, and / or other types of vehicles (e.g., unmanned and / or accommodating one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), departments of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicle" (standard number J3016-201806, published June 15, 2018; standard number J3016-201609, published September 30, 2016; and previous and future versions of this standard). Mobile vehicle 1300 may have the capability to perform functions at one or more of the autonomous driving levels from Level 3 to Level 5. For example, depending on the embodiment, mobile vehicle 1300 may have the capability of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). 【0165】 The mobile vehicle 1300 may include components such as the mobile vehicle's chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components. The mobile vehicle 1300 may include a propulsion system 1350, such as an internal combustion engine, a hybrid power unit, a fully electric engine, and / or another propulsion system type. The propulsion system 1350 may be connected to the drivetrain of the mobile vehicle 1300, which may include a transmission, in order to enable propulsion for the mobile vehicle 1300. The propulsion system 1350 may be controlled in response to receiving a signal from a throttle / accelerator 1352. 【0166】 A steering system 1354, which may include a steering wheel, may be used to steer the vehicle 1300 (for example, along a desired course or route) when the propulsion system 1350 is operating (for example, when the vehicle is moving). The steering system 1354 may receive signals from the steering actuator 1356. The steering wheel may also be an option for fully automated (level 5) functionality. 【0167】 The brake sensor system 1346 may be used to operate the vehicle brakes in response to receiving signals from the brake actuator 1348 and / or the brake sensor. 【0168】 The controller 1336, which may include one or more system-on-a-chip (SoC) 1304 (Figure 13C) and / or GPUs, can provide signals (e.g., expressions of commands) to one or more components and / or systems of the vehicle 1300. For example, the controller can send signals to actuate the vehicle brakes via one or more brake actuators 1348, actuate the steering system 1354 via one or more steering actuators 1356, and actuate the propulsion system 1350 via one or more throttle / accelerators 1352. The controller 1336 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist the driver in driving the vehicle 1300. The controller 1336 may include a first controller 1336 for autonomous driving functions, a second controller 1336 for functional safety functions, a third controller 1336 for artificial intelligence functions (e.g., computer vision), a fourth controller 1336 for infotainment functions, a fifth controller 1336 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1336 may handle two or more of the aforementioned functions, and two or more controllers 1336 may handle a single function, and / or any combination thereof. 【0169】 The controller 1336 can provide signals for controlling one or more components and / or systems of the mobile vehicle 1300 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example and without limitation, global navigation satellite system sensors 1358 (e.g., global positioning system sensors), RADAR sensors 1360, ultrasonic sensors 1362, LIDAR sensors 1364, inertial measurement unit (IMU) sensors 1366 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1396, stereo cameras 1368, wide-view cameras 1370 (e.g., fisheye cameras), infrared cameras 1372, surround cameras 1374 (e.g., 360-degree cameras), long-range and / or medium-range cameras 1398, speed sensors 1344 (e.g., for measuring the speed of a moving vehicle 1300), vibration sensors 1342, steering sensors 1340, brake sensors (e.g., as part of a brake sensor system 1346), and / or other sensor types. 【0170】 One or more of the controllers 1336 may receive inputs (represented, for example, by input data) from the instrument cluster 1332 of the mobile vehicle 1300 and provide outputs (represented, for example, by output data, display data, etc.) via a human-machine interface (HMI) display 1334, an audible annunciator, a loudspeaker, and / or other components of the mobile vehicle 1300. The outputs may include information such as mobile vehicle velocity, speed, time, map data (e.g., HD map 1322 in Figure 13C), location data (e.g., the location of the mobile vehicle 1300, such as on the map), direction, the locations of other mobile vehicles (e.g., occupied grids), and information about objects and the status of objects as grasped by the controller 1336. For example, the HMI display 1334 may display information regarding the presence of one or more objects (e.g., road signs, warning signs, changes in traffic signals, etc.) and / or driving operations that the moving vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting exit 34B within 3.22 km (2 miles), etc.). 【0171】 The mobile vehicle 1300 further includes a network interface 1324 that can communicate over one or more networks using one or more wireless antennas 1326 and / or a modem. For example, the network interface 1324 may have the capability to communicate over LTE, WCDMA®, UMTS, GSM, CDMA2000, etc. The wireless antennas 1326 can also enable communication between objects in the environment (e.g., mobile vehicles, mobile devices, etc.) using local area networks such as Bluetooth®, Bluetooth® LE, Z-Wave, ZigBee, and / or low-power wide-area networks (LPWANs) such as LoRaWAN, SigFox. 【0172】 Figure 13B shows examples of camera positions and fields of view of the exemplary autonomous vehicle 1300 of Figure 13A according to several embodiments of the present disclosure. The cameras and their respective fields of view are exemplary embodiments and are not intended to limit the scope. For example, additional and / or alternative cameras may be included, and / or cameras may be placed in different positions on the mobile vehicle 1300. 【0173】 The camera type may include, but is not limited to, a digital camera that can be used with components and / or systems of the mobile vehicle 1300. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or other ASILs. Depending on the embodiment, the camera type may have the capability of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may have the capability to use a roll shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include an RCCC (red clear clear clear) color filter array, an RCCB (red clear clear blue) color filter array, an RBGC (red blue green clear) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras having RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to increase light sensitivity. 【0174】 In some applications, one or more cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono-camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video). 【0175】 One or more of the cameras may be mounted in custom-designed (3D-printed) mounting parts to eliminate stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capability. Referring to side mirror mounting parts, the side mirror parts may be custom 3D-printed so that the camera mounting plate conforms to the shape of the side mirror. In some examples, the camera may be integrated within the side mirror. For side-view cameras, the camera may also be integrated within four struts located at each corner of the cabin. 【0176】 A camera having a field of view that includes a portion of the environment in front of the moving vehicle 1300 (e.g., a forward-facing camera) may be used for surround view to help identify the forward path and obstacles and, with the help of one or more controllers 1336 and / or control SoCs, to help provide information essential for generating an occupied grid and / or determining a preferred moving vehicle path. The forward-facing camera may also be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The forward-facing camera may also be used for ADAS functions and systems, including other functions such as lane departure warning (LDW), autonomous cruise control (ACC), and / or traffic sign recognition. 【0177】 Various cameras may be used in forward-facing configurations, including, for example, a monocular camera platform that includes a CMOS (complementary metal oxide semiconductor) color imaging device. Another example may be a wide-view camera 1370, which can be used to capture objects entering the view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although only one wide-view camera is shown in Figure 13B, any number of wide-view cameras 1370 may be present in the mobile vehicle 1300. In addition, long-range cameras 1398 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. Long-range cameras 1398 may also be used for object detection and classification, as well as basic object tracking. 【0178】 One or more stereo cameras 1368 may also be included in the forward-facing configuration. The stereo camera 1368 may include an integrated control unit with an expandable processing unit that may provide a programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet® interface on a single chip. Such a unit may be used to generate a 3D map of the moving vehicle's environment, including distance estimates for all points in the image. An alternative stereo camera 1368 may include a compact stereo vision sensor that includes two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the moving vehicle to an object and activate autonomous emergency braking and lane departure warning functions using the generated information (e.g., metadata). Other types of stereo cameras 1368 may be used in addition to or instead of those described herein. 【0179】 A camera having a field of view including a portion of the environment on the sides of the mobile vehicle 1300 (e.g., a side-view camera) may be used for surround view, providing information used to create and update the occupancy grid and generate side impact collision warnings. For example, surround cameras 1374 (e.g., four surround cameras 1374 as shown in Figure 13B) may be positioned on the mobile vehicle 1300. The surround cameras 1374 may include wide-view cameras 1370, fisheye cameras, 360-degree cameras, and / or similar. For example, four fisheye cameras may be positioned in front of, behind, and on the sides of the mobile vehicle. In an alternative configuration, the mobile vehicle may use three surround cameras 1374 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround view camera. 【0180】 A camera having a field of view that includes a portion of the environment behind the mobile vehicle 1300 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creation and updating of the occupancy grid. A wide variety of cameras may be used, including, but not limited to, cameras suitable as forward-facing cameras (e.g., long-range and / or medium-range cameras 1398, stereo camera 1368), infrared cameras 1372, etc., as described herein. 【0181】 Figure 13C is a block diagram of an exemplary system architecture of the exemplary autonomous vehicle 1300 of Figure 13A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted together. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. 【0182】 Each component, feature, and system of the mobile vehicle 1300 in Figure 13C is illustrated as being connected via a bus 1302. Bus 1302 may include a Controller Area Network (CAN) data interface (or, referred to as the "CAN bus"). CAN may also be a network within the mobile vehicle 1300 used to assist in the control of various features and functions of the mobile vehicle 1300, such as the operation of brakes, acceleration, steering, and windshield wipers. The CAN bus may be configured to have dozens or hundreds of nodes, each having its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering angle, ground speed, engine revolutions per minute (RPM), button position, and / or other mobile vehicle status indicators. The CAN bus may be ASIL B compliant. 【0183】 Bus 1302 is described herein as a CAN bus, but this is not intended to limit it. For example, FlexRay and / or Ethernet® may be used in addition to, or as an alternative to, a CAN bus. In addition, a single line is used to represent bus 1302, but this is not intended to limit it. There may be any number of buses 1302, which may include, for example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet® buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1302 may be used to perform different functions and / or for redundancy. For example, a first bus 1302 may be used for collision avoidance and a second bus 1302 may be used for operation control. In any example, each bus 1302 may communicate with any of the components of the mobile vehicle 1300, and two or more buses 1302 may communicate with the same component. In some examples, each SoC 1304, each controller 1336, and / or each computer within the mobile vehicle may have access to the same input data (e.g., input from sensors in the mobile vehicle 1300) and may be connected to a common bus such as a CAN bus. 【0184】 The mobile vehicle 1300 may include one or more controllers 1336, such as those described herein with respect to Figure 13A. The controllers 1336 may be used for a variety of functions. The controllers 1336 may be connected to any of the various other components and systems of the mobile vehicle 1300 and may be used for the control of the mobile vehicle 1300, the artificial intelligence of the mobile vehicle 1300, infotainment for the mobile vehicle 1300, and / or the like. 【0185】 The mobile vehicle 1300 may include a system-on-a-chip (SoC) 1304. The SoC 1304 may include a CPU 1306, a GPU 1308, a processor 1310, a cache 1312, an accelerator 1314, a data store 1316, and / or other components and features not shown. The SoC 1304 may be used to control the mobile vehicle 1300 in various platforms and systems. For example, the SoC 1304 may be coupled in a system (e.g., the system of the mobile vehicle 1300) having an HD map 1322 that can obtain map refreshes and / or updates from one or more servers (e.g., server 1378 in Figure 13D) via a network interface 1324. 【0186】 The CPU 1306 may include a CPU cluster or CPU complex (also referred to as "CCPLEX"). The CPU 1306 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 1306 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 1306 may include four dual-core clusters, each cluster having its own dedicated L2 cache (e.g., 2MBL2 cache). The CPU 1306 (e.g., CCPLEX) may be configured to support concurrent cluster operation, allowing any combination of the CPU 1306 clusters to be active at any given time. 【0187】 The CPU1306 can implement power management capabilities that include one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be gated when a core is not actively executing instructions by executing WFI / WFE instructions; each core may be independently power-gated; each core cluster may be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster may be independently power-gated when all cores are power-gated. The CPU1306 can further implement enhanced algorithms for managing power states, where acceptable power states and expected wake-up times are specified, and the hardware / microcode determines the best power state to input to the cores, clusters, and CCPLEX. The processing core may support a simplified power state input sequence in software where the work is offloaded to the microcode. 【0188】 The GPU1308 may include an integrated GPU (or, as referred to herein, "iGPU"). The GPU1308 may be programmable and efficient for parallel workloads. In some embodiments, the GPU1308 may be able to use an enhanced tensor instruction set. The GPU1308 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with a storage capacity of at least 96KB), and two or more of the streaming microprocessors may share a cache (e.g., an L2 cache with a storage capacity of 512KB). In some embodiments, the GPU1308 may include at least eight streaming microprocessors. The GPU1308 may be able to use a Computation Application Programming Interface (API). In addition, the GPU1308 may be able to use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA). 【0189】 The GPU1308 can be power-optimized for optimal performance in automotive and embedded use cases. For example, the GPU1308 can be manufactured on a FinFET (Fin field-effect transistor). However, this is not intended to be a limitation, and the GPU1308 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. Not limited to, for example, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed-precision NVIDIA tensor cores for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads with a mixture of computation and addressing operations. A streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination between concurrent threads. A streaming microprocessor may also include a combined L1 data cache and shared memory unit to simplify programming while improving performance. 【0190】 In some examples, the GPU1308 may include high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of 900 GB / s. In some examples, in addition to or instead of HBM memory, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used. 【0191】 The GPU1308 can incorporate unified memory technology, including access counters, to enable more precise movement of memory pages to the processor that most frequently accesses them, thereby improving the efficiency of shared memory ranges across processors. In some examples, address translation service (ATS) support can be used to allow the GPU1308 to directly access the CPU1306 page table. In such examples, when the GPU1308 memory management unit (MMU) experiences a miss, an address translation request can be sent to the CPU1306. In response, the CPU1306 can examine its page table for virtual-to-real-address mapping and send the translation back to the GPU1308. As such, unified memory technology can enable a single, unified virtual address space for both the CPU1306 and GPU1308 memory, thereby simplifying GPU1308 programming and porting of applications to the GPU1308. 【0192】 In addition, the GPU1308 may include an access counter that can record how often the GPU1308 accesses the memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses that page most frequently. 【0193】 The SoC1304 may include any number of caches 1312, including those described herein. For example, cache 1312 may include an L3 cache available to both the CPU 1306 and the GPU 1308 (e.g., connected to both the CPU 1306 and the GPU 1308). Cache 1312 may include a write-back cache that can record line states, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4MB or more, depending on the embodiment, although a smaller cache size may be used. 【0194】 The SoC1304 may include an arithmetic logic unit (ALU) that can be used to perform processing for any of the various tasks or operations of the vehicle 1300 (for example, a processing DNN). In addition, the SoC1304 may include a floating-point unit (FPU) (or other mass coprocessor or numerical coprocessor type) for performing mathematical operations within the system. For example, the SoC104 may include one or more FPUs integrated as execution units within the CPU1306 and / or GPU1308. 【0195】 SoC 1304 may include one or more accelerators 1314 (e.g., a hardware accelerator, a software accelerator, or a combination thereof). For example, SoC 1304 may include a hardware acceleration cluster that may include an optimized hardware accelerator and / or a large on-chip memory. The large on-chip memory (e.g., 4MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other operations. The hardware acceleration cluster may be used to complement the GPU 1308 and to offload a portion of the tasks of the GPU 1308 (e.g., to free up more cycles of the GPU 1308 for other tasks). As an example, the accelerator 1314 may be used for target workloads that are stable enough to be suitable for acceleration (e.g., perception, convolutional neural networks (CNNs)). As used herein, the term “CNN” may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., as used for object detection). 【0196】 Accelerator 1314 (e.g., hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may also be an accelerator configured and optimized to perform image processing functions (e.g., CNN, RCNN, etc.). The DLA may further be optimized for a specific set of neural network types and floating-point operations, as well as for inference. The design of the DLA can provide more performance per millisecond than a general-purpose GPU and significantly exceed the performance of a CPU. The TPU can perform several functions, including, for example, single-instance convolutional functions supporting INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions. 【0197】 DLA can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection, identification, and detection using data from microphones; CNNs for facial recognition and mobile vehicle owner identification using data from camera sensors; and / or CNNs for security and / or safety-related events. 【0198】 DLA can perform any function of GPU1308, and by using inference accelerators, for example, a designer can target either DLA or GPU1308 for any function. For example, a designer can focus on CNN and floating-point arithmetic processing on DLA, and leave other functions to GPU1308 and / or other accelerators 1314. 【0199】 The acceleration device 1314 (for example, a hardware acceleration cluster) may include, or may be referred to herein as, a computer vision acceleration device and a programmable vision accelerator (PVA). The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA can include, but is not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors. 【0200】 The RISC core can interact with an image sensor (for example, the image sensor of any of the cameras described herein), an image signal processor, and / or the like. Each RISC core can include any amount of memory. The RISC core can use any of several protocols depending on the embodiment. In some examples, the RISC core can execute a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core can include an instruction cache and / or tightly coupled RAM. 【0201】 DMA can enable PVA components to access system memory independent of the CPU 1306. DMA can support any number of features used to bring optimizations to the PVA, including but not limited to supporting multidimensional addressing and / or circular addressing. In some examples, DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping. 【0202】 A vector processor may also be a programmable processor that can be designed to efficiently and flexibly execute the programming of computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), or very-long instruction word (VLIW) digital signal processor. A combination of SIMD and VLIW can increase throughput and speed. 【0203】 Each vector processor may include an instruction cache and be linked to dedicated memory. As a result, in some examples, each vector processor may be configured to run independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA can run the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can run different computer vision algorithms simultaneously on the same image, or even run different algorithms sequentially on the image or parts of an image. In particular, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In addition, a PVA may include additional error correction code (ECC) memory to enhance overall system safety. 【0204】 The accelerator 1314 (for example, a hardware accelerator cluster) may include a computer vision network on-chip and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 1314. In some examples, the on-chip memory may include at least 4 MB of SRAM consisting of eight field-configurable memory blocks, which may be accessible by both the PVA and DLA, for example, and not limited to. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA can access the memory via a backbone that provides the PVA and DLA with high-speed access to the memory. The backbone may include a computer vision network on-chip that interconnects the PVA and DLA to the memory (for example, using an APB). 【0205】 A computer vision network on-chip may include an interface that determines whether both the PVA and DLA are activatable and enable signals before any control signals / addresses / data are transmitted. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may be used. 【0206】 In some embodiments, the SoC1304 may include a real-time ray tracing hardware accelerator, such as the one described in Patent Document 1, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and size of objects (e.g., in a world model) to generate real-time visualization simulations for RADAR signal interpretation, acoustic propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison to LIDAR data for localization and / or other functions, and / or other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations. 【0207】 The accelerator 1314 (e.g., a hardware accelerator cluster) has diverse applications for autonomous driving. The PVA may also be a programmable vision accelerator that can be used in critical processing stages in ADAS and autonomous vehicles. The PVA's capabilities are suitable for areas of algorithms requiring predictable processing at low power and low latency. In other words, the PVA performs well in semi-high density or high density typical computations, even on small data sets, where predictable execution time is required along with low latency and low power. Therefore, because the PVA is efficient in object detection and integer computation, in relation to a platform for autonomous vehicles, the PVA is designed to run classic computer vision algorithms. 【0208】 For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. While semi-global matching-based algorithms may be used in some examples, this is not intended to be a limitation. Numerous applications for Level 3-5 autonomous driving require motion estimation / stereo matching on the fly (e.g., SFM (structure from motion), pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions with input from two monocular cameras. 【0209】 In some applications, PVA can be used to perform high-density optical flow by processing raw RADAR data (e.g., using 4D Fast Fourier Transform) to provide processed RADAR data. In other applications, PVA is used for flight depth processing, for example, by processing raw flight data to provide processed flight data. 【0210】 DLA can be used to run any type of network to enhance control and driving safety, for example, a neural network that outputs a confidence value for each object detection. Such confidence values ​​can be interpreted as probabilities or as providing the relative "weight" of each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only detections that exceed the threshold as true positives. In an automatic emergency braking (AEB) system, a false positive detection would cause the moving vehicle to automatically apply the emergency brakes, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for the AEB. DLA can run a neural network that devolves the confidence values. The neural network can accept at least a subset of parameters as its input, such as bounding box dimensions, ground plane estimation acquired (e.g., from another subsystem), object orientation, distance, inertial measurement unit (IMU) sensor output correlated with 3D position estimation acquired from the neural network and / or other sensors (e.g., LIDAR sensor 1364 or RADAR sensor 1360), and others. 【0211】 The SoC1304 may include a data store 1316 (for example, memory). The data store 1316 may also be the on-chip memory of the SoC1304 and can store neural networks that will run on the GPU and / or DLA. In some examples, the data store 1316 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 1312 may comprise an L2 or L3 cache 1312. References to the data store 1316 may include references to memory associated with the PVA, DLA, and / or other accelerators 1314, as described herein. 【0212】 The SoC1304 may include one or more processors 1310 (e.g., embedded processors). The processors 1310 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. The boot and power management processor may also be part of the SoC1304 boot sequence and can provide runtime power management services. The boot power and management processor can provide clock and voltage programming, assistance with system low-power state transitions, management of SoC1304 thermal and temperature sensors, and / or management of SoC1304 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC1304 may use the ring oscillators to detect the temperatures of the CPU 1306, GPU 1308, and / or accelerator 1314. If the temperature is determined to have exceeded a threshold, the boot and power management processor may enter a temperature fault routine and place the SoC1304 into a lower power state and / or put the mobile vehicle 1300 into chauffeur safe shutdown mode (for example, safely shut down the mobile vehicle 1300). 【0213】 The processor 1310 may further include a set of integrated processors capable of performing the functions of an audio processing engine. The audio processing engine may also be an audio subsystem enabling full hardware support for multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM. 【0214】 The processor 1310 may further include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timer and interrupt controllers), various I / O controller peripherals, and routing logic. 【0215】 The processor 1310 may further include a safety cluster engine, which includes a dedicated processor subsystem for handling safety management in automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic for detecting any differences between their operations. 【0216】 The processor 1310 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management. 【0217】 The processor 1310 may further include a high dynamic range signal processor, which may include an image signal processor, a hardware engine that is part of the camera processing pipeline. 【0218】 The processor 1310 may include a video image synthesizer, which may be a processing block (e.g., implemented on a microprocessor) that implements the post-video processing functions required by a video playback application to produce the final image for the player window. The video image synthesizer can perform lens distortion correction with the wide-view camera 1370, the surround camera 1374, and / or the in-cabin monitoring camera sensor. The in-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of a high-level SoC configured to identify in-cabin events and respond appropriately. The in-cabin system can perform lip reading to activate cellular service and make calls, compose emails, change the destination of a moving vehicle, activate or change the infotainment system and settings of a moving vehicle, or provide voice-activated web surfing. Certain features are available to the driver only when operating in autonomous mode and are otherwise disabled. 【0219】 The video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion occurs in the video, the noise reduction reduces the weight of the information provided by adjacent frames and appropriately weights the spatial information. If an image or a portion of an image does not contain motion, the temporal noise reduction performed by the video image synthesizer can reduce the noise in the current image using information from the previous image. 【0220】 The video image synthesizer can also be configured to perform stereo rectification on the input stereo lens frame. Furthermore, the video image synthesizer can be used for user interface compositing when the operating system desktop is in use, so that the GPU1308 is not required to continuously render new surfaces. Even when the GPU1308 is powered on and actively performing 3D rendering, the video image synthesizer can be used to offload the GPU1308 to improve performance and responsiveness. 【0221】 The SoC1304 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions to receive video and input from a camera. The SoC1304 may further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a specific role. 【0222】 The SoC1304 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC1304 may be used to process data from cameras (connected, for example, via Gigabit Multimedia Serial Link and Ethernet®), sensors (e.g., LiDAR sensor 1364, RADAR sensor 1360, etc., which may be connected via Ethernet®), data from bus 1302 (e.g., speed of vehicle 1300, steering wheel position, etc.), and data from GNSS sensor 1358 (connected, for example, via Ethernet® or CAN bus). The SoC1304 may further include a dedicated high-performance mass storage controller which may include its own DMA engine and which may be used to free up CPU 1306 from routine data management tasks. 【0223】 The SoC1304 may also be an inter-terminal platform with a flexible architecture that extends to automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and, together with deep learning tools, provides a platform for a flexible, reliable driving software stack. The SoC1304 can be faster, more reliable, more energy-efficient, and more space-efficient than conventional systems. For example, when the accelerator 1314 is coupled with the CPU 1306, the GPU 1308, and the data store 1316 can provide a fast and efficient platform for autonomous vehicles at levels 3-5. 【0224】 Therefore, this technology brings capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, many CPUs cannot execute real-time complex object detection algorithms, which are required for in-vehicle ADAS applications and actual Level 3-5 autonomous vehicles. 【0225】 In contrast to conventional systems, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, the technologies described herein enable multiple neural networks to run simultaneously and / or sequentially, and the results to be combined to enable Level 3–5 autonomous driving capabilities. For example, a DLA or a CNN running on a dGPU (e.g., GPU1320) may include text and word recognition, enabling a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of signs and passing that semantic understanding to a route planning module running on the CPU complex. 【0226】 As another example, multiple neural networks may run simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of a flashing light and the text "Caution: Flashing light indicates frozen conditions" may be interpreted independently or collectively by several neural networks. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing light indicates frozen conditions" may be interpreted by a second deployed neural network, informing the vehicle's route planning software (preferably running on a CPU complex) that frozen conditions are present when flashing light is detected. The flashing light may be identified by informing the vehicle's route planning software of the presence (or absence) of the flashing light, and by operating a third deployed neural network through multiple frames. All three neural networks can run simultaneously within the DLA and / or on the GPU1308, for example. 【0227】 In some applications, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of the legitimate driver and / or owner of the vehicle 1300. An always-on sensor processing engine may be used to unlock the vehicle and turn on the lights when the owner approaches the driver's side door, and, in security mode, to stop the vehicle when the owner leaves the vehicle. In this way, the SoC 1304 provides security against theft and / or vehicle hijacking. 【0228】 In another example, a CNN for emergency vehicle detection and identification can detect and identify emergency vehicle sirens using data from microphone 1396. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, SoC 1304 uses a CNN for classifying environmental and urban sounds, as well as for classifying visual data. In a preferred embodiment, a CNN running on DLA is trained to identify the relative terminal velocity of emergency vehicles (for example, by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the moving vehicle is operating, as identified by GNSS sensor 1358. Thus, for example, when operating in Europe, the CNN would attempt to detect European sirens, and when in the United States, the CNN would attempt to identify only North American sirens. After an emergency vehicle is detected, the control program may be used, with the assistance of the ultrasonic sensor 1362, to perform an emergency vehicle safety routine, such as slowing down the vehicle, stopping it at the side of the road, parking the vehicle, and / or idling the vehicle, until the emergency vehicle has passed. 【0229】 The vehicle may include a CPU 1318 (e.g., a separate CPU, or dCPU) which can be connected to the SoC 1304 via a high-speed interconnect (e.g., PCIe). The CPU 1318 may include, for example, an x86 processor. The CPU 1318 may be used to perform any of a variety of functions, including, for example, mediating the consequences of a potential mismatch between ADAS sensors and the SoC 1304, and / or monitoring the status and condition of the controller 1336 and / or the infotainment SoC 1330. 【0230】 The mobile vehicle 1300 may include a GPU 1320 (e.g., a separate GPU, or dGPU) which can be connected to the SoC 1304 via a high-speed interconnect (e.g., NVIDIA NVLINK). The GPU 1320 can provide additional artificial intelligence capabilities, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based on input from the mobile vehicle 1300's sensors (e.g., sensor data). 【0231】 Mobile vehicle 1300 may further include a network interface 1324 which may include one or more wireless antennas 1326 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas and Bluetooth® antennas). The network interface 1324 may be used to enable wireless connectivity to a cloud over the Internet (e.g., with server 1378 and / or other network devices), to other mobile vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other mobile vehicles, a direct link may be established between two mobile vehicles, and / or an indirect link may be established (e.g., over a network and over the Internet). The direct link may be provided using a mobile vehicle-to-mobile communication link. The mobile vehicle-to-mobile communication link may provide mobile vehicle 1300 information about mobile vehicles in close proximity to mobile vehicle 1300 (e.g., mobile vehicles in front of, beside, and / or behind mobile vehicle 1300). This function may also be part of the joint adaptive cruise control function of the mobile vehicle 1300. 【0232】 The network interface 1324 may include an SoC that provides modulation and demodulation functions and enables the controller 1336 to communicate over a wireless network. The network interface 1324 may include a radio frequency front end for upconversion from baseband to radio frequency and downconversion from radio frequency to baseband. Frequency conversion can be performed through well-known processes and / or using a superheterodyne process. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communication over LTE, WCDMA®, UMTS, GSM, CDMA2000, Bluetooth®, Bluetooth® LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols. 【0233】 The mobile unit 1300 may further include a data store 1328 which may include storage outside the chip (for example, outside the SoC 1304). The data store 1328 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data. 【0234】 The vehicle 1300 may further include a GNSS sensor 1358. The GNSS sensor 1358 (e.g., GPS, an assisted GPS sensor, a differential GPS (DGPS) sensor, etc.) assists in mapping, perception, occupy grid generation, and / or route planning functions. Any number of GNSS sensors 1358 may be used, including, but not limited to, a GPS using a USB connector with Ethernet® to a serial (RS-232) bridge. 【0235】 The mobile vehicle 1300 may further include a RADAR sensor 1360. The RADAR sensor 1360 may be used by the mobile vehicle 1300 for long-range mobile vehicle detection, even in darkness and / or severe weather conditions. The RADAR functional safety level may be ASIL B. In some examples, the RADAR sensor 1360 may use CAN and / or bus 1302 for control and to access object tracking data (for example, to transmit data generated by the RADAR sensor 1360) using Ethernet® access for accessing raw data. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor 1360 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor may be used. 【0236】 The RADAR sensor 1360 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, and short-range side coverage. In some examples, the long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view achieved by two or more independent scans, such as within a range of 250m. The RADAR sensor 1360 can help distinguish between static and moving objects and may be used by ADAS systems for emergency brake assist and forward collision warning. The long-range RADAR sensor may include monostatic multimodal RADARs having multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In one example with six antennas, the four central antennas may create a focused beam pattern designed to record around the moving vehicle 1300 at high speed with minimal interference from traffic in adjacent lanes. The other two antennas can widen the field of view, enabling rapid detection of moving vehicles entering or leaving the lane of the moving vehicle 1300. 【0237】 As an example, a medium-range RADAR system may include a range of up to 1360 m (forward) or 80 m (rear) and a field of view of up to 42 degrees (forward) or 1350 degrees (rear). A short-range RADAR system may include, but is not limited to, RADAR sensors designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and beside the moving vehicle. 【0238】 Short-range radar systems can be used in ADAS systems for blind spot detection and / or lane change assistance. 【0239】 The mobile vehicle 1300 may further include ultrasonic sensors 1362. Positioned on the front, rear, and / or sides of the mobile vehicle 1300, the ultrasonic sensors 1362 may be used for parking assistance and / or for creating and updating the occupancy grid. A wide variety of ultrasonic sensors 1362 may be used, and different ultrasonic sensors 1362 may be used for detection of different ranges (e.g., 2.5m, 4m). The ultrasonic sensors 1362 may operate at a functional safety level of ASIL B. 【0240】 The mobile vehicle 1300 may include a LiDAR sensor 1364. The LiDAR sensor 1364 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor 1364 may also have a functional safety level of ASIL B. In some examples, the mobile vehicle 1300 may include multiple LiDAR sensors 1364 (e.g., two, four, six, etc.) that can use Ethernet® (for example, to provide data to a Gigabit Ethernet® switch). 【0241】 In some examples, the LIDAR sensor 1364 may have the ability to provide a list of objects and their distances within a 360-degree field of view. A commercially available LIDAR sensor 1364 may have an advertised range of approximately 1300m, for example, with an accuracy of 2cm to 3cm and support for 1300Mbps Ethernet® connectivity. In some examples, one or more non-protruding LIDAR sensors 1364 may be used. In such examples, the LIDAR sensor 1364 may be implemented as a small device that can be incorporated into the front, rear, side, and / or corners of a mobile vehicle 1300. In such examples, the LIDAR sensor 1364 may have a range of 200m even for low-reflection objects and can provide a field of view up to 120 degrees horizontal and 35 degrees vertical. A front-mounted LIDAR sensor 1364 may be configured for a horizontal field of view between 45 and 135 degrees. 【0242】 In some applications, LiDAR technologies such as 3D flash LiDAR may also be used. 3D flash LiDAR uses a laser flash as a source to illuminate the area around a moving vehicle up to approximately 200m. The flash LiDAR unit includes receptors that record the laser pulse travel time and reflected light on each pixel, sequentially corresponding to the range from the moving vehicle to the object. Flash LiDAR can enable the generation of high-precision and distortion-free images of the surroundings with every laser flash. In some applications, four flash LiDAR sensors may be deployed, one on each side of the moving vehicle 1300. Available 3D flash LiDAR systems include solid-state 3D steering array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than a blower. The flash LiDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-documented intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 1364 can be made less susceptible to motion blur, vibration, and / or shock. 【0243】 The mobile vehicle may further include an IMU sensor 1366. In some examples, the IMU sensor 1366 may be positioned in the center of the rear axle of the mobile vehicle 1300. The IMU sensor 1366 may include, but is not limited to, an accelerometer, magnetometer, gyroscope, magnetic compass, and / or other sensor types. In some examples, such as in a 6-axis application, the IMU sensor 1366 may include an accelerometer and a gyroscope, while in a 9-axis application, the IMU sensor 1366 may include an accelerometer, a gyroscope, and a magnetometer. 【0244】 In some embodiments, the IMU sensor 1366 may be implemented as a miniature, high-performance GPS-Aided Inertial Navigation System (GPS / INS) that combines a micro-electro-mechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 1366 may enable the moving vehicle 1300 to estimate its direction of travel without requiring input from a magnetic sensor by directly observing and correlating velocity changes from the GPS to the IMU sensor 1366. In some embodiments, the IMU sensor 1366 and the GNSS sensor 1358 may be combined in a single integrated unit. 【0245】 The mobile vehicle may include a microphone 1396 placed inside and / or around the mobile vehicle 1300. The microphone 1396 may, among other things, be used for emergency vehicle detection and identification. 【0246】 The mobile vehicle may further include any number of camera types, including a stereo camera 1368, a wide-view camera 1370, an infrared camera 1372, a surround camera 1374, a long-range and / or medium-range camera 1398, and / or other camera types. The cameras may be used to capture image data around the entire exterior surface of the mobile vehicle 1300. The type of camera used will depend on the embodiment and requirements of the mobile vehicle 1300, and any combination of camera types may be used to achieve the required coverage around the mobile vehicle 1300. In addition, the number of cameras may vary depending on the embodiment. For example, the mobile vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may, as an example, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet®. Each camera is described in more detail herein in relation to Figures 13A and 13B. 【0247】 The mobile vehicle 1300 may further include a vibration sensor 1342. The vibration sensor 1342 can measure vibrations of components of the mobile vehicle, such as axles. For example, a change in vibration may indicate a change in the road surface. In another example, when two or more vibration sensors 1342 are used, the difference in vibration may be used to determine friction or slippage of the road surface (for example, when the difference in vibration is between a power-driven axle and a free-rotating axle). 【0248】 The mobile vehicle 1300 may include an ADAS system 1338. In some examples, the ADAS system 1338 may include a System of Control (SoC). The ADAS system 1338 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions. 【0249】 The ACC system may utilize a radar sensor 1360, a lithium-ion sensor 1364, and / or a camera. The ACC system may include longitudinal ACC and / or transverse ACC. Longitudinal ACC monitors and controls the distance of vehicle 1300 to the vehicle immediately in front of it and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Transverse ACC performs distance maintenance and advises vehicle 1300 to change lanes when necessary. Transverse ACC is related to other ADAS applications such as LCA and CWS. 【0250】 CACC uses information from other vehicles that can be received from other vehicles via a wireless link through the network interface 1324 and / or wireless antenna 1326, or indirectly via a network connection (e.g., via the Internet). Direct links may be provided by vehicle-to-vehicle (V2V) communication links, while indirect links may be infrastructure-to-vehicle (I2V) communication links. Generally, the V2V communication concept provides information about the vehicle immediately ahead (e.g., the vehicle immediately in front of vehicle 1300, in the same lane as vehicle 1300), while the I2V communication concept provides information about traffic further ahead. A CACC system may include either or both I2V and V2V information sources. Given information about the vehicle ahead of vehicle 1300, CACC can be more reliable, and CACC has the potential to make traffic flow smoother and reduce road congestion. 【0251】 The FCW system is designed to warn the driver of hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or radar sensor 1360, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibration components. The FCW system can provide warnings in the form of audible, visual warnings, vibration, and / or quick brake pulses. 【0252】 An AEB system can detect an imminent forward collision with another moving vehicle or other object and automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system may use a forward-facing camera and / or radar sensor 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver to take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system may automatically apply the brakes as part of an effort to prevent, or at least mitigate, the impact of the anticipated collision. The AEB system may include techniques such as dynamic brake support and / or impending collision braking. 【0253】 The LDW system warns the driver when the vehicle 1300 crosses a lane marking by providing visual, audible, and / or tactile warnings, such as vibration of the steering wheel or seat. The LDW system does not activate when the driver indicates an intentional lane departure by activating the turn signal. The LDW system may use a forward-facing camera connected to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration components. 【0254】 The LKA system is a modified form of the LDW system. The LKA system provides steering input or braking to correct the moving vehicle 1300 when it begins to drift out of its lane. 【0255】 The BSW system detects and warns the driver of a moving vehicle in the vehicle's blind spot. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses the turn signal. The BSW system can use a rear-facing camera and / or radar sensor 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration component. 【0256】 The RCTW system can provide visual, audible, and / or haptic notifications when an object is detected outside the range of the rear camera while the vehicle 1300 is reversing. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a collision. The RCTW system may use one or more rear-facing RADAR sensors 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibration component. 【0257】 Conventional ADAS systems warn the driver and allow the driver to determine whether a safe condition truly exists and act accordingly. However, conventional ADAS systems have sometimes tended to produce misjudgment results that, while not usually catastrophic, can be troubling and distracting to the driver. In the autonomous vehicle 1300, however, if the results are contradictory, the vehicle 1300 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 1336 or the second controller 1336). For example, in some embodiments, the ADAS system 1338 may also be a backup and / or secondary computer for providing perceptual information to a backup computer rationality module. The backup computer rationality monitor can run a variety of redundant software on hardware components to detect failures in perceptual and dynamic driving tasks. The output from the ADAS system 1338 may be provided to the supervisory MCU. If the outputs from the primary and secondary computers are contradictory, the supervisory MCU must decide how to reconcile the contradiction to ensure safe operation. 【0258】 In some implementations, a primary computer may be configured to provide a supervising MCU with a reliability score indicating the reliability of the primary computer in a selected outcome. If the reliability score exceeds a threshold, the supervising MCU may follow the primary computer's instructions, regardless of whether the secondary computer gives conflicting or inconsistent results. If the reliability score does not meet the threshold, and the primary and secondary computers produce different results (e.g., conflicting results), the supervising MCU may mediate between the computers to determine an appropriate outcome. 【0259】 The supervisory MCU may be configured to run a neural network trained and configured to determine, based on the outputs from the primary and secondary computers, when a secondary computer is providing a false alarm. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer is reliable and when it is not. For example, when the secondary computer is a radar-based forward crossing (FCW) system, the neural network in the supervisory MCU can learn when the FCW is identifying metal objects that are not actually dangerous, such as sewer grates or manhole covers that trigger an alarm. Similarly, when the secondary computer is a camera-based lane departure warning (LDW) system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest operation. In embodiments involving a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running a neural network with associated memory. In a preferred embodiment, the supervisory MCU may comprise and / or be included as a component of the SoC1304. 【0260】 In other examples, ADAS system 1338 may include a secondary computer that performs ADAS functions using conventional rules of computer vision. As such, the secondary computer may use classical computer vision rules (if-then), and the presence of a neural network within the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identities make the entire system more fault-tolerant, particularly to failures caused by software (or software-hardware interface) functions. For instance, if a software bug or error exists in the software running on the primary computer, and non-identical software code running on the secondary computer produces the same overall result, the supervisory MCU may have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer did not cause a critical error. 【0261】 In some examples, the output of the ADAS system 1338 may be supplied to the perception block and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 1338 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information when identifying the object. In other examples, the secondary computer may have its own neural network, which is trained as described herein and therefore reduces the risk of misjudgment. 【0262】 The mobile vehicle 1300 may further include an infotainment SoC 1330 (for example, an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system does not have to be an SoC and may include two or more separate components. The infotainment SoC 1330 may include a combination of hardware and software that can be used to provide the mobile vehicle 1300 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, fuel level, total mileage, brake fuel level, oil level, door open / close, air filter information, and other mobile vehicle-related information). For example, the infotainment SoC 1330 may also include wireless, disc player, navigation system, video player, USB and Bluetooth® connectivity, car computer, in-car entertainment, Wi-Fi, steering wheel audio control unit, hands-free voice control, heads-up display (HUD), HMI display 1334, telematics device, control panel (for example, for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 1330 may be further used to provide information (for example, visual and / or audible) to the user of the vehicle, such as information from the ADAS system 1338, autonomous driving information such as planned vehicle operation, trajectory, surrounding environment information (for example, intersection information, vehicle information, road information, etc.), and / or other information. 【0263】 The infotainment SoC 1330 may include GPU functionality. The infotainment SoC 1330 can communicate with other devices, systems, and / or components of the vehicle 1300 via bus 1302 (e.g., CAN bus, Ethernet®, etc.). In some examples, the infotainment SoC 1330 may be coupled to a supervisory MCU so that the infotainment system's GPU can perform certain self-drive functions in the event of a primary controller 1336 (e.g., the primary and / or backup computer of the vehicle 1300) failure. In such examples, the infotainment SoC 1330 can put the vehicle 1300 into a chauffeur-safe stop mode as described herein. 【0264】 The mobile vehicle 1300 may further include an instrument cluster 1332 (e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 1332 may include a controller and / or a supercomputer (e.g., a separate controller or supercomputer). The instrument cluster 1332 may include a set of instruments such as a speedometer, fuel level indicator, oil pressure indicator, tachometer, odometer, turn signals, gear shift position indicator, seat belt warning light, parking brake warning light, engine fault light, airbag (SRS) system information, lighting control device, safety system control device, and navigation information. In some examples, information may be displayed and / or shared between the infotainment SoC 1330 and the instrument cluster 1332. In other words, the instrument cluster 1332 may be included as part of the infotainment SoC 1330, and vice versa. 【0265】 Figure 13D is a system diagram of communication between the cloud-based server of Figure 13A and an exemplary autonomous vehicle 1300 according to some embodiments of the present disclosure. System 1376 may include a server 1378, a network 1390, and a mobile vehicle including the mobile vehicle 1300. Server 1378 may include a plurality of GPUs 1384(A) to 1384(H) (collectively referred to herein as GPU 1384), PCIe switches 1382(A) to 1382(H) (collectively referred to herein as PCIe switch 1382), and / or CPUs 1380(A) to 1380(B) (collectively referred to herein as CPU 1380). The GPUs 1384, CPUs 1380, and PCIe switches may be interconnected by high-speed interconnects, such as, for example, NVLink interfaces 1388 and / or PCIe connections 1386 developed by NVIDIA. In some examples, the GPU1384 is connected via NVLink and / or NVSwitch SoCs, and the GPU1384 and PCIe switch 1382 are connected via PCIe interconnects. Eight GPU1384s, two CPU1380s, and two PCIe switches are illustrated, but this is not intended to be an limitation. Depending on the embodiment, each server 1378 may contain any number of GPU1384s, CPU1380s, and / or PCIe switches. For example, server 1378 may contain eight, sixteen, thirty-two, and / or more GPU1384s, respectively. 【0266】 Server 1378 can receive image data from mobile vehicles via network 1390, representing images showing unexpected or altered road conditions, such as recently commenced road construction. Server 1378 can transmit map information 1394, including information about traffic and road conditions, to mobile vehicles via network 1390, including information about neural networks 1392, updated neural networks 1392, and / or map information 1394. Updates to map information 1394 may include updates to HD map 1322, including information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural networks 1392, updated neural networks 1392, and / or map information 1394 may have arisen from new training and / or experience represented in data received from any number of mobile vehicles in the environment, and / or based on training performed in a data center (for example, using server 1378 and / or other servers). 【0267】 Server 1378 may be used to train a machine learning model (e.g., a neural network) based on training data. The training data may be generated by a mobile device and / or in a simulation (e.g., using a game engine). In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or otherwise pre-processed, while in other instances, the training data is not tagged and / or pre-processed (e.g., if the neural network does not require supervised learning). Training may be performed according to any one or more classes of machine learning techniques, including but not limited to the following: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, associative learning, transfer learning, feature learning (including key component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including pre-dictionary learning), rule-based machine learning, anomaly detection, and variations or combinations thereof. After the machine learning model has been traced, it may be used by the mobile vehicle (for example, transmitted to the mobile vehicle via network 1390), and / or the machine learning model may be used by server 1378 to remotely monitor the mobile vehicle. 【0268】 In some examples, Server 1378 can receive data from a mobile vehicle and apply it to a state-of-the-art real-time neural network for real-time intelligent inference. Server 1378 may include deep learning supercomputers and / or dedicated AI computers powered by GPU 1384, such as the DGX and DGX Station Machines developed by NVIDIA. However, in some examples, Server 1378 may include deep learning infrastructure that uses only CPU-powered data centers. 【0269】 The deep learning infrastructure of server 1378 can have the capability for high-speed real-time inference, which can be used to evaluate and verify the condition of the processor, software, and / or associated hardware in mobile vehicle 1300. For example, the deep learning infrastructure can receive periodic updates from mobile vehicle 1300, such as images of a sequence and / or objects located within images of that sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with objects identified by mobile vehicle 1300, and if the results do not match and the infrastructure concludes that the AI ​​in mobile vehicle 1300 is not functioning properly, server 1378 can send a signal to mobile vehicle 1300 instructing the mobile vehicle's fail-safe computer to infer control, notify passengers, and complete a safe parking operation. 【0270】 For inference, server 1378 may include GPU 1384 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT). The combination of a GPU-powered server and inference accelerator can enable real-time responsiveness. In other examples, such as when performance is not a major requirement, a server powered by a CPU, FPGA, and other processors may be used for inference. 【0271】 Exemplary computing devices Figure 14 is a block diagram of an example of a computing device 1400 suitable for use in implementing some embodiments of the present disclosure. The computing device 1400 may include an interconnection system 1402 that indirectly or directly connects the following devices: memory 1404, one or more central processing units (CPUs) 1406, one or more graphics processing units (GPUs) 1408, a communication interface 1410, input / output (I / O) ports 1412, input / output components 1414, a power supply unit 1416, one or more presentation components 1418 (e.g., a display), and one or more logical units 1420. In at least one embodiment, the computing device 1400 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). As an unrestricted example, one or more of the GPUs 1408 may include one or more vGPUs, one or more of the CPUs 1406 may include one or more vCPUs, and / or one or more of the logical units 1420 may include one or more virtual logical units. As such, the computing device 1400 may include individual components (e.g., an entire GPU dedicated to the computing device 1400), virtual components (e.g., a portion of a GPU dedicated to the computing device 1400), or a combination thereof. 【0272】 The various blocks in Figure 14 are shown connected by lines via the interconnection system 1402, but this is not intended to be restrictive and is simply for clarity. For example, in some embodiments, a presentation component 1418, such as a display device, could be considered an I / O component 1414 (for example, if the display is a touchscreen). As another example, the CPU 1406 and / or GPU 1408 may include memory (for example, memory 1404 may represent a storage device in addition to the memory of the GPU 1408, CPU 1406, and / or other components). In other words, the computing devices in Figure 14 are merely illustrative. Categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types are all intended to fall within the scope of the computing devices in Figure 14 and are therefore not distinguished. 【0273】 The interconnection system 1402 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1402 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a VESA (video electronics standards association) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or other types of buses or links. In some embodiments, direct connections exist between components. For example, the CPU 1406 may be directly connected to the memory 1404. Furthermore, the CPU 1406 may be directly connected to the GPU 1408. Where direct or point-to-point connections exist between components, the interconnection system 1402 may include PCIe links for implementing the connections. In these examples, the PCI bus does not need to be included in the computing device 1400. 【0274】 Memory 1404 may include any of various computer-readable media. The computer-readable media may be any available media accessible by the computing device 1400. The computer-readable media may include both volatile and non-volatile media, and removable and non-removable media. For example, but not limited to, the computer-readable media may include computer storage media and communication media. 【0275】 Computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1404 may store computer-readable instructions (e.g., representing programs and / or program elements), such as an operating system. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by computing device 1400. In this specification, computer storage media does not include signals themselves. 【0276】 Computer storage media include any information distribution medium that can implement computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transfer mechanisms. The term “modulated data signal” may refer to a signal that has been modified in a manner that has one or more of its characteristic sets or encodes information within the signal. For example, but not limited to, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included in the scope of computer-readable media. 【0277】 The CPU 1406 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 1400 to execute one or more of the methods and / or processes described herein. The CPU 1406 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) each capable of processing a large number of software threads simultaneously. The CPU 1406 may include any type of processor, and depending on the type of computing device 1400 in which it is implemented, it may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 1400, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1400 may include one or more CPUs 1406 in one or more microprocessors or auxiliary coprocessors, such as a computing coprocessor. 【0278】 In addition to or instead of the CPU 1406, the GPU 1408 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 1400 to execute one or more of the methods and / or processes described herein. One or more of the GPUs 1408 may be an integrated GPU (for example, with one or more of the CPUs 1406, and / or one or more of the GPUs 1408 may be discrete GPUs). In embodiments, one or more of the GPUs 1408 may be coprocessors of one or more of the CPUs 1406. The GPUs 1408 may be used by the computing device 1400 to render graphics (for example, 3D graphics) or to perform general-purpose computing. For example, the GPUs 1408 may be used for GPU-based general-purpose computing (GPGPU). It may be used for a GPU. The GPU 1408 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU 1408 can generate pixel data for an output image in response to rendering commands (for example, rendering commands from CPU 1406 received via the host interface). The GPU 1408 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of memory 1404. The GPU1408 may include two or more GPUs operating in parallel (for example, via a link). The link can connect directly to the GPUs (for example, using NVLINK) or via a switch (for example, using NVSwitch). When coupled together, each GPU1408 can generate pixel data or GPGPU data for different parts of an output or different outputs (for example, the first GPU for the first image and the second GPU for the second image). Each GPU may have its own memory or may share memory with other GPUs. 【0279】 In addition to or instead of the CPU 1406 and / or GPU 1408, the logical unit 1420 may be configured to execute at least some computer-readable instructions to control one or more of the computing devices 1400 to execute one or more of the methods and / or processes described herein. In embodiments, the CPU 1406, GPU 1408, and / or the logical unit 1420 can execute any combination of methods, processes, and / or parts thereof discretely or congruently. One or more of the logical units 1420 may be part of and / or integrated with one or more of the CPU 1406 and / or GPU 1408, and / or one or more of the logical units 1420 may be discrete components of the CPU 1406 and / or GPU 1408 or otherwise external to them. In embodiments, one or more of the logical units 1420 may be coprocessors of one or more of the CPU 1406 and / or one or more of the GPU 1408. 【0280】 Examples of the logic unit 1420 include one or more processing cores and / or components thereof, such as Tensor Cores (TCs), Tensor Processing Units (TPUs), Pixel Visual Cores (PVCs), Vision Processing Units (VPUs), Graphics Processing Clusters (GPCs), Texture Processing Clusters (TPCs), Streaming Multiprocessors (SMs), Tree Traversal Units (TTUs), Artificial Intelligence Accelerators (AIAs), Deep Learning Accelerators (DLAs), Logical Units (ALUs), Application-Specific Integrated Circuits (ASICs), Floating-Point Units (FPUs), Input / Output (I / O) elements, Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) elements, and / or similar. 【0281】 The communication interface 1410 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1400 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 1410 may include components and functions to enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth®, Bluetooth® LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet® or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. 【0282】 I / O port 1412 can enable the computing device 1400 to be logically connected to other devices, including I / O components 1414, presentation components 1418, and / or other components, some of which can be built into (e.g., integrated into) the computing device 1400. Exemplary I / O components 1414 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, etc. I / O components 1414 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some cases, the input may be transmitted to appropriate network elements for further processing. The NUI may implement any combination of voice recognition, stylus recognition, face recognition, biometric recognition, on-screen and beside-screen gesture recognition, air gestures, head and target tracking, and touch recognition related to the display of the computing device 1400 (as described in more detail below). The computing device 1400 may include depth cameras, such as stereoscope camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 1400 may include accelerometers or gyroscopes that enable motion detection (for example, as part of an inertia measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by the computing device 1400 to render immersive augmented reality or virtual reality. 【0283】 The power supply device 1416 may include a hardwired power supply device, a battery power supply device, or a combination thereof. The power supply device 1416 can provide power to the computing device 1400 to enable the components of the computing device 1400 to operate. 【0284】 The presentation component 1418 may include a display (e.g., a monitor, touch screen, television screen, head-up display device (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 1418 can receive data from other components (e.g., GPU 1408, CPU 1406, etc.) and output data (e.g., as images, videos, sounds, etc.). 【0285】 Exemplary data center Figure 15 shows an exemplary data center 1500 that may be used in at least one embodiment of the present disclosure. The data center 1500 may include a data center infrastructure layer 1510, a framework layer 1520, a software layer 1530, and / or an application layer 1540. 【0286】 As shown in Figure 15, the data center infrastructure layer 1510 may include a resource orchestrator 1512, grouped computing resources 1514, and node computing resources ("node CRs") 1516(1) to 1516(N), where "N" represents any integer or natural number. In at least one embodiment, the node CRs 1516(1) to 1516(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and / or cooling modules. In some embodiments, one or more nodes CR1516(1) to 1516(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR1516(1) to 15161(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or similar, and / or one or more nodes CR1516(1) to 1516(N) may correspond to a virtual machine (VM). 【0287】 In at least one embodiment, the grouped computing resources 1514 may include a separate group of nodes CR1516 housed in one or more racks (not shown), or a number of racks housed in data centers in various geographical locations (also not shown). The separate group of nodes CR1516 within the grouped computing resources 1514 may include grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CR1516, including CPUs, GPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination. 【0288】 The resource orchestrator 1522 can configure or otherwise control one or more nodes CR1516(1) to 1516(N) and / or grouped computing resources 1514. In at least one embodiment, the resource orchestrator 1522 may include a software design infrastructure ("SDI") management entity for the data center 1500. The resource orchestrator 1522 may include hardware, software, or any combination thereof. 【0289】 In at least one embodiment, as shown in Figure 15, the framework layer 1520 may include a job scheduler 1560, a configuration manager 1534, a resource manager 1536, and / or a distributed file system 1538. The framework layer 1520 may include a framework to support the software 1532 of the software layer 1530 and / or one or more applications 1542 of the application layer 1540. The software 1532 or applications 1542 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. The framework layer 1520 may be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which may use the distributed file system 1538 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 1532 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 1500. The configuration manager 1534 may have the ability to configure different layers, for example, a software layer 1530 and a framework layer 1520 including Spark and a distributed file system 1538 to support large-scale data processing. The resource manager 1536 may have the ability to manage clustered or grouped computing resources mapped or allocated for support of the distributed file system 1538 and the job scheduler 1560. In at least one embodiment, the clustered or grouped computing resources may include computing resources 1514 grouped in the data center infrastructure layer 1510. The resource manager 1036 can coordinate with the resource orchestrator 1512 to manage these mapped or allocated computing resources. 【0290】 In at least one embodiment, the software 1532 included in the software layer 1530 may include software used by at least a portion of nodes CR1516(1) to 1516(N), grouped computing resources 1514, and / or the distributed file system 1538 of the framework layer 1520. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software. 【0291】 In at least one embodiment, the application 1542 included in the application layer 1540 may include one or more types of applications used by at least a portion of the nodes CR1516(1) to 1516(N), the grouped computing resources 1514, and / or the distributed file system 1538 of the framework layer 1520. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments. 【0292】 In at least one embodiment, any of the configuration manager 1534, resource manager 1536, and resource orchestrator 1512 may implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. Self-rewriting actions may free the data center operator of data center 1500 from making potentially poor configuration decisions and possibly avoiding underutilized and / or underperforming parts of the data center. 【0293】 The data center 1500 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters by a neural network architecture using the software and / or computing resources described herein with respect to the data center 1500. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described herein with respect to the data center 1500 by using weight parameters calculated via one or more training techniques, not limited to those described herein. 【0294】 In at least one embodiment, the data center 1500 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) for performing training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as services that enable users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services. 【0295】 Exemplary network environment A network environment suitable for use in implementing the embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Each client device, server, and / or other device type (e.g., each device) may be implemented as one or more instances of the computing device 1400 in Figure 14, for example, each device may include similar components, features, and / or functionalities of the computing device 1400. In addition, if backend devices (e.g., servers, NAS, etc.) are implemented, they may be included as part of the data center 1500, examples of which are further detailed herein with respect to Figure 15. 【0296】 Components of a network environment may communicate with one another via the network, either wired, wirelessly, or both. A network may include multiple networks, or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. If a network includes a wireless telecommunications network, its components, such as base stations, towers, or access points (and other components), may provide wireless connectivity. 【0297】 Compatible network environments may include one or more peer-to-peer network environments (in which case servers may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to the server can be implemented on any number of client devices. 【0298】 In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, or a combination thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support the software in the software layer and / or one or more applications in the application layer. The software or applications may each include web-based service software or applications. In the embodiment, one or more of the client devices may use the web-based service software or applications (for example, by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework that may use a distributed file system for, for example, large-scale data processing (e.g., “big data”). 【0299】 A cloud-based network environment may provide cloud computing and / or cloud storage that implements any combination of the computing and / or data storage functions (or one or more of them) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (e.g., one or more data centers that may be distributed across states, territories, countries, or the world). If the connection to the user (e.g., a client device) is relatively close to the edge server, the core server may delegate at least a portion of its functionality to the edge server. The cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment). 【0300】 A client device may include at least some of the components, features, and functionalities of the exemplary computing device 1400 described herein with respect to Figure 14. As an example, and not limited to, a client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, airship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, instrument, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device. 【0301】 This disclosure may be described in general terms with computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by computers or other machines, such as personal digital assistants or other handheld devices. Generally, a program module, including routines, programs, objects, components, and data structures, refers to code that performs a specific task or implements a specific abstract data type. This disclosure may be implemented in a variety of configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network. 【0302】 In this specification, any “and / or” statement relating to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, “element A, element B, and / or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one element A, at least one element B, or at least one element A and at least one element B. Furthermore, “at least one of element A and element B” may include at least one element A, at least one element B, or at least one element A and at least one element B. 【0303】 The subject matter of this disclosure is described in a manner that is specific in order to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors intend that the claimed subject matter may be carried out in other ways, including different steps or combinations of steps similar to those described herein, in conjunction with other current or future technologies. Furthermore, the terms “step” and / or “block” may be used herein to imply different elements of the way in which they are used, but these terms should not be construed as implying any particular order among the various steps disclosed herein unless the order of the individual steps is expressly stated and, when so, expressed.

Claims

[Claim 1] A method carried out by a processor, A step of calculating the trajectory of the autonomous machine in spacetime that is expected to be occupied by the autonomous machine, starting from a future position corresponding to each of at least two time steps if the autonomous machine in the environment were to move along a first trajectory, and performing one or more safety procedures which are procedures for controlling the autonomous machine to avoid collisions with one or more objects in the environment, A step of comparing one or more future locations corresponding to the motion of the one or more objects with the calculated trajectory at each time step to evaluate whether the calculated trajectory at each time step indicates a future collision with the one or more objects, A step of determining whether the first trajectory should be used when performing one or more operations using the autonomous machine, based at least on the evaluation for each of the two time steps: Methods that include... [Claim 2] The method according to claim 1, wherein the evaluation includes the step of calculating, using the calculated trajectory at each time step, the time and location at which the autonomous machine will stop in accordance with the execution of the one or more safety procedures. [Claim 3] The method according to claim 1, wherein the one or more future locations are calculated based at least on evaluating one or more predicted trajectories calculated for the one or more objects to the one or more future locations. [Claim 4] The method according to claim 1, wherein the one or more future locations are calculated at least on the basis of evaluating the direct paths of the one or more objects from the position of the one or more objects at each time step to the one or more future locations. [Claim 5] The method according to claim 1, wherein, after a period of time in which the one or more objects occupy the one or more future locations, the autonomous machine is controlled using a second trajectory different from the first trajectory, based on the comparison performed at least on whether the autonomous machine arrives at the one or more future locations in spacetime during the execution of the one or more safety procedures. [Claim 6] The method according to claim 1, wherein the first trajectory is determined to be unsuitable for consideration when controlling the autonomous machine, at least on the basis that the distance from the future position corresponding to each time step to the one or more future locations is less than a threshold. [Claim 7] A step of assigning a score to the first trajectory that corresponds to the distance from the future position corresponding to each time step to the one or more future locations, wherein the use of the first trajectory involves performing one or more actions using the autonomous machine, based at least on the score assigned to the first trajectory. The method according to claim 1, further comprising: [Claim 8] The step of assigning a score to each of the multiple trajectories of the autonomous machine, A step of selecting the first orbit, at least based on the fact that the score of the first orbit is higher than the scores of each of the other orbits among the plurality of orbits. The method according to claim 1, further comprising: [Claim 9] The method according to claim 1, wherein the one or more objects include one or more of pedestrians, animals, or stationary obstacles. [Claim 10] The system calculates the trajectory of the autonomous machine in spacetime that is expected to be occupied by the autonomous machine, which is a procedure for controlling the autonomous machine to avoid collisions with one or more objects in the environment, starting from a future position corresponding to each of at least two time steps if the autonomous machine in the environment were to move along a first trajectory, and performing one or more safety procedures, which are procedures for controlling the autonomous machine to avoid collisions with one or more objects in the environment. The calculated trajectory at each time step is compared with one or more future locations corresponding to the motion of the one or more objects to evaluate whether the calculated trajectory at each time step indicates a future collision with the one or more objects. Based at least on the evaluation for each of the two time steps, determine whether the first trajectory should be used when performing one or more operations using the autonomous machine. A processor having one or more processing units for that purpose. [Claim 11] The processor according to claim 10, wherein one or more processing units further evaluate the second trajectory of the autonomous machine using the comparison, at least on the basis that the first trajectory and the second trajectory include the same lateral path. [Claim 12] The processor according to claim 10, wherein the comparison is performed on at least the one or more future locations and the shapes representing the autonomous machine in the calculated trajectory at each time step. [Claim 13] The processor according to claim 10, wherein the comparison is performed at least on whether the autonomous machine arrives at the one or more future locations in spacetime after a time has passed in which the one or more objects occupy the one or more future locations, at the time of execution of the one or more safety procedure. [Claim 14] The processor according to claim 10, wherein the calculated trajectory at each time step is defined by one or more paths branching off at the future position of the first trajectory. [Claim 15] The processor according to claim 10, wherein the calculated trajectory for each time step is calculated from one or more first velocities of the autonomous machine at each time step, and the one or more future locations are calculated from one or more second velocities of the one or more objects at each time step. [Claim 16] One or more processors, One or more memory devices, which are executed by the one or more processors, A step of calculating the trajectory of the autonomous machine in spacetime that is expected to be occupied by the autonomous machine, starting from a future position corresponding to each of at least two time steps if the autonomous machine in the environment were to move along a first trajectory, and performing one or more safety procedures which are procedures for controlling the autonomous machine to avoid collisions with one or more objects in the environment, A step of comparing the calculated trajectory at each time step with one or more future locations corresponding to the motion of the one or more objects to evaluate whether the calculated trajectory at each time step indicates a future collision with the one or more objects, and A step of determining whether the first trajectory should be used when performing one or more operations using the autonomous machine, based at least on the evaluation for each of the two time steps. One or more memory devices that store instructions causing one or more processors to perform a method including the above, A system that includes these features. [Claim 17] The system according to claim 16, wherein the evaluation includes the step of calculating, using the calculated trajectory for each time step, the time and place at which the autonomous machine would stop if the autonomous machine were to perform the one or more safety procedures within the first trajectory. [Claim 18] The system according to claim 16, wherein the evaluation is based at least on the step of calculating one or more trajectories of the one or more objects to the one or more future locations. [Claim 19] The system according to claim 18, wherein the one or more trajectories include the direct paths of the one or more objects from the starting position of the one or more trajectories to the one or more future locations. [Claim 20] The aforementioned system Control systems for fully or semi-autonomous machines, Perceptual systems of fully or semi-autonomous machines, A system for performing simulation operations. A system for performing deep learning operations. Systems implemented using edge devices, Systems implemented using robots A system that incorporates one or more virtual machines (VMs). A system implemented at least partially in a data center, or A system that is at least partially implemented using cloud computing resources. The system according to claim 16, which is included in at least one of the following.

Images