Processor and method for selecting a trajectory for an autonomous machine performed by the processor - Patents.com

Abstract

Embodiments of the present disclosure relate to behavior planning for autonomous vehicles. The techniques described herein select a preferred trajectory for an autonomous vehicle based on evaluation of multiple hypothetical trajectories by different components within a planning system. Various components provide an optimization score for each trajectory according to the component's priority, and the scores from the multiple components may form a final optimization score. This scoring system allows competing priorities of different components (e.g., comfort, minimum travel time, fuel economy) to be considered together. In an illustrative example, the trajectory with the highest combined score may be selected for implementation. As such, an iterative approach that evaluates various factors may be used to identify an optimal or preferred trajectory for an autonomous vehicle when navigating an environment.

Description

[Technical field]

[0001] The present invention relates to behavior planning for autonomous vehicles. [Background technology]

[0002] Designing a system to drive a vehicle autonomously and safely without supervision is extremely difficult. An autonomous vehicle should at least have the ability to perform as the functional equivalent of an attentive driver - utilizing a perception and action system with a remarkable ability to identify and react to moving and static obstacles in a complex environment - to avoid collisions with other objects or structures along the vehicle's path. Thus, the ability to detect obstacles, paths, waiting conditions, road or lane geometry, and / or other information about the environment is often essential for an autonomous driving perception system.

[0003] The driving perception system's decisions can be used for behavior planning, e.g., determining the vehicle's trajectory along a route. As such, behavior planning for autonomous vehicles is a challenging but essential task, especially for complex urban scenarios. For example, in order for an autonomous vehicle to understand its surroundings and make safe and effective behavior decisions, the autonomous vehicle must process these various environmental inputs along with the current vehicle trajectory and route information to calculate the vehicle's successive or future trajectory. However, conventional systems for behavior planning often rely on inflexible decisions that do not account for constantly changing variables in the environment. For example, conventional methods may select a route with the lowest travel time and then choose a lane plan that follows the route. However, this inflexible planning does not enable the planning system to optimally adjust the trajectory based on changing environmental conditions. For example, by determining the vehicle's behavior separately from other decisions, such as collision avoidance, adjustments to the current plan may be reactive rather than proactive. As such, in a collision avoidance example, control decisions may be determined using output from the behavior planner, and the collision avoidance functionality may generate updated control decisions when it determines that a collision may occur. As a result, the vehicle's trajectory is not determined using collision avoidance as a factor, but rather is overwritten by the vehicle's collision avoidance functionality. Thus, these conventional systems determine vehicle behavior that is suboptimal and does not consider dynamic inputs from various different functionalities of the vehicle, such as collision avoidance or lane planning. Summary of the Invention [Means for solving the problem]

[0004] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure relate to behavior planning for autonomous vehicles. Disclosed are systems and methods relating to planning trajectories for autonomous vehicles that have several advantages over previous trajectory planning techniques. For example, in contrast to conventional techniques, the techniques described herein allow multiple hypothetical trajectories to be simultaneously evaluated by different components within a planning system. Each of these various components - corresponding, for example, to collision avoidance, vehicle maneuver type, route planning, lane planning, yield information, etc. - may provide an optimization score for each trajectory according to the component's priority. As such, the scores from the multiple components may form a final optimization score, and the trajectory with the highest combined score may be selected for implementation. This scoring system allows competing priorities of different components (e.g., comfort, minimum travel time, fuel economy, safety, etc.) to be considered together.

[0005] In further contrast to conventional systems such as those mentioned above, the techniques described herein enable safer and more efficient planning by providing information that can be used by multiple system components to quantify the consequences of not following a currently preferred or selected lane plan or route. For example, conventional approaches may recognize that failing to change lanes within the next 5 seconds may cause a route to be missed, but these conventional approaches are not known to quantify or encode this failure when generating the vehicle's potential future trajectory output. In accordance with the present disclosure, by quantifying this failure, a preferred or optimal route may be generated and used by the planning system when determining the trajectory to follow. As a result, the behavior planning system of the present disclosure can determine the vehicle's potential trajectory using information from various system components and can update the potential trajectory - e.g., by considering each of these various systems - at each iteration or time step to reflect the dynamic nature of environmental variables, route planning information, and / or lane planning information.

[0006] The present systems and methods for behavior planning for autonomous vehicles are described in detail below with reference to the accompanying drawings. [Brief description of the drawings]

[0007] [Figure 1] 1 is an illustration of an example behavior planning architecture for an autonomous vehicle, according to some embodiments of the present disclosure. [Diagram 2] 1 is an illustration of an example rail plan and corresponding speed calculations, according to some embodiments of the present disclosure. [Diagram 3] 1 is an illustration of example environmental conditions around an autonomous vehicle, according to some embodiments of the present disclosure. [Figure 4] 13 is an illustration of an example rail plan and corresponding speed calculations as adjusted according to environmental conditions, in accordance with some embodiments of the present disclosure. [Diagram 5] 1 is a flow diagram illustrating a method for selecting a trajectory for an autonomous vehicle, according to some embodiments of the present disclosure. [Figure 6] 1 is a flow diagram illustrating a method for selecting a trajectory for an autonomous vehicle, according to some embodiments of the present disclosure. [Figure 7] 1 is a flow diagram illustrating a method for selecting a trajectory for an autonomous vehicle, according to some embodiments of the present disclosure. [Figure 8A] 1 is an illustration of an example autonomous vehicle, according to some embodiments of the present disclosure. [Figure 8B] 8B is an illustration of camera positions and fields of view for the example autonomous vehicle of FIG. 8A, according to some embodiments of the present disclosure. [Figure 8C] FIG. 8B is a block diagram of an example system architecture of the example autonomous vehicle of FIG. 8A in accordance with some embodiments of the present disclosure. [Figure 8D] FIG. 8B is a system diagram of communication between a cloud-based server and the example autonomous vehicle of FIG. 8A, according to some embodiments of the present disclosure. [Figure 9]FIG. 1 is a block diagram of an example computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 10] FIG. 1 is a block diagram of an exemplary data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Systems and methods are disclosed for planning trajectories of an autonomous vehicle or another type of autonomous machine, such as, but not limited to, those described herein. The present disclosure may be described with respect to an exemplary autonomous vehicle 800 (examples of which are described herein with respect to FIGS. 8A-8D and alternatively referred to herein as "vehicle 800" or "self-vehicle 800"), but this is not intended to be limiting. For example, the systems and methods described herein may be used by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more advanced driver assistance systems (ADAS)), robots and robotic platforms, warehouse vehicles, off-road vehicles, flying ships, boats, and / or other vehicle types. Additionally, the present disclosure may be described with respect to autonomous driving, but this is not intended to be limiting. For example, the systems and methods described herein may be used in robotics (e.g., behavior planning for robots), aviation systems (e.g., behavior planning for drones or other aircraft), marine systems (e.g., behavior planning for ships), simulation environments (e.g., for testing or validating behavior planning systems of virtual vehicles within a virtual simulation environment), and / or other technology fields, for example, route planning, lane planning, behavior planning, and / or control decisions.

[0009] The techniques described herein may employ an architecture that allows each planning component to use minimal computational resources. In some embodiments, the trajectory planning system described herein may employ three different planning components, each with a different planning horizon. As a result, computational efficiency may be improved by providing each planning component with planning information with only enough detail to accomplish the necessary tasks of traversing the respective planning horizon. In at least one embodiment, the three different planning components include a route planner, a lane planner, and a behavior planner. Each component may produce a result and / or a result set and communicate it to the next component in the planning pipeline. The route planner may communicate a route set to the lane planner, which may communicate a lane plan to the behavior planner. The behavior planner may develop and evaluate multiple trajectories. The lane plan may take the form of a lane graph with time rewards associated with the graph nodes. This type of lane plan may differ from a traditional lane plan that includes one or more lane sequences. The selected or optimized trajectory can be sent from the behavior planner to a motion controller, which is responsible for implementing the planned trajectory.

[0010] The route planner may have geographic information that covers a large geographic area (e.g., a metropolitan area, a state, a country, a region) but has a relatively small amount of detail (e.g., less detail than found in geographic information used for lane planning and trajectory planning). The planning horizon of the route planner may be the entire area between a start point (e.g., a current location) and a destination point. The route planner may calculate multiple routes to a destination using a lower-detail map of a relatively large area. For example, a large geographic area may be greater than 50 square miles, greater than 500 square miles, greater than 1000 square miles, or even larger. In one aspect, a single route is not initially selected as optimal, but instead, each route may be scored according to an estimated time of travel to the route destination, or by some other scoring mechanism that quantifies the optimal or preferred route. Since a nearly infinite number of routes may be possible, various methods may be used to identify the top routes. For example, 5, 10, 50, and / or another number of best routes may be identified, as measured by estimated time of travel and / or other factors. These routes may be communicated to a lane planner, which, in addition to other components such as a behavior planner, may determine a preferred route from among the plurality.

[0011] The lane planner may generate a lane plan based on one or more routes received from the route planner. The lane planner may use a second geographic map that includes more detail than is found in the map used by the route planner. The second geographic map may cover a smaller geographic area than the map used by the route planner. For example, the smaller geographic area may be less than 50 square miles. In some embodiments, the lane plan may take the form of an annotated lane graph that may have a planning horizon of several miles. The lane graph may generally indicate the lanes available to the autonomous vehicle on a route (e.g., a road). Annotations on the lane graph may indicate temporal scores for various positions or points on the lane graph. For example, temporal scores may be provided every 10 meters, 20 meters, or some other interval. The temporal scores may communicate an overall value of moving the autonomous vehicle to a particular point on the lane graph.

[0012] The behavior planning component can then take these scores into account when evaluating various trajectories. For example, in some situations, one lane may perform as well as any other available lane. As illustrated, an autonomous vehicle traveling down an interstate 20 miles from an exit dictated by a route may use either lane without meaningfully impacting travel time. In this situation, the temporal scores of each lane may be very similar or even the same. On the other hand, as the autonomous vehicle approaches the exit, the temporal score in the exit lane will increasingly indicate a preference for being in the exit lane over the temporal scores in the non-exit lanes. At the exit, the difference between the temporal score in the exit lane and the temporal scores in the other lanes may correspond to the time penalty incurred by missing the exit. Thus, if missing the exit would cause a 5-minute increase in travel time to the destination, then the difference in the temporal scores may reflect this 5-minute increase.

[0013] The behavior planning component may take these different temporal scores into account when planning the trajectory. The behavior planner may have a smaller planning horizon than the route planner or lane planner. In general, the behavior planner may plan the vehicle's movements with respect to the next 0.5 seconds, 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 10 seconds, 20 seconds, etc., and / or based on a distance measure, e.g., the next 5m, 10m, 50m, 100m, 200m, 300m, etc. The distance covered will vary depending on the speed of the vehicle's movement. The behavior planner may use a map that shows a high level of detail over a small area. For example, the small area may be less than the area in the map used by the lane planner or route planner. For example, the map used by the behavior planner may cover less than 100 square meters. The richer detail may include objects (e.g., other vehicles, pedestrians) detected by sensors associated with the autonomous vehicle. In one aspect, at least some of these objects may not be within the information available to the lane planner. The selected actual trajectory may govern the autonomous vehicle's movement for only a short period of time, e.g., one second, or a short distance, e.g., five meters. New trajectories may be continually evaluated and implemented to adjust for changing conditions in the autonomous vehicle's environment (e.g., moving vehicles, moving pedestrians, waiting conditions, etc.).

[0014] The behavior planner may first select one or more desired actions for the autonomous vehicle to take. The actions may be based on an annotated lane graph provided by the lane planner. The action or selected actions may be based on achieving a safe, efficient, or best outcome as indicated by a momentary score included in the annotated lane graph. Possible actions include, but are not limited to, lane following, lane change speed adaptation, overtaking, double lane change, and lane change behavior. The behavior planner may generate multiple possible or hypothetical trajectories that accomplish one or more desired actions. Various components within the behavior planner may score the hypothetical trajectories and / or potentially eliminate trajectories. For example, if the collision avoidance component determines that a hypothetical trajectory will or may result in a collision, then the hypothetical trajectory may be eliminated from further consideration. In response, the comfort module may assign a score to each trajectory based on providing a comfortable ride to passengers aboard the autonomous vehicle. Other modules or components may similarly generate and assign scores such that a selected or preferred trajectory is selected from a plurality of trajectories having the highest or best score.

[0015] As a result, the techniques described herein may improve upon current techniques by using an iterative approach to identify optimal trajectories. In a first iteration, an initial plurality of hypothetical trajectories may be evaluated with the trajectory having the highest optimization score acting as a seed for generating additional hypothetical trajectories for evaluation. In a second iteration, a second plurality of hypothetical trajectories may be generated by slightly modifying various parameters of the seed trajectory. The second plurality of hypothetical trajectories may then be evaluated to determine whether one of these trajectories has a higher optimization score than the seed trajectory. The hypothetical trajectory having the highest optimization score may be selected for implementation.

[0016] Referring to FIG. 1, FIG. 1 is an example behavior planning architecture 100 according to some embodiments of the present disclosure. It should be understood that this and other configurations described herein are described merely as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by an entity may be implemented by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory.

[0017] The behavior planning architecture 100 can use different planning components (e.g., three planning components), each having a different planning horizon. In one embodiment, the different planning components include a route planner 110, a lane planner 120, and a behavior planner 140. Each component can produce a result and / or a result set and communicate it to the next component in the planning pipeline. The route planner 110 can communicate a route set to the lane planner 120, which can communicate a lane plan to the behavior planner 140. The behavior planner 140 can develop and evaluate multiple trajectories, and a selected or preferred trajectory can be sent from the behavior planner to a motion controller 180 that is responsible for implementing the planned trajectory.

[0018] The route planner 110 may use geographic information 112 that, in an embodiment, covers a geographic area (e.g., a town, metropolitan area, state, country, region, etc.) but has a relatively small amount of detail. The route planner's planning horizon may be the entire area between a start point (e.g., a current location) and a destination point (e.g., a final location). The technique may calculate multiple routes to a destination using a lower-detail map of a relatively large area (e.g., a GNSS-based map compared to a high-definition (HD) map). The route planner 110 may calculate multiple routes to a destination using a lower-detail map of a relatively large area. For example, the large geographic area may be greater than 50 square miles, greater than 500 square miles, greater than 1000 square miles, or even larger. However, this is not intended to be limiting, and in some embodiments, maps of any level of detail may be used. In one aspect, a single route is not initially selected as optimal or preferred; instead, each route of the multiple routes may be scored according to the estimated time of travel to the route destination or by some other scoring mechanism that quantifies the preferred route. As such, and because a nearly infinite number of routes may be possible, various methods may be used to identify several potential routes from the multiple routes. For example, the 10 best routes, as measured by the estimated time of travel and / or other factors, may be identified. These routes may be communicated to lane planner 120, thereby providing the lane planner 120 with multiple routes.

[0019] In one or more embodiments, the route planner 110 may be provided or issued a driving mission 116. The driving mission 116 may be represented as a sequence of one or more waypoints, which may be encoded as GPS points and / or locations (e.g., stop exactly at this bus stop for drop-off or pick-up) in a larger lane graph 130. The driving mission 116 may also end with a parking area (a request to park in a particular parking area). According to one or more embodiments, a parking area is a subset of the larger lane graph 130 that typically includes a loop structure around the parking lot and optionally map knowledge of the parking space itself.

[0020] The route planner 110 may be designed to be modular and separable from other planning steps. Architectural designs may efficiently support long distance routes. The route planner 110 may pass the resulting route plan as well as some basic state information to the lane planner 120. The route plan may include one or more GPS traces that are determined to be potential approximate routes to use to reach a waypoint or parking area from the current location. The GPS points in the route plan may also have expected time rewards associated with them (or, preferably, the time rewards may be expressed as expected time remaining to the destination). The route plan may be derived using an abbreviated hierarchy 118 that allows the shortest route to be determined within milliseconds, even for continent-wide routes. The GPS traces of the route plan may provide approximate guidance to the lane planner 120, rather than strict assertions about specific lanes or roads. The route planner 110 may provide multiple routes or route variations that are within a threshold travel time of each other, and in an embodiment, each route may be associated with an estimated travel time. In one embodiment, each route segment is assigned a travel time that gives the lane planner 120 and other components information that may be used to calculate the cost of missing an exit and being forced to take an alternate route. In some embodiments, travel times may also be calculated by other components, such as the lane planner 120, which may be calculated using distance and speed limits, but may be refined with real-time traffic data, construction data, weather data, and / or the like.

[0021] The route planner 110 may also include a state management component 114 to handle basic state management. For example, the state management component 114 may ensure that, at cold start, sensors and calculations are up and running and that localization has found the location of the autonomous vehicle before it starts moving (e.g., before the mission control system starts navigating the autonomous vehicle). Until then, the route planner 110 may be in a START state. Then, when the autonomous vehicle (or its behavior planner 140) is moving on a known lane graph, the route planner 110 may switch to a DRIVING state. The route planner 110 may also switch to a PARK state when a parking area is reached in a drive mission. Special SUMMON and LIMP (warning) states also exist. Crawling occurs when something unexpected occurs and the autonomous vehicle is slowing down or moving carefully to a stop; for example, a LIMP state may be initiated if localization declares that the autonomous vehicle is off the map or unable to localize.

[0022] The lane planner 120 may generate a lane plan based on one or more routes received from the route planner. The lane planner 120 may use a second geographic map that includes more detail than is found in the map used by the route planner. The second geographic map may cover a smaller geographic area than the map used by the route planner. For example, the smaller geographic area may be less than 129.5 square kilometers (50 square miles). The lane plan may take the form of an annotated lane graph, which may have a planning horizon of several miles. The lane graph may represent lanes available to the autonomous vehicle on a route (e.g., a road). Annotations on the lane graph may indicate temporal scores for various locations on the lane graph. For example, temporal scores may be provided every 10 meters, 20 meters, or some other interval on the lane graph. The temporal scores may be referred to as sequential time rewards for potential future locations. A larger or better reward may be associated with a faster path towards the route destination, or in other embodiments, a lower score may indicate a better or faster path. The scale of the reward may be in units of time, e.g., seconds.

[0023] The temporal scores communicate the overall value of moving the autonomous vehicle to a particular point on the lane graph. The behavior planner 140 can then take these scores into account when evaluating various trajectories. For example, in some situations, one lane may perform as well as any other available lane. As illustrated, an autonomous vehicle traveling down an interstate 20 miles from a route-directed exit may use either lane without meaningfully impacting travel time. In this situation, the temporal scores of each lane may be very similar or even the same. On the other hand, as the autonomous vehicle approaches the exit, the temporal scores in the exit lane will increasingly indicate a preference for being in the exit lane over the temporal scores in the non-exit lanes. At the exit, the difference between the temporal scores in the exit lane and the temporal scores in the other lanes may correspond to the time penalty incurred by missing the exit. Thus, if missing an exit would cause a 5 minute increase in travel time to the destination, then the momentary score difference would reflect this 5 minute increase.

[0024] Lane planner 120 may include lane progression predictor 122 and parking lane planner 126. Lane planner 120 may receive route plan and baseline state information from route planner 110. The purpose of lane planning may be to provide information about how beneficial it is to be in various lanes nearby the autonomous vehicle. For example, if the autonomous vehicle is approaching a position where one lane separates from the other lanes and an exit, the lane planner may provide information about how much more time may be lost if that exit is missed. The output of lane planner 120 may be represented as an expected time reward encoded on a larger lane graph 130 associated with local lane graph 132. This encoding may provide more flexibility than simply a straight command or recommendation of which lane to be in, as it allows behavior planner 140 to weigh the value of being in a particular lane against the difficulty of the maneuver to get there.

[0025] The lane planner 120 works with a large lane graph 130, which may correspond to a light representation of lane positions and expected times to transition between them. The lane planner 120 may use the route plan GPS trace provided by the route planner 110 as a guide to cut out a corridor (e.g., a few hundred meters wide) of the large lane graph 130 around the proposed route. The lane planner 120 may then target an area centered around some distance (e.g., a few miles) along the proposed route and use it as a temporary proxy destination. The proxy destination may now have an expected time reward from the route plan that may be propagated back to all nodes in the large lane graph 130 within the corridor. The lane planner 120 considers lane changes and the probability that they will not succeed immediately or at all. This means that the expected time reward may require a probability calculation. For example, each node may be associated with a set of actions (e.g., lane keeping, branching, lane change), which probabilistically result in a new lane position (e.g., a different lane if the lane change is successful, or further along the same lane if the lane change is unsuccessful).

[0026] According to some embodiments, this probability can be expressed as a Markov Decision Process (MDP) that solves for the value of being in any node starting from a given fixed value in the proxy destination. MDPs address things like the increasing pressure to make a lane change in time when an autonomous vehicle approaches a crack in the road, or estimating the feasibility of entering a different lane and then returning to a preferred lane.

[0027] There are many algorithms for estimating the value of states in an MDP (in this case at nodes in a large lane graph), e.g., classical value iteration. To be efficient, a heuristic can be used to remove all cycles in the large lane graph, which reduces the value iteration to a single path of dynamic programming. A heuristic can be used when a cycle exists to remove edges that are the furthest away from the autonomous vehicle on the cycle. This heuristic avoids removing edges that are part of the shortest distance between the autonomous vehicle and the destination, but are part of a cycle. However, breaking a cycle may not ignore the stochastic adjustments contributed by unnecessary cycles on the way to the destination. For example, if an autonomous vehicle fails to change lanes and has to circle around a cloverleaf intersection and try the exact same lane change again, this may result in an unnecessary cycle. Breaking the cycle removes this possibility, but since using unnecessary cycles may be undesirable, this may be an acceptable approximation. The graph may be ordered and multiple paths to the same destination may be considered. Also, reflective edges that point back to the same node, e.g., stopping and waiting to make a lane change, can also be handled correctly (nodes that have no dependencies other than reflective edges can be determined).

[0028] First, Dijkstra's algorithm - and / or another shortest path algorithm type - can be run to find the distances and shortest paths to all nodes using any action. Then the rewards of the nodes are determined or edges on cycles are iteratively removed in a single graph traversal. Dijkstra is then run over the large lane graph to find the shortest optimistic paths (assuming lane changes are successful if they are possible). It produces an ordering of the nodes in the lane planning scope by the optimistic time to arrive at them. Edges can also be ordered, for example, by the time to arrive at the end of the corresponding edge, so that edges can be ordered the same as the nodes they point to.

[0029] Backward edges are used to keep track, for each node, which nodes it depends on to determine its expected time (its actions and all the nodes it points to for failed actions). Similarly, edges pointing to undetermined nodes can be tracked.

[0030] All nodes with zero dependencies may be found and determined. When a node is determined, the back edges may be used to reduce the dependency counts of other nodes. If these updates find new nodes with zero dependencies, the new nodes may also be queued. All determined nodes and edges pointing to determined nodes may also be removed from the previously computed node and edge order.

[0031] If there are no nodes with zero dependencies, the furthest undetermined edge may be removed according to the results of Dijkstra's algorithm (and / or another shortest path algorithm type). The dependencies between the nodes may then be updated. In some embodiments, one or more nodes may be created with zero dependencies, meaning that there are no connections to other nodes. This represents a dead end, and the time associated with these nodes may be determined as the time equivalent to a mission failure. When all nodes have been determined, the process stops. The complexity of this algorithm is essentially that of Dijkstra's algorithm plus that of a single graph traversal.

[0032] The lane planner 120 uses a large lane graph 130 derived from map data that is also linked to a local lane graph 132 of lanes. In one or more embodiments, the lane planner 120 can derive values ​​for the entire large lane graph 130. However, the behavior planner 140 can use (solely or primarily) values ​​from a selection of local lane graphs 132 that are close to the autonomous vehicle. The lane progression predictor 122 can generate lane progression estimates based on live perceptions, e.g., lane speeds and congestion levels in different lanes and the speeds of the nearest vehicles. The progression predictions are used to modify the general expected traversal time. The expected traversal time allows the lane planner to determine that a lane change to a currently faster lane would be beneficial.

[0033] Lane planner 120 can change functions when route planner 110 changes state (e.g., to PARK, SUMMON, or LIMP, as non-limiting examples). For parking, lane planner 120 can generate a lane plan using parking lane planner 126. Parking lane planner 126 can use large lane graph cycles of parking lots to motivate behavior planner 140 to explore parts of parking lots that have not been explored for the longest time. Parking can have any number of phases. Non-limiting examples of phases include parking, wandering and maneuvering to a parking spot. Parking wandering is similar to driving around a lane graph in an urban area, except the objective is to drive around cyclically until a space is found. The lane planner can achieve this by setting values ​​such that a good driving pattern around a loop is achieved. After line of sight to an empty space is established, a parking maneuver is initiated by behavior planner 140.

[0034] Similarly, a call may have any number of phases as well. If the autonomous vehicle is on a known lane graph, then lane planning to the destination may be performed as in a normal DRIVING state. However, if the autonomous vehicle is not yet on a known lane graph, for example, in a parking space slightly off the mapped path, or does not have a standard path to follow the mapped path, then the parking lane planner 126 may be used to find a path from where the autonomous vehicle starts to one of the closest poses on the lane graph. If there is no map at all, except for simply an approximate direction to the caller, then that approximate direction is simply passed to the behavior planner 140. If detailed map information is unavailable or insufficiently detailed map information is available, then the LIMP mode may work as well.

[0035] The behavior planner 140 can consider one or more actions and proceed through one or more iterations of hypothesis generation and evaluation of more detailed implementations of those actions, which may be represented as precise motion plans (pose trajectories spanning several seconds into the future) and evaluated by the motion planner 172. The behavior planner 140 includes an action generator 142, a longitudinal preconstraint component 150, a hypothesis generation 160, and an action plan selector 170.

[0036] The behavior planner 140 may have a smaller planning horizon than the route planner or lane planner. In general, the behavior planner 140 may plan the vehicle's movements for the next few seconds. The distance covered will vary depending on the speed of the vehicle's movements. For example, the behavior planner may have a planning horizon of 50 m, 100 m, 200 m, 300 m, or the like. The behavior planner 140 may use a map that shows a high level of detail over a small area. For example, the small area may be less than the area in the map used by the lane planner 120 or the route planner 110. For example, the map used by the behavior planner may cover less than 100 square meters. The richer detail may include objects (e.g., other vehicles, pedestrians) detected by sensors associated with the autonomous vehicle. In one aspect, at least some of these objects may not be within the information available to the lane planner 120. The actual trajectory selected may govern the autonomous vehicle's motion for only a short period of time, e.g., one second. New trajectories may be continually evaluated and implemented to adjust for changing conditions (e.g., moving vehicles, moving pedestrians) in the autonomous vehicle's environment.

[0037] The computational flow for each action evaluated by the behavior planner 140 may be initiated by the action generator 142. The action generator 142 may first select one or more desired actions for the autonomous vehicle to take. The actions may then be based on the annotated lane graph provided by the lane planner 120. The action or actions selected may be based on achieving the best result as indicated by a momentary score included in the annotated lane graph. Possible actions include, but are not limited to, lane following, lane change speed adaptation, overtaking, lane change behavior, stop on the shoulder, parking loitering, parking end, call out, and slow down. As with the previous components, the action generator 142 may select or score multiple actions. In some embodiments, the actions may be represented as nominal rails, and / or each action may be associated with multiple nominal rails.

[0038] One or more of the actions may be initiated in the first phase by a nominal lateral rail or lane. The world model 164 may provide an explicit list of all lanes in an exploded format, including, for example, all junctions of the lane the autonomous vehicle is currently in, as well as lanes into which the autonomous vehicle may consider changing with one or more successive lane changes. Each lane the autonomous vehicle is currently in provides a lane following action. Each lane into which the autonomous vehicle may possibly change provides a lane change action. Similarly, the autonomous vehicle may consider overtaking actions with the intent to exit and return to its own lane while temporarily in another lane, which may even be that of oncoming traffic. The autonomous vehicle may also consider stopping on the shoulder, which is a special lane in the world model. The autonomous vehicle may also consider a U-turn. Each of these actions may even have nominal rails and lane edges from the world model, and are often tied to lanes in a map that provides information about the path beyond the live perception range.

[0039] For parking, the parking prowling state is based on the same action implementation as lane keeping via incentives provided by the parking lane planner 126. During parking prowling, the behavior planner monitors for the presence of vacant parking spaces within the line of sight. This may be provided by live perception (e.g., using neural networks to detect parking spaces) or a combination of mapped parking spaces and obstacle perception, or both redundantly. This process may be translated by the parking planner 144 into a motion plan - which may differ slightly from the nominal rail - and may provide a goal quadrilateral when a vacant parking space is detected. The parking planner 144 may perform a search for a pose trajectory between the current pose and a goal pose without obstacle interference. In one or more embodiments, the algorithm A * / D * A search algorithm from the family may be used to search for a pose trajectory. The search algorithm may be used to determine whether the space is actually kinematically feasible to arrive at, and if so, to determine one or more sequences of poses to arrive there. In one or more embodiments, this search may be performed in the current obstacle environment (e.g., at a particular time). This means that motion effects, such as a cyclist trying to quickly overtake the autonomous vehicle between the autonomous vehicle and the space, have not yet been fully taken into account. For this reason, the plan from the parking planner 144 is considered as a nominal rail for the rest of the process, which will take these motion effects into account to ensure that the actions of the autonomous vehicle are safe (note that other actors may of course move quickly even when the autonomous vehicle is driving slowly). Nevertheless, the parking planner 144 is powerful because it may discover things such as a three-point turn to enter the space that would not have been found otherwise.

[0040] The calling component 146 may work in a similar manner to the parking planner 144, but in reverse. The parking planner 144 is used to find a motion plan that moves the autonomous vehicle to a nearby portion of the lane graph. On the lane graph, the autonomous vehicle proceeds as usual using lane following actions. In the absence of a map, or if the autonomous vehicle is in LIMP mode, the autonomous vehicle may only have a general sense of which compass heading to travel in. In this case, the autonomous vehicle may simply use that compass heading to form a nominal rail to guide the rest of the process. The computational flow of each action then proceeds similarly. At the end of the process followed by the action generator 142, each action may be associated with a nominal rail and / or lane edges and lanes from the map.

[0041] The longitudinal preconstraint component 150 may receive nominal rails corresponding to the actions generated by the action generator 142. Each of these nominal rails may be evaluated by the longitudinal preconstraint component 150. The longitudinal preconstraint component 150 considers basic longitudinal constraints per lane (or per nominal rail). Different constraints may be considered by different components. The components of the longitudinal preconstraint component 150 include curve speed adaptation 152, speed regulation 154, distance maintenance 156, safety braking 158, and / or yield planner 159. The output of the longitudinal preconstraint component 150 may include acceleration constraints that limit the speed associated with each nominal rail / lane that arrived as an input. As such, the longitudinal preconstraint component 150 may enforce the lane / rail with acceleration constraints, speed constraints, or distance constraints.

[0042] Curve Speed ​​Adaptation 152 may be performed by considering the centripetal forces that may occur in the future when following the curvature of a given lane / rail at a certain speed. This occurs currently in the relevant map lanes, but may also occur in the live perceived lanes when their curvature in range becomes accurate enough for this task. The result of Curve Speed ​​Adaptation 152 may be a calculated acceleration constraint based on a tolerance range of centripetal forces. The centripetal force constraint may be based on passenger comfort, safety, and / or other factors.

[0043] Speed ​​adjustment 154 may implement standard compliance with speed limits when using this lane. Distance maintenance 156 and safety braking 158 may perform several longitudinal control duties related to vehicles and other contenders at high speeds and long distances. Distance maintenance 156 component may generate acceleration constraints that enforce the distance between another moving object and the autonomous vehicle. Safety braking 158 component may generate similar constraints that ensure the ability to stop the autonomous vehicle via braking to avoid potential collisions that may occur under different modeled conditions. Most of the obstacle avoidance related longitudinal control may be handled by motion planner 172. Aside from the long distances associated with high speeds, the perceived granularity of things such as the exact lateral position and range of the lead vehicle in the lane may not be fine enough to guarantee motion planning around the lateral of the lead vehicle. At the highest distance and speed associated with oncoming traffic at full speed or at full speed encountering a stopped vehicle or hazard, the granularity of the decision can be to slow down or not to slow down because something appears to be slow in the autonomous vehicle's current lane. The logic is similar for slow traffic or other conditions in lanes next to the autonomous vehicle's current lane that prompt the autonomous vehicle to slow down. When the distance is high, Obstacle-In-Path-Assignment (OIPA) can be used to act on the obstacle. When an obstacle approaches, fine adjustments such as lateral nudging instead of braking can be selected by the motion planner 172. This architecture also allows the use of already tuned longitudinal limiters for high-speed highway driving without significant modification.

[0044] The yield planner 159 may handle yield on a per lane (or lane-by-lane basis), in the sense that some yields do not need to consider exactly where the autonomous vehicle is laterally within the lane to know that yield is required. This includes handling stops or yield lines where others have right-of-way. This existence of such conditions is provided in the world model 164 via wait conditions. Wait conditions are encoded in sections of the own path between the entrance and exit lines, competitor paths that have similar competing sections, and wait states that may be considered right-of-way situations. The world model 164 may also provide competitor path associations by which the autonomous vehicle's behavior and control module can deduce that other competitors have right-of-way (or that the autonomous vehicle has right-of-way). Wait conditions may include a subset of the above, for example, a traffic light for timing an on-ramp that has no direct link to a competitor path and only a stop line with a wait state associated with it. The architectural separation of concerns achieved in the wait condition is that it encodes an expectation, rule, or convention. It can then be the responsibility of the behavior planner 140 to monitor and implement the yield and determine the actual kinematic feasibility or any hazards. The yield planner 159 considers the deceleration required to stop at a line with a STOP_AT_ENTRY wait condition (e.g., induced by a red light). The yield planner 149 can also address a TAKE_WAY_TRANSIENT (e.g., induced by a yellow light) and calculate what level of deceleration would be required to stop at the line or before some margin of overrun. This allows the yield planner 149 to decide between scenarios where, for example, the light turns yellow almost past the line, where a stop may not be mandatory, and scenarios when it is appropriate to stop.

[0045] The Yield Planner 159 also handles YIELD_CONTENTION wait states that may indicate that the autonomous vehicle is expected to yield to a competing path. These cases include crossing pedestrians, stop signs where a competitor has the right of way, yield signs, approaching traffic for an unprotected left turn, right-of-way in Europe, and many more cases.

[0046] These cases all have in common the need to consider competitors in the relevant competitor path and ensure that the autonomous vehicle does not impede them, and also ensure that it is clear to them that the autonomous vehicle is yielding. To do so, the yield planner 159 may perform a simplified (and therefore computationally efficient) form of motion planning. In one or more embodiments, the yield planner uses a simplified model in which the autonomous vehicle and competing actors move along their respective lanes and require a full range of some margin added to each lane for passing. The yield planner 159 may calculate for a particular pair of autonomous vehicle and competitor longitudinal profiles whether their claimed sets will intersect if their future is rolled out (more on this later in the section on motion planning). In essence, the claimed set is the set of points in front of any actor that is "claimed" by its stopping distance. Even if a physical intersection does not exist, a claimed intersection may imply a dangerous situation is occurring that may be difficult to control, and at a minimum may upset others or call into question the intentions of the yield, so it may be better and more conservative to consider the claimed intersection rather than simply the physical intersection. The acceleration plan that results in the claimed intersection (or the actual physical intersection) may be output as a longitudinal constraint.

[0047] In cases where the autonomous vehicle has a yield obligation, the yield planner 159 can analyze whether the choice of longitudinal progression interferes with any of the competitor's reasonable choices, i.e., the autonomous vehicle would remain well out of reach regardless of whether the competitor slows down or speeds up, or chooses a left turn or goes straight, etc.

[0048] The yield planner 159 may also consider joint wait conditions, also called wait groups. For example, because there is no safe intermediate place to wait for crossing traffic from the right, the yield planner 159 may need to ensure that crossing traffic from both the left and right can be cleared before attempting to clear only the first traffic. Another such example is a crossing pedestrian after a left turn when performing an unprotected left. In this case, the yield planner 159 may have to ensure that it can clear both the oncoming traffic and the crossing pedestrian at the same time. This is signaled in the world model by grouping those wait conditions into wait groups.

[0049] The yield planner 159 may also handle the distinction between the cases YIELD_ENTRY and YIELD_CONTENTION. The former expects the autonomous vehicle to stay at the entry line until it is ready to clear the conflict without delay. In the latter case, the autonomous vehicle may creep forward, constrained only by the actual conflict itself, allowing it to drive forward into the intersection while waiting and spacing out the oncoming traffic. In this situation, the constraint may exclude all speeds that are inconsistent with creeping forward.

[0050] The yield planner 159 may also handle multi-way stop intersections signaled by the STOPPED_FIRST_HAS_PRECEDENCE wait state. In this case (the canonical example is a four-way stop intersection in the United States), the right of way is derived from who stopped first heading into the intersection. In this regard, the yield planner 159 relies on a combination of short-term prediction and an analysis of who stopped first heading into the intersection. After the right of way is determined, the same analysis of yield, yield from entrance or continue on, is applied. If both the autonomous vehicle and a contender are stopped at the intersection, the short-term prediction uses history to predict the future movement of the contender. In a situation where another contender has stopped before the autonomous vehicle, the prediction should be that it will move now (or that some contender will move). If the autonomous vehicle is the first of all contenders to be stopped, the prediction of all contenders should be that they will stay where they were. This is powerful because it allows a policy to be almost as simple as "if no one else is passing, don't pass, otherwise don't wait". Note that it depends on having a sufficient amount of training data for the relevant cases being considered. This form of short-term prediction can also be used in motion planning, similarly helping to predict whether to yield in unstructured cases.

[0051] For the waiting state NEGOTIATE, the yield planner 159 delegates responsibility regarding visible contenders to the motion planner 172. For this state, there may be no explicit expectation of precedence of either party, but instead expect to negotiate the situation according to what is kinematically feasible, safe, and efficient. The motion planner 172 will do what is kinematically feasible and safe given the predictions and avoid the intersections claimed, but will also not yield. This will effectively go down as many paths as possible, generating aggressive assertive behavior. However, the yield planner 159 does consider unseen actors on the contenders' paths. This ensures a straightforward discussion of visibility.

[0052] The world model 164 represents visibility in three ways: as unknown regions in a radial distance map, as polygonal occlusion boundaries, and as unknown path occupancy. In a full implementation of the perception and mapping system, these may all be consistent with occlusion areas appearing as regions with their boundaries, and may also affect path occupancy without conflict. Occlusion boundaries may be generated by end-to-end trained neural networks as are path occupancies. However, this may not be available in some implementations, and therefore planning may use path occupancy. Alternative embodiments may use occlusion boundaries to generate the expanding claim set of the motion planner 172. When the yield planner 159 finds a contender path with no TAKE_WAY wait states and with unknown occupied vertices (or short distances), it guesses which contenders have the worst case speeds (total interval from lowest to highest) at those vertices and performs yield planning with these hypothetical contenders. This provides the architectural ability to deal with all occlusion contenders coming down known contender paths.

[0053] The competitor paths can be known from a map or by live detection, or both. One implementation of this could be based on analyzing the visibility by projecting vertices from the mapped competitor paths, on paths where competitors are expected to be, to the sensor, and comparing distances to a depth map. This can in principle handle occlusions from both static and dynamic occluders.

[0054] The combined output from the various components of the longitudinal preconstraint component 150 may be a plurality of longitudinally constrained nominal rails. The rails generated by the action generator 142 may be combined with a plurality of different acceleration plans to generate a number of trajectories along the rails. The plurality of longitudinally constrained nominal rails includes rails that satisfy the various constraints calculated by the longitudinal preconstraint component 150. The plurality of longitudinally constrained nominal rails may be passed to the hypothesis generation component 160 for further refinement.

[0055] Motion planning goes through hypothesis generation and evaluation (motion planner 172). The idea is that motion planner 172 evaluates a given trajectory (complete planned pose into the future of the autonomous vehicle) very thoroughly for quality, and since the space of all trajectories is too large to search, hypothesis generation 160 provides likely or plausible trajectories.

[0056] According to an embodiment, a high degree of variability can be used for the motion planner 172 hypothesis generation. In some embodiments, the central component is a path fan generator 162 that generates laterally varying choices starting from and around a longitudinally constrained nominal rail while maintaining its curvature. This is adapted to the intent of lane-navigating while nudging and turning to pass static and dynamic obstacles. The longitudinal variations within the longitudinal constraints can be added to each lateral choice, yielding a 2D grid of motion planner trajectories. This is used for standard lane-following cases and actions such as parking prowling, parking tail, calling out and creeping (for example and without constraints) along with it.

[0057] An example trajectory is shown in FIG. 2. The nominal rail may correspond to the right lane in which the autonomous vehicle is located. As shown, a series of cones 224 present obstacles to be avoided. The path fan generator 162 may generate different laterally varying choices associated with the nominal rail. The varying choices include path 202, path 204, path 206, path 208, path 210, path 212, path 214, path 216, path 218, path 220, and path 222. Each path may be associated with a speed profile as shown in profile graph 230. As shown, paths 218, 220, and 222 come to a complete stop within a short distance. In contrast, path 202 would allow a higher speed to be maintained.

[0058] Lane changes may be performed in three stages, depicted as follows: 1) following the current lane while hoping for a gap in the other lane, 2) following the current lane while performing speed adaptation to aggressively try to get into the gap, and finally pushing laterally to try to use the gap or create a gap. Note that multiple iterations over hypothesis generation and evaluation may be used, building on the results of previous rounds. For lane changes, one iteration of regular lane following (path fan generation and motion planner evaluation) may be performed first. For the first stage of lane change, that is all that may be needed. For the speed adaptation stage, the best lateral choice is determined or chosen and taken into account. This may be done using the best lateral choice and a few additional choices (e.g., one to the left and one to the right of the best choice) to perform a search for the best speed adaptation. This may be done, for example, by an S-curve generator 163, by generating a two-dimensional family of longitudinal S-curves swept by acceleration amount and switching time. The hypotheses are then passed through the motion planner 172 for evaluation. During the pushing phase, the best results may be used to explore the amount of lateral pushing.

[0059] Figures 3 and 4 illustrate hypothesis generation in the context of a lane change constrained by a nearby vehicle. Figure 3 includes a forward view 302 showing a vehicle in the left lane directly ahead of the autonomous vehicle. A driver's side rear-facing view 304 shows a car in the left lane directly behind the autonomous vehicle. A passenger's side rear-facing view 308 shows a cone that the autonomous vehicle has passed. And a rear view 310 also shows a car in the left lane directly behind the autonomous vehicle. These views can be used to determine the location of nearby vehicles, which can then be used to search for a hypothetical trajectory consistent with a lane change into the space between the trailing and leading vehicles in the left lane.

[0060] 4 illustrates possible tracks within a nominal rail generated by track fan generator 162. These tracks include track 402, track 404, track 406, track 408, track 410, track 412, track 414, track 416, track 418, track 420, and track 422. A speed profile graph 430 illustrates the speeds available for some of the tracks.

[0061] Hypothesis generation 160 is flexible and can use other less structured approaches such as dynamic programming to search for free-form paths, implemented by a dynamic programming component 166, or a linear-quadratic regulator (LQR) / obstacle-aware MPC control 168 that follows an initial path (e.g., nominal rail) and iteratively adjusts it to stay away from obstacles. This can be done either in static scenes where the maneuver planner ignores maneuvers, trusting that it will then ascertain and avoid dangerous maneuver-induced situations, or by using the claim set directly in the search.

[0062] Each hypothetical trajectory may be evaluated by the plan evaluation component 171 to identify the highest quality or optimal trajectory, where quality includes many considerations but may be quantified by an optimization score. Different qualities may have different weights when calculating the score. At a high level, an ideal drive may be traced back to a period of five categories. Objectives may include, but are not limited to, maximizing comfort and making progress (reaching the destination with the least expenditure of resources such as time, money, fuel and clothing). Another objective may be maximizing crash safety (obstacles), other things being equal, staying in lanes (path), and operating while following applicable rules and conventions (waiting conditions). Some of these priorities are tied to the motion plan in a tangible way. Progression and smoothness may be evaluated directly from the motion plan. For other aspects, it may be difficult to make a direct link. For example, a good strategy for distance keeping may be to stay close to avoid cut-ins from vehicles that are not even visible yet, which is not directly discernible from analyzing the obstacles in this scene instance. However, direct kinematic constraints from obstacles around the autonomous vehicle can be evaluated in the short-term prediction, as described further below.

[0063] In one embodiment, the optimization score is normalized to time. The sequential time rewards of potential future positions may be used directly in the score and as a starting point, while other components contributing to the score may add time as a penalty. For example, a predicted collision would add 100 hours, which may represent a significant adjustment or offset (e.g., a penalty) and may cause this trajectory not to be selected. Lesser, but still undesirable conditions may result in lesser adjustments. The adjustments may be scaled or normalized according to the route distance or estimated time of travel. In this way, adjustments resulting from undesirable features of the route will add more time for longer routes and less time for shorter routes. Normalization may occur by calculating the adjustment as a percentage of the time to the destination. For example, some conditions on the route may result in an adjustment of 2% of the remaining estimated time to reach the destination.

[0064] The motion planner 172 also has a period of preference to stay close to the nominal rail or between the edges of the lane under consideration. These preferences may be used to calculate the optimization score. When lane following, there may be a strong preference not to span the edge of the lane line, even if the lane edge is not a physical obstacle. However, if spanning the edge of the lane line is the only reasonable way to avoid a collision, it may be more preferred. However, this is different from making a planned controlled lane change. Specifically, in a planned controlled lane change, there is time to initiate the indicator signal and give the other time to notice the signal. For that reason, the motion planner 172 considers trajectories differently in the context of lane following and lane change, even though they may be the same trajectory. The optimization score of a trajectory may increase the closer the lane center is followed. When the trajectory crosses a lane line, the optimization score may decrease. The amount of score decrease may depend on the context of the crossing lanes. For example, crossing a dashed line may involve a smaller penalty than crossing a solid or double solid line. Crossing into oncoming traffic may result in a much larger penalty.

[0065] The motion planner 172 may also include some periods that are informed by yield requirements to other actors, but at least some of the waiting conditions may be addressed by the yield planner 159. The motion planner 172 may also include periods that prefer not sharing a lane side-by-side, or even side-by-side with a competitor in another lane. This helps with behaviors such as accommodating a lane-sharing motorcycle or side-by-side lane sharing when pushing through a competitive low-speed merge, while not initiating those conditions. Similarly, it may prefer to be able to pass a parked vehicle or be in someone's blind spot, but make those conditions temporary if possible.

[0066] The core of motion planning may include analyzing motion plans for obstacle competitors. The analysis may include monitoring intersections between claim sets. Instead of considering whether there is a collision between objects in the present or future, motion planning according to the disclosed embodiment considers whether there is an intersection between claim sets. This is a way to address how to upgrade static motion planning to an environment with competitors moving at speeds that do not allow for near-instantaneous stopping. A claim set is a shape in space-time that an actor will trace while attempting to safely stop and line up laterally with the road. An actor virtually "claims" this set as it needs to maintain collision safety. The idea is that even if the bodies do not intersect, an intersection of claim sets would be difficult to allow while maintaining some form of controlled safety. On the contrary, if the claim sets are respected by each other, the actors can stay within them. If the actors do, the claim sets will run to the end in space-time and will not expand. This allows the static motion planning problem to be upgraded to one with motion. A moving actor cannot stop immediately, but a claim set can. Another benefit of the present approach is that it does not rely on training data or predictions on nearby or actual collision situations that are rare or difficult to encounter. The present approach also ensures collision safety beyond simply escaping without a physical collision.

[0067] In one or more embodiments, the safety force field 184 approach may be operational within the behavior planner 140 and control system of the autonomous vehicle and is not interrupted by other functions. It reactively prevents forbidden momentary control and changes it to permitted control. However, the design is not to have it triggered in normal operation. The motion planner 172 deals with similar ground constraints in a more proactive manner. It essentially asks the question "If I move like this into the future and the competitor moves as predicted subject to that, will the constraints be triggered?". Then, a motion plan that is unlikely to trigger the constraints in the future may be preferred, or a motion plan that triggers them later in the future may be preferred. Thus, the autonomous vehicle can start decelerating or making lateral adjustments much earlier and therefore drive more smoothly and safely. Note that the ground constraints allow the self-vehicle, and therefore the operation, to be very assertive. The motion planner 172 can generate deterministic behavior unless the predictions suggest that the autonomous vehicle will not enter a dangerous situation and prior constraints by the yield planner 159 occur.

[0068] According to an embodiment, the motion planner 172 can use a CUDA implementation to evaluate multiple future claimed intersections. CUDA is a parallel computing platform and application programming interface model created by NVIDIA Corporation. CUDA allows software developers and software engineers to use CUDA-enabled graphics processing units for general-purpose processing. This CUDA implementation considers multiple choices of parallel trajectories, typically a two-dimensional family of trajectories (e.g., a 10x10 grid) that vary both across lateral paths and by longitudinal velocity. The motion planner 172 then checks, for each of those future trajectories and for each time step, whether the predicted actor states lead to a claimed intersection associated with the autonomous vehicle. This is done efficiently with a slightly more conservative constraint than the claimed intersection of space-time. Instead, other actors should be clear longitudinally along the path of the autonomous vehicle (this is similar to a simple check that the claim sets do not intersect when viewed from the side, looking only at a projection that preserves longitudinal and temporal directions), or the claim sets are separated when viewed from a top view that projects temporal directions. An approximation of the claim sets of all actors as they are projected in time is used (2D claim sets). Collision checks are then performed on polygonal representations of those 2D claim sets. Note that static obstacles can be used directly as 2D claim sets. The above process means that collision checks are performed between polygonal representations of multiple scene configurations in parallel. A longitudinal pre-constraint is available to the motion planner 172 per action, which can be used before the motion planner 172 evaluation to modify hypothesis generation as well as after evaluation to essentially block or strongly favor avoiding faster trajectories.

[0069] The Last Safe Arrival (LSA) planner 174 works together with the motion planner 172. The LSA planner 174 constraint arises from a more conservative assumption than the SFF 184 that may be applied to pedestrians who have no reason to believe they are cautious or may behave erratically. It is derived by assuming that contenders (pedestrians) may accelerate in any way and do not start decelerating until an autonomous vehicle literally occupies their path. This results in a constraint similar to the claimed intersection from the SFF 184 that is applied as a constraint on the planner (LSA planner 174) that forecasts instantaneously and into the future above the core constraint. If the same forecast as the core constraint is used, which is a fairly conservative but safe starting point for pedestrians, then this amounts to checking that none of the current or future claim sets claim a point after the LSA time that an autonomous vehicle has not already claimed. This can be done efficiently by checking the furthest ahead of each claim set plus each arrival depending on the current stopping distance. This runs in parallel with the motion planner evaluation of the same set of trajectories and may be applied to pedestrians only.

[0070] The short-term prediction component 161 can generate predicted locations of obstacles using both analytical and data-driven approaches. The analytical approach takes the obstacle fences given by the world model 164 and predicts that they will continue along a velocity vector given some incremental adjustment back to the lane structure (note that there may be multiple lane choices per competitor, such as left turn, right turn, U-turn, or straight ahead). The strength of analytical prediction is that low but non-zero probabilities can be encoded and associated with things that are known to be relevant but may be rare in the training data, such as cutting in or emergency braking.

[0071] Data-driven methods use the history of perceived scenes to predict what a competitor will do, the future being trained from episodes known as training data. Real data can be particularly useful for this form of training. According to an embodiment, this type of training can be done on the output of an automatic system, thus benefiting from large amounts of data without a manually generated ground truth.

[0072] The plan selector 175 selects a plan based on the ratings or scores provided by the plan evaluation component 171. In one instance, the trajectory with the highest quality score is selected. The plan selector 175 may also provide longitudinal conditioning 176 and lateral conditioning 178, both of which provide small smoothing to the selected trajectory.

[0073] The selected plan may be evaluated by SFF 184, LSA 186, Lane Keeping Assistant (LKA) 188, and (Automatic Emergency Braking) AEB 190. The motion planner 172 or other components may take into account inputs from these components when evaluating the trajectory or, at least, compliance with the constraints they impose. For example, the safety braking component 158 ​​may generate constraints intended to avoid automatic emergency braking.

[0074] The techniques described herein can improve upon current techniques by identifying optimal trajectories using an iterative approach. In a first iteration, an initial plurality of hypothetical trajectories may be evaluated with the trajectory having the highest optimization score acting as a seed for generating additional hypothetical trajectories for evaluation. In a second iteration, a second plurality of hypothetical trajectories may be generated by slightly modifying various parameters of the seed trajectory. For example, the seed trajectory may be communicated to the track fan generator 162 and used as an input instead of the nominal rail. The track fan generator may then generate a second plurality of trajectories that are low in lateral variation (lower lateral irregularities than those used in the first iteration at the nominal rail) and low in longitudinal variation. The second plurality of hypothetical trajectories may then be evaluated to determine which one of these trajectories has a higher optimization score than the seed trajectory. The hypothetical trajectory with the highest optimization score may be selected for implementation.

[0075] A Model Predictive Controller (MPC) 180 takes the chosen trajectory and selects the instantaneous lateral and longitudinal acceleration control 195 using a more refined vehicle model. The MPC control can work by calculating the future trajectory based on the control sequence and the model, and iteratively adjusting the control sequence with a nonlinear optimizer to optimize a cost function. In this case, the cost function can be designed to find a trade-off between smoothness and following the required trajectory (this stage is not aware of obstacles, but it can be set to closely follow the trajectory, and the smoothness of the trajectory should be ensured by the motion planning stage). The SFF 184 and other components can also be used to gate unacceptable acceleration choices.

[0076] 5-7, each block of methods 500, 600, and 700 described herein includes computational processes that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. The methods may also be implemented as computer usable instructions stored on a computer storage medium. The methods may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, methods 500, 600, and 700 are described with respect to behavior planning system 100 of FIG. 1, by way of example. However, these methods may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to those described herein.

[0077] 5 is a flow diagram illustrating a method 500 for selecting a trajectory for an autonomous vehicle according to some embodiments of the disclosure. The method 500 includes, at block 502, generating a lane graph including a first set of time rewards for potential future positions of the autonomous machine. The method 500 includes, at block 504, generating a plurality of possible trajectories that enable the autonomous machine to complete an autonomous machine action. The method 500 includes, at block 506, generating an optimization metric for the possible trajectories using the first set of time rewards as an input. The method 500 includes, at block 508, selecting a trajectory for implementation using the optimization metric. The method 500 includes, at block 510, implementing the trajectory using the autonomous machine.

[0078]

[0046] Referring now to Figure 6, Figure 6 is a flow diagram illustrating a method 600 for selecting a trajectory for an autonomous vehicle, according to some embodiments of the present disclosure. The method 600 includes, at block 602, generating a scene prediction including predicted future object positions of objects detected by one or more sensors corresponding to the autonomous machine;

[0079] The method 600 includes, at block 602, generating a plurality of possible trajectories for an autonomous machine action, the plurality of possible trajectories having a plurality of longitudinal conditions and a plurality of lateral conditions within a nominal lateral rail corresponding to the autonomous machine action. The method 600 includes, at block 604, evaluating the plurality of possible trajectories using the scene prediction and a first series of time rewards corresponding to a plurality of potential future positions of the autonomous machine to generate an optimization measure for the plurality of possible trajectories. The method 600 includes, at block 606, selecting a starting parameter set trajectory used to generate additional trajectories for evaluation using the optimization measure. The method 600 includes, at block 608, selecting an individual trajectory from the additional trajectories using the optimization measure. The method 600 includes, at block 610, implementing the individual trajectory for the autonomous machine.

[0080] Referring to FIG. 7, FIG. 7 is a flow diagram illustrating a method 700 for selecting a trajectory of an autonomous vehicle according to some embodiments of the present disclosure. The method 700 includes, at block 702, generating a plurality of possible trajectories for an autonomous machine action, the plurality of possible trajectories having a plurality of longitudinal conditions and a plurality of lateral conditions. The method 700 includes, at block 704, evaluating a trajectory within the plurality of possible trajectories using a scene prediction to generate an optimization metric for the trajectory. The method 700 includes, at block 706, selecting an individual trajectory from the plurality of possible trajectories using the optimization metric. The method 700 includes, at block 702, implementing an individual trajectory for the autonomous machine, where the plurality of longitudinal conditions are based at least on the scene prediction, and the plurality of lateral conditions are based at least on a nominal rail corresponding to a direction of travel of the autonomous machine.

[0081] Exemplary Autonomous Vehicle 8A is a diagram of an example autonomous vehicle 800 according to some embodiments of the present disclosure. Autonomous vehicle 800 (alternatively referred to herein as "vehicle 800") may include, but is not limited to, passenger vehicles, such as cars, trucks, buses, first responder vehicles, shuttles, electric or mopeds, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, submarines, drones, and / or other types of vehicles (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806 published June 15, 2018, Standard No. J3016-201609 published September 30, 2016, and previous and future versions of this standard). Mobile vehicle 800 may be capable of functionality according to one or more of autonomous driving levels 3 through 5. For example, mobile vehicle 800 may be capable of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the embodiment.

[0082] The mobile vehicle 800 may include components such as a chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the mobile vehicle. The mobile vehicle 800 may include a propulsion system 850, such as an internal combustion engine, a hybrid power plant, a fully electric engine, and / or another propulsion system type. The propulsion system 850 may be connected to a drive train of the mobile vehicle 800, which may include a transmission, to enable propulsion of the mobile vehicle 800. The propulsion system 850 may be controlled in response to receiving a signal from a throttle / accelerator 852.

[0083] A steering system 854, which may include a steering wheel, may be used to steer the vehicle 800 (e.g., along a desired course or route) when the propulsion system 850 is operating (e.g., when the vehicle is moving). The steering system 854 may receive signals from a steering actuator 856. The steering wheel may be an option for fully automated (Level 5) functionality.

[0084] Brake sensor system 846 may be used to operate vehicle brakes in response to receiving a signal from a brake actuator 848 and / or a brake sensor.

[0085] A controller 836, which may include one or more system on chip (SoC) 804 (FIG. 8C) and / or a GPU, may provide signals (e.g., representations of commands) to one or more components and / or systems of the vehicle 800. For example, the controller may send signals to operate vehicle brakes via one or more brake actuators 848, operate steering system 854 via one or more steering actuators 856, and operate propulsion system 850 via one or more throttle / accelerators 852. The controller 836 may include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) to process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist a driver in driving the vehicle 800. The controllers 836 may include a first controller 836 for autonomous driving functions, a second controller 836 for functional safety functions, a third controller 836 for artificial intelligence functions (e.g., computer vision), a fourth controller 836 for infotainment functions, a fifth controller 836 for redundancy in emergency situations, and / or other controllers. In some instances, a single controller 836 may handle two or more of the aforementioned functions, and two or more controllers 836 may handle a single function, and / or any combination thereof.

[0086] Controller 836 may provide signals to control one or more components and / or systems of vehicle 800 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data may be received from, for example, and without limitation, global navigation satellite system sensors 858 (e.g., global positioning system sensors), RADAR sensors 860, ultrasonic sensors 862, LIDAR sensors 864, inertial measurement unit (IMU) sensors 866 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 896, stereo cameras 868, wide-view cameras 870 (e.g., fisheye cameras), infrared cameras 872, surround cameras 874 (e.g., 360 degree cameras), long-range and / or medium-range cameras 898, speed sensors 844 (e.g., for measuring the speed of the moving vehicle 800), vibration sensors 842, steering sensors 840, brake sensors (e.g., as part of a brake sensor system 846), and / or other sensor types.

[0087] One or more of the controllers 836 may receive input (e.g., represented by input data) from the instrument cluster 832 of the vehicle 800 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 834, an audible annunciator, a loudspeaker, and / or other components of the vehicle 800. The output may include information such as vehicle velocity, speed, time, map data (e.g., HD map 822 of FIG. 8C), position data (e.g., the position of the vehicle 800 on a map, etc.), direction, the positions of other vehicles (e.g., occupancy grid), information about objects and object situations as perceived by the controller 836, etc. For example, the HMI display 834 may display information regarding the presence of one or more objects (e.g., road signs, warning signs, changing traffic signals, etc.) and / or a driving maneuver that the moving vehicle has performed, is performing, or will perform (e.g., changing lanes now, taking exit 34B in 2 miles, etc.).

[0088] The mobile vehicle 800 further includes a network interface 824 capable of communicating over one or more networks using one or more wireless antennas 826 and / or a modem. For example, the network interface 824 may be capable of communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The wireless antennas 826 may also enable communication between objects in the environment (e.g., mobile vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc., and / or low power wide-area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0089] 8B is an illustration of camera positions and fields of view of the example autonomous vehicle 800 of FIG. 8A, according to some embodiments of the present disclosure. The cameras and their respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at different positions on the vehicle 800.

[0090] The camera type of the camera may include, but is not limited to, a digital camera that may be adapted for use with the components and / or systems of the mobile vehicle 800. The camera may be capable of operating at automotive safety integrity level (ASIL) B and / or at another ASIL. The camera type may be capable of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some instances, the color filter array may include a red clear clear clear (RCCC) color filter array, a red clear clear blue (RCCB) color filter array, a red blue green clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras with RCCC, RCCB, and / or RBGC color filter arrays, may be used in an effort to increase light sensitivity.

[0091] In some instances, one or more of the cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more of the cameras (e.g., all of the cameras) may simultaneously record and provide image data (e.g., video).

[0092] One or more of the cameras may be mounted in a mounting part, such as a custom designed (3D printed) part, to filter out stray light and reflections from within the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capabilities. With reference to a side mirror mounting part, the side mirror part may be custom 3D printed such that the camera mounting plate fits the shape of the side mirror. In some instances, the camera may be integrated into the side mirror. For side view cameras, the cameras may also be integrated into four posts at each corner of the cabin.

[0093] A camera (e.g., a forward-facing camera) with a field of view that includes a portion of the environment in front of the vehicle 800 may be used for surround view to help identify the forward path and obstacles and, with the aid of one or more controllers 836 and / or control SoCs, provide information essential to generating an occupancy grid and / or determining a preferred vehicle path. Forward-facing cameras may be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras may also be used for ADAS functions and systems, including other functions such as lane departure warning ("LDW"), autonomous cruise control ("ACC"), and / or traffic sign recognition.

[0094] A variety of cameras may be used in a forward-facing configuration, including, for example, a monocular camera platform including a complementary metal oxide semiconductor (CMOS) color imager. Another example may be a wide-view camera 870 that may be used to capture objects entering the view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although only one wide-view camera is shown in FIG. 8B, there may be any number of wide-view cameras 870 in the vehicle 800. In addition, a long-range camera 898 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, especially for objects for which the neural network has not yet been trained. The long-range camera 898 may also be used for object detection and classification, as well as basic object tracking.

[0095] One or more stereo cameras 868 may also be included in the forward-facing configuration. The stereo camera 868 may include an integrated control unit with an extensible processing unit that may provide programmable logic (FPGA) and a multi-core microprocessor with CAN or Ethernet interfaces integrated on a single chip. Such a unit may be used to generate a 3D map of the vehicle's environment, including distance estimates for all points in the image. An alternative stereo camera 868 may include a compact stereo vision sensor that may include two camera lenses (one left and one right) and an image processing chip that can measure distances from the vehicle to objects of interest and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 868 may be used in addition to or instead of those described herein.

[0096] A camera having a field of view that includes a portion of the environment to the side of the moving vehicle 800 (e.g., a side-view camera) may be used for surround view, providing information used to create and update the occupancy grid and to generate side impact collision warnings. For example, surround cameras 874 (e.g., four surround cameras 874 as shown in FIG. 8B) may be positioned on the moving vehicle 800. The surround cameras 874 may include a wide-view camera 870, a fish-eye camera, a 360-degree camera, and / or the like. For example, four fish-eye cameras may be positioned at the front, back, and sides of the moving vehicle. In an alternative arrangement, the moving vehicle may use three surround cameras 874 (e.g., left, right, and rear) and may utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround view camera.

[0097] A camera having a field of view that includes the portion of the environment behind the moving vehicle 800 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creating and updating an occupancy grid. As described herein, a wide variety of cameras may be used, including, but not limited to, cameras that are also suitable as forward-facing cameras (e.g., long-range and / or mid-range camera 898, stereo camera 868, infrared camera 872, etc.).

[0098] FIG. 8C is a block diagram of an example system architecture of the example autonomous vehicle 800 of FIG. 8A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted altogether. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities may be implemented by hardware, firmware, and / or software. For example, various functions may be implemented by a processor executing instructions stored in a memory.

[0099] Each of the components, features, and systems of the mobile vehicle 800 of FIG. 8C are shown as connected via a bus 802. The bus 802 may include a controller area network (CAN) data interface (alternatively referred to as a "CAN bus"). The CAN may be a network within the mobile vehicle 800 that is used to help control various features and functions of the mobile vehicle 800, such as actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0100] The bus 802 is described herein as being a CAN bus, but this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or as an alternative to a CAN bus. Additionally, a single line is used to represent the bus 802, but this is not intended to be limiting. There may be any number of buses 802, which may include, for example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some instances, two or more buses 802 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 802 may be used for collision avoidance functions and a second bus 802 may be used for actuation control. In any instance, each bus 802 may communicate with any of the components of the vehicle 800, and two or more buses 802 may communicate with the same components. In some examples, each SoC 804, each controller 836, and / or each computer in the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 800) and may be connected to a common bus, such as a CAN bus.

[0101] Mobile vehicle 800 may include one or more controllers 836, such as those described herein with respect to FIGURE 8A. Controller 836 may be used for a variety of functions. Controller 836 may be coupled to any of a variety of other components and systems of mobile vehicle 800 and may be used for control of mobile vehicle 800, artificial intelligence of mobile vehicle 800, infotainment for mobile vehicle 800, and / or the like.

[0102] The mobile vehicle 800 may include a system on a chip (SoC) 804. The SoC 804 may include a CPU 806, a GPU 808, a processor 810, a cache 812, an accelerator 814, a data store 816, and / or other components and features not shown. The SoC 804 may be used to control the mobile vehicle 800 in a variety of platforms and systems. For example, the SoC 804 may be combined in a system (e.g., the system of the mobile vehicle 800) having an HD map 822 that can obtain map refreshes and / or updates via a network interface 824 from one or more servers (e.g., server 878 of FIG. 8D ).

[0103] CPU 806 may include a CPU cluster or CPU complex (alternatively referred to as a "CCPLEX"). CPU 806 may include multiple cores and / or L2 caches. For example, in some embodiments, CPU 806 may include eight cores in a coherent multiprocessor configuration. In some embodiments, CPU 806 may include four dual-core clusters, each with its own dedicated L2 cache (e.g., a 2MBL2 cache). CPU 806 (e.g., CCPLEX) may be configured to support concurrent cluster operation allowing any combination of CPU 806 clusters to be active at any given time.

[0104] The CPU 806 may implement power management capabilities including one or more of the following features: individual hardware blocks may be automatically clock gated when idle to conserve dynamic power, each core clock may be gated when the core is not actively executing instructions by execution of WFI / WFE instructions, each core may be independently power gated, each core cluster may be independently clock gated when all cores are clock gated or power gated, and / or each core cluster may be independently power gated when all cores are power gated. The CPU 806 may further implement an enhanced algorithm for managing power states where allowed power states and expected wake-up times are specified and the hardware / microcode determines the best power state for entering the cores, clusters, and CCPLEXes. The processing cores may support simplified power state entry sequences in software where work is offloaded to the microcode.

[0105] GPU808 may include an integrated GPU (alternatively referred to herein as an "iGPU"). GPU808 may be programmable and efficient for parallel workloads. In some instances, GPU808 may use an enhanced tensor instruction set. GPU808 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache having at least 96 KB storage capacity) and two or more of the streaming microprocessors may share a cache (e.g., an L2 cache having 512 KB storage capacity). In some embodiments, GPU808 may include at least eight streaming microprocessors. GPU808 may use a computational application programming interface (API). In addition, GPU808 may use one or more parallel computing platforms and / or programming models (e.g., CUDA from NVIDIA Corporation).

[0106] The GPU 808 may be power optimized for best performance in automotive and embedded use cases. For example, the GPU 808 may be fabricated on FinFETs (Fin field-effect transistors). However, this is not intended to be limiting and the GPU 808 may be fabricated using other semiconductor fabrication processes. Each streaming microprocessor may incorporate several mixed-precision processing cores partitioned into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores may be partitioned into four processing blocks. In such an example, each processing block may be assigned 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA tensor cores for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads having a mix of computational and addressing operations. Streaming microprocessors may include independent thread scheduling capabilities to allow finer grain synchronization and coordination among concurrent threads. Streaming microprocessors may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0107] The GPU 808 may include high bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide about 900GB / sec peak memory bandwidth in some instances. In some instances, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used in addition to or in place of the HBM memory.

[0108] The GPU 808 may include unified memory technology including access counters to enable more accurate movement of memory pages to the processors that access them most frequently, thereby improving the efficiency of storage ranges shared between processors. In some instances, address translation service (ATS) support may be used to enable the GPU 808 to directly access the CPU 806 page tables. In such instances, when the GPU 808 memory management unit (MMU) experiences a miss, an address translation request may be sent to the CPU 806. In response, the CPU 806 may consult its page tables for a virtual-to-real mapping of the address and send the translation back to the GPU 808. As such, the unified memory technology may enable a single unified virtual address space for both the CPU 806 and GPU 808 memories, thereby simplifying GPU 808 programming and porting of applications to the GPU 808.

[0109] In addition, GPU 808 may include access counters that can record the frequency of accesses of GPU 808 to the memory of other processors. The access counters can help ensure that memory pages are moved to the physical memory of the processors that are accessing the pages most frequently.

[0110] The SoC 804 may include any number of caches 812, including those described herein. For example, the cache 812 may include an L3 cache available to (e.g., connected to) both the CPU 806 and the GPU 808. The cache 812 may include a write-back cache that can record line state, such as by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending on the embodiment, although smaller cache sizes may be used.

[0111] The SoC 804 may include an arithmetic logic unit (ALU) that may be utilized in performing processing for any of the various tasks or operations (e.g., processing DNNs) of the vehicle 800. In addition, the SoC 804 may include a floating point unit (FPU) (or other math co-processor or numeric co-processor type) for performing mathematical operations within the system. For example, the SoC 104 may include one or more FPUs integrated as execution units within the CPU 806 and / or GPU 808.

[0112] The SoC 804 may include one or more accelerators 814 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 804 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other operations. The hardware acceleration cluster may be used to complement the GPU 808 and to offload some of the tasks of the GPU 808 (e.g., to free up more cycles for the GPU 808 to perform other tasks). As an example, the accelerator 814 may be used for target workloads that are stable enough to be suitable for acceleration (e.g., perception, convolutional neural networks (CNNs), etc.). As used herein, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and Faster RCNNs (e.g., as used for object detection).

[0113] The accelerator 814 (e.g., a hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured and optimized to perform image processing functions (e.g., CNN, RCNN, etc.). The DLA may further be optimized for a specific set of neural network types and floating point operations, as well as inference. The design of the DLA may provide more performance per millimeter than a general-purpose GPU, greatly exceeding the performance of a CPU. The TPU may perform several functions, including, for example, single instance convolution functions, supporting INT8, INT16, and FP16 data types for both features and weights, as well as post-processor functions.

[0114] The DLA can quickly and efficiently run neural networks, in particular CNNs, on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object identification and detection using data from camera sensors, CNNs for distance estimation using data from camera sensors, CNNs for emergency vehicle detection and identification using data from microphones, CNNs for face recognition and moving vehicle owner identification using data from camera sensors, and / or CNNs for security and / or safety related events.

[0115] The DLA can perform any function of the GPU 808, and by using an inference accelerator, for example, a designer can target either the DLA or the GPU 808 for any function. For example, a designer can focus on processing CNNs and floating point operations on the DLA, and offload other functions to the GPU 808 and / or other accelerators 814.

[0116] The accelerator 814 (e.g., hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA may provide a balance between performance and flexibility. For example, each PVA may include, but is not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors, for example.

[0117] The RISC cores may interact with an image sensor (e.g., an image sensor of any of the cameras described herein), an image signal processor, and / or the like. Each RISC core may include any amount of memory. The RISC cores may use any of a number of protocols, depending on the embodiment. In some instances, the RISC cores may run a real-time operating system (RTOS). The RISC cores may be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores may include an instruction cache and / or tightly coupled RAM.

[0118] The DMA may allow components of the PVA to access system memory independent of the CPU 806. The DMA may support any number of features used to provide optimizations to the PVA, including but not limited to supporting multi-dimensional addressing and / or circular addressing. In some instances, the DMA may support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0119] The vector processor may be a programmable processor that may be designed to efficiently and flexibly execute computer vision algorithm programming and provide signal processing capabilities. In some instances, the PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as, for example, a single instruction, multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW may increase throughput and speed.

[0120] Each vector processor may include an instruction cache and may be coupled to dedicated memory. As a result, in some instances, each vector processor may be configured to execute independently of other vector processors. In other instances, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other instances, the vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on sequential images or portions of an image. In particular, any number of PVAs may be included in a hardware accelerated cluster, and any number of vector processors may be included in each PVA. In addition, the PVAs may include additional error correcting code (ECC) memory to increase overall system security.

[0121] The accelerator 814 (e.g., a hardware acceleration cluster) may include a computer vision network-on-chip and SRAM to provide high bandwidth, low latency SRAM for the accelerator 814. In some instances, the on-chip memory may include, for example and without limitation, at least 4 MB of SRAM consisting of eight field configurable memory blocks that may be accessible by both the PVA and DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, controllers, and multiplexers. Any type of memory may be used. The PVA and DLA may access the memory through a backbone that provides the PVA and DLA with high speed access to the memory. The backbone may include a computer vision network-on-chip that interconnects the PVA and DLA to the memory (e.g., using the APB).

[0122] The computer vision network-on-chip may include an interface that determines, prior to transmission of any control signals / addresses / data, that both the PVA and DLA provide ready and valid signals. Such an interface may provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst type communication for continuous data transfer. This type of interface may follow ISO26262 or IEC61508 standards, although other standards and protocols may be used.

[0123] In some instances, the SoC 804 may include a real-time ray tracing hardware accelerator, such as described in U.S. Patent Application Serial No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and scale of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, for acoustic propagation synthesis and / or analysis, for simulation of SONAR systems, for general wave propagation simulation, for comparison against LIDAR data for localization and / or other functions, and / or for other uses. In some examples, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing related operations.

[0124] The accelerator 814 (e.g., a hardware accelerator cluster) has a variety of applications for autonomous driving. The PVA may be a programmable vision accelerator that can be used for critical processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are well suited to the domain of algorithms that require predictable processing at low power and low latency. In other words, the PVA works well with semi-dense or dense regular computations on small data sets that require predictable execution times with low latency and low power. Hence, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms, since the PVA is efficient in object detection and operating on integer computations.

[0125] For example, according to one embodiment of the present technology, the PVA is used to perform computer stereo vision. A semi-global matching based algorithm may be used in some instances, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require motion estimation / stereo matching on the fly (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions with input from two monocular cameras.

[0126] In some instances, PVA may be used to perform dense optical flow by processing raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR. In other instances, PVA is used for time of flight depth processing, for example, by processing raw time of flight data to provide processed time of flight data.

[0127] DLA can be used to implement any type of network to enhance control and driving safety, including, for example, a neural network that outputs a measure of the confidence of each object detection. Such confidence value can be interpreted as a probability or as providing a relative "weight" of each detection compared to other detections. This confidence value allows the system to make further decisions regarding which detections should be considered true positive detections rather than false positive detections. For example, the system can set a confidence threshold and consider only detections above the threshold as true positive detections. In an automatic emergency braking (AEB) system, a false positive detection would cause a moving vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered to trigger AEB. DLA can implement a neural network that regresses the confidence value. The neural network may receive as its inputs at least some subset of parameters, such as bounding box dimensions, ground plane estimates obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 866 outputs that correlate with the vehicle 800 orientation, distance, and 3D position estimates of the object obtained from the neural network and / or other sensors (e.g., LIDAR sensor 864 or RADAR sensor 860), and others.

[0128] The SoC 804 may include a data store 816 (e.g., a memory). The data store 816 may be an on-chip memory of the SoC 804 and may store the neural network to be executed on the GPU and / or DLA. In some instances, the data store 816 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 812 may comprise an L2 or L3 cache 812. References to the data store 816 may include references to memory associated with the PVA, DLA, and / or other accelerators 814, as described herein.

[0129] The SoC 804 may include one or more processors 810 (e.g., embedded processors). The processors 810 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. The boot and power management processor may be part of the SoC 804 boot sequence and may provide run-time power management services. The boot power and management processor may provide clock and voltage programming, assist with system low power state transitions, manage the SoC 804 thermal and temperature sensors, and / or manage the SoC 804 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 804 may use the ring oscillator to detect the temperature of the CPU 806, the GPU 808, and / or the accelerator 814. If the temperature is determined to exceed a threshold, the boot and power management processor may enter a temperature fault routine, place the SoC 804 in a lower power state and / or place the vehicle 800 in a Chauffeur safe shutdown mode (e.g., bring the vehicle 800 to a safe shutdown).

[0130] The processor 810 may further include a set of embedded processors that may perform the functions of an audio processing engine. The audio processing engine may be an audio subsystem that allows full hardware support of multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In some instances, the audio processing engine is a dedicated processor core that includes a digital signal processor with dedicated RAM.

[0131] The processor 810 may further include an always-on processor engine that can provide the necessary hardware features to support low power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0132] The processor 810 may further include a safety cluster engine that includes a processor subsystem dedicated to handling safety management of automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In a safety mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic to detect any differences between their operations.

[0133] The processor 810 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0134] The processor 810 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0135] The processor 810 may include a video image compositor, which may be a processing block (e.g., implemented in a microprocessor) that implements video post-processing functions required by the video playback application to produce a final image for the player window. The video image compositor may perform lens distortion correction on the wide-view camera 870, on the surround camera 874, and / or on the in-cabin surveillance camera sensor. The in-cabin surveillance camera sensor is preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify in-cabin events and respond appropriately. The in-cabin system may perform lip reading to activate cellular service and make phone calls, dictate emails, change the destination of the vehicle, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are available to the driver only when operating in autonomous mode and are disabled otherwise.

[0136] The video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, when motion occurs in the video, the noise reduction reduces the weight of information provided by adjacent frames and appropriately weights the spatial information. When an image or part of an image does not contain motion, the temporal noise reduction performed by the video image synthesizer may use information from previous images to reduce noise in the current image.

[0137] The video image composer may also be configured to perform stereo rectification on the input stereo lens frames. The video image composer may further be used for user interface compositing when the operating system desktop is in use and the GPU 808 is not required to continuously render new surfaces. Even when the GPU 808 is powered on and actively performing 3D rendering, the video image composer may be used to offload the GPU 808 for improved performance and responsiveness.

[0138] The SoC804 may further include a mobile industry processor interface (MIPI) camera serial interface, a high-speed interface for receiving video and input from a camera, and / or a video input block that may be used for camera and related pixel input functions. The SoC804 may further include an input / output controller that may be controlled by software and that may be used to receive I / O signals that are not committed to a specific role.

[0139] The SoC 804 may further include a wide range of peripheral interfaces to enable peripherals, audio codecs, power management, and / or communication with other devices. The SoC 804 may be used to process data from cameras (e.g., connected via gigabit multimedia serial links and Ethernet), sensors (e.g., LIDAR sensor 864, RADAR sensor 860, etc., which may be connected via Ethernet), data from bus 802 (e.g., speed of mobile vehicle 800, steering wheel position, etc.), and data from GNSS sensor 858 (e.g., connected via Ethernet or CAN bus). The SoC 804 may further include a dedicated high-performance mass storage controller, which may include its own DMA engine and may be used to offload the CPU 806 from routine data management tasks.

[0140] The SoC 804 may be an end-to-end platform with a flexible architecture that spans levels 3-5 of automation, thereby providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The SoC 804 may be faster, more reliable, and more energy- and space-efficient than traditional systems. For example, when the accelerator 814 is combined with the CPU 806, the GPU 808, and the data store 816, the SoC 804 may provide a fast and efficient platform for levels 3-5 of autonomous vehicles.

[0141] Thus, the present technology provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms may be executed on a CPU, which may be configured using a high-level programming language, such as the C programming language, to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and for practical Level 3-5 autonomous vehicles.

[0142] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, the techniques described herein allow multiple neural networks to run simultaneously and / or sequentially and the results to be combined to enable Level 3-5 autonomous driving capabilities. For example, the CNN running on the DLA or dGPU (e.g., GPU820) can include text and word recognition that allows the supercomputer to read and understand traffic signs, including signs for which the neural network was not specifically trained. The DLA can further include neural networks that can identify, interpret, and provide a semantic understanding of the signs and pass the semantic understanding to a route planning module running on the CPU complex.

[0143] As another example, multiple neural networks may be run simultaneously, as required for level 3, 4, or 5 operation. For example, a warning sign consisting of "Caution: Flashing lights indicate icy conditions" along with a lightning flash may be interpreted by several neural networks, independently or collectively. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" may be interpreted by a second deployed neural network that informs the vehicle's route planning software (preferably running on a CPU complex) that icy conditions exist when a flashing light is detected. The flashing light may be identified by running a third deployed neural network through multiple frames, informing the vehicle's route planning software of the presence (or absence) of the flashing light. All three neural networks may be run simultaneously, such as within the DLA and / or on the GPU 808.

[0144] In some instances, a CNN for facial recognition and vehicle owner identification can use data from the camera sensor to identify the presence of a legitimate driver and / or owner of the vehicle 800. The always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in security mode, to stop operation of the vehicle when the owner leaves the vehicle. In this manner, the SoC 804 provides security against theft and / or vehicle hijacking.

[0145] In another example, a CNN for emergency vehicle detection and identification can detect and identify emergency vehicle sirens using data from microphone 896. In contrast to conventional systems that use general classifiers to detect sirens and manually extract features, SoC 804 uses CNNs for classification of environmental and urban sounds, as well as classification of visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative terminal velocity of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the mobile vehicle is operating, as identified by the GNSS sensor 858. Thus, for example, when operating in Europe, the CNN will attempt to detect European sirens, and when in the United States, the CNN will attempt to identify only North American sirens. After an emergency vehicle is detected, a control program can be used to perform emergency vehicle safety routines, such as slowing down the mobile vehicle, stopping it at the side of the road, parking the mobile vehicle, and / or idling the mobile vehicle, with the assistance of ultrasonic sensor 862, until the emergency vehicle has passed.

[0146] The vehicle may include a CPU 818 (e.g., a separate CPU, or a dCPU) that may be coupled to the SoC 804 via a high-speed interconnect (e.g., PCIe). The CPU 818 may include, for example, an X86 processor. The CPU 818 may be used to perform any of a variety of functions, including, for example, reconciling potentially inconsistent results between the ADAS sensors and the SoC 804, and / or monitoring the status and health of the controller 836 and / or the infotainment SoC 830.

[0147] Vehicle 800 may include a GPU 820 (e.g., a discrete GPU, or a dGPU) that may be coupled to SoC 804 via a high-speed interconnect (e.g., NVLINK from NVIDIA Corporation). GPU 820 may provide additional artificial intelligence functionality, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based on input (e.g., sensor data) from sensors of vehicle 800.

[0148] The mobile vehicle 800 may further include a network interface 824 that may include one or more wireless antennas 826 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 824 may be used to enable wireless connections with the cloud over the Internet (e.g., with the server 878 and / or other network devices), with other mobile vehicles, and / or with computing devices (e.g., passenger client devices). To communicate with other mobile vehicles, a direct link may be established between the two mobile vehicles and / or an indirect link may be established (e.g., through a network and via the Internet). A direct link may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link may provide the mobile vehicle 800 information about mobile vehicles in its vicinity (e.g., mobile vehicles in front of, beside, and / or behind the mobile vehicle 800). This function may be part of a cooperative adaptive cruise control function of the mobile vehicle 800.

[0149] The network interface 824 may include a SoC that provides modulation and demodulation functions and enables the controller 836 to communicate over a wireless network. The network interface 824 may include a radio frequency front end for upconversion from baseband to radio frequency and downconversion from radio frequency to baseband. The frequency conversion may be performed through well known processes and / or may be performed using a superheterodyne process. In some instances, the radio frequency front end functions may be provided by a separate chip. The network interface may include wireless functions for communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0150] Mobile vehicle 800 may further include a data store 828, which may include off-chip (e.g., off-SoC 804) storage. Data store 828 may include one or more memory elements, including RAM, SRAM, DRAM, VRAM, flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0151] The vehicle 800 may further include a GNSS sensor 858. The GNSS sensor 858 (e.g., a GPS, an aided GPS sensor, a differential GPS (DGPS) sensor, etc.) aids in mapping, perception, occupancy grid generation, and / or route planning functions. Any number of GNSS sensors 858 may be used, including, but not limited to, a GPS using a USB connector with an Ethernet to serial (RS-232) bridge, for example.

[0152] The mobile vehicle 800 may further include a RADAR sensor 860. The RADAR sensor 860 may be used by the mobile vehicle 800 for long-range mobile vehicle detection, even in darkness and / or severe weather conditions. The RADAR functional safety level may be ASIL B. In some instances, the RADAR sensor 860 may use the CAN and / or bus 802 for control and to access object tracking data (e.g., to transmit data generated by the RADAR sensor 860), with access to Ethernet to access raw data. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor 860 may be suitable for front, rear, and side RADAR use. In some instances, a pulsed Doppler RADAR sensor is used.

[0153] The RADAR sensor 860 may include different configurations, such as long range with a narrow field of view, short range with a wide field of view, and short range side coverage. In some instances, the long range RADAR may be used for adaptive cruise control functions. Long range RADAR systems may provide a wide field of view achieved by two or more independent scans, such as within a 250m range. The RADAR sensor 860 may help distinguish between static and moving objects and may be used by ADAS systems for emergency brake assist and forward collision warning. The long range RADAR sensor may include a monostatic multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and high speed CAN and FlexRay interfaces. In one instance with six antennas, the center four antennas may create a focused beam pattern designed to record the surroundings of the moving vehicle 800 at high speeds with minimal interference from traffic in adjacent lanes. The other two antennas may widen the field of view, allowing for rapid detection of moving vehicles entering or leaving the lane of the moving vehicle 800.

[0154] As an example, a medium-range RADAR system may include a range of up to 860 meters (front) or 80 meters (rear) and a field of view of up to 42 degrees (front) or 850 degrees (rear). A short-range RADAR system may include, but is not limited to, a RADAR sensor designed to be mounted on either end of a rear bumper. When mounted on either end of a rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and adjacent to a moving vehicle.

[0155] Short-range RADAR systems may be used in ADAS systems for blind spot detection and / or lane change assist.

[0156] The mobile vehicle 800 may further include ultrasonic sensors 862. The ultrasonic sensors 862, which may be located at the front, rear, and / or sides of the mobile vehicle 800, may be used for parking assist and / or for creating and updating an occupancy grid. A wide variety of ultrasonic sensors 862 may be used, with different ultrasonic sensors 862 being used for different ranges of detection (e.g., 2.5m, 4m). The ultrasonic sensors 862 may operate at a functional safety level of ASIL B.

[0157] The mobile vehicle 800 may include a LIDAR sensor 864. The LIDAR sensor 864 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 864 may be functional safety level ASIL B. In some instances, the mobile vehicle 800 may include multiple (e.g., 2, 4, 6, etc.) LIDAR sensors 864 that may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0158] In some instances, the LIDAR sensor 864 may have the ability to provide a list of objects and their distances in a 360-degree field of view. A commercially available LIDAR sensor 864 may have an advertised range of about 800 m, for example, with an accuracy of 2 cm to 3 cm, and with support for an 800 Mbps Ethernet connection. In some instances, one or more non-protruding LIDAR sensors 864 may be used. In such instances, the LIDAR sensor 864 may be implemented as a small device that may be integrated into the front, rear, sides, and / or corners of the vehicle 800. In such instances, the LIDAR sensor 864 may have a range of 200 m even for low-reflecting objects, and provide up to 120 degrees horizontal and 35 degrees vertical field of view. A front-mounted LIDAR sensor 864 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0159] In some instances, LIDAR technology such as 3D flash LIDAR may also be used. 3D flash LIDAR uses a laser flash as a transmitter to illuminate the surroundings of the vehicle up to about 200 m. The flash LIDAR unit includes a receptor that records the laser pulse transit time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR may allow a highly accurate and distortion-free image of the surroundings to be generated with every laser flash. In some instances, four flash LIDAR sensors may be deployed, one on each side of the vehicle 800. Available 3D flash LIDAR systems include solid-state 3D steering array LIDAR cameras that have no moving parts other than the blower (e.g., non-scanning LIDAR devices). Flash LIDAR devices may use 5 nanosecond Class I (eye-safe) laser pulses per frame and may capture reflected laser light in the form of a 3D range point cloud and coregistered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 864 may be less susceptible to motion blur, vibration, and / or shock.

[0160] The vehicle may further include an IMU sensor 866. In some instances, the IMU sensor 866 may be positioned at the center of the rear axle of the vehicle 800. The IMU sensor 866 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some instances, such as in a 6-axis application, the IMU sensor 866 may include an accelerometer and a gyroscope, while in a 9-axis application, the IMU sensor 866 may include an accelerometer, a gyroscope, and a magnetometer.

[0161] In some embodiments, the IMU sensor 866 may be implemented as a miniature, high-performance GPS-Aided Inertial Navigation System (GPS / INS) that combines micro-electro-mechanical system (MEMS) inertial sensors, highly sensitive GPS receivers, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. As such, in some instances, the IMU sensor 866 may enable the vehicle 800 to estimate heading without requiring input from a magnetic sensor by directly observing and correlating changes in velocity from the GPS to the IMU sensor 866. In some instances, the IMU sensor 866 and the GNSS sensor 858 may be combined in a single integrated unit.

[0162] The vehicle may include a microphone 896 placed in and / or around the vehicle 800. The microphone 896 may be used for emergency vehicle detection and identification, among other things.

[0163] The vehicle may further include any number of camera types, including stereo cameras 868, wide view cameras 870, infrared cameras 872, surround cameras 874, long and / or mid-range cameras 898, and / or other camera types. The cameras may be used to capture image data around the entire exterior of the vehicle 800. The type of cameras used depends on the embodiment and requirements of the vehicle 800, and any combination of camera types may be used to achieve the required coverage around the vehicle 800. Additionally, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may support, by way of example only, Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each camera is described in further detail herein with respect to FIG. 8A and FIG. 8B.

[0164] The mobile vehicle 800 may further include a vibration sensor 842. The vibration sensor 842 may measure vibrations of components of the mobile vehicle, such as an axle. For example, a change in vibration may indicate a change in the surface of the road. In another example, when two or more vibration sensors 842 are used, the difference in vibration may be used to determine friction or slippage of the road surface (e.g., when the difference in vibration is between a powered axle and a free-running axle).

[0165] The mobile vehicle 800 may include an ADAS system 838. In some instances, the ADAS system 838 may include a SoC. The ADAS system 838 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0166] The ACC system may use a RADAR sensor 860, a LIDAR sensor 864, and / or a camera. The ACC system may include longitudinal ACC and / or lateral ACC. The longitudinal ACC monitors and controls the distance to the vehicle directly ahead of the vehicle 800 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. The lateral ACC performs distance keeping and advises the vehicle 800 to change lanes when necessary. The lateral ACC is related to other ADAS applications such as LCA and CWS.

[0167] CACC uses information from other moving vehicles, which may be received from other moving vehicles via a wireless link via the network interface 824 and / or the wireless antenna 826, or indirectly via a network connection (e.g., via the Internet). A direct link may be provided by a vehicle-to-vehicle (V2V) communication link, while an indirect link may be an infrastructure-to-vehicle (I2V) communication link. In general, a V2V communication concept provides information about the immediately preceding moving vehicle (e.g., the moving vehicle immediately ahead of the moving vehicle 800, in the same lane as the moving vehicle 800), while an I2V communication concept provides information about the traffic further ahead. A CACC system may include either or both I2V and V2V information sources. Given the information of the moving vehicles ahead of the moving vehicle 800, CACC may be more reliable and may potentially make traffic flow smoother and reduce congestion on the roads.

[0168] The FCW system is designed to warn the driver of hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback such as a display, speaker, and / or vibration components. The FCW system can provide warnings in the form of audio, visual alerts, vibrations, and / or quick brake pulses.

[0169] An AEB system can detect an imminent forward collision with another moving vehicle or other object and automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. The AEB system can use a forward-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, the AEB system typically first alerts the driver to take corrective action to avoid the collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes as part of an effort to prevent, or at least mitigate, the effects of the predicted collision. The AEB system can include techniques such as Dynamic Brake Support and / or Collision Imminent Braking.

[0170] The LDW system provides visual, audible, and / or tactile warnings, such as vibrations in the steering wheel or seat, to alert the driver when the mobile vehicle 800 crosses a lane marking. The LDW system is not activated when the driver indicates an intentional lane departure by activating a turn signal. The LDW system may use a forward-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0171] The LKA system is a modification of the LDW system. The LKA system provides steering input or braking to correct the vehicle 800 if it begins to wander out of its lane.

[0172] The BSW system detects and warns the driver of a moving vehicle in the vehicle's blind spot. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use rear-facing cameras and / or RADAR sensors 860 coupled to dedicated processors, DSPs, FPGAs, and / or ASICs electrically coupled to driver feedback, e.g., displays, speakers, and / or vibration components.

[0173] The RCTW system may provide visual, audible, and / or tactile notifications when an object is detected outside the range of the rear camera when the vehicle 800 is backing up. Some RCTW systems include AEB to ensure vehicle brakes are applied to avoid a collision. The RCTW system may use one or more rear-facing RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, e.g., a display, speaker, and / or vibration components.

[0174] Because conventional ADAS systems alert the driver and allow the driver to determine whether a safety condition truly exists and act accordingly, conventional ADAS systems can be prone to producing false positive results that are usually not catastrophic but can be annoying and distracting to the driver. However, in an autonomous vehicle 800, the vehicle 800 itself must decide whether to listen to the results from the primary computer or the secondary computer (e.g., the first controller 836 or the second controller 836) when the results are conflicting. For example, in some embodiments, the ADAS system 838 may be a backup and / or secondary computer to provide perception information to a backup computer rationality module. The backup computer rationality monitor can run redundant and diverse software on hardware components to detect failures in perception and dynamic driving tasks. Output from the ADAS system 838 may be provided to a supervisory MCU. When the outputs from the primary and secondary computers conflict, the supervisory MCU must decide how to reconcile the conflict to ensure safe operation.

[0175] In some instances, the primary computer may be configured to provide a reliability score to the overseer MCU indicating the reliability of the primary computer in a selected outcome. If the reliability score exceeds a threshold, the overseer MCU may follow the instructions of the primary computer regardless of whether the secondary computers provide conflicting or inconsistent results. If the reliability score does not meet the threshold, and if the primary and secondary computers show different (e.g., conflicting) results, the overseer MCU may arbitrate between the computers to determine the appropriate outcome.

[0176] The supervisory MCU may be configured to execute a neural network that is trained and configured to determine conditions under which the secondary computer provides a false alarm based on the outputs from the primary and secondary computers. Thus, the neural network in the supervisory MCU may learn when the output of the secondary computer may be trusted and when it may not be trusted. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU may learn when the FCW identifies a metal object that is not actually a hazard, such as a sewer grate or manhole cover, that triggers an alarm. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU may learn to ignore the LDW when a bicyclist or pedestrian is present and lane departure is, in fact, the safest maneuver. In embodiments that include a neural network executing on the supervisory MCU, the supervisory MCU may include at least one of a DLA or a GPU suitable for executing the neural network with associated memory. In a preferred embodiment, the director MCU may comprise and / or be included as a component of the SoC804.

[0177] In other instances, the ADAS system 838 may include a secondary computer that performs ADAS functions using classical rules of computer vision. As such, the secondary computer may use classical computer vision rules (if-then) and the presence of a neural network in the supervisory MCU may improve reliability, safety and performance. For example, the diverse implementations and intentional non-identity may make the overall system more fault-tolerant, especially to failures caused by software (or software-hardware interface) functions. For example, if there is a software bug or error in the software running on the primary computer and non-identical software code running on the secondary computer provides the same overall result, the supervisory MCU may have greater confidence that the overall result is correct and that a bug in the software or hardware on the primary computer has not caused a critical error.

[0178] In some instances, the output of the ADAS system 838 may be fed to a perception block of the primary computer and / or a dynamic driving task block of the primary computer. For example, if the ADAS system 838 indicates a forward collision warning due to an object immediately ahead, the perception block may use this information when identifying the object. In other instances, the secondary computer may have its own neural network that is trained as described herein, thus reducing the risk of false positives.

[0179] The mobile vehicle 800 may further include an infotainment SoC 830 (e.g., an in-vehicle infotainment system (IVI)). Although shown and described as an SoC, the infotainment system need not be an SoC and may include two or more separate components. The infotainment SoC 830 may include a combination of hardware and software that may be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear park assist, wireless data system, mobile vehicle related information such as fuel level, total distance traveled, brake fuel level, oil level, door opening / closing, air filter information, etc.) to the mobile vehicle 800. For example, the infotainment SoC 830 may be a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, a car computer, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, a heads-up display (HUD), an HMI display 834, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 830 may further be used to provide information (e.g., visual and / or audible) to a user of the vehicle, such as information from an ADAS system 838, autonomous driving information such as planned vehicle maneuvers, trajectory, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0180] The infotainment SoC 830 may include GPU functionality. The infotainment SoC 830 may communicate with other devices, systems, and / or components of the mobile vehicle 800 via the bus 802 (e.g., CAN bus, Ethernet, etc.). In some instances, the infotainment SoC 830 may be coupled to the supervisor MCU such that the infotainment system's GPU can perform some self-drive functions in the event that the primary controller 836 (e.g., the primary and / or backup computer of the mobile vehicle 800) fails. In such instances, the infotainment SoC 830 may place the mobile vehicle 800 in a Chauffeur safe stop mode as described herein.

[0181] The mobile vehicle 800 may further include an instrument cluster 832 (e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 832 may include a controller and / or a supercomputer (e.g., a separate controller or a supercomputer). The instrument cluster 832 may include a set of instruments such as a speedometer, fuel level, oil pressure, a tachometer, an odometer, turn signals, a gear shift position indicator, a seat belt warning light, a parking brake warning light, an engine malfunction light, an airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some instances, information may be displayed and / or shared between the infotainment SoC 830 and the instrument cluster 832. In other words, the instrument cluster 832 may be included as part of the infotainment SoC 830, or vice versa.

[0182] 8D is a system diagram of communication between a cloud-based server and the example autonomous vehicle 800 of FIG. 8A in accordance with some embodiments of the present disclosure. System 876 may include a server 878, a network 890, and a mobile vehicle including the mobile vehicle 800. Server 878 may include multiple GPUs 884(A)-884(H) (collectively referred to herein as GPUs 884), PCIe switches 882(A)-882(H) (collectively referred to herein as PCIe switches 882), and / or CPUs 880(A)-880(B) (collectively referred to herein as CPUs 880). GPUs 884, CPUs 880, and PCIe switches may be interconnected with a high-speed interconnect, such as, but not limited to, an NVLink interface 888 and / or a PCIe connection 886 developed by NVIDIA Corporation. In some instances, the GPUs 884 are connected via an NVLink and / or NVSwitch SoC, and the GPUs 884 and PCIe switch 882 are connected via a PCIe interconnect. Although eight GPUs 884, two CPUs 880, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each server 878 may include any number of GPUs 884, CPUs 880, and / or PCIe switches. For example, the servers 878 may each include 8, 16, 32, and / or more GPUs 884.

[0183] Server 878 can receive image data from the mobile vehicle over network 890, the image data representing images showing unexpected or changed road conditions, such as recently started road construction. Server 878 can transmit neural network 892, updated neural network 892, and / or map information 894, including information about traffic and road conditions, to the mobile vehicle over network 890. Updates to map information 894 can include updates to HD map 822, such as information about construction sites, potholes, detours, flooding, and / or other obstacles. In some instances, neural network 892, updated neural network 892, and / or map information 894 may have arisen from new training and / or experience represented in data received from any number of mobile vehicles in the environment and / or based on training performed at a data center (e.g., using server 878 and / or other servers).

[0184] The server 878 may be used to train a machine learning model (e.g., a neural network) based on the training data. The training data may be generated by a moving vehicle and / or generated in a simulation (e.g., using a game engine). In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or undergoes other pre-processing, while in other instances, the training data is not tagged and / or pre-processed (e.g., if the neural network does not require supervised learning). The training may be performed according to any one or more classes of machine learning techniques, including, but not limited to, the following classes: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multi-linear subspace learning, manifold learning, representation learning (including preliminary dictionary learning), rule-based machine learning, anomaly detection, and variations or combinations thereof. After the machine learning model has been traced, the machine learning model may be used by the vehicle (e.g., transmitted to the vehicle via network 890) and / or the machine learning model may be used by server 878 to remotely monitor the vehicle.

[0185] In some instances, the server 878 can receive data from the moving vehicles and apply the data to state-of-the-art real-time neural networks for real-time intelligent inference. The server 878 can include deep learning supercomputers and / or dedicated AI computers powered by GPUs 884, such as the DGX and DGX Station machines developed by NVIDIA Corporation. However, in some instances, the server 878 can include a deep learning infrastructure that uses only CPU-powered data centers.

[0186] The deep learning infrastructure of the server 878 may be capable of rapid real-time inference and may use that capability to evaluate and validate the health of the processor, software, and / or associated hardware within the mobile vehicle 800. For example, the deep learning infrastructure may receive periodic updates from the mobile vehicle 800, such as sequences of images and / or objects that the mobile vehicle 800 has located within the images of the sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them with objects identified by the mobile vehicle 800, and if the results do not match and the infrastructure concludes that the AI ​​within the mobile vehicle 800 is not functioning properly, the server 878 may send a signal to the mobile vehicle 800 to infer control, notify passengers, and command the failsafe computer of the mobile vehicle 800 to complete a safe parking maneuver.

[0187] For inference, the server 878 may include a GPU 884 and one or more programmable inference accelerators (e.g., TensorRT from NVIDIA Corporation). The combination of a GPU-powered server and inference acceleration can enable real-time responsiveness. In other instances, such as when less performance is required, servers powered by CPUs, FPGAs, and other processors may be used for inference.

[0188] Exemplary Computing Device 9 is a block diagram of an example computing device 900 suitable for use in implementing some embodiments of the present disclosure. The computing device 900 may include an interconnect system 902 that indirectly or directly couples the following devices: memory 904, one or more central processing units (CPUs) 906, one or more graphic processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, input / output components 914, a power supply 916, one or more presentation components 918 (e.g., displays), and one or more logic units 920. In at least one embodiment, the computing device 900 may include one or more virtual machines (VMs) and / or any of its components may include virtual components (e.g., virtual hardware components). As non-limiting examples, one or more of GPUs 908 may include one or more vGPUs, one or more of CPUs 906 may include one or more vCPUs, and / or one or more of logic units 920 may include one or more virtual logic units. As such, computing device 900 may include discrete components (e.g., an entire GPU dedicated to computing device 900), virtual components (e.g., a portion of a GPU dedicated to computing device 900), or a combination thereof.

[0189] Although the various blocks in FIG. 9 are shown connected via interconnect system 902 with lines, this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component 918, such as a display device, may be considered an I / O component 914 (e.g., where the display is a touch screen). As another example, CPU 906 and / or GPU 908 may include memory (e.g., memory 904 may represent a storage device in addition to the memory of GPU 908, CPU 906, and / or other components). In other words, the computing devices of FIG. 9 are merely exemplary. Categories such as "workstations," "servers," "laptops," "desktops," "tablets," "client devices," "mobile devices," "handheld devices," "gaming consoles," "electronic control units (ECUs)," "virtual reality systems," and / or other device or system types are all intended to be within the scope of the computing devices of FIG. 9 and therefore are not to be distinguished.

[0190] The interconnect system 902 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 902 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 906 may be directly connected to the memory 904. Additionally, the CPU 906 may be directly connected to the GPU 908. When there are direct or point-to-point connections between components, the interconnect system 902 may include a PCIe link to implement the connections. In these examples, a PCI bus need not be included in the computing device 900.

[0191] Memory 904 may include any of a variety of computer readable media. Computer readable media may be any available media that can be accessed by computing device 900. Computer readable media may include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media.

[0192] Computer storage media may include both volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, memory 904 may store computer readable instructions (e.g., representing programs and / or program elements), such as an operating system. Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store the desired information and that may be accessed by computing device 900. As used herein, computer storage media does not include the signals themselves.

[0193] Computer storage media may embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media may include wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0194] The CPU 906 may be configured to execute at least some of the computer readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and / or processes described herein. The CPU 906 may include one or more (e.g., 1, 2, 4, 8, 28, 72, etc.) cores each capable of processing multiple software threads simultaneously. The CPU 906 may include any type of processor, and may include different types of processors depending on the type of computing device 900 implemented (e.g., a processor with fewer cores for a mobile device and a processor with more cores for a server). For example, depending on the type of computing device 900, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). Computing device 900 may include one or more CPUs 906 within one or more microprocessors or auxiliary coprocessors, such as computational coprocessors.

[0195] In addition to or instead of the CPU 906, the GPU 908 may be configured to execute at least some of the computer readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and / or processes described herein. One or more of the GPUs 908 may be integrated GPUs (e.g., with one or more of the CPUs 906 and / or one or more of the GPUs 908 may be discrete GPUs. In an embodiment, one or more of the GPUs 908 may be coprocessors of one or more of the CPUs 906. The GPU 908 may be used by the computing device 900 to render graphics (e.g., 3D graphics) or perform general purpose computations. For example, the GPU 908 may be configured to perform general purpose computing on a GPU (GPGPU). A graphics processing unit (GPU) may be used for processing graphics data. The GPU 908 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU 908 may generate pixel data for an output image in response to rendering commands (e.g., rendering commands from the CPU 906 received via a host interface). The GPU 908 may include graphics memory, e.g., display memory, for storing the pixel data or any other suitable data, e.g., GPGPU data. The display memory may be included as part of the memory 904. GPU 908 may include two or more GPUs operating in parallel (e.g., via links). The links may connect the GPUs directly (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When coupled together, each GPU 908 may generate pixel data or GPGPU data for a different portion of the output or for a different output (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with the other GPUs.

[0196] In addition to or instead of the CPU 906 and / or GPU 908, the logic unit 920 may be configured to execute at least some of the computer readable instructions to control one or more of the computing devices 900 to perform one or more of the methods and / or processes described herein. In an embodiment, the CPU 906, the GPU 908, and / or the logic unit 920 may execute any combination of the methods, processes and / or portions thereof, either discretely or jointly. One or more of the logic units 920 may be part of and / or integrated with one or more of the CPU 906 and / or GPU 908, and / or one or more of the logic units 920 may be a discrete component to or otherwise external to the CPU 906 and / or GPU 908. In an embodiment, one or more of the logic units 920 may be a co-processor of one or more of the CPU 906 and / or GPU 908.

[0197] Examples of logic unit 920 include one or more processing cores and / or components thereof, such as, for example, a Tensor Core (TC), a Tensor Processing Unit (TPU), a Pixel Visual Core (PVC), a Vision Processing Unit (VPU), a Graphics Processing Cluster (GPC), a Texture Processing Cluster (TPC), a Streaming Multiprocessor (SM), a Tree Traversal Unit (TTU), an Artificial Intelligence Accelerator (AIA), a Deep Learning Accelerator (DLA), an Arithmetic Logic Unit (ALU), an Application Specific Integrated Circuit (ASIC), a Floating Point Unit (FPU), an Input / Output (I / O) element, a Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) element, and / or the like.

[0198] The communications interface 910 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 900 to communicate with other computing devices over electronic communications networks, including wired and / or wireless communications. The communications interface 910 may include components and functionality to enable communication over any of a number of different networks, such as a wireless network (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), a wired network (e.g., communicating over Ethernet or InfiniBand), a low power wide area network (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0199] The I / O ports 912 may enable the computing device 900 to be logically coupled to other devices, including I / O components 914, presentation components 918, and / or other components, some of which may be built-in (e.g., integrated) to the computing device 900. Exemplary I / O components 914 include a microphone, a mouse, a keyboard, a joystick, a game pad, a game controller, a satellite dish, a scanner, a printer, a wireless device, and the like. The I / O components 914 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by a user. In some cases, the input may be sent to an appropriate network element for further processing. The NUI may implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and next to the screen, air gestures, head and eye tracking, and touch recognition in conjunction with the display of the computing device 900 (as described in more detail below). The computing device 900 may include a depth camera, such as a stereoscopic camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 900 may include an accelerometer or gyroscope (e.g., as part of an inertia measurement unit (IMU)) to enable detection of motion. In some instances, the output of the accelerometer or gyroscope may be used by the computing device 900 to render immersive augmented or virtual reality.

[0200] The power supply 916 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 916 may provide power to the computing device 900 to enable the components of the computing device 900 to operate.

[0201] The presentation component 918 may include a display (e.g., a monitor, a touch screen, a television screen, a heads-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component 918 can receive data from other components (e.g., GPU 908, CPU 906, etc.) and output data (e.g., as images, video, sound, etc.).

[0202] Exemplary Data Center 10 illustrates an example data center 1000 that may be used in at least one embodiment of the present disclosure. The data center 1000 may include a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and / or an application layer 1040.

[0203] 10, the data center infrastructure layer 1010 may include a resource orchestrator 1012, grouped computational resources 1014, and node computational resources ("node CRs") 1016(1)-1016(N), where "N" represents any integer, natural number. In at least one embodiment, the node CRs 1016(1)-1016(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and / or cooling modules, etc. In some embodiments, one or more of the nodes CR 1016(1)-1016(N) may correspond to a server having one or more of the aforementioned computing resources. Additionally, in some embodiments, the nodes CR 1016(1)-1016(N) may include one or more virtual components, such as a vGPU, a vCPU, and / or the like, and / or one or more of the nodes CR 1016(1)-1016(N) may correspond to a virtual machine (VM).

[0204] In at least one embodiment, the grouped computing resources 1014 may include separate groups of nodes CR 1016 housed in one or more racks (not shown), or multiple racks housed in data centers in various geographic locations (also not shown). The separate groups of nodes CR 1016 in the grouped computing resources 1014 may include grouped computing, network, memory, or storage resources that may be configured or assigned to support one or more workloads. In at least one embodiment, several nodes CR 1016 including CPUs, GPUs, and / or other processors may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0205] The resource orchestrator 1022 may configure or otherwise control one or more nodes CR 1016(1)-1016(N) and / or grouped computational resources 1014. In at least one embodiment, the resource orchestrator 1022 may include a software design infrastructure ("SDI") management entity of the data center 1000. The resource orchestrator 1022 may include hardware, software, or some combination thereof.

[0206] In at least one embodiment, as shown in FIG. 10 , framework layer 1020 may include a job scheduler 1032, a configuration manager 1034, a resource manager 1036, and / or a distributed file system 1038. Framework layer 1020 may include frameworks to support software 1032 in software layer 1030 and / or one or more applications 1042 in application layer 1040. Software 1032 or applications 1042 may include web-based service software or applications, such as those offered by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. Framework layer 1020 may be a type of free and open source software web application framework, such as, but not limited to, Apache Spark™ (hereinafter “Spark”), which may use a distributed file system 1038 for large-scale data processing (e.g., “big data”). In at least one embodiment, the job scheduler 1032 may include a Spark driver to facilitate scheduling of workloads supported by various tiers of the data center 1000. The configuration manager 1034 may be capable of configuring different tiers, for example, the software tier 1030 and the framework tier 1020 including Spark and a distributed file system 1038 to support large scale data processing. The resource manager 1036 may be capable of managing clustered or grouped computing resources that are mapped or assigned in support of the distributed file system 1038 and the job scheduler 1032. In at least one embodiment, the clustered or grouped computing resources may include the computing resources 714 grouped in the data center infrastructure tier 1010. The resource manager 1036 may coordinate with the resource orchestrator 1012 to manage these mapped or assigned computing resources.

[0207] In at least one embodiment, software 1032 included in software layer 1030 may include software used by at least a portion of nodes CR 1016(1)-1016(N), grouped computational resources 1014, and / or distributed file system 1038 of framework layer 1020. The one or more types of software may include, but are not limited to, Internet web page searching software, email virus scanning software, database software, and streaming video content software.

[0208] In at least one embodiment, the applications 1042 included in the application layer 1040 may include one or more types of applications used by at least a portion of the nodes CRs 1016(1)-1016(N), the grouped computational resources 1014, and / or the distributed file system 1038 of the framework layer 1020. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0209] In at least one embodiment, any of configuration manager 1034, resource manager 1036, and resource orchestrator 1012 can implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. The self-rewriting actions can free data center operators of data center 1000 from making potentially poor configuration decisions and possibly avoiding underutilized and / or underperforming portions of the data center.

[0210] Data center 1000 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculation of weight parameters via a neural network architecture using the software and / or computing resources described above with respect to data center 1000. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1000, for example, by using the weight parameters calculated via one or more training techniques, including but not limited to those described herein.

[0211] In at least one embodiment, data center 1000 may use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or corresponding virtual computing resources) for training and / or performing inference using such resources. Additionally, one or more of such software and / or hardware resources may be configured as services, such as image recognition, speech recognition, or other artificial intelligence services, to enable a user to train or perform inference on information.

[0212] Example Network Environment A network environment suitable for use in implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other back-end devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be implemented with one or more instances of computing device 900 of FIG. 9 - e.g., each device may include similar components, features, and / or functionality of computing device 900. In addition, if a back-end device (e.g., server, NAS, etc.) is implemented, the back-end device may be included as part of data center 1000, examples of which are further detailed herein with respect to FIG. 10.

[0213] Components of a network environment may communicate with each other via a network, which may be wired, wireless, or both. A network may include multiple networks or a network of networks. By way of example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. When a network includes a wireless telecommunications network, components such as base stations, communication towers, or access points (as well as other components) may provide wireless connectivity.

[0214] Compatible network environments may include one or more peer-to-peer network environments, in which case a server may not be included in the network environment, and one or more client-server network environments, in which case one or more servers may be included in the network environment. In a peer-to-peer network environment, the functionality described herein with respect to a server may be implemented in any number of client devices.

[0215] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support software in the software layer and / or one or more applications in the application layer. The software or applications may include web-based service software or applications, respectively. In an embodiment, one or more of the client devices may use the web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open source software web application framework that may use a distributed file system for large-scale data processing (e.g., "big data"), for example.

[0216] A cloud-based network environment may provide cloud computing and / or cloud storage performing any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these various functions may be distributed to multiple locations from a central or core server (e.g., one or more data centers that may be distributed across a state, territory, country, or the world). When a connection to a user (e.g., a client device) is relatively close to an edge server, the core server may delegate at least a portion of the functionality to the edge server. A cloud-based network environment may be private (e.g., restricted to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0217] A client device may include at least some of the components, features, and functionality of the exemplary computing device 900 described herein with respect to Figure 9. By way of illustration, and not by way of limitation, a client device may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a boat, an airship, a virtual machine, a drone, a robot, a handheld communications device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these depicted devices, or any other suitable device.

[0218] The present disclosure may be described in the general context of computer code or machine usable instructions, including computer executable instructions, such as program modules, being executed by a computer or other machine, such as a personal digital assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. The present disclosure may be implemented in a variety of configurations, including handheld devices, consumer electronics, general purpose computers, more specialized computing devices, etc. The present disclosure may also be implemented in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0219] As used herein, the term "and / or" with respect to two or more elements should be construed to mean only one element or a combination of elements. For example, "element A, element B, and / or element C" may include element A only, element B only, element C only, elements A and element B, elements A and element C, elements B and element C, or elements A, B, and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Furthermore, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0220] The subject matter of the present disclosure has been described with specificity to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. Instead, the inventors intend that the claimed subject matter may be implemented in other ways, including different steps or combinations of steps similar to those described in this document, in conjunction with other current or future technologies. Furthermore, although the terms "step" and / or "block" may be used herein to connote different elements of the method used, these terms should not be construed as implying any particular order between the various steps disclosed herein unless and when the order of the individual steps is explicitly described.

Claims

1. 1. A method, executed by one or more processors, of selecting a trajectory for an autonomous machine, comprising: generating a lane graph including a first set of time rewards for potential future positions of the autonomous machine; generating a plurality of possible trajectories that enable the autonomous machine to complete autonomous machine actions; generating an optimization measure for the possible trajectories using the first set of time rewards as an input; selecting a trajectory for implementation using the optimization measure; and implementing the trajectory using the autonomous machine. selecting a route between a starting point and a route destination using a geographic data set of a first geographic area as an input; the first series of time rewards includes a series of sequential time rewards calculated using lane information for a second geographic area that is smaller than the first geographic area; The method, wherein the plurality of possible trajectories is generated using detailed lane information for a third geographic area that is smaller than the second geographic area.

2. 2. The method of claim 1, wherein the first series of temporal rewards is calculated without scene information from a local world model representing objects detected by a sensor at the autonomous machine.

3. The method of claim 1 , comprising generating a scene prediction including predicted future object positions for objects detected by sensors on the autonomous machine.

4. 2. The method of claim 1 , wherein the autonomous machine action is selected using the first set of temporal rewards as an input.

5. 2. The method of claim 1, wherein the lane graph includes a second series of time rewards for potential future positions of the autonomous machine on a second lane of a route.

6. 11. A method, performed by one or more processors, of selecting a trajectory for an autonomous machine, the method comprising: generating a scene prediction including predicted future object positions of objects detected by one or more sensors corresponding to the autonomous machine; generating a plurality of possible trajectories for autonomous machine actions, the plurality of possible trajectories having a plurality of longitudinal and lateral conditions within a nominal lateral rail corresponding to the autonomous machine actions; evaluating the plurality of possible trajectories using the scene prediction and a first series of time rewards corresponding to a plurality of potential future positions of the autonomous machine to generate an optimization measure for the plurality of possible trajectories; selecting a starting parameter set trajectory used to generate additional trajectories for evaluation using the optimization measure; selecting individual trajectories from the additional trajectories using the optimization measure; and implementing the individual trajectories for the autonomous machine.

7. The method of claim 6 , further comprising: generating vertical constraints for the autonomous machine actions using the scene predictions.

8. 7. The method of claim 6, further comprising: selecting an additional autonomous machine action for detailed planning; generating an additional plurality of trajectories for the additional autonomous machine action; and evaluating the additional trajectory in the additional plurality of trajectories using the scene prediction and the first set of time rewards to generate the optimization measure for the additional trajectory.

9. 7. The method of claim 6, wherein the selection of the autonomous machine action is performed using a lane graph as selection input data.

10. 7. The method of claim 6, further comprising the step of selecting a route between a starting point and a route destination using the geographic data set of the first geographic area as an input.

11. The method of claim 10 , wherein the first series of time rewards is calculated using lane information for a second geographic area that is smaller than the first geographic area.

12. 7. The method of claim 6, wherein the first set of time rewards corresponds to an annotated lane graph including a second set of time rewards for potential future positions of the autonomous machine.

13. 7. The method of claim 6, wherein the first set of temporal rewards is calculated without scene information from a local world model representing objects detected by the one or more sensors at the autonomous machine.

14. generating a plurality of possible trajectories for an autonomous machine action, the plurality of possible trajectories having a plurality of longitudinal conditions and a plurality of lateral conditions; evaluating a trajectory within the plurality of possible trajectories using a scene prediction to generate an optimization measure for the trajectory; selecting a respective trajectory from the plurality of possible trajectories using the optimization metric; and implementing said individual trajectories of an autonomous machine; and and one or more processing units for performing a processor, wherein the plurality of longitudinal conditions are based at least on the scene prediction and the plurality of lateral conditions are based at least on a nominal rail corresponding to a heading of the autonomous machine.

15. 15. The processor of claim 14, wherein the one or more processing units are further for generating an annotated lane graph including a first series of time rewards for potential future positions of the autonomous machine on a first lane in the direction of travel of the autonomous machine, each time reward representing a travel time to a route destination.

16. 15. The processor of claim 14, wherein the one or more processing units are further for: using the optimization measure to select a trajectory of a starting parameter set to be used to generate additional trajectories for evaluation; and evaluating the additional trajectories using the scene prediction to generate the optimization measure for the additional trajectory.

17. The processor of claim 16 , wherein the additional trajectories include a combination of new longitudinal and lateral conditions not found in the plurality of possible trajectories.

Images