An unstructured vehicle route planner

Abstract

The techniques discussed herein may comprise an autonomous vehicle guidance system that generates a path for controlling an autonomous vehicle based at least in part on a static object map and / or one or more dynamic object maps. The guidance system may identify a path based at least in part on determining a cost map associated with a set of nodes and the static and / or dynamic objects, among other costs, and may prune the set of nodes and create additional nodes from the remaining nodes until a computational or other limit is reached. The path output by the technique may be associated with the least expensive node of the set of generated nodes.

Description

[Technical field]

[0001] The present invention relates to an unstructured vehicle route planner. [Background technology]

[0002] Related Applications This application claims priority to U.S. Patent Application No. 16 / 872,284, filed May 11, 2020, which is incorporated by reference herein for all intents and purposes.

[0003] Autonomous vehicles may rely on various en-route indicators, such as roadway lights, lane markings, and / or the like, to navigate. However, autonomous vehicles may fail to navigate accurately and / or efficiently when such indicators are obscured (e.g., by snow, debris, sand), degraded (e.g., out of lights, worn lane markings), and / or disabled (e.g., an obstacle partially blocks a lane, traffic signs and / or traffic cones indicate an alternate lane that conflicts with the original lane markings). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent Application Serial No. 15 / 900,318 [Patent Document 2] U.S. Patent Application Serial No. 16 / 147,177 [Patent Document 3] U.S. Patent Application Serial No. 16 / 399,743 [Patent Document 4] U.S. Patent Application Serial No. 16 / 011,436 [Patent Document 5] U.S. Patent Application Serial No. 15 / 963,833 [Patent Document 6] U.S. Patent Application Serial No. 15 / 807,521 [Patent Document 7] U.S. Patent Application Serial No. 16 / 420,050 [Patent Document 8] U.S. Patent Application Serial No. 16 / 206,877 [Patent Document 9] U.S. Patent Application Serial No. 16 / 504,147 [Patent Document 10] U.S. Patent Application Serial No. 16 / 517,506 [Brief description of the drawings]

[0005] The detailed description will be set forth with reference to the accompanying drawings, in which the left-most digit of a reference number identifies the figure in which the reference number first appears. The same reference numbers in different drawings indicate similar or identical items.

[0006] [Figure 1] FIG. 1 illustrates an autonomous vehicle and an example scenario in which lane references (whether previously mapped or detected) may not be reliable for determining commands for controlling the vehicle's movement. [Diagram 2] FIG. 1 is a block diagram of an example autonomous vehicle architecture with a guidance system for unstructured path planning. [Figure 3A] FIG. 1 illustrates a first example dynamic data predicting a motion of a dynamic object at a first time and an nth time associated with a first mode of operation of an autonomous vehicle. [Figure 3B] FIG. 11 illustrates a second example dynamic data predicting the movement of a dynamic object at a first time and an nth time associated with a second mode of operation of an autonomous vehicle. [Figure 4A] 1 is a diagram illustrating an example process for generating a path for controlling an autonomous vehicle based at least in part on environmental data, which may be generated from sensor data and may include an occupancy map and / or a speed probability map indicating predicted positions of static and / or dynamic objects (hereinafter dynamic object predicted positions). [Figure 4B]1 is a diagram illustrating an example process for generating a path for controlling an autonomous vehicle based at least in part on environmental data, which may be generated from sensor data and may include an occupancy map and / or a speed probability map indicating predicted positions of static and / or dynamic objects (hereinafter dynamic object predicted positions). [Figure 4C] 1 is a diagram illustrating an example process for generating a path for controlling an autonomous vehicle based at least in part on environmental data, which may be generated from sensor data and may include an occupancy map and / or a speed probability map indicating predicted positions of static and / or dynamic objects (hereinafter dynamic object predicted positions). [Figure 5A] FIG. 1 is a top-down view of the results of successive node creation and pruning operations according to the techniques discussed herein. [Figure 5B] FIG. 1 is a top-down view of the results of successive node creation and pruning operations according to the techniques discussed herein. [Figure 5C] FIG. 1 is a top-down view of the results of successive node creation and pruning operations according to the techniques discussed herein. [Figure 5D] FIG. 1 is a top-down view of the results of successive node creation and pruning operations according to the techniques discussed herein. [Figure 6A] 1 is a top-down representation of an environment and a contiguous set of nodes generated by the techniques discussed herein. [Figure 6B] 1 is a top-down representation of an environment and a contiguous set of nodes generated by the techniques discussed herein. [Figure 6C] 1 is a top-down representation of an environment and a contiguous set of nodes generated by the techniques discussed herein. [Figure 6D] 1 is a top-down representation of an environment and a contiguous set of nodes generated by the techniques discussed herein. [Figure 6E]1 is a top-down representation of an environment and a contiguous set of nodes generated by the techniques discussed herein. [Figure 6F] 1 is a top-down representation of an environment and a contiguous set of nodes generated by the techniques discussed herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Techniques discussed herein may comprise an autonomous vehicle guidance system that generates a route for controlling an autonomous vehicle based at least in part on a static object map (e.g., a map showing space occupied by static objects and / or "free space" in an environment) and / or one or more dynamic object maps (e.g., a map showing the likelihood that locations in an environment will be occupied by moving objects or stationary objects that may move at a future time). An autonomous vehicle including a guidance system discussed herein may thereby be able to accurately and efficiently navigate around both static and dynamic objects (moving or potentially moving) even when the environment is "unstructured" (i.e., the environment does not include path indicators or the path indicators have been degraded, obscured, and / or are generally no longer in use), such as an environment that does not include markings, roads, sidewalks, and / or other similar structures that may help identify boundaries for operating the autonomous vehicle, and / or in situations where road lanes do not have a wide enough open area for the vehicle to operate in.

[0008] An autonomous vehicle equipped with the guidance system discussed herein may detect that indicators for generating a path for the vehicle are unavailable (or determine that the vehicle is unable to proceed along predetermined or accepted instructions, such as lane criteria identified by a map stored in memory otherwise accessible to the autonomous vehicle) and determine a path for controlling the autonomous vehicle based at least in part on detecting the indicators being unavailable. Additionally or alternatively, the guidance system discussed herein may continuously generate an uncertain path that is not based (at least primarily) on the indicators at substantially the same time that a designated guidance system (trained to generate paths using the structure and / or indicators) generates a path based at least in part on the indicators. In some examples, if a path generated by the designated guidance system results in a collision or is otherwise inoperable (e.g., the path causes the autonomous vehicle to stop and / or send a message to a teleoperation system), the autonomous vehicle may use the uncertain path if the uncertain path is collision-free.

[0009] As used herein, a route may comprise at least a start position and an end position associated within an environment. In some examples, the start position may identify a current location (e.g., position and / or heading) of the autonomous vehicle, and / or the end position may identify a target location of the autonomous vehicle. The route may additionally or alternatively identify one or more curves between the start position and the end position and / or a vehicle position associated with the start position and / or the end position. For example, the start position may identify a location within a first lane of a road, and the end position may identify a location within a second lane two city blocks, a right turn, and an additional city block away from the start position. In other words, a route may be a high-level mission of an autonomous vehicle. As used herein, a curve may include a straight line.

[0010] Techniques discussed herein may include accepting a route, such as accepting an end location (which may be specified, e.g., by a planning component, a remote operator, and / or a passenger), determining a current position of the autonomous vehicle (e.g., based at least in part on sensor data received from one or more sensors of the autonomous vehicle, such as a global positioning sensor (GPS), LIDER, RADAR, camera, such as those that may be used in the location location system described herein), and determining a route between a start (current) position associated with the end location and an end position based at least in part on a map stored by the autonomous vehicle. Techniques may include determining, at least in part, based on the map and / or the vehicle's determination of the presence of lanes, drivable surfaces, and / or other indicators, curves associated with at least a portion of the route. For example, the autonomous vehicle may determine the curves based on the shape of the roads and / or free space between the start and end positions.

[0011] The autonomous vehicle may generate a path by techniques discussed herein to reach an end position specified by the route. In some examples, the start and end positions discussed herein may be associated with a portion of the route. Where the route may specify roads, such as to be driven by the autonomous vehicle, the path may specify how the autonomous vehicle reaches an end position (e.g., which lane, position within the lane, direction of travel) over an intermediate distance (e.g., 10 meters, 20 meters, 50 meters, 100 meters). The autonomous vehicle may additionally or alternatively generate a trajectory for controlling the autonomous vehicle based at least in part on the path. The trajectory may be used by the autonomous vehicle to control the movement of the autonomous vehicle over a shorter time period / distance than the path (e.g., 5 ms, 10 ms, 0.5 s) based at least in part on the reversal limit technique and the position and / or location specified by the path. In such examples, the trajectory may comprise any one or more of, for example, a waypoint and / or a desired speed, acceleration, heading, guidance angle, etc. associated with one or more waypoints.

[0012] In some examples, the techniques may include accepting sensor data from sensors of the autonomous vehicle and generating a static object map and / or a dynamic object map based at least in part on the sensor data. As discussed in more detail in U.S. Patent No. 6,399,433 entitled “Creating Clean Maps including Semantic Information,” U.S. Patent No. 6,399,433 entitled “Radar Spatial Estimation,” U.S. Patent No. 6,399,433 entitled “Predicting an Occupancy Associated with Occluded Region,” U.S. Patent No. 6,399,433 entitled “Occlusion Aware Planning,” and U.S. Patent No. 6,399,433 entitled “Data Segmentation Using Masks,” all of which are incorporated herein by reference. In some instances, the map may additionally or alternatively include probabilities associated with locations in the occupancy map (e.g., indicating the probability that a corresponding location in the environment is or is not occupied by the object).

[0013] A dynamic object map may include a map of an environment indicating the likelihood that a respective portion of the environment is likely to be occupied by a moving object. For example, a portion of the dynamic object map may indicate the likelihood that a dynamic object will occupy a corresponding portion of the environment at a future time. A current position of a dynamic object may be indicated in a static object map or in a first frame of the dynamic object map associated with a current time. In some examples, a dynamic object map may include two or more frames, each of which may be associated with a different time (e.g., the current and / or one or more future times). In some examples, the static object map and / or the dynamic object map may additionally or alternatively include an indication of an object's speed and / or heading, such as may be provided by object tracking, direct speed measurements (e.g., RADAR), etc. Generating dynamic object maps based at least in part on sensor data is discussed in more detail at least in U.S. Patent No. 6,363,433 entitled "Probabilistic Heat Maps for Behavior Prediction," U.S. Patent No. 6,363,433 entitled "Trajectory Prediction on Top-Down Scenes and Associated Model," U.S. Patent No. 6,363,433 entitled "Probabilistic Risk Assessment for Trajectory Evaluation," U.S. Patent No. 6,363,433 entitled "Prediction on Top-Down Scenes Based on Action Date," and U.S. Patent No. 6,363,433 entitled "Predicting an Occupancy Associated with Occluded Region," all of which are incorporated herein by reference. Static object maps and dynamic object maps may be collectively referred to herein as environmental data.

[0014] The technique may additionally or alternatively include determining a cost map based at least in part on the environmental data and determining a set of nodes associated with the next time step. In some cases, the cost map may include or be associated with a multivariate space. For example, the cost map and / or a position in the multivariate space may be associated with a position in the environment, a heading of the vehicle, a speed of the vehicle, and / or a curvature of the trajectory. Determining the nodes associated with the next time step may include determining a location in the multivariate space, which may identify a position in the environment, a heading of the vehicle, a speed of the vehicle, and / or a curvature of the trajectory that is reachable from a current position of the vehicle in an interval of time from the current time to the next time step (e.g., 500 ms, 100 ms, 1 sec, etc. from the current time). Determining the cost map includes determining a cost associated with a position in the multivariate space, the cost based at least in part on occupancy indicated by the static object map, predicted occupancy indicated by one or more frames of the dynamic object map, proximity of the position to indications in the static object map and / or the dynamic object map, a visit cost based on the difference of the vehicle's current position and trajectory, or previous node positions and / or trajectories, from a trajectory required to reach the portion of the multivariate space, etc. In some examples, at least a portion of the cost map may be pre-computed.

[0015] The techniques discussed herein may include determining a succession of waves / sets of nodes that extend from the vehicle's current position in the multivariate space to the nth time step, where n is a positive integer. For example, if the time steps are each 100 milliseconds and n is 20, the techniques may include determining 20 sets of nodes that extend away from the vehicle's position, with the 20th set of nodes being associated with a time that is 2 seconds in the future. In some examples, the set of nodes may be associated with a time step, e.g., the oth 100 millisecond interval, and the nodes generated may be based at least in part on one or more nodes selected for exploration. For example, for the first set of nodes, the selected nodes may be the vehicle's current position, heading, speed, and / or trajectory, and the first set of nodes may be generated based thereon. However, the second set of nodes may be generated at least in part based on one or more nodes selected from the first set for future exploration, e.g., determining at least a portion of the second set of nodes may be based at least in part on a node selected from the first set, i.e., a position, heading, speed, and / or trajectory associated with the selected node. In some examples, the generated nodes in the new set may be based on nodes selected from any of the previously generated sets of nodes (e.g., nodes from the immediately preceding set, nodes from sets generated in two or more generations prior). For example, the selection may be based at least in part on a cost associated with one or more nodes (e.g., determined at least in part based on a cost map, similarity cost), such as by selecting p lowest-cost nodes according to the costs associated with the nodes, where p is a positive integer.

[0016] The technique may include determining a hash associated with each node of the set and each node of the nodes, and pruning any node that has a hash that matches the hash of another node. For example, if two nodes have matching hashes, the node that has a higher cost and / or a shorter distance along the route may be discarded as part of the pruning. The technique may additionally or alternatively include determining p lowest cost nodes of the nodes that remain after pruning. Determining the next set of nodes may be based at least in part on the p nodes selected from the pruned set.

[0017] In some examples, hashes for positions in the multivariate space may be pre-computed before run-time and hashes for nodes may be determined based at least in part on lookups from a hash table. In some examples, hashes may be determined relative to a portion of the multivariate space having a dimension associated with a minimum spacing between nodes (resolution). For example, a hash table may be associated with a portion having a dimension of cubic centimeters and an orientation angle (in the multivariate space that includes potential positions and orientations of vehicles). Increasing the size of the resolution may increase the number of hash collisions (matches) that occur, effectively increasing the number of nodes that are discarded during the pruning stage and increasing the diversity of nodes that remain after pruning. Reducing the resolution may have the opposite effect.

[0018] The techniques may include determining the set of nodes until a time limit, a computation limit, and / or an nth set of nodes is determined (i.e., a node set cap is applied). The guidance system may determine one or more adjacent paths through the set of nodes when the set of nodes is generated and / or when a time limit, computation limit, or node limit is reached. In some examples, the guidance system may trace one or more potential paths through the set of nodes as they are generated and trace back the lowest cost path (i.e., from the node at the end of the potential path in the last tier of nodes to the start in the first tier of nodes and / or at the current position of the vehicle) to ensure that it is collision-free. In any event, the guidance system may determine the adjacent path through the set of nodes associated with the lowest cost and / or that makes the greatest progress along the route.

[0019] For example, the guidance system may perform a search for a path from a start position to an end position of at least a portion of a route. Performing the search may include determining a contiguous set of connections between nodes of different sets of nodes from the start position to the end position. In some examples, the search may be configured with a rule set that may include, for example, one or more rules that specify boundaries within which to determine the path (e.g., the boundaries may be determined at least in part based on sensor data and / or a map), node connection rules (e.g., a node may have only one parent node), and the like. In some examples, the search may include determining a directed graph between nodes of the set of nodes. The directed graph may include connections (e.g., edges) between a first node and a second node and / or weights (e.g., costs) associated with the connections.

[0020] In various examples, a path can be determined based at least in part on determining all connections between all nodes in a region and exploring a set of connections between them that provide a total minimum cost, which may include a cost associated with each movement primitive determined on the path. In at least some examples, such connections may be limited to only those connections between nodes that are feasible. In additional and / or alternative examples, computational resources may be reserved based at least in part on informing the search direction without enumerating all possible connections. Additionally, it is understood that such connections need not be limited to nodes in successive layers. As a non-limiting example, such path exploration may include exploring nodes between any one or more of a first layer to a second layer, a third layer, a fourth layer, etc. However, in at least some examples, nodes may be generated based on parent nodes regardless of the layer in which the node was generated.

[0021] In some examples, before performing a full search of the path, the techniques may include determining whether a previous path and / or previous connections determined by the guidance system are feasible (e.g., satisfy current constraints such as speed, maximum guidance angle, and / or bounds; are collision-free; have a cost less than a cost threshold).

[0022] At least a portion of determining adjacent paths or cost maps may include determining a likelihood of collision associated with a position in the multivariate space. For example, determining the likelihood of collision may include representing the autonomous vehicle as two circles having a diameter equal to a width of the autonomous vehicle, where a first circle may be centered on a front axle of the autonomous vehicle and a second circle may be centered on a rear axle of the autonomous vehicle, and determining a distance between the representation and the nearest static object and / or a portion of the environment predicted to be occupied by a dynamic object at a future time. In some examples, the representation of the occupied space shown by the autonomous vehicle and / or occupancy map may be expanded to account for error (e.g., the diameter of the circle and / or the area shown to be occupied in the occupancy may be expanded to half the autonomous vehicle width).

[0023] In some examples, the path determined by the guidance system may be a coarse path. For example, the coarse path may identify the position, heading, speed, and / or curvature of the way for the vehicle to follow at 1 second or 500 millisecond intervals, but the vehicle components may require or be capable of control over finer time intervals (e.g., 10 milliseconds, 100 milliseconds). In other words, the coarse path may not be smooth enough for the vehicle to follow without significant error. In some examples, a first type of processor (e.g., a graphics processing unit (GPU)) may determine a cost map, generate nodes, prune nodes, and / or determine a path, and a second type of processor may smooth the path generated by the GPU and / or determine a trajectory for controlling the vehicle based at least in part on the smooth path.

[0024] The guidance system discussed herein may identify a route as feasible and / or determine a confidence score associated with the route based at least in part on the cost discussed herein. The guidance system may output a route and / or a confidence score, which the autonomous vehicle may use to control the movement of the autonomous vehicle, for example, by generating a trajectory based at least in part on the route. In some examples, the guidance system may output a main route and / or an uncertain route. For example, the guidance system may determine the uncertain route based at least in part on generating a set of route candidates, the set including two groups of route candidates based at least in part on a threshold distance (e.g., the two groups may be two different homotopic groups), and selecting a main route from the first group and an uncertain route from the second group. In some examples, the main route may be selected as the main route based at least in part on determining that the main route is associated with a first total cost that is less than a second total cost associated with the uncertain route. The dominant path may be associated with a first total cost, and / or the uncertain path may be associated with a second total cost that is less than a cost threshold, and / or may be the minimum cost of the respective group associated therewith.

[0025] The techniques discussed herein may improve vehicle safety by improving the vehicle's ability to predict the movement and / or behavior of objects in the vehicle's surroundings and to plan a collision-free and economical path for the vehicle. The techniques may also enable the vehicle to operate in situations where road and / or lane markers are not present or are disabled.

[0026] Example scenario FIG. 1 illustrates an example scenario 100 including a vehicle 102. In some instances, the vehicle 102 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire journey, and the driver (or passenger) is not expected to control the vehicle at all times. However, in other examples, the vehicle 102 may be a fully or partially autonomous vehicle having any other level or classification. It is contemplated that the techniques discussed herein may apply beyond robotic control, such as autonomous vehicles. For example, the techniques discussed herein may apply to mining, manufacturing, augmented reality, and the like. Additionally, while the vehicle 102 is depicted as a land vehicle, the vehicle 102 may be a spacecraft, a watercraft, and the like. In some examples, the vehicle 102 may be represented in a simulation as a simulated vehicle. For simplicity, the discussion herein does not distinguish between simulated and real-world vehicles. Thus, references to a "vehicle" may refer to a simulated vehicle and / or a real-world vehicle.

[0027] According to the techniques discussed herein, the vehicle 102 may accept sensor data from sensors 104 of the vehicle 102. For example, the sensors 104 may include location sensors (e.g., Global Positioning System (GPS) sensors), inertial sensors (e.g., accelerometer sensors, gyroscope sensors, etc.), magnetic sensors (e.g., compasses), position / speed / acceleration sensors (e.g., speedometers, drive system sensors), depth position sensors (e.g., LIDER sensors, RADAR sensors, sonar sensors, time of flight (ToF) cameras, depth cameras, ultrasonic and / or sonar sensors, and / or other depth-sensing sensors), image sensors (e.g., cameras), audio sensors (e.g., microphones), and / or environmental sensors (e.g., barometers, hygrometers, etc.).

[0028] The sensors 104 may generate sensor data, which may be accepted by a computing device 106 associated with the vehicle 102. However, in other examples, some or all of the sensors 104 and / or computing devices 106 may be separate from and / or located remotely from the vehicle 102, and data capture, processing, command, and / or control may be communicated to / from the vehicle 102 by one or more remote computing devices over wired and / or wireless networks.

[0029] The computing device 106 may comprise a memory 108 that stores a perception component 110, a planning component 112, a guidance system 114, and / or a controller 116. In some examples, the planning component 112 may comprise the guidance system 114. The perception component 110, the planning component 112, the guidance system 114, and / or the controller 116 may include one or more machine learned (ML) models and / or other computer executable instructions. In general, the perception component 110 can determine what is in the environment surrounding the vehicle 102, and the planning component 112 can determine how to operate the vehicle 102 depending on the information received from the perception component 110. For example, the planning component 112 may determine the trajectory 118 based at least in part on the sensory data and / or other information, such as, for example, one or more maps, localization information (e.g., where the vehicle 102 is relative to features detected by the map and / or sensory component 110 in the environment), and / or a route generated by the guidance system 114.

[0030] The trajectory 118 may comprise instructions for the controller 116 of the autonomous vehicle 102 to operate the drive components of the vehicle 102 to achieve a guidance angle and / or guidance rate, which may result in a vehicle position, vehicle speed, and / or vehicle acceleration that tracks the path generated by the guidance system. For example, the trajectory 118 may comprise a target heading, target guidance angle, target guidance rate, target position, target speed, and / or target acceleration that the controller tracks as part of the path. For example, a coarse path generated by the guidance system 114 according to techniques discussed herein may indicate vehicle position, heading, speed, and / or entry / exit curvature at millisecond time intervals 500, and a smooth path output by the guidance system 114 may comprise such points at 10 millisecond or 100 millisecond intervals, which may correspond to the time intervals associated with the trajectory 118. In some examples, the controller may comprise software and / or hardware for operating the drive components of the vehicle 102 sufficiently to track the trajectory 118. For example, the controller 116 may include one or more proportional-integral-derivative (PID) controllers.

[0031] In the example scenario 100, the autonomous vehicle 102 has accepted and / or determined a route 120 that defines a start position 122, an end position 124, and a curve between the start position 122 and the end position 124 (note that the curve may comprise a straight line and / or one or more curves). For example, the planning component 112 may have determined the route 120 based at least in part on sensor data and an end position accepted as part of the mission (e.g., from an occupant, from a command center). As used herein, reference to a "position" may include both a location and / or a position (e.g., a position and / or a heading / heading of the vehicle). In some examples, the route may not comprise an end position 124 and may additionally or alternatively comprise a target position, such as a target lane, a target relative position (e.g., 10 feet from a road edge), a target object (e.g., follow the vehicle, follow the occupant, move toward a person calling the vehicle), etc.

[0032] As the vehicle maneuvers to reach end position 124, autonomous vehicle 102 may encounter scenarios such as example scenario 100 where a planner relying on lane criteria to generate a path (e.g., determined at least in part based on a map and / or locating autonomous vehicle 102) may be unable to generate a path accurately and / or efficiently. For example, various objects (e.g., blocking vehicle 126, toolbox 128, and fallen traffic cone 130) cumulatively block all three lanes of a depicted road, which may cause another planner to stop the vehicle and / or command teleoperation because the lanes do not have enough room for the autonomous vehicle.

[0033] However, the guidance system 114 discussed herein may generate the path 132 based at least in part on the environmental data 134 generated from the sensor data captured by the sensor 104. For example, the perception component 112 may generate all or a portion of the environmental data 134, which may include static data and / or dynamic data. For example, the static data may indicate the likelihood that an object is present at a location in the environment, and the dynamic data may indicate the likelihood that an object occupies or will occupy a location in the environment. In some instances, the dynamic data may include multiple frames associated with different time steps spaced up to a prediction limit (i.e., the maximum time / distance for which the dynamic data is predicted).

[0034] Techniques discussed herein may additionally or alternatively include determining dynamic data based at least in part on a mode of operation of the vehicle. For example, the dynamic data may include a first prediction associated with a first vehicle mode and a first time, and a second prediction associated with a second vehicle mode and a second time. 2 and a second prediction associated with a time of the first prediction. The vehicle modes may include mission level modes, such as passenger entry, passenger transfer, and passenger exit, route or trajectory level modes, such as maintain trajectory, slow down to a stop, shift lanes, perform a right turn, and / or signal modes, such as activate speaker, activate turn signals, flash lights, etc. The behavior and signals of the autonomous vehicle may affect decisions and behaviors made by other entities near the autonomous vehicle 102, thereby affecting the predicted movement of other objects.

[0035] In some examples, the perception component 110 may accept sensor data from the sensors 104 and determine data related to objects near the vehicle 102, such as static data and / or dynamic data. The depicted examples of environmental data 134 including static and / or dynamic data are top-down representations of such data, however, any representation of static and / or dynamic data is contemplated, such as heat maps, object classifications associated with detected objects, instance segmentations, semantic segmentations, two-dimensional and / or three-dimensional bounding boxes, trajectories, etc. The perception component 110 may additionally or alternatively determine route data specifying the destination of the vehicle, global map data identifying characteristics of roads (e.g., features detectable with different sensor modalities useful for locating an autonomous vehicle), local map data identifying characteristics detected in the vicinity of the vehicle (e.g., locations and / or dimensions of buildings, trees, fences, fire hydrants, stop signs, and any other features detectable with various sensor modalities), tracking data (e.g., environmental representations, object detection, and / or tracking as discussed herein), and / or the like.

[0036] In some examples, the perception component 110 may comprise a hardware and / or software pipeline that may include one or more GPUs, ML models, Kalman filters, etc. In some examples, the perception component 110 may monitor as much of the environment surrounding the autonomous vehicle as possible, which may be limited by sensor capabilities, object and / or environmental occlusions (e.g., buildings, height changes, objects in front of other objects), and / or environmental effects such as fog, snow, etc. For example, sensor data may include RADAR data that the perception component 110 may accept as input. The perception component 110 may be configured to detect as much information about objects and the environment as possible to avoid failing to account for events or object behaviors that should be taken into account by the planning component 112 in determining the trajectory 118.

[0037] Object classifications, which may be part of the environmental data 134 and determined by the perception component 110, may distinguish between different object types, such as, for example, passenger vehicles, pedestrians, bicyclists, delivery trucks, trailer trucks, traffic signs, etc. Tracking may include historical, current, and / or predicted object positions, speeds, accelerations, and / or headings. The data produced by the perception component 110 may be collectively referred to as perception data. Once and / or as the perception component 110 generates the perception data, the perception component 110 may provide the perception data to the planning component 112 and / or the guidance system 114. In some examples, the perception data may include the output of a sensor discrete pipeline (e.g., vision, LIDER, RADAR) and / or a hybrid sensor pipeline (e.g., vision-LIDER, RADAR-LIDER).

[0038] The planning component 112 may use the sensory data received from the perception component 110 and / or the path received from the guidance system 114 to determine one or more trajectories, control the movement of the vehicle 102 to traverse a path or route, and / or otherwise control the operation of the vehicle 102, although any such operation may be performed within various other components (e.g., localization may be performed by a localization component, which may be based at least in part on the sensory data). For example, the planning component 112 may determine a route for the vehicle 102 from a first location to a second location, determine a smooth trajectory from a rough trajectory received from a guidance system, substantially simultaneously generate a plurality of potential trajectories for controlling the movement of the vehicle 102 according to a backstopping limit technique (e.g., 1 microsecond, 0.5 seconds) to control the vehicle to traverse the route (e.g., to avoid any of the detected objects) based at least in part on the path and sensory data and / or simulated sensory data (which may further include predictions regarding detected objects in such data), and select one of the potential trajectories as a trajectory 118 of the vehicle 102 that may be used to generate drive control signals that may be sent to the controller 116 for actuating drive components of the vehicle 102. FIG. 1 illustrates an example of such a trajectory 118 represented as arrows indicating heading, speed, and / or acceleration, and the trajectory itself may include instructions for the controller, which in turn may actuate the drive system of the vehicle 102.

[0039] In some examples, the guidance system 114 may be a secondary guidance system to the nominal guidance system of the planning component 112. The planning component 112 may accept a trajectory or path from the nominal guidance system and may accept a path generated by the guidance system 114 discussed herein. The path accepted from the guidance system 114 may be treated by the planning component 114 as an uncertain path - for example, the planning component 112 may rely on the output of the nominal guidance system of the planning component 112 unless or until the output of the nominal guidance system causes the vehicle 102 to stop or command teleoperation or fails to output a feasible, collision-free, and / or economical path or trajectory. In such cases, the planning component 112 may switch using the path output by the guidance system 114. In additional or alternative examples, the planning component 112 may use the route output by the guidance system 114 to always generate a trajectory within certain regions (e.g., within areas without lane references) based at least in part on sensor data and / or perception data, etc.

[0040] The guidance system 114 may include one or more GPUs or may be communicatively coupled to one or more GPUs (e.g., via a publish-subscribe messaging system, via a data bus), and the techniques discussed herein may be parallelized and spread across threads of the GPUs, although it is contemplated that the techniques discussed herein may include at least portions that are sequential.

[0041] Exemplary System FIG. 2 illustrates a block diagram of an example system 200 implementing the techniques discussed herein. In some cases, the example system 200 may include a vehicle 202, which may represent the vehicle 102 of FIG. 1. In some cases, the vehicle 202 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire operation, and the driver (or passenger) is not expected to control the vehicle at all times. However, in other examples, the vehicle 202 may be a fully or partially autonomous vehicle having any other level or classification. Additionally, in some cases, the techniques described herein may be usable by non-autonomous vehicles.

[0042] Vehicle 202 may include vehicle computing device 204, sensors 206, emitters 208, network interfaces 210, and / or drive components 212. Vehicle computing device 204 may represent computing device 106, and sensor 206 may represent sensor 104. System 200 may additionally or alternatively include computing device 214.

[0043] In some instances, the sensors 206 may represent sensors 104 and may include LIDER sensors, RADAR sensors, ultrasonic transducers, sonar sensors, location sensors (e.g., Global Positioning System (GPS), compass, etc.), inertial sensors (e.g., Inertial Measurement Units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), image sensors (e.g., Red-Green-Blue (RGB) cameras, Infrared (IR) cameras, intensity cameras, depth cameras, time-of-flight cameras, etc.), microphones, wheel encoders, environmental sensors (e.g., thermometers, hygrometers, light sensors, pressure sensors, etc.), etc. The sensors 206 may include multiple instances of each of these or other types of sensors. For example, the RADAR sensors may include individual RADAR sensors installed at the corners, front, rear, sides, and / or top of the vehicle 202. As another example, the cameras may include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle 202. The sensors 206 may provide input to the vehicle computing device 204 and / or the computing device 214.

[0044] The vehicle 202 may also include emitters 208 for emitting light and / or sound, as described above. The emitters 208 in this example may include internal audio and visual emitters for communicating with occupants of the vehicle 202. By way of example and not limitation, the internal emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The emitters 208 in this example may also include external emitters. By way of example and not limitation, the external emitters in this example include lights for signaling the direction of vehicle action or other indicators (e.g., indicator lights, signs, light arrays, etc.) and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which may include acoustic beam steering technology.

[0045] The vehicle 202 may also include a network interface 210 that enables communication between the vehicle 202 and one or more other local or remote computing devices. For example, the network interface 210 may facilitate communication with other local computing devices and / or drive components 212 on the vehicle 202. The network interface 210 may also additionally or alternatively enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). The network interface 210 may additionally or alternatively enable the vehicle 202 to communicate with a computing device 214. In some examples, the computing device 214 may comprise one or more nodes of a distributed computing system (e.g., a cloud computing architecture).

[0046] The network interface 210 may include physical and / or logical interfaces for connecting the vehicle computing device 204 to another computing device or a network, such as the network 216. For example, the network interface 210 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 200.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing devices. In some instances, the vehicle computing device 204 and / or the sensors 206 may send sensor data to the computing device 214 over the network 216 at a particular frequency, such as in near real-time, after the lapse of a predetermined period of time, or the like.

[0047] In some cases, the vehicle 202 may include one or more drive components 212. In some cases, the vehicle 202 may have a single drive component 212. In some cases, the drive component 212 may include one or more sensors to detect the surroundings of the drive component 212 and / or the vehicle 202. By way of example and not limitation, the sensors of the drive component 212 may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive component, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive component, cameras or other image sensors, ultrasonic sensors that acoustically detect objects in the surroundings of the drive component, LIDER sensors, RADAR sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive component 212. In some cases, the sensors on the drive component 212 may overlap or supplement corresponding systems (e.g., sensors 206) of the vehicle 202.

[0048] The drive components 212 may include many of the vehicle systems, including a high-voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use by other vehicle systems, an induction system including an induction motor and an induction rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking forces to mitigate loss of traction and maintain control, an HVAC system, lighting fixtures (e.g., lighting fixtures such as head / tail lights for illuminating the exterior surroundings of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high-voltage junctions, high-voltage cables, charging systems, charging ports, etc.). Additionally, the drive components 212 may include a drive component controller that may accept and pre-process data from sensors to control the operation of various vehicle systems. In some cases, the drive component controller may include one or more processors and memory communicatively coupled to the one or more processors. The memory may store one or more components for performing various functions of the drive components 212. Moreover, the drive components 212 may also include one or more communication connections that enable each drive component to communicate with one or more other local or remote computing devices.

[0049] Vehicle computing device 204 may include a processor 218 and a memory 220 communicatively coupled to one or more processors 218. Memory 220 may represent memory 108. Computing device 214 may also include a processor 222, and / or a memory 224. Processor 218 and / or 222 may be any suitable processor capable of processing data and executing instructions to perform operations as described herein. By way of example and not limitation, processor 218 and / or 222 may include one or more central processing units (CPUs), graphics processing units (GPUs), integrated circuits (e.g., application specific integrated circuits (ASICs)), gate arrays (e.g., field programmable gate arrays (FPGAs)), and / or any other device or portion of a device that processes electronic data to convert the electronic data into other electronic data that may be stored in registers and / or memory.

[0050] Memory 220 and / or 224 may be examples of non-transitory computer-readable media. Memory 220 and / or 224 may store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods and functions attributed to the various systems described herein. In various implementations, memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which those illustrated in the accompanying figures are merely examples relevant to the discussion herein.

[0051] In some cases, memory 220 and / or memory 224 may store a localization component 226, a perception component 228, a planning component 230, a guidance system 232 (of which zero or more portions may be hardware, such as a GPU, CPU, and / or other processing unit), a map 234, and / or a system controller 236. The perception component 228 may represent the perception component 110, the planning component 230 may represent the planning component 112, the guidance system 232 may represent the guidance system 114, and the system controller 236 may represent the controller 116.

[0052] In at least one example, the localization component 226 may include hardware and / or software to accept data from the sensors 206 to determine the position, speed, and / or orientation (e.g., one or more of x-position, y-position, z-position, roll, pitch, or yaw) of the vehicle 202. For example, the localization component 226 may include and / or request / accept a map 234 of the environment and be able to continuously determine the location, speed, and / or orientation of the autonomous vehicle within the map 234. In some instances, the localization component 226 may accept image data, LIDER data, RADAR data, IMU data, GPS data, wheel encoder data, etc., utilizing SLAM (simultaneous localization and mapping), CLAMS (calibration, localization and mapping, simultaneously), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc., to accurately determine the location, position, and / or speed of the autonomous vehicle. In some cases, the localization component 226 may provide data to various components of the vehicle 202 to determine an initial position of the autonomous vehicle for generating a trajectory and / or for generating map data as discussed herein. In some examples, the localization component 226 may provide the location and / or orientation of the vehicle 202 relative to the environment and / or sensor data associated therewith to the perception component 228.

[0053] In some instances, the perception component 228 may comprise a primary perception system and / or a prediction system implemented in hardware and / or software. The perception component 228 may detect objects in the environment surrounding the vehicle 202 (e.g., identify that an object is present), classify the objects (e.g., determine an object type associated with a detected object), segment the sensor data and / or other representation of the environment (e.g., identify a portion of the sensor data and / or representation of the environment as associated with a detected object and / or object type), determine characteristics associated with the object (e.g., a trajectory identifying a current, predicted, and / or previous position, heading, speed, and / or acceleration associated with the object), etc. Data determined by the perception component 228 is referred to as perception data.

[0054] The planning component 230 can accept the location and / or orientation of the vehicle 202 from the localization component 226 and / or the sensory data from the perception component 228, and can determine instructions for controlling the operation of the vehicle 202 based at least in part on any of this data. In some examples, determining the instructions can include determining the instructions based at least in part on a format associated with a system with which the instructions are associated (e.g., a first instruction for controlling the movement of the autonomous vehicle can be formatted in a first format of messages and / or signals (e.g., analog, digital, pneumatic, kinematic) that the system controller 236 and / or the drive component 212 can perform / trigger analysis, and a second instruction for the emitter 208 can be formatted according to a second format associated therewith).

[0055] The guidance system 232 may operate on the vehicle 202 and / or on the computing device 214. In some examples, the guidance system 232 may be upstream from (providing input to) the planning component 230 in the pipeline and downstream from (accepting input from) at least some of the components of the perception component 228. The guidance system 232 may be configured to pass all or a portion of the output of the guidance system 232 to the planning component 230, or may be configured to pass none of the output of the guidance system 232 to the planning component 230. In some examples, the guidance system 232 may be part of the perception component 228. The guidance system 232 may be configured to generate a smooth path and a coarse path that the planning component 230 may use to generate a trajectory that tracks the smooth path, as discussed herein, and in additional or alternative examples, the guidance system 232 may be able to determine both a coarse path and a smooth path and may directly calculate the smooth path.

[0056] Memory 220 and / or 224 may additionally or alternatively store a mapping system (e.g., generating a map based at least in part on sensor data), a planning system, a ride management system, etc. Although localization component 226, perception component 228, planning component 230, guidance system 232, map 234, and / or system controller 236 are illustrated as being stored in memory 220, any of these components may include processor-executable instructions, machine-learned models (e.g., neural networks), and / or hardware, and all or a portion of any of these components may be stored on memory 224 or configured as part of computing device 214.

[0057] As described herein, the localization component 226, the perception component 228, the planning component 230, the guidance system 232, and / or other components of the system 200 may comprise one or more ML models. For example, the localization component 226, the perception component 228, the planning component 230, and / or the guidance system 232 may each comprise a different ML model pipeline. In some examples, the ML model may comprise a neural network. A typical neural network is a biologically inspired algorithm that passes input data through a series of connected layers to produce an output. Each layer in a neural network may also include another neural network, or may include any number of layers (whether convolutional or not). As can be understood in the context of the present disclosure, the neural network may utilize machine learning, which may refer to a broad class of such algorithms in which an output is generated based on learned parameters.

[0058] Although discussed in the context of neural networks, any type of machine learning may be used consistent with this disclosure. For example, machine learning algorithms may include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression spline (MARS), locally estimated scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least-angle regression (LARS)), decision tree algorithms (e.g., classification and regression tree (CART), iterative dichotomiser 3 (ID3), Chi-squared automatic interaction detection (CHAID), and others). detection, decision stumps, conditional decision trees), Bayesian algorithms (e.g., Naïve Bayes, Gaussian Naïve Bayes, Multinomial Naïve Bayes, average one-dependence estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-median, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., Perceptron, Backpropagation, Hopfield network, Radial Basis Function Network (RBFN)), deep learning algorithms (e.g., Deep Boltzmann Machine (DBM)),The architectures may include neural networks such as CNNs, deep belief networks (DBNs), convolutional neural networks (CNNs), stacked autoencoders, dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (blending), gradient boosting machines (GBMs), gradient boosting regression trees (GBRTs), random forests), support vector machines (SVMs), supervised learning, unsupervised learning, semi-supervised learning, etc. Additional examples of architectures include neural networks such as ResNet-50, ResNet-101, VGG, DenseNet, PointNet, etc. In some examples, the ML models discussed herein may include PointPillars, a second feature layer, a top-down feature layer (see, e.g., U.S. Patent Publication 2009 / 0133991, the entirety of which is incorporated herein by reference), and / or VoxelNet. Architecture latency optimizations may include MobilenetV2, Shufflenet, Channelnet, Peleenet, etc. The ML models may include residual blocks, such as Pixor, in some examples.

[0059] In some examples, the ML model may comprise a neural network, such as, for example, a convolutional neural network (CNN). As described herein, a typical neural network is a biologically inspired algorithm that passes input data through a series of connected layers to produce an output. Each layer in a neural network may also include another neural network, or may include any number of layers (whether convolutional or not). As can be understood in the context of the present disclosure, a neural network may utilize machine learning, which can refer to a broad class of such algorithms in which an output is generated based on learned parameters. As discussed in the context of neural networks, any type of machine learning may be used consistent with the present disclosure. In some examples, the guidance system 232 and / or the ML model discussed herein may comprise processor-executable instructions, hardware, and / or some combination thereof (e.g., field programmable gate array (FPGA), application specific integrated circuit (ASIC)) stored in and / or accessible to memory of the computing device 204.

[0060] The memory 220 can additionally or alternatively store one or more system controllers 236, which may be configured to control guidance, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 202. These system controllers 236 can communicate with and / or control corresponding systems of the drive component 212 and / or other components of the vehicle 202. For example, the planning component 230 can generate instructions based at least in part on sensory data generated by the perception component 228 and / or the guidance system 232, validate the sensory data and / or the path generated by the guidance system 232, and / or send instructions to the system controller 236. The system controller 236 can control operation of the vehicle 202 based at least in part on instructions received from the planning component 230.

[0061] 2 is illustrated as a distributed system, it should be noted that in alternative examples, components of the vehicle 202 may be associated with the computing device 214 and / or components of the computing device 214 may be associated with the vehicle 202. That is, the vehicle 202 may perform one or more of the functions associated with the computing device 214 and vice versa.

[0062] Example Dynamic Data 3A and 3B depict examples of dynamic data 300 and 302 that may be determined by a perception component based at least in part on sensor data associated with an object in an environment surrounding a vehicle. The perception component may comprise one or more ML models that accept sensor data and predict the movement of the object. For example, the one or more ML models may output a series of frames, each frame associated with a different future time. A distinct portion (e.g., pixel, region) of the frame may indicate a likelihood that the object will occupy a portion of the environment corresponding to the distinct portion at the time associated with the frame. The depiction illustrates the distinct portion as a small square and a zero or less than threshold likelihood as an empty space. In other words, only portions of the dynamic data 300 and 302 with significant (e.g., greater than a lower threshold) likelihood are depicted. Probabilities associated with distinct portions are depicted as shading of the square depicting the distinct portion - the darker the shading, the greater the likelihood. In some examples, the likelihood may be a posterior probability output associated with a portion of the environment based at least in part on the sensor data and the architecture of the ML model. The ML model may additionally or alternatively be trained to determine a velocity associated with each distinct portion and a confidence score associated with this velocity.

[0063] Both FIG. 3A and FIG. 3B depict a prediction for an object represented by dynamic data 300 and 302 moving in a direction 308 at time t=0 and a prediction that a vehicle is oriented in a direction 310 at least at time t=0. The vehicle's position 312 is represented as "x." For example, dynamic data 300 and / or 302 may be associated with a vehicle attempting to reverse out of a parking spot in front of the vehicle. Dynamic data 300 may include frames 304(1)-(n) and dynamic data 302 may include frames 306(1)-(n), each of which may be associated with a different future time. In some examples, the interval of time associated with the frames may be the same interval as the interval between node tiers (e.g., 500 milliseconds in one example, although other time periods are contemplated). For example, the first time, i.e., t=0, may be associated with the current time and the nth time may be associated with a prediction limit, such as t=2 seconds. Depending on the period between frames, there may be any number of intervening frames.

[0064] In some examples, the ML model may accept as input, in addition to sensor data, a signal indicating the mode of the vehicle. For example, the mode signal may include the output of the trajectory by the planner (e.g., either the trajectory itself or an abstraction thereof, such as "stay course", "turn right", etc.), a previously determined path, a mission level mode (e.g., drop passenger, pick up passenger, transport passenger), an emitter mode (e.g., speaker / horn activation, turning light status), etc.

[0065] Dynamic data 300 and dynamic data 302 illustrate the differences in predictions that may result from different vehicle modes provided as inputs. Figure 3A represents dynamic data that may be output by the ML model based on a vehicle mode indicating the vehicle is approaching a stop, and Figure 3B represents dynamic data that may be output by the ML model based on a vehicle mode indicating the vehicle is continuing on its trajectory. In short, Figure 3A includes a prediction that a dynamic object may complete a turn maneuver in front of the vehicle, and Figure 3B includes a prediction that the dynamic object may come to a stop.

[0066] The techniques discussed herein may include a first ML model that outputs nominally dynamic object predictions that do not consider vehicle modes. In other words, the first ML model is not trained using vehicle modes and the first ML model does not accept vehicle modes as inputs. The techniques may further include training a second ML model to accept vehicle modes as inputs, and / or the techniques may include training different ML models to handle different vehicle modes and selecting an ML model based on the current vehicle mode. Additionally or alternatively, one or more ML models may immediately generate an output, which may be concatenated and provided to a guidance system. For example, the dynamic data 300 and 302 may be generated by the same ML model or different ML models, but either or both of the dynamic data 300 and 302 may be provided to a guidance system. In at least one example, a vehicle mode may be associated with the dynamic data generated therefrom. As used herein, dynamic data may refer to a single frame of dynamic object data identifying the likelihood that an object occupies or will occupy a portion of the environment at a time associated with the frame, a frame generated from input that included a vehicle mode, a frame generated from input that did not include a vehicle mode, multiple frames (e.g., all of dynamic data 300, etc.), and / or collective dynamic data across multiple vehicle modes (e.g., dynamic data 300 and dynamic data 302).

[0067] Example Process 4A-4C illustrate a pictorial flow diagram of an example process 400 for determining a path for controlling an autonomous vehicle based at least in part on environmental data, which may be generated from sensor data and may include occupancy maps showing static and / or dynamic objects predicting position and / or velocity maps. The operations discussed herein may be used in combination, separately, sequentially, and / or in parallel and may be performed by the same device or different devices. For example, a computing device of the autonomous vehicle may accomplish at least some of the operations and / or a remote computing device (e.g., of a distributed computing service, of a teleoperation system) may accomplish at least some of the operations, but in at least some examples, the autonomous vehicle may accomplish all of the operations. Hardware and / or software components of a guidance system may be configured to accomplish at least a portion of the example process 400. In additional or alternative examples, a remote computing device may accomplish at least a portion of the example process 400. In some examples, at least some of the operations of the process may be pre-computed by other computing devices, such as the autonomous vehicle and / or a distributed computing system.

[0068] 4A-4C continue with the example scenario 100 illustrated in FIG.

[0069] In operation 402, the exemplary process 400 may include accepting a route associated with at least a start position in the environment by any of the techniques discussed herein. In some examples, accepting the route may additionally or alternatively include accepting a start position and a target (e.g., soft target) lane or other criteria. For example, the other criteria may include highway entrances or exits, objects (e.g., following another vehicle or a passenger who has temporarily left the vehicle), relative positions in an unstructured environment such as an open field, and the like. An exemplary route is depicted in FIG. 1 and is reproduced in FIG. 4A. In some examples, operation 402 may include determining a route based at least in part on the vehicle's current location (e.g., determined at least in part based on sensor data) and / or an end position (e.g., an end point of the route, a midpoint along the route). The route may additionally or alternatively include one or more curves (which may include one or more straight lines), and the curves may be determined at least in part based on curvatures associated with roads and / or other paths to the end location. Such curves may match and / or be determined at least in part based on indicators associated with maps and / or roads stored in memory (e.g., determining shapes associated with medians, double yellow lines, shoulder indications, sidewalks, etc.). In various examples, such a route may additionally or alternatively include desired lanes to drive in, streets to take, etc.

[0070] At operation 404, the example process 400 may include accepting environmental data associated with the environment surrounding the vehicle by any of the techniques discussed herein. The environmental data may include data about static objects, static data 406, and / or data about dynamic objects, dynamic data 408. The static data 406 and / or dynamic data 408 may be accepted from a perception component of the vehicle and / or from a localization component and / or mapping component of the vehicle. It should be noted that the dynamic data 408 may sometimes indicate that an object is stationary. Such an object may be included in the dynamic data 408 when the perception component associated a track with the object (e.g., the object was previously moving) and / or if the perception component classified the object as an object classification associated with a dynamic object identifier. Such object classifications may include, for example, bicyclists, pedestrians, vehicles, animals, etc., while buildings, signs, etc. would typically not be associated with a dynamic object identifier and associated sensor data that may be mixed into the static data 406. Further details regarding path planning using static object occupancy maps may be found in US Pat. No. 6,399,433 entitled "Unstructured Vehicle Path Planner," which is incorporated herein in its entirety.

[0071] In some examples, the size of the portion of the environmental data indicated as likely to be occupied may be expanded to further avoid collisions. In some cases, operation 404 may additionally or alternatively include determining a distance map, where a location in the distance map indicates the distance from the location to the closest portion indicated as occupied in the environmental data. Note that a distance map may be determined for each time step of the dynamic data 408, one or more of the time steps of the dynamic data 408, and / or a mixed distance map may be determined (e.g., an average of multiple distance maps). The static data 406 and / or the dynamic data 408 may indicate the likelihood and / or a binary indication that a portion of the environment is / is not occupied. For example, the static data 406 may indicate the presence of a static object by indicating a maximum likelihood (e.g., 1 in an example where the likelihood ranges from 0 to 1, with 1 being a confident indication that the object is present) and the dynamic data 408 may indicate a likely position of the object by a likelihood between 0 and 1 (not including 1), although other systems may be used to indicate static and dynamic objects, such as using a logit. In some examples, the distance map may be calculated based at least in part on filtering the environmental data (i.e., the static data 406 and / or the dynamic data 408) using a likelihood threshold to determine a binary mask. In other words, determining the distance may be based at least in part on the distance from a point in the environment to the nearest portion of the environmental data that exhibits a likelihood equal to or greater than the threshold. Note that the position of the vehicle is indicated with an "x".

[0072] At operation 410, the example process 400 may include determining a cost map based at least in part on the environmental data by any of the techniques discussed herein. For example, operation 410 may include concatenating or otherwise combining the static data 406 and the dynamic data 408 and / or generating a cost map 412 for one or more time steps and / or vehicle modes. The time interval between the cost maps may be the same as or different from the time interval between the dynamic data 408 time steps and / or the time steps between the layers of nodes generated at operation 414. In other words, the dynamic data, cost maps, and node generation rates may be generated relative to time steps that are synchronous or asynchronous. Thus, interpolation or projection may be required to generate the cost map 412 or to determine costs associated with the nodes. In at least one example where interpolation may not be used, the number of cost maps 412 may be equal to the maximum number of layers to be generated such that a cost map is generated for each time step in which a node is generated. Additionally or alternatively, the number of cost maps may be less than the number of layers of nodes.

[0073] In some examples, determining the cost map includes determining costs associated with reaching a position, heading, velocity, acceleration (e.g., longitudinal and / or lateral acceleration), and / or entry / exit curvature in the environment, although for simplicity, the discussion herein focuses primarily on reaching a position. Note that the cost map may be associated with a multivariate space associated with these different metrics (e.g., position, heading, velocity, curvature), which is greatly simplified in the figures. In practice, this means that a first cost may be associated with position, heading, and velocity, while different costs may be associated with the same position and heading but different velocities.

[0074] Determining the cost may be based at least in part on a pre-computed field of visit costs indicating weights associated with maneuvers required to reach different portions of the multivariate space. The pre-computed field of visit costs may include multiple fields for different speeds (e.g., 5 miles per hour, 15 miles per hour, etc.). For example, the cost of reaching a position and heading 10 feet ahead of the vehicle may be higher than if the speed at that position is higher or lower than the vehicle's current speed (e.g., if the vehicle is traveling 30 miles per hour, a first cost to reach the same position and heading at 25 miles per hour or 35 miles per hour will be less than a second cost to reach the same position and heading at 10 miles per hour or 50 miles per hour).

[0075] Determining the cost may additionally or alternatively be based at least in part on determining a distance from the position in the cost map to a target reference, such as a target lane, determining a distance to a static object or a dynamic object (which may be based on a distance map determined in operation 404), and / or determining a likelihood of collision based at least in part on the static data 406, the dynamic data 408, and / or the distance map determined in operation 404. For example, the cost may be based on a distance to the nearest object and / or a speed and / or direction of travel of the nearest dynamic object. The cost may additionally or alternatively be based on a route or The deviation cost may be based at least in part on deviations in position, heading, speed, and / or curvature specified by the position in the multivariate space from other reference points (e.g., target lanes) and / or deviation costs based at least in part on displacement along the route. For example, the cost may decrease as the displacement along the route increases. Determining the displacement along the route may include projecting the point onto the route and determining a portion of the route that would be effectively traversed. The deviation cost may additionally or alternatively be based at least in part on deviations in the position, heading, speed, etc. in the multivariate space from the vehicle's current position, heading, speed, etc.

[0076] In some examples, determining the cost may be based at least in part on an object classification associated with the object. For example, because the behavior and / or importance of objects may vary, different cost functions may be associated with different types of objects. For example, a cost function associated with a static object may not include a component that accounts for movement, and a cost function associated with a vehicle may differ from a cost function associated with a pedestrian.

[0077] In some examples, operation 410 may include determining a cost associated with less than the entirety of the multivariate space. For example, operation 410 may limit the cost determination based at least in part on a current vehicle mode, which may include a speed of the vehicle, and / or the cost determination may be based on operation 414 and / or operation 43. 2 In other words, all, a portion, or no cost map may be calculated before one or more nodes are generated. For example, a portion of the cost map may be determined based at least in part on a distance map determined based at least in part on environmental data and / or pre-calculated visit cost fields, while other portions of the cost determination (e.g., collision probability cost, similarity cost, route displacement cost) may be determined once the nodes are determined or as part of the node generation operation.

[0078] In some examples, operation 410 (and any of the other operations that discuss dynamic data) may include determining a portion of the dynamic data 408 associated with a drive corridor of the vehicle. The drive corridor may be associated with a previous path, route, or current trajectory of the vehicle. Determining the portion of the dynamic data 408 may include determining a portion of the dynamic data 408 that falls into the drive corridor at one or more of the time steps. In some cases, buffers on both sides of the drive corridor may be added to include additional dynamic data 408 for safety. In some examples, determining the cost map may be based on this portion of the dynamic data 408 rather than all of the dynamic data 408, although if sufficient computational bandwidth is available, all of the dynamic data 408 may be used.

[0079] Returning to FIG. 4B, at operation 414, the example process 400 may include determining a set of nodes for the next time step (or next distance) by any of the techniques discussed herein. FIG. 4B depicts a simplified version of the cost map (simplified cost map 416) so that the representation of the nodes may be more clearly seen. Note that the number and density of nodes are only depicted for this example. The number and density of nodes and the size of the time steps may be larger or smaller than depicted and described herein. Determining the set of nodes for the next time step may be based at least in part on the position of one or more nodes of any previous layer. In the depicted example, the next time step is the first time step, i.e., t=1, so that node 418 was determined based on the current position of the vehicle at time t=0, indicated by "x". The subsequent layer of nodes associated with the second time step, i.e., t=2, may be based at least in part on the position of one or more of nodes 418 in the multivariate space. If each time step corresponds to 100 milliseconds of time, then the nodes 418 may correspond to different positions, headings, velocities, and / or curvatures at 100 milliseconds in the future. A second tier of nodes may correspond to 200 milliseconds in the future, etc., until computation or memory allocation is exhausted, until a target location is reached, and / or until a threshold number of nodes / time steps set, i.e., n, is reached.

[0080] In some examples, the nodes may be associated with control policies, such as, for example, maintain heading and speed, decelerate at a specified deceleration with 5 degree continuous changes in guidance angle, maintain heading and speed but nudge away from the object, etc. The nodes themselves may comprise data structures indicating the control policy associated with the node, the position in the multivariate space that would result from implementing the control policy, the cost to reach the node, the predicted cost of the following node, a similarity cost, and / or the identity of a parent node associated with the node (e.g., the node on which the node is based - e.g., a parent node may be the starting point of the control policy). Determining the set of nodes may include generating a tree with each node having one parent, although in additional or alternative examples, determining the set of nodes may include generating a directed acyclic graph (DAG) or any other type of data structure to relate the nodes. In some examples, the control policy may be dynamically generated by a planning component of the vehicle based at least in part on sensor data, perception data, and / or map data, and / or the control policy may be one of a number of standard control policies stored in memory. In some examples, a standard control policy may be associated with different sensory data and / or map data conditions. For example, a control policy for slow, sharp turns may be unavailable while traveling on a highway above a certain speed, while a different control policy for emergency high speed maneuvers may be associated with highway driving.

[0081] The cost of reaching a node may be the sum of the cost of reaching the node from its parent node (e.g., based at least in part on a cost map that may take into account deviation costs, collision costs) and the cost associated with the parent node. In other words, the cost of reaching a node may be the summed costs of each of the nodes that hierarchically precede the node.

[0082] The similarity cost of a node may be determined based at least in part on the node's proximity to the nearest node or nodes in the multivariate space and / or the match score between the control policies. This similarity cost may be calculated before or after node generation. In at least one example, the similarity cost may be determined and added to a total cost associated with the node after collisions and / or hash pruning, as discussed further below.

[0083] In some examples, determining the nodes may include searching for a solution in a multivariate space that maximizes a combination of displacement along the route and lateral / azimuth diversity between solutions (or satisfies a diversity heuristic) and minimizes a cost based at least in part on a cost map within a given time interval. For example, the search algorithm may include algorithms such as D*, D*lite, Focused Dynamic A*, A*, LPA*, Dijkstra's algorithm, etc., although other search algorithms for exploring and / or generating directed graphs and / or weighted directed graphs may be used. In some examples, the search may be based at least in part on building a directed graph or tree based at least in part on a sample node and a rule set. For example, the rule set may specify that two nodes in the same layer may not be connected and / or there may be a minimum and / or maximum distance between connected nodes, etc. In some examples, one of the algorithms discussed above may be modified to include a cost parameter as discussed herein. In at least one example, the distances from the static data 406 and / or the dynamic data 408, their values, and / or the cost map may be cost parameters that are updated as part of a search algorithm.

[0084] In some examples, operation 414 may return m nodes identified by the search algorithm as having increased displacement and lateral / azimuth diversity at lower cost, where m is a positive integer. "Increased" and "lower" mean that the displacement, diversity, and cost values ​​may or may not have been maximum or minimum values ​​themselves, but have been identified by the search algorithm as balancing the displacement, diversity, and / or cost. In other words, the search algorithm may attempt to optimize the node's displacement, diversity, and cost. For layers beyond the first layer, the search algorithm may explore up to m nodes per node in the previous layer.

[0085] At operation 420, the example process 400 may include determining whether the node 422 is likely to cause a collision by any of the techniques discussed herein. For example, this operation may include representing the vehicle at the node 422 in a multivariate space. This may include representing the vehicle as two circles having a diameter equal to the width of the autonomous vehicle, where a first circle may be centered on the front axle of the autonomous vehicle (i.e., front circle 424) and a second circle may be centered on the rear axle of the autonomous vehicle (rear circle 426), and determining a distance between the representation and the nearest static object and / or a portion of the environment predicted to be occupied by a dynamic object at a future time. In some examples, the node 422 may be positioned at a distance from the vehicle to the node 422. 2 is the position of the rear axle (for example, rear circle 42 6 ) although additional or alternative placements are contemplated (e.g., center of the overlap region of the circles, center of the front circle 424). In some examples, the representation of the occupied space shown by the autonomous vehicle and / or occupancy map may be expanded to account for error (e.g., the diameter of the circle and / or the area shown as occupied within the occupancy may be expanded to half the autonomous vehicle width).

[0086] Determining whether the node is likely to incur a collision may include determining a shortest distance 428 from either circle to the nearest static or dynamic object. Determining a distance to a dynamic object may include determining a distance to a portion of a cost map associated with the environmental data and / or a likelihood at or above a likelihood threshold. If the distance 428 is less than the distance threshold, operation 420 may proceed to operation 430. If not, operation 420 may proceed to operation 432.

[0087] At operation 430, the example process 400 may include pruning a node that may be likely to cause a collision and / or associating a collision possibility with the node by any of the techniques discussed herein. Pruning a node may include removing the node from the tree or graph or associating an indication with the node that the node will cause a collision. The collision indicator may prevent the node from being used as a parent of a subsequent node and from being selected as part of a final path. Associating a collision possibility with the node may include modifying a weight associated with the node (e.g., by increasing the cost) as part of a cost determination operation at operation 444.

[0088] At operation 432, the example process 400 may include determining a hash associated with a node in the set of nodes generated at operation 414, according to any of the techniques discussed herein. FIG. 4B shows an example hash 4 of node 422 in the set of nodes 418. 3 4, where the hash 434 is represented by a string of numbers and / or letters depending on the type of hash. In some examples, the hash associated with a multivariate space may be pre-computed by a hashing function, with a number of bits allocated to each dimension of the multivariate space.

[0089] For example, if the hashing function is configured to output 64-bit integers, then the x-position in the state space may be represented by a 16-bit hash, the y-position by 16 bits, the velocity by 8 bits, and the yaw by 8 bits. The hash function may be designed to generate a hash that represents a portion of a multivariate space having a dimension determined at least in part based on the resolution 436 set as a parameter of the hash function. Any node located within a dimension of that portion of the multivariate space will generate the same hash 436. 4 The resolution 436 may be used to control the diversity of nodes generated - the larger the resolution 436, the greater the node diversity (i.e., more nodes will generate the same hash), but this may reduce the total number of nodes after pruning, so the resolution 436 may be adjusted so that there are tens of solutions (e.g., 20, 30, 50), although other numbers are contemplated (e.g., 5, 100, hundreds).

[0090] According to the illustrated example, both node 422 and node 438 have hash 4 due to the size of resolution 436. 3 4. In at least one example, operation 432 may include storing a list of node hash values ​​that have been generated so far during the search. In some examples, the node hash may be calculated for the vehicle using a hashing function, while in additional or alternative examples, the node hash may be calculated by looking up a hash in a lookup table from a set of pre-calculated hashes based on the node's location in the multivariate space.

[0091] 4C, at operation 440, the example process 400 may include determining whether the hash of the node matches the hash of any other node, according to any of the techniques discussed herein. If the hash of the node matches the hash of another node, operation 440 may proceed to operation 442 where at least one of the nodes is pruned; otherwise, operation 442 may proceed to operation 443 where at least one of the nodes is pruned. 40 may proceed to operation 444. In some examples, operations 440 and 442 may include two stages. In a first stage, hashes of newly generated nodes ("node candidates") may be compared against each other and pruned, and in a second stage, hashes of the remaining node candidates may be compared to previously generated nodes.

[0092] action 4 4 In 2, the exemplary process 400 may include pruning one of the nodes associated with the matching hash by any of the techniques discussed herein. In at least one example, the most recently generated node may be discarded. In an additional or alternative example, the exemplary process 400 may include determining a cost associated with each node with a matching hash and discarding all but the node associated with the lowest cost. Discarding a node may remove the node from consideration for use as part of a path and / or prevent subsequent layers of the node from being generated based on the pruned node location in the multivariate space. If the pruned node is a parent node (i.e., a node that has one or more child nodes), its child nodes may also be pruned. In some examples, if the two hashes collide, the newer of the nodes may be pruned if the two nodes were generated in different node generation operations. For example, if a node candidate collides with a previously generated node (e.g., in stage 2 of hash-based pruning), the older of the two nodes (i.e., the node generated in association with the previous node set) may be preserved and the newer node may be pruned. 4C illustrates one example of a pruned tier of nodes 446. It should be noted that although a hash-based pruning method is described herein, other suitable pruning techniques may also be used.

[0093] At operation 444, the example process 400 may include determining a cost associated with the node according to any of the techniques discussed herein. In some examples, operation 444 may be part of operations 410, 414, and / or 420. For example, operation 410 may include determining a first portion of a cost associated with the node, and operation 444 may include determining a second portion of a cost associated with the node. In additional or alternative examples, operation 410 may be reduced or eliminated (e.g., in examples where the generation of nodes is not based on cost), and operation 444 may include one or more portions of operation 410 discussed above. In yet another example, generating nodes may be based at least in part on determining costs associated with different portions of the multivariate space, and / or pruning nodes may be based at least in part on determining costs associated with the nodes. Other costs, such as collision costs, deviation costs, visit costs, etc., may be generated in operations 410, 414, and / or 420, any of which may be calculated in operation 444 (as long as subsequent operations, such as node generation in examples where node generation is cost-based, do not rely on them).

[0094] At least one additional cost that may be calculated after the nodes are generated is the similarity cost. The similarity cost may include a weight (weighted / unweighted) applied to the sum of the costs discussed above, which increases the cost of nodes that are close in the multivariate space. However, if the resolution is adjusted correctly, the similarity cost may be small or not used at all.

[0095] In operation 448, the exemplary process 400 may include determining whether a computational limit has been met, by any of the techniques discussed herein. For example, operation 448 may include determining whether a computational or memory allocation has been exhausted, determining that a target location in the environment has been reached by one or more nodes, and / or determining that a threshold number of sets of nodes / time steps, i.e., n, has been reached. In other words, if the nth node set has been generated, operation 448 may transition to operation 452, otherwise operation 448 may transition to operation 450 and finally return to any one or more of operations 414, 420, 430, 432, 440, 442, 444, and / or 448. For example, the exemplary process 400 may iteratively repeat at least some of operations 414, 420, 430, 432, 440, 442, 444, and / or 448 until one of the computational limits is met.

[0096] At operation 450, the example process 400 may include selecting the lowest-cost p lowest-cost node for further exploration (unless a limit is met), where p is a positive integer, by any of the techniques discussed herein. The lowest-cost p node may be selected from among the pruned set of nodes. Note that the lowest-cost p node may include nodes from the set of most recently generated nodes and / or the set of previously generated nodes. In some instances, this may cause the algorithm to revert to exploring older nodes for paths if the most recently explored node becomes too costly (e.g., the difference cost starts to become high, causing the algorithm to look for a new path) or introduces collisions.

[0097] The lowest cost p node may be selected (i.e., at operation 414) to determine a subsequent layer of nodes. Determining a subsequent layer of nodes (at operation 414) may be based at least in part on exploring nodes among the p selected nodes. Exploring nodes may include determining up to m nodes based at least in part on the nodes (see node 500 determined at least in part based on node 422 in FIG. 5A). As discussed above, exploring nodes may include determining positions in a multivariate space that would result from implementing different control policies with the explored node as a starting point.

[0098] The number of nodes explored by the search algorithm for each node in the previous layer, i.e., m, may be greater than or equal to the number p of nodes selected in operation 450. The next layer of nodes may include up to p·m nodes. In some examples, p may be tens, hundreds, or any other number of nodes. Determining the lowest cost p nodes is performed in operation 44. 4 The method may include ranking, sorting, and / or identifying the lowest cost p nodes according to their cost determined in, however, in some examples, the nodes may be sorted by cost as they are generated according to a counting sorting method that may be accomplished by a GPU processing unit.

[0099] In some examples, different cores or threads of a first type processor, such as a GPU, may handle different ones of the node explorations out of the explored p nodes. For example, a first core or thread may handle generating up to m nodes based on a first node of the p nodes, a second core or thread may handle generating up to m nodes based on a second node of the p nodes, etc. It will be understood that the cost determination (e.g., the cost map and / or a particular cost associated with the node), hash determination, and / or other steps may be parallelized as well, but the second type processor may handle any of these or other operations. In some examples, once the nodes, hashes, and / or costs are determined, the nodes, hashes, and / or costs may be sent to a second type processor different from the first type processor for processing, which may include operations 430, 432, 434, 448, and / or 450, but the first type or processor may accomplish one or more of these operations.

[0100] In operation 452, the example process 400 may include determining a lowest-cost end node and / or outputting a path associated therewith by any of the techniques discussed herein. In some examples, the search algorithm may gradually track and / or generate a total running cost of paths and different paths as part of the search algorithm (e.g., this total cost may be identified by the furthest node along one path in a graph or tree as the cost to reach the node), while some search algorithms may determine one or more paths after storing up to n node set data. In at least one example, operation 452 may include determining the lowest-cost node associated with the most recent time step (e.g., the node associated with 5 seconds in the future if 5 seconds is the maximum time step reached) or a displacement along the route. In some examples, the cost itself may reflect the displacement, since the displacement may be part of the cost determination (e.g., reducing the overall cost), and operation 452 may simply select the lowest-cost node. In yet another example, determining the lowest-cost node may include determining the lowest-cost node (e.g., lowest-cost end node 454) from among the most recently generated (last) set of nodes rather than from all of the generated nodes.

[0101] Additionally or alternatively, determining the path may include iteratively determining different combinations of connections between nodes of different node layers and selecting connections to add to the directed graph based at least in part on reducing the running total cost of the combinations of connections. In some examples, the search algorithm may do this incrementally as nodes are generated, depending on the algorithm used. For example, the search algorithm may store these connections in a directed graph or tree, including connections (e.g., edges) between first and second nodes and / or weights (e.g., costs) associated with the connections. The costs may be based at least in part on the costs determined in operations 410, 414, 420, and / or 444.

[0102] In some examples, operation 452 may include determining a path associated with a lowest-cost node. In examples where each node has only one parent (e.g., nodes generated according to a tree structure), determining the path 456 may include outputting the lowest-cost node and all of its parents back to the starting position. In examples where the nodes are generated according to a graph, determining the path may include tracing the lowest-cost path from the node. Outputting the path may include outputting the lowest-cost node, a cost associated with the lowest-cost node, one or more intervening nodes between the lowest-cost node and the starting position and / or costs associated therewith, and / or a trace through the nodes that form the lowest-cost node to the starting position. In some examples, operation 452 may additionally or alternatively include outputting multiple paths. For example, operation 452 may output the r lowest-cost paths, where r is a positive integer, and / or at least two paths in situations where significantly different paths are created (e.g., one path goes left and another path goes right).

[0103] For example, path 456 can be a curve, such as a Bezier curve as illustrated in FIG. 4C , while the path output by the guidance system can include a coarse path, which can include a list of coordinates of the nodes connected by the path (e.g., positions, heading, speed specified thereby), linear connections between the nodes, etc. In examples where operation 452 includes determining a coarse path, the example process 400 can further include sending the coarse path to another component, such as a smoothing component of the planning component, which accepts the coarse path and smooths the coarse path into a smooth path having curvature and acceleration that is drivable by the particular type of vehicle being controlled.

[0104] Exemplary Iterative Node Layer Generation 5A-5D depict a top-down view of the results of successive node generation and pruning operations in a multivariate space according to the techniques discussed herein. For simplicity, static objects are represented binary rather than according to their cost mappability, as per the above discussion. Note that the dynamic data associated with different time steps changes depending on the frame of dynamic data associated with the respective time step. Also, note that while FIGS. 4A-5D depict only the node layers and paths generated for one vehicle mode, it is contemplated that the guidance system may determine paths associated with at least the current vehicle mode or the nominal vehicle mode (default) and / or up to one path per vehicle mode. FIGS. 5A-5D may depict at least some further iterations of operations 414, 420, 430, 432, 440, 442, 444, 448, and / or 450 of the exemplary process 400 according to the example depicted in FIGS. 4A-4C.

[0105] 5A depicts a top-down representation of a second set of nodes 500 generated from the lowest cost p nodes of the first set of nodes (e.g., the pruned set of nodes 446). The depiction includes nodes connected to nodes 422 by lines. Nodes 500 represent nodes that may be generated based at least in part on nodes 422 as part of operation 414. Nodes 500 are shown as being generated from nodes 422 by lines. In at least one example, different ones of the lowest cost p nodes may have been assigned to different processing units of a GPU for exploration. For example, a core or thread of a GPU may have been assigned to node 422 and generated node 500 in response.

[0106] FIG. 5B depicts a top-down representation of the result of pruning the pruned set of nodes 446 and the second set of nodes 500 (i.e., second pruned node 504). Note that the second pruned node 504 includes nodes generated in both the first and second sets of nodes. The lowest cost p-node of the second pruned node 504 may be selected and used to generate the third set of nodes depicted in FIG. 5C. may be pruned as the third pruned node 508. FIG. 5D depicts a top-down representation of the third pruned node 508 remaining after the second pruned node 504 and the third set of nodes 506 are pruned.

[0107] Figures 6A-6F depict top-down representational node creation in an environment with two static objects. Figures 6A-6F depict free space (Free Space 600) in white, occupied space (Occupied Space 602) in black, and nodes in grey. Figure 6A depicts the environment and a node 604 associated with a starting position.

[0108] FIG. 6B illustrates the nodes generated after six iterations of node generation.

[0109] FIG. 6C illustrates the nodes generated after 10 iterations of node generation.

[0110] Figure 6D illustrates the nodes generated after 15 iterations of node generation. Figure 6D also demonstrates that the lowest cost p node for exploration can be a node from the set of nodes generated many previous iterations.

[0111] 6E illustrates the nodes generated after 20 iterations of node generation. Note that there are new nodes (indicated at 606) that pass through static objects, but there are also new nodes that have been added - these are indicated at 608 - that are explorations of earlier iterations of node generation.

[0112] FIG. 6F illustrates the nodes generated after 25 iterations of node generation and the output path 610 associated with the lowest cost end node, shown as a solid line.

[0113] Example clauses A. Accepting a predicted position of a dynamic object in an environment and a probability that the dynamic object will occupy the predicted position at a future time; determining a path for controlling a vehicle based at least in part on data of the environment, determining a cost map based at least in part on the environmental data; determining a first node based at least in part on a current position of the vehicle in the environment, the first node being associated with a first candidate trajectory for controlling movement of the vehicle over a first time interval; determining a subset of the first nodes based at least in part on a pruning technique; and determining a cost map associated with the subset of the first nodes. determining a route including determining a cost associated with a node of a first subset of nodes based at least in part on the cost map and a similarity of the node to the additional node; determining two or more nodes of the subset having a lowest cost among the costs associated with the first subset of nodes based at least in part on the costs; determining a second node based at least in part on the two or more nodes; and determining a route based at least in part on a second cost associated with the second node and the cost associated with the subset of first nodes.

[0114] B. The method of paragraph A, wherein determining the nodes of the second nodes is based at least in part on determining a future position, speed, and heading of the vehicle based at least in part on projecting a position, speed, and heading associated with one of the two or more nodes according to a control policy, wherein each node of the second nodes is associated with a different control policy.

[0115] C. The method of either paragraph A or B, wherein the pruning technique includes determining a hash associated with the first node, determining that one or more of the first nodes are associated with redundant hashes, and discarding one or more of the first nodes based at least in part on determining that the one or more of the first nodes are associated with redundant hashes.

[0116] D. The method of any one of paragraphs A-C, wherein the path is a coarse path defining at least one of a position, heading, or velocity of the vehicle over a number of first time intervals, and the operation further includes refining the coarse path based at least in part on determining a curvature of the path in addition to at least one of a position, heading, or velocity over a number of second time intervals, the second time intervals being shorter than the first time interval.

[0117] E. The method of any one of paragraphs A-D, wherein the coarse path is determined by a first type processing unit and the refined path is determined by a second type processing unit different from the first type processing unit.

[0118] F. The method of any one of paragraphs A-E, wherein the future time is a first future time and the probability is a first probability, and the data of the environment further includes a second map indicative of a second predicted position of the dynamic object and a second probability that the dynamic object will occupy the second predicted position at a second future time that is later than the first future time, and wherein a cost associated with the first subset of nodes is based at least in part on the first probability and the second cost is based at least in part on the second probability.

[0119] G. one or more processors, and a method, when executed by the one or more processors, for causing a system to receive a predicted position of a dynamic object in an environment and a probability that the dynamic object will occupy the predicted position at a future time, determine a path for controlling a vehicle based at least in part on data of the environment, determining a cost map based at least in part on data of the environment, determining a first node based at least in part on a current position of the vehicle in the environment, determining a subset of the first nodes based at least in part on a pruning technique, and determining a cost associated with the subset of the first nodes, wherein the cost map and one or more other and a memory storing processor-executable instructions to cause operations including determining a cost including determining a cost associated with a node based at least in part on a similarity of the nodes of a first subset of nodes to the node; determining two or more nodes of the subset having a lowest cost among the costs associated with the first subset of nodes based at least in part on the cost; determining a second node based at least in part on the two or more nodes; and determining a path based at least in part on one or more of the second cost associated with the second node and the costs associated with the first subset of nodes.

[0120] H. The system of paragraph G, wherein determining the nodes of the second nodes is based at least in part on determining a future position, speed, and heading of the vehicle based at least in part on projecting a position, speed, and heading associated with one of the two or more nodes in accordance with a control policy, each node of the second nodes being associated with a different control policy.

[0121] I. The system of either paragraph G or H, wherein the path is a coarse path defining at least one of a position, heading, or velocity of the vehicle over a number of first time intervals, and the operation further includes refining the coarse path based at least in part on defining a curvature of the path in addition to at least one of a position, heading, or velocity over a number of second time intervals, the second time intervals being shorter than the first time intervals.

[0122] J. The system of any one of paragraphs G-I, wherein the coarse path is determined by a first type processing unit and the smooth path is determined by a second type processing unit different from the first type processing unit.

[0123] K. the future time is a first future time, the probability is a first probability, and the data of the environment further includes a second map indicative of a second predicted position of the dynamic object and a second probability that the dynamic object will occupy the second predicted position at a second future time that is later than the first future time; 1. node of The system of any one of paragraphs G-J, wherein the cost associated with the subset is based at least in part on a first probability and the second cost is based at least in part on a second probability.

[0124] L. The system of any one of paragraphs G-K, wherein determining the cost map includes determining a cost associated with at least one of a position, heading, speed, or curvature in a parameter space associated with the environment, and wherein determining the cost is based at least in part on determining a visit cost based at least in part on a deviation of at least one of the position, heading, speed, or curvature from a current position, heading, speed, or curvature of the vehicle or target lane, or on determining a proximity of a static object to a position, a proximity or probability of a position to a predicted position, or on determining a deviation of the position, heading, speed, or curvature from a route or path criteria.

[0125] M. The system of any one of paragraphs G-L, wherein determining a cost associated with a node of the first subset of nodes further includes determining a weight based at least in part on at least one of: a similarity of the first node to a second node of the first subset of nodes, a difference in a lateral position of the first node compared to a lateral position of one or more other nodes of the first subset of nodes, or a distance along a route associated with the first node.

[0126] N. When executed by one or more processors, the method includes having the one or more processors receive a predicted position of a dynamic object in an environment, a probability that the dynamic object will occupy the predicted position at a future time, determining a path for controlling the vehicle based at least in part on the data of the environment, determining a cost map based at least in part on the data of the environment, determining a first node based at least in part on a current position of the vehicle in the environment, determining a subset of the first nodes based at least in part on the pruning technique, and determining a cost associated with the subset of the first nodes, the cost map and one or more other nodes. A non-transitory computer-readable medium storing processor-executable instructions to cause operations including determining a cost including determining a cost associated with the nodes based at least in part on similarity of the nodes of the first subset of nodes; determining two or more nodes of the subset having a lowest cost among costs associated with the first subset of nodes based at least in part on the cost; determining a second node based at least in part on the two or more nodes; and determining a route based at least in part on one or more of the second cost associated with the second node and the cost associated with the first subset of nodes.

[0127] O. The non-transitory computer-readable medium of paragraph N, wherein determining the node of the second node is based at least in part on determining a future position, speed, and heading of the vehicle based at least in part on projecting a position, speed, and heading associated with one of the two or more nodes according to a control policy, wherein the node of the second node is associated with a different control policy.

[0128] P. The non-transitory computer-readable medium of either paragraph N or O, wherein the pruning technique includes determining a set of hashes associated with a first node and determining nodes among the first node associated with the redundant hashes as redundant nodes, the subset of nodes excluding the redundant nodes.

[0129] Q. The non-transitory computer-readable medium of any one of paragraphs N-P, wherein the path is a coarse path defining at least one of a position, heading, or velocity of the vehicle over a number of first time intervals, and the operation further includes refining the coarse path based at least in part on defining a curvature of the path in addition to at least one of the position, heading, or velocity over a number of second time intervals, the second time intervals being shorter than the first time interval.

[0130] R. the future time is a first future time, the probability is a first probability, and the data of the environment further includes a second map indicating a second predicted position of the dynamic object and a second probability that the dynamic object will occupy the second predicted position at a second future time that is later than the first future time; 1. node of The non-transitory computer-readable medium of any one of paragraphs N-Q, wherein the cost associated with the subset is based at least in part on a first probability and the second cost is based at least in part on a second probability.

[0131] S. The non-transitory computer-readable medium of any one of paragraphs N-R, wherein determining the cost map includes determining a cost associated with at least one of a position, heading, speed, or curvature in a parameter space associated with the environment, and determining the cost is based at least in part on determining a visit cost based at least in part on a deviation of at least one of a position, heading, speed, or curvature from a current position, heading, speed, or curvature of the vehicle or target lane, determining a proximity of the position to a static object, a proximity or probability of the position to a predicted position, or determining a deviation of the position, heading, speed, or curvature from a route or path criteria.

[0132] T. The non-transitory computer-readable medium of any one of paragraphs N-S, wherein determining a cost associated with a node of the first subset of nodes further includes determining a weight based at least in part on at least one of: a similarity of the first node to a second node of the first subset of nodes, a difference in a lateral position of the first node compared to a lateral position of one or more other nodes of the first subset of nodes, or a distance along a route associated with the first node.

[0133] U. The method of any one of paragraphs A-F, wherein the method further includes determining environmental data based at least in part on sensor data captured by one or more sensors of the vehicle, where determining the environmental data includes accepting an indication of a mode of operation of the vehicle, determining a first prediction of at least one of a position, heading, or velocity of a dynamic object in the environment based at least in part on the mode of operation of the vehicle and the sensor data, and determining a first heat map based at least in part on the first prediction.

[0134] conclusion Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

[0135] The components described herein represent instructions that may be stored on any type of computer-readable medium and that may be implemented in software and / or hardware. All of the methods and processes described above may be embodied in and fully automated through software code components and / or computer-executable instructions executed by one or more computers or processors, hardware, or any combination thereof. Alternatively, some or all of the methods may alternatively be embodied in dedicated computer hardware.

[0136] At least some of the processes discussed herein are illustrated as logical flow graphs, each operation of which represents a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more non-transitory computer-readable storage media that, when executed by one or more processors, cause a computer or autonomous vehicle to perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0137] Unless otherwise stated, conditional language such as "may," "could," "may," or "might," among others, is understood in context to suggest that some examples include certain features, elements, and / or steps, but not other examples. Thus, such conditional language is not intended to generally imply that certain features, elements, and / or steps are in any way required in one or more examples, or that one or more examples necessarily include logic for determining, with or without user input or prompting, whether certain features, elements, and / or steps are included or should be performed in any particular example.

[0138] Conjunctive language such as the phrase "at least one of X, Y, or Z," unless otherwise stated, should be understood to present that the item, term, etc. may be either X, Y, or Z, or any combination thereof, including multiples of each element. Unless expressly described as singular, "a" means singular and plural.

[0139] Any routine description, element, or block in the flow diagrams described herein and / or depicted in the accompanying figures should be understood to potentially represent a module, segment, or portion of code that includes one or more computer executable instructions for implementing a particular logical function or element within the routine. Alternative implementations are included within the scope of the examples described herein, where elements or functions may be omitted or performed out of order from that shown or discussed, including substantially synchronously, in reverse order, with additional operations, or omitting operations, depending on the functions involved as would be understood by those skilled in the art. It should be noted that terms may substantially indicate ranges. For example, substantially simultaneously may indicate that two activities occur within a time range of each other, substantially the same dimensions may indicate that two elements have dimensions within a range of each other, and so forth.

[0140] It should be understood that many variations and modifications may be made to the examples described above and that the elements thereof are among the other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the accompanying claims.

Claims

1. 1. A system comprising: one or more processors; When executed by the one or more processors, the system comprises: receiving a predicted position of a dynamic object in an environment and a probability that said dynamic object will occupy said predicted position at a future time; determining a path for controlling a vehicle based at least in part on data of the environment; determining a cost map based at least in part on data of the environment, the cost map indicating predicted positions of dynamic objects within the environment and probabilities that the dynamic objects will occupy the predicted positions at future times; determining a first node based at least in part on a current position of the vehicle within the environment; determining a first subset of nodes based at least in part on the pruning technique; determining a cost associated with the first subset of nodes, the cost determining including determining a cost associated with the nodes based at least in part on the cost map and an affinity of the nodes in the first subset of nodes to one or more other nodes; determining two or more nodes of the subset having a lowest cost among the costs associated with the first subset of nodes based at least in part on the costs; determining a second node based at least in part on the two or more nodes; determining the path based at least in part on one or more of a second cost associated with the second node and the costs associated with a subset of the first nodes; determining a route, a memory storing processor-executable instructions for performing operations including A system comprising:

2. Determining a node among the second nodes includes: determining a future position, speed, and heading of the vehicle based at least in part on projecting a position, speed, and heading associated with one of the two or more nodes according to a control policy; Based at least in part on Each of the second nodes is associated with a different control policy. The system of claim 1 .

3. The path is a coarse path that defines at least one of a position, a heading, or a velocity of the vehicle over a number of first time intervals, and the operation includes: refining the coarse path based at least in part on determining a curvature of the path in addition to at least one of the position, the heading, or the velocity over a number of second time intervals, the second time intervals being shorter than the first time interval. The system of claim 1 or 2, further comprising:

4. the coarse path is determined by a first type processing unit; the smooth path is determined by a second type of processing unit different from the first type of processing unit. The system of claim 3.

5. the future time is a first future time, the probability is a first probability, and the data of the environment further includes a second map indicating a second predicted position of the dynamic object and a second probability that the dynamic object will occupy the second predicted position at a second future time that is later than the first future time; the costs associated with the first subset of nodes are based at least in part on the first probability; the second cost is based at least in part on the second probability. A system according to any one of claims 1 to 4.

6. Determining the cost map includes determining a cost associated with at least one of a position, a heading, a velocity, or a curvature in a parameter space associated with the environment, wherein determining the cost includes: determining a visit cost based at least in part on a deviation of at least one of the position, heading, speed, or curvature of the vehicle or target lane from a current position, heading, speed, or curvature; or determining the proximity of the position to a static object, the proximity of the position to the predicted position, or the probability; or Determining deviations of said position, said heading, said speed, or said curvature from a route or path reference. The system of claim 1 , wherein the system is based at least in part on

7. Determining the costs associated with the nodes of the first subset of nodes includes: an affinity of the first node to a second node of the subset of the first nodes; a difference in the lateral position of the first node compared to the lateral positions of one or more other nodes in the subset of first nodes; or a distance along a route associated with the first node The system of claim 1 , further comprising determining a weight based at least in part on at least one of:

8. The pruning technique includes: determining a hash associated with the first node; determining that the one or more nodes of the first node are associated with a redundant hash; discarding the one or more of the first nodes based at least in part on determining that the one or more of the first nodes are associated with redundant hashes; and 8. The system according to claim 1 , comprising:

9. When executed by one or more processors, the one or more processors are receiving a predicted position of a dynamic object in an environment and a probability that said dynamic object will occupy said predicted position at a future time; determining a path for controlling a vehicle based at least in part on data of the environment; determining a cost map based at least in part on data of the environment, the cost map indicating predicted positions of dynamic objects within the environment and probabilities that the dynamic objects will occupy the predicted positions at future times; determining a first node based at least in part on a current position of the vehicle within the environment; determining a first subset of nodes based at least in part on the pruning technique; determining a cost associated with the first subset of nodes, the cost determining including determining a cost associated with the nodes based at least in part on the cost map and an affinity of the nodes in the first subset of nodes to one or more other nodes; determining two or more nodes of the subset having a lowest cost among the costs associated with the first subset of nodes based at least in part on the costs; determining a second node based at least in part on the two or more nodes; determining the path based at least in part on one or more of a second cost associated with the second node and the costs associated with a subset of the first nodes; determining a route, A non-transitory computer-readable medium storing processor-executable instructions for causing a processor to perform operations including:

10. Determining a node among the second nodes includes: determining a future position, speed, and heading of the vehicle based at least in part on projecting a position, speed, and heading associated with one of the two or more nodes according to a control policy; Based at least in part on Nodes of the second nodes are associated with different control policies.

10. The non-transitory computer readable medium of claim 9.

11. The pruning technique includes: determining a set of hashes associated with the first node; determining a node among the first nodes associated with a redundant hash as a redundant node; Including, the subset of nodes excludes the redundant node; 11. The non-transitory computer readable medium of claim 9 or 10.

12. The path is a coarse path that defines at least one of a position, a heading, or a velocity of the vehicle over a number of first time intervals, and the operation is refining the coarse path based at least in part on determining a curvature of the path in addition to at least one of the position, the heading, or the velocity over a number of second time intervals, the second time intervals being shorter than the first time interval.

10. The non-transitory computer readable medium of claim 9, further comprising:

13. the future time is a first future time, the probability is a first probability, and the data of the environment further includes a second map indicating a second predicted position of the dynamic object and a second probability that the dynamic object will occupy the second predicted position at a second future time that is later than the first future time; the costs associated with the first subset of nodes are based at least in part on the first probability; the second cost is based at least in part on the second probability.

13. The non-transitory computer readable medium of claim 9.

14. Determining the cost map includes determining a cost associated with at least one of a position, a heading, a velocity, or a curvature in a parameter space associated with the environment, wherein determining the cost includes: determining a visit cost based at least in part on a deviation of at least one of a position, heading, speed, or curvature of the vehicle or target lane from a current position, heading, speed, or curvature of the vehicle or target lane; determining the proximity of the position to a static object, the proximity of the position to the predicted position, or the probability; or Determining deviations of said position, said heading, said speed, or said curvature from a route or path reference.

14. The non-transitory computer readable medium of claim 9, wherein the non-transitory computer readable medium is based at least in part on

15. Determining the costs associated with the nodes of the first subset of nodes includes: an affinity of the first node to a second node of the subset of the first nodes; a difference in the lateral position of the first node compared to the lateral positions of one or more other nodes in the subset of first nodes; or determining a weight based at least in part on at least one of a distance along a route associated with the first node; and determining a first heat map based at least in part on the first prediction; 15. The non-transitory computer readable medium of claim 9, further comprising:

Images