In-path obstacle detection and avoidance system

Abstract

The vehicle may include various sensors for detecting objects in the environment. In some cases, the object may be within the planned path of the vehicle's travel. In these cases, departing from the planned path may be dangerous for the occupants, so the vehicle may determine operating parameters associated with passing through the object while maintaining a position within the planned path, if possible, based on the object's dimensions, the vehicle's dimensions, and semantic information about the object.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Patent Application No. 16 / 722,931, entitled “IN-PATH OBSTACLE DETECTION AND AVOIDANCE SYSTEM,” filed December 20, 2019, which is incorporated by reference in its entirety. [Background technology]

[0002] An autonomous vehicle can use sensor data to capture data of an environment. To effectively navigate an environment, an autonomous vehicle uses the sensor data to detect objects in the environment and avoid collisions. To associate the sensor data with objects, segmentation techniques can be used. However, segmentation of three-dimensional data presents unique challenges that can result in an erroneous determination that an object is obstructing the path of the autonomous vehicle, causing a potentially unsafe response by the autonomous vehicle. Summary of the Invention

[0003] The detailed description will now be made with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears, and the use of the same reference number in different figures indicates similar or identical components or features. [Brief description of the drawings]

[0004] [Figure 1] 1 is a pictorial diagram of a situation in which a vehicle encounters an object that partially obstructs a lane. [Diagram 2] 2 is an exemplary pictorial diagram illustrating passing distance zones associated with the vehicle of FIG. 1. [Diagram 3] 2 is an exemplary pictorial diagram illustrating the vehicle of FIG. 1 detecting and passing an object that partially obstructs a lane. [Figure 4]2 is an exemplary pictorial diagram illustrating the vehicle of FIG. 1 detecting and passing an object that partially obstructs a lane. [Diagram 5] 2 is an exemplary pictorial diagram illustrating the vehicle of FIG. 1 detecting and passing an object that partially obstructs a lane. [Figure 6] 2 is an exemplary pictorial diagram illustrating the vehicle of FIG. 1 detecting and passing over an object on a ground surface. [Figure 7] FIG. 2 illustrates an exemplary process for detecting and navigating an object that partially obstructs a vehicle's path. [Figure 8] FIG. 2 illustrates another exemplary process for detecting and navigating an object that partially obstructs a vehicle's path. [Figure 9] FIG. 2 illustrates another exemplary process for detecting and navigating an object that partially obstructs a vehicle's path. [Figure 10] FIG. 1 illustrates an example process for detecting objects and determining surfaces associated with a planned path of a vehicle. [Figure 11] FIG. 1 is a block diagram of an example system for implementing the techniques described herein, in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] This disclosure describes methods, apparatus, and systems for detecting objects (or portions thereof) near or within the path of an autonomous vehicle and determining an appropriate proximity of the vehicle with respect to the detected objects. For example, the systems discussed herein may determine an appropriate passing distance to an object on the roadway, an object within the vehicle's planned path, an object near or adjacent to the planned path, and / or an object overhanging the planned path based on the vehicle's speed, the type or class of object (e.g., pedestrian, foliage, paper, rock, etc.), the size and / or location of the object, and the size along the surface of the vehicle.

[0006] For example, some incidents that drivers encounter sporadically include tree branches and foliage that encroach on and / or overhang the roadway, as well as small objects along the roadway surface, such as paper / trash, large rocks, or other debris. In these instances, a human driver may be able to visually assess whether the object will contact any part of the vehicle, and if so, the risk of damage to the vehicle from contacting the object. In this manner, the human driver may determine an appropriate passing distance to either leave or not leave an occupied lane. For example, the driver may avoid contact with a rock by positioning the vehicle to allow the rock to pass safely under the chassis and between the wheels. However, conventional autonomous vehicle sensor systems and perception pipelines may detect a rock in the roadway, identify the rock as an obstacle to be avoided, and have the vehicle drive around the rock to avoid the obstacle entirely. In some cases, driving around a rock may actually increase the risk of damage and / or a collision because the vehicle may move into another lane (e.g., oncoming traffic) when driving around the rock could otherwise be accomplished within the lane. The systems discussed herein are configured to detect objects (here, rocks), evaluate the objects, and determine that the objects can be avoided without leaving the vehicle's current lane or path, thereby improving the overall safety of the autonomous vehicle.

[0007] As another illustrative example, foliage often overhangs the road, especially in poorly maintained or little-used streets. In some cases, a human driver may drive a vehicle into the foliage or allow the foliage to "nudge" the vehicle without damaging the vehicle. This type of maneuver is especially common in narrow, one-way road conditions where avoidance may not even be possible. In this example, a conventional autonomous vehicle sensor system and perception pipeline may detect foliage overhanging the road, identify the foliage as an obstacle to be avoided, and prevent the vehicle from proceeding. In this situation, a vehicle using a conventional system may become stuck if the vehicle is prevented from backing up (e.g., if another vehicle is approaching behind the autonomous system).

[0008] In one situation, the vehicle discussed herein may determine that the foliage is at a distance from / apart from the ground surface (e.g., clearance or height) that the vehicle can pass under without contact. In a second situation, the vehicle discussed herein may determine that the foliage belongs to a type or class of object that the vehicle can contact without being damaged (e.g., leaves and small branches bend to allow the vehicle to pass). In the second situation, the vehicle may select operating parameters (such as less than 5 miles per hour, less than 10 miles per hour, less than 15 miles per hour, etc.) that allow the vehicle to proceed through the foliage despite the contact. In a third situation, the vehicle may determine a portion of the vehicle that may contact the foliage. For example, in the third situation, the vehicle may position itself such that the roof of the vehicle may contact the foliage, but the sensitive sensor located on the top of the vehicle will not.

[0009] As discussed above, the vehicles discussed herein can make informed decisions related to object avoidance based on the semantic class of the object (e.g., wall or structure, foliage, rock, plant, vehicle, vehicle door, debris or clutter, bike, traffic light or cone, etc.), characteristics of the class, the location of the object (e.g., object height, object distance from ground level, object shape, etc.), the shape of the vehicle, and the location and vulnerability of components relative to the vehicle. This type of object avoidance may enable the vehicle to operate in additional situations or environments and increase the overall safety of the vehicle because the vehicle will more frequently maintain its position within an appropriate traffic lane.

[0010] FIG. 1 illustrates a pictorial representation of a situation 100 in which a vehicle 102 encounters an in-path object or in-path or in-lane traversable object 104. In the present example, the vehicle 102 is driving in a lane 106 of a road, and the vehicle captures data representing the object 104 that overhangs the lane 106. In this example, the vehicle 102 may determine that the object 104 is not touching the ground surface, generally indicated by 108. For example, the vehicle 102 may segment and classify the captured data representing the object 104. The vehicle 102 may also generate a bounding box associated with the region of space occupied by the object 104. The vehicle may then determine the maximum distance of the object from the ground surface (e.g., the distance between the highest point of the bounding box and the ground) and the minimum distance of the object 104 from the ground surface (e.g., the distance between the lowest point of the object and the ground). The vehicle 102 may then determine that the object is not in contact with the ground when the minimum distance is approximately greater than zero.

[0011] Because the object 104 is not in contact with the ground surface 108, the vehicle 102 may determine a minimum distance 110 of the object 104 from the ground surface 108 (e.g., the bottom of the object). The vehicle 102 may also store a number of passing distance regions or thresholds (e.g., margin distances from the vehicle). For example, as discussed in more detail below with respect to FIG. 2, a first passing distance region may define a first distance between the vehicle 102 and an object, such as the object 104, at which the vehicle 102 may pass the object while traveling under normal operating conditions. Thus, if the distance 110 is greater than the first distance associated with the first passing distance region, the vehicle 102 may pass without changing the operating conditions.

[0012] The vehicle 102 may also include a second passing distance region that defines a second distance between the vehicle 102 and the object 104 that the vehicle 102 may pass through after adjusting the operating parameters, such as reducing the speed below the first speed threshold. Again, if the distance 110 is greater than the second distance associated with the second passing distance region, the vehicle 102 may pass at the first reduced speed (or under other pre-determined operating parameters). The vehicle 102 may also include a third passing distance zone that defines a third distance between the vehicle 102 and the object that the vehicle 102 may pass through after adjusting the operating parameters, such as reducing the speed below the second threshold. Again, if the distance 110 is greater than the third distance associated with the third passing distance zone, the vehicle 102 may pass at the second reduced speed. It should be understood that a vehicle may include any number of consecutive passing distance regions or thresholds, and that each consecutive passing distance region may be a reduced distance from the vehicle 102 and may include a corresponding reduced speed.

[0013] In some cases, the distance 110 may be less than the third pass distance region. In this example, the third pass distance region may be a final pass distance region such that the third pass distance threshold may be a minimum distance to pass the object 104 without contact. The vehicle 102 may also be configured to determine a class of the object 104. For example, in the illustrated example, the vehicle may determine that the object belongs to a class that includes foliage, and the foliage class has a low risk of damage upon contact. Alternatively, if the class of object was a parked automobile, the vehicle 102 may classify the parked automobile into a class of object that has a high risk of damage upon contact. In this example, the vehicle 102 may determine that the vehicle 102 may contact the foliage 104 when the vehicle 102 travels under a tree. In this case, the leaf 104 belongs to a class of objects that have a low risk of damage (or a risk of damage below a risk threshold), and the contact may be less than the contact threshold, so the vehicle 102 may still decide to proceed. In some cases, the vehicle 102 may compare the risk of contact to the risk of following the outside of the lane 106. For example, moving into oncoming traffic on a busy road may be much more likely to cause damage to the vehicle 102 than contacting the leaf 104.

[0014] In some cases, a vehicle may include subclasses, each including a different risk of contact. For example, a branch class may include a subclass for branches having a first risk of damage, a subclass for branches smaller than a half-inch having a second risk of damage, and a third class for branches larger than a half-inch. Each subclass may also include an acceptable avoidance distance and / or a speed of contact (if contact is deemed necessary). It should be understood that the number of subclasses may vary based on the class, and that the speed at which the vehicle 102 contacts the branch 104 may be determined based on the class / subclass, the amount of objects 104 likely to contact the vehicle 102, and / or the location at which the objects 104 are likely to contact the vehicle 102.

[0015] FIG. 2 shows an example diagram 200 illustrating passing distance regions 202, 204, 206 associated with the vehicle 102 of FIG. 1. As discussed above, the vehicle 102 may store multiple passing distance regions, such as passing distance regions 202-206. Each of the passing distance regions 202-206 may include a boundary area or distance that may indicate a set of operating parameters associated with the vehicle 102 when the distance of an object to the vehicle is within the region 202-206. In the current example, the vehicle 102 may operate using parameters associated with the closest or nearest passing distance region 202-206 that contains the object. Thus, if the object is farther from the vehicle 102 than the first passing distance region 202, the vehicle 102 may operate using parameters (e.g., normal operating parameters) associated with the first distance region 202. If the object is within the first passing distance region 202 but farther than the second passing distance region 204, the vehicle 102 may operate under a second set of parameters or conditions. Similarly, if the object is within the second passing distance region 204 but farther than the third passing distance region 206, the vehicle 102 may operate under a third set of parameters or conditions. Finally, if the object is within the third passing distance region 206, the vehicle 102 may operate under a fourth set of parameters or conditions (which may include stopping and / or remote operator control in some cases).

[0016] In the illustrated example, the first passing distance region 202 and the second passing distance region 204 are represented by rectangular bounding boxes around the vehicle 102. The rectangular bounding boxes may be determined based on corresponding distances from outer extremities of the vehicle 102 (e.g., the outermost positions of the vehicle 102 on each side—front, rear, left, right, top, bottom). As such, the first passing distance region 202 and the second passing distance region 204 do not necessarily take into account changes in the shape or dimensions of the vehicle 102 itself. In this manner, the vehicle 102 may make action decisions more quickly and using fewer computational resources when an object is farther away than the first passing distance region 202 or the second passing distance region 204.

[0017] In the current example, a third passing distance region 206, defining a third distance between the vehicle 102, may be defined by the distance from the outside of the vehicle at each point around the outside. Thus, the vehicle 102 may be able to pass an object that protrudes into the road from the side but avoids the rearview mirror, or that extends above the surface of the road but is positioned to avoid the wheels and / or frame.

[0018] 3 shows an example diagram 300 illustrating the vehicle 102 of FIG. 1 detecting and passing an object 302 that is partially obstructing the vehicle's path. In the current example, the vehicle 102 may detect the object 302 along the surface of the road. For example, the vehicle may segment and classify captured data representing the object 302. The vehicle may also generate a bounding box 304 associated with the region of space occupied by the object 302. The vehicle 102 may then determine a maximum distance 306 of the object 302 from the ground surface (e.g., the distance between the highest point of the bounding box and the ground). The vehicle 102 may also determine that the object 302 is avoidable based on the third passing distance region 206 exceeding the maximum distance 306 (e.g., the object 302 does not enter the third passing distance region) at the position of the object 302 relative to the vehicle 102 while the object 302 is within the first passing distance region 202 and the second passing distance region 204. For example, as illustrated, the height of the object 302 allows the vehicle 102 to pass over the object 302 without contacting it if the wheels avoid the object 302. In this example, the vehicle 102 may also determine the semantic class of the object 302 to be a rock or otherwise inanimate object. In another example, if the object 302 is, for example, a small animal, the vehicle 102 may stop motion because the animal may move. However, if the object 302 is inanimate and can safely pass under the chassis of the vehicle 102, the vehicle 102 may proceed to drive over the rock 302 at a reduced speed (or operating under third driving parameters). In some examples, the bounding box 304, the position of the object 302, and the height of the object 306 may be provided to a planner system of the vehicle 102, which may modify the vehicle's planned path and / or operating parameters.

[0019] FIG. 4 shows an example diagram 400 illustrating the vehicle 102 of FIG. 1 detecting and passing an object 402 that partially obstructs a lane. In the current example, the vehicle 102 may detect the object 402 along the surface of the road. In this case, the vehicle 102 may still determine the maximum distance (e.g., height) of the object 402 from the ground. However, unlike the example of FIG. 3, the vehicle 102 may determine that the object 402 is high (e.g., has a height) that would contact the bottom of the vehicle 102 if the vehicle attempted to pass over it. In this case, the vehicle 102 may determine that the vehicle 102 cannot avoid the object 402 while remaining in the vehicle's path according to the first passing distance threshold 202 and the second passing distance threshold 204. However, the vehicle 102 may determine, based on the dimensions of the third passing distance threshold 206, the dimensions of the object 402, and the width of the path, that the object 402 may not be able to pass under the frame or between the wheels, but may pass under the rearview mirror or other protruding sensor 404 extending from the exterior of the vehicle while the vehicle maintains its position within the vehicle's planned path. Again, the vehicle 102 may initially determine the semantic class of the object 402 to be a rock before the object 402 enters the passing distance thresholds 202 and 204.

[0020] FIG. 5 shows an example diagram 500 illustrating the vehicle 102 of FIG. 1 detecting and passing an object 502 that partially obstructs the path. In the current example, the vehicle 102 may determine that the object 502 extends into the path a distance that would cause the object 502 to enter the first passing distance threshold 202, the second passing distance threshold 204, and the third passing distance threshold 206. In this example, the vehicle 102 may also determine that the semantic class of the object is foliage (e.g., bushes). For example, the class or subclass may be leaves and twigs, which have a low risk of damaging the vehicle 102. The vehicle 102 may also determine that there is a large amount of oncoming traffic and that crossing the lane line is not advisable. For example, the vehicle or planning system may assign a "cost" to different trajectories / actions, including contacting the object 502 and driving into oncoming traffic. Based on the cost, the vehicle 102 may then decide to contact the object 502. In some cases, deviating from the most direct route may have a first cost, braking or swerving may have a second cost (based on passenger discomfort, etc.), etc. In this example, the vehicle 102 may generate multiple candidate trajectories, calculate a cost for each, and then select the trajectory with the lowest cost as the planned path for the vehicle 102.

[0021] In this situation, the vehicle 102 may determine, based at least in part on the semantic classification and the determined area 504 of the contact (e.g., rearview mirror or protruding sensor 404), that the vehicle 102 may proceed despite the contact with the foliage 502. In this example, the vehicle 102 may operate within a limited set of parameters, such as staying below a speed threshold.

[0022] FIG. 6 shows an example diagram 600 illustrating the vehicle 102 of FIG. 1 detecting and passing an object 602 on the ground. In the current example, the vehicle 102 may detect the object 602 along the surface of the road or below the ground surface. In this case, the vehicle 102 may determine the maximum distance (e.g., depth) of the object 602 from the ground. The vehicle 102 may also determine that the depth is greater than a depth threshold above which the vehicle 102 may drive its wheels. The vehicle 102 may also determine that the object 602 is avoidable (e.g., the object 602 may pass between the wheels, similar to the rock 302 of FIG. 3) based on the third passing distance threshold 206. In this example, the vehicle 102 may also determine the semantic class of the object 602 to be a hole or obstacle in the ground surface itself. However, because the object 602 is inanimate and can safely pass between the wheels of the vehicle 102, the vehicle 102 may proceed to drive over the object 602 at a reduced speed (or operating under third driving parameters).

[0023] 4-6, the vehicle 102 may determine that an object may be passed within the lane based on the third passing distance zone 206, as discussed above, but it should be understood that in each of these examples, the vehicle 102 may adjust its position while remaining within the lane to appropriately avoid the object as shown.

[0024] 7-8 are flow diagrams illustrating an example process associated with determining vehicle operating parameters for objects in the vehicle's path or lane of FIGS. 1-6. The process is illustrated as a collection of blocks in a logical flow diagram that represent sequences of operations, some or all of which may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, perform the operations noted. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types.

[0025] The order in which the operations are described should not be construed as a limitation. Any number of the described blocks can be combined in any order and / or in parallel to implement a process or alternative processes. Not all of the blocks need to be executed. For purposes of discussion, the processes herein are described with reference to the frameworks, architectures, and environments described in the examples herein, but the processes may be implemented in a wide variety of other frameworks, architectures, or environments.

[0026] 7 illustrates an example process 700 for detecting and passing an object that is partially obstructing the path of a vehicle. The vehicle may determine a path within a lane of operation to avoid an object in the path based on the semantic class of the object, characteristics of the class, the object's location relative to the path and / or ground surface, the object's shape or size, and the vehicle's shape or size. As discussed above, this type of object avoidance may enable the vehicle to operate in additional situations or environments and increase the overall safety of the vehicle because the vehicle more frequently maintains a position within an appropriate traffic lane.

[0027] At 702, the vehicle may capture sensor data including a set of points indicative of an obstacle to the vehicle's path. For example, the vehicle may capture sensor data including LIDAR (Light Detection and Ranging) data, RADAR data, SONAR data, image data (red-green-blue, infrared, intensity, depth, etc.), audio data, infrared data, position data, depth data, or any combination thereof.

[0028] At 704, the vehicle may associate the set of points with an object. For example, the vehicle may process the captured sensor data using various machine learning models and / or neural networks to determine the presence of an object. For example, the object may be detected by the processes and techniques discussed in U.S. Patent Application No. 15 / 963,833, the entirety of which is incorporated herein by reference. In one specific example, the vehicle may perform segmentation on the vehicle and / or identify a bounding box region around the object to define object edges.

[0029] At 706, the vehicle may determine that the object is within a planned path of the vehicle. For example, the vehicle may determine that the object is located along a path within which the vehicle is currently operating based at least in part on a set of points or the object's location relative to a stored scene or map of the physical environment.

[0030] At 708, the vehicle may determine one or more distances of the object from a surface associated with the planned path. For example, the vehicle may identify the ground surface and determine the maximum distance of the object from the ground surface (e.g., the top of the object) and the minimum distance of the object from the ground surface (e.g., the bottom of the object). In some cases, the vehicle may determine a ground horizon associated with the ground surface, as discussed in U.S. Patent Application No. 16 / 698,055 or U.S. Patent Application No. 15 / 622,905, both of which are incorporated by reference in their entireties.

[0031] At 710, the vehicle may determine whether the object intersects with a first passage area associated with the vehicle based in part on the distance. In some cases, the first passage area may be a first predetermined distance from a particular point along each side (e.g., left, right, top, bottom) of the vehicle. In some cases, the first predetermined area may be uniform on each side of the vehicle, while in other cases, the first predetermined distance may vary. For example, the first passage area may be farther from the top of the vehicle than from along the sides of the vehicle. In one example, the vehicle may determine whether the vehicle can drive under the object by determining whether a minimum distance is greater than the first passage area along the top surface of the vehicle. Similarly, in another example, the vehicle may determine whether the vehicle can drive over the object by determining whether a maximum distance is greater than the first passage area along the bottom surface of the vehicle. If the object does not intersect with the first traversal region, process 700 may proceed to 712, where the vehicle may proceed to pass the object using a first set of operating parameters (e.g., normal operating parameters). In some examples, a planning system of the vehicle may receive the object data and determine operating parameters and / or update a planned path for the vehicle based at least in part on the object. However, if the object does intersect with the first traversal region, process 700 may proceed to 714.

[0032] At 714, the vehicle may determine, based in part on the distance, whether the object intersects with a second traverse area associated with the vehicle. Similar to the first traverse area, the second traverse area may be a second predetermined distance from a particular point along each side (e.g., left, right, top, bottom) of the vehicle.

[0033] In the current example, the second predetermined distance may be less than the first predetermined distance associated with the first passage area (e.g., the acceptable distance for passing according to the second passage area is closer than the acceptable distance for passing according to the first passage distance area). In one example, the vehicle may determine whether the vehicle can drive under the object by determining whether the minimum distance is greater than the second passage area along the top surface of the vehicle. Similarly, in another example, the vehicle may determine whether the vehicle can drive over the object by determining whether the maximum distance is greater than the second passage area along the bottom surface of the vehicle. If the object does not intersect with the second passage area, the process 700 may proceed to 716, where the vehicle may proceed to pass the object using a second set of operating parameters. The second set of operating parameters may include, for example, a reduced speed or modify the planned path. However, if the object does intersect with the second passage area, the process 700 may proceed to 718.

[0034] At 718, the vehicle may determine whether the object intersects with a third passing area associated with the vehicle based in part on the distance. Unlike the first passing area or the second passing area, the third passing area may be a third pre-determined distance from a plurality of points along each side (e.g., left, right, top, bottom) of the vehicle such that the third passing area reflects the shape of the vehicle itself. In the current example, the third pre-determined distance may be less than the first pre-determined distance and the second pre-determined distance (e.g., the acceptable distance for passing according to the third passing area is closer than the acceptable distance for passing according to the first passing distance area or the second passing area). In one example, the vehicle may determine whether the vehicle can drive under the object by determining whether the minimum distance is greater than the third passing area along the top surface of the vehicle. In another example, the vehicle may determine whether the vehicle can drive over the object by determining whether the maximum distance is less than the third passing area along the bottom surface of the vehicle. In some cases, the vehicle may determine a pass area, such as a location relative to the vehicle, to drive over the object. For example, the vehicle may determine that the object fits under a portion of the vehicle (because the third pass distance threshold corresponds to the shape of the vehicle) and align the portion with the object. If the object does not cross the third pass distance threshold, process 700 may proceed to 716, where the vehicle may proceed to pass the object using a third set of operating parameters. The third set of operating parameters may include, for example, a further reduced speed than the second set of operating parameters. However, if the object does cross the second pass distance threshold, process 700 may proceed to 722.

[0035] At 722, the vehicle may determine a semantic class of the object. For example, the vehicle may perform classification on the object using various machine learning models and / or neural networks. In some cases, the vehicle may also assign a subclass to the object. In some examples, the vehicle may perform classification on the set of points using various machine learning models and / or neural networks. For example, one or more neural networks may generate any number of learned inferences or heads, such as a semantic class head, a direction to object instance center head, and other learned output heads. In some cases, the neural network may be an end-to-end trained network architecture. In one example, the semantic class estimation may include segmenting and / or classifying the extracted deep convolutional features into semantic data (e.g., stiffness, hardness, safety risk, risk of position change, class or type, potential direction of travel, etc.). In other cases, the semantic class of an object may be determined as discussed in US patent application Ser. No. 15 / 820,245, which is incorporated herein by reference in its entirety.

[0036] At 724, the vehicle may determine whether the semantic class criteria are met or exceeded. For example, the vehicle may store various class characteristics associated with each class and may determine based on the class characteristics whether the vehicle may proceed even if the object is within the third passing distance threshold. For example, if the object belongs to an inanimate or non-inanimate class, is likely to damage the vehicle (e.g., leaves, paper, trash, and small twigs may not be likely, but a bicycle or scooter may be), and the amount of contact between the vehicle and the object is less than the contact threshold (e.g., less than 1.0 inches, less than 2.0 inches, or less than 5.0 inches of contact), the process 700 may proceed to 726 and the vehicle may proceed to pass the object using a fourth set of operating parameters. In some cases, the fourth set of operating parameters may be more restrictive, such as a much reduced speed, than the third set of operating parameters. If the semantic class does not meet or exceed the progression criteria (e.g., the object is animate, too hard, or rigid), process 700 moves to 728 and the vehicle issues an alert to an operator (e.g., a remote vehicle operator).

[0037] At 728, the vehicle may issue alerts to a remote operator, who may be able to control or otherwise guide the vehicle using image data generated by the vehicle and a remote control or steering device. In this manner, the remote operator can make informed decisions similar to those made by a human driver given the same type of information related to objects in the planned route.

[0038] 8 illustrates another example process 800 for detecting and passing an object that is partially obstructing the vehicle's path. As discussed above, an autonomous vehicle may determine a path (such as within a lane of operation) to avoid an object within a lane based on the semantic class of the object, stored or known information associated with the class, the object's location with respect to one or more planes associated with the vehicle's path and / or the physical environment, the object's dimensions, and the vehicle's dimensions. As discussed above, this type of object avoidance may enable the vehicle to operate in additional situations or environments while also increasing the overall safety of the vehicle because the vehicle stays within the appropriate traffic lane more often.

[0039] At 802, the vehicle may capture sensor data, including a set of points, representing an object obstructing the vehicle's planned path. For example, the vehicle may include various sensor systems, such as sensor systems configured to capture LIDAR data, RADAR data, SONAR data, image data, audio data, position data, depth data, or any combination thereof.

[0040] At 804, the vehicle may determine that the object is within the vehicle's planned path. For example, the vehicle may associate the set of points with a map or scene of the physical environment. The vehicle may determine that at least a portion of the set of points is within the vehicle's planned path based on the location of the set of points within the map or scene. In some cases, the vehicle may associate, segment, and classify the set of points to identify the object and locate the object within a stored scene representing the environment. In some examples, the vehicle may generate a top-down view or map of the physical environment and the planned path, such as those shown in FIGS. 2-6. In other examples, the vehicle may also determine a bounding box or region associated with the object and determine whether the bounding box intersects with a corridor (e.g., a three-dimensional representation of the planned path). The vehicle may then use the location of the object (or bounding box) and / or the geometry of the object to determine that the object is within the vehicle's path.

[0041] In another example, the vehicle may determine that at least a portion of a set of points (or bounding box) is within a corridor or planned path of the vehicle based on intersections between the points or bounding box and the corridor. For example, the vehicle may identify the ground surface and left and right sides of the planned path (e.g., vertical planes that bound the corridor) and determine that the object is located between the left and right sides of the corridor. As discussed herein, the left and right sides of the corridor are variable based on the vehicle's speed, angle of travel, etc.

[0042] At 806, the vehicle may determine a surface associated with the physical environment or corridor. For example, the surface may be a horizontal ground plane associated with the Earth's surface, or a left or right side of the corridor.

[0043] At 808, the vehicle may determine a maximum distance of the object from a surface (e.g., the ground plane) and a minimum distance associated with the object from the surface. For example, the vehicle may determine a height of the object based on the maximum distance associated with the object from the ground plane. For example, if the minimum distance is approximately zero, the object represented by the object is located along the ground plane and the maximum distance is equal to the height of the object.

[0044] Similarly, the vehicle may determine an offset from the ground plane (e.g., a clearance below an object represented by a set of points) based on a minimum distance between the ground plane and the object. For example, if the minimum distance is approximately greater than zero, the represented object is an overhanging object, and the minimum distance represents a clearance below the overhang.

[0045] In other examples, such as when the plane is a vertical side plane of the planned route corridor, the vehicle may use a maximum distance to determine a first incursion distance from the right side of the vehicle's corridor and a minimum distance to determine a second incursion distance from the other side of the corridor in a manner similar to that for the ground surface. In some specific examples, the vehicle may determine a maximum distance and a minimum distance from both of the vertical side planes of the corridor.

[0046] At 810, the vehicle may determine whether the minimum distance is greater than or equal to a top surface threshold associated with the vehicle (e.g., the bottom of the object is above the vehicle's top surface by more than a predefined distance). For example, if the surface is the ground, the vehicle may determine whether the vehicle can safely pass under the overhang. For example, as discussed above with respect to FIG. 7, the vehicle may compare various passing distance regions to the minimum distance to determine operational parameters that may be used when passing over an overhanging object. It should also be understood that the regions may vary based on the dimensions of the vehicle's top surface and / or may be a fixed distance from a point along the vehicle's top surface.

[0047] If the minimum distance is greater than or equal to the upper surface threshold (e.g., one or more of the pass-through distance regions discussed above), the process 800 may proceed to 812 and the vehicle may pass the object because the object is an overhang (e.g., not aligned with the road surface) and the vehicle has enough clearance to pass without contact. In some cases, the planning system may receive the clearance information and / or a first set of operation parameters and plan a trajectory associated with passing the object based on the first set of operation parameters and the position of the object within the planned path. In the current example, at 812, the vehicle (or vehicle planning system) may utilize a first set of operation parameters, such as normal operating parameters, when passing the object. In some examples, such as the example of FIG. 7, the first set of operation parameters may depend on the pass-through distance region or the closeness of the object to the vehicle. However, if the minimum distance is not greater than or equal to the upper surface threshold, the process 800 proceeds to 814.

[0048] At 814, the vehicle may determine whether the maximum distance is equal to or less than a bottom threshold associated with the vehicle (e.g., the top of the object is below the bottom of the vehicle by less than a predefined distance). For example, if the surface is the ground, the vehicle may determine whether the vehicle can safely pass over the object along the road surface. In some examples, as discussed above with respect to FIG. 7, the vehicle may compare various passing distance regions to the maximum distance to determine operational parameters that may be used when passing over the surface object. It should also be understood that the regions or thresholds may vary based on the dimensions of the bottom of the vehicle and / or may be fixed distances from a point along the top of the vehicle. For example, an object passing under a protruding sensor may have a larger bottom threshold than an object passing under the vehicle's chassis. In some cases, the vehicle may also determine the width of a surface object passing between the wheels to ensure that the object has enough room for the vehicle to pass over the object without impacting the wheels.

[0049] If the maximum distance is less than or equal to a bottom threshold (e.g., one or more of the pass-through distance regions discussed above), process 800 may proceed to 812, and the vehicle may pass the object because the object is a small surface object and the vehicle has enough room to pass without contacting it. However, if the maximum distance is not greater than or equal to the bottom threshold, process 800 may proceed to 816.

[0050] At 816, the vehicle may determine a semantic class of the object represented by the set of points. In some examples, the vehicle may perform classification on the set of object points using various machine learning models and / or neural networks. For example, one or more neural networks may generate any number of learned inferences or heads, such as a semantic class head, a direction head to the object instance center, and other learned output heads (such as target range, target orientation, target speed, object bounding box, etc.). In some cases, the neural network may be an end-to-end trained network architecture, such as using stochastic gradient descent. In some cases, the appropriate true output in the form of an image map may include a semantic pixel-by-pixel classification (e.g., leaves, rocks, vehicles, animals, trash, etc.) and a bounding box representation. In some examples, the semantic class estimation may include segmenting and / or classifying the extracted deep convolutional features into semantic data (e.g., stiffness, hardness, safety risk, risk of position change, class or type, potential direction of travel, etc.).

[0051] In some cases, semantic classification may be performed prior to or in parallel with determining the maximum and minimum distances from the plane. In some cases, such as when an object is classified as paper or plastic trash, the vehicle may ignore a set of points since the object does not pose a risk of damage upon contact and may be likely to move (e.g., be blown away by the wind), and thus computational resources may be conserved.

[0052] At 818, the vehicle may determine whether the semantic class criteria are met or exceeded. For example, the vehicle may include types of classes (e.g., paper, plastic, other soft debris, leaves, small branches, etc.) that are considered safe classes and pose a risk of damage below a threshold level. In other examples, the vehicle may compare class characteristics, such as stiffness, hardness, etc., to various criteria or thresholds to determine if the object is safe. In one particular example, the criteria may include risk of damage upon contact, or various scales of risk of damage upon contact, based on, for example, the speed of the vehicle. In the current example, if the semantic class criteria are met or exceeded, the process 800 proceeds to 820, where the vehicle passes through the object using a second set of operating parameters. However, if the semantic class criteria are not met or exceeded, the process 800 proceeds to 822, where the vehicle may issue a warning to the operator. For example, the vehicle may notify an operator within the vehicle and / or a remote operator via one or more networks. In this manner, the vehicle may be manually steered around the object, thereby ensuring the safety of the occupants.

[0053] 9 illustrates another example process 900 for detecting and passing an object that is partially obstructing the vehicle's path. As discussed above, an autonomous vehicle may determine a path within a lane of operation to avoid an object within the lane based on the semantic class of the object, characteristics of the class, the object's location relative to one or more planes associated with the lane and / or the physical environment, the object's dimensions, and the vehicle's dimensions. As discussed above, this type of object avoidance may enable the vehicle to operate in additional situations or environments while also increasing the overall safety of the vehicle because the vehicle stays within the appropriate traffic lane more often.

[0054] At 902, the vehicle may capture sensor data including a set of points indicative of an obstacle to a path of the vehicle. For example, the vehicle may include a variety of sensor systems, such as a sensor system configured to capture LIDAR data, RADAR data, SONAR data, image data, audio data, position data, depth data, or any combination thereof.

[0055] At 904, the vehicle may identify an object within the set of points and determine a semantic class of the object. For example, the vehicle may perform techniques such as object detection, filtering, classification, segmentation, feature extraction, sparse feature representation, pattern detection, white space detection, pixel correlation, feature mapping, etc. to identify an object within the set of points and locate the object with respect to a map or scene of the physical environment. In some cases, the object detection, classification, and segmentation may include machine learning models and / or neural networks. One or more neural networks may generate any number of learned inferences or heads.

[0056] At 906, the vehicle may determine that an object is within the vehicle's path. For example, the vehicle may determine that at least a portion of an object within an area of ​​the vehicle's lane occupies or is within the vehicle's path. In some cases, the vehicle may determine vertical side planes associated with the edges of the lane and a ground plane associated with the surface of the road. The vehicle may then determine that at least a portion of the object is within the space defined by the identified planes.

[0057] At 908, the vehicle may determine the maximum distance of the object from the ground surface and the minimum distance of the object from the ground surface. For example, if the minimum distance is approximately zero, the object is located along the ground surface and the maximum distance is equal to the height of the object. However, if the minimum distance is greater than zero, the object is offset from the ground surface and the object overhangs the roadway. For example, if the minimum distance of the object from the ground surface places the object above the vehicle (e.g., greater than a threshold distance from the top surface of the vehicle), the vehicle may safely pass under the overhanging object. Alternatively, if the maximum distance of the object from the ground surface places the object below the vehicle (e.g., less than a threshold distance from the bottom surface of the vehicle), the vehicle may safely pass over the ground object. In some cases, the vehicle may also determine an area of ​​the vehicle (e.g., wheels or chassis) that is in line with the object. The vehicle may then determine whether the object may pass under the object based on the intersection of the particular area of ​​the vehicle with the object.

[0058] At 910, the vehicle may determine the maximum distance of the object from a surface associated with a side of the vehicle's planned path (e.g., the right or left side of the corridor) and the minimum distance of the object from the side surface. In some cases, the vehicle may determine the maximum and minimum distance of the object from a vertical surface associated with both sides of the corridor. In this manner, the vehicle may determine the incursion distance of the object into the vehicle's planned path from either the right or left side of the corridor.

[0059] At 912, the vehicle may determine a path for the vehicle based at least in part on the maximum distance from ground level, the minimum distance from ground level, the maximum distance from a side surface, the minimum distance from a side surface, the vehicle's dimensions, and the semantic class of the object. For example, the vehicle may compare various distances to one or more thresholds (e.g., the passing distance thresholds discussed above with respect to FIGS. 1-7) to determine if the vehicle can safely pass the object. In some cases, the vehicle may also determine a risk of damage to the vehicle based at least in part on the semantic class of the object. For example, if the object is a newspaper lying in a lane, the risk of damage may be low and the vehicle may proceed. However, if the object is a child's bicycle, the risk of damage (e.g., to either the object, the bicycle, or the vehicle) may be high and the vehicle may not proceed.

[0060] 10 illustrates an example process for detecting objects and determining surfaces associated with a planned path of a vehicle. For example, a vehicle may include one or more lidar devices configured to capture a dataset, generate a voxel space, determine voxels associated with a ground surface, identify objects, and generate bounding boxes associated with the objects.

[0061] At 1002, a vehicle may capture lidar data. In some examples, the vehicle may receive multiple lidar points from multiple lidar sensors operating in conjunction with the autonomous vehicle's perception system. In some examples, data may be acquired from two or more lidar sensors or fused into a single lidar data set that represents the physical environment 1004.

[0062] At 1006, the vehicle may associate the lidar data with a voxel space. For example, while example 108 illustrates a voxel space that includes five voxels in each dimension (e.g., x, y, z), any number of voxels may be included within the voxel space. In some examples, the voxel space may correspond to a physical environment, such as an area around an origin or virtual origin of the lidar dataset. For example, the voxel space may represent an area that is 10 meters wide, 10 meters long, and 10 meters high. Further, each voxel in the voxel space (e.g., voxel 1010) may represent a physical area, such as 25 centimeters in each dimension. As may be understood in the context of this disclosure, the voxel space may represent any area of ​​the environment, and individual voxels may similarly represent any volume. In some examples, a voxel may be of uniform size throughout voxel space, but in some examples, the volume of a voxel may vary based on the location of the voxel relative to the origin of the data. For example, the density of the lidar data may decrease as the distance from the lidar sensor increases, so the size of a voxel in voxel space may increase in proportion to the distance from the voxel to the lidar sensor (or an origin representing multiple lidar sensors).

[0063] In some examples, the vehicle may map individual points of the point cloud to individual voxels. In some examples, operation 106 may include, for example, subtracting motion vectors associated with the lidar data to convert the lidar data to a stationary reference point, for example, if the lidar data is captured by a moving platform such as an autonomous vehicle. That is, in some examples, the lidar data may be associated with a voxel space that is fixed with respect to the overall map (as opposed to a voxel space that is fixed with respect to the moving vehicle, for example). In some examples, at 1006, the vehicle may discard or omit voxels that contain no data or that contain less than a threshold number of points to create a sparse voxel space. Additionally, in some examples, operation 106 may include aligning the vehicle's attitude (e.g., vehicle orientation) and associated lidar data to the voxel map, for example, to compensate for or adjust for any errors associated with the vehicle's position with respect to the voxel space.

[0064] At 1012, the vehicle may determine one or more voxels associated with the ground surface. In some examples, the ground surface may correspond to a surface drivable by the autonomous vehicle, as discussed above. As shown at 114, a single voxel, e.g., voxel 1010, is shown containing lidar data 1016, which may represent a statistical accumulation of data, including a number of data points, an average intensity, an average x value, an average y value, an average z value, and a covariance matrix based on the lidar data. In such examples, the lidar data 1016 is shown as a number of points for illustrative purposes, but each voxel 1010 may store only a statistical accumulation of those points. In some examples, at 1012, the vehicle may fit a plane 1018 to the lidar data 1016, which may include determining a plane approximation of the lidar data 1016 (e.g., based at least in part on the covariance matrix, e.g., by performing an eigenvalue decomposition or principal component analysis on the covariance matrix). For example, the vehicle may include performing a principal component analysis or eigenvalue decomposition on the lidar data 1016 represented in the voxel 1010 to fit a plane 1018 to the data 1016. In some examples, the vehicle may include determining a planar approximation of the lidar data 1016 represented in the voxel 1010 based at least in part on data associated with voxels neighboring the voxel 1010. The vehicle may also determine a normal vector 1020 associated with the plane 1018. Additionally, the vehicle may determine a reference direction, which may correspond to an orientation of the autonomous vehicle, and may include determining whether the normal vector 1020 is within a threshold amount or orientation with respect to the reference direction (such as to aid in the orientation of a vertical or planar surface).

[0065] As a non-limiting example, determining the ground surface may include determining a dot product between a vector in an elevation dimension (e.g., a reference direction) of such a device carrying a lidar system and a normal vector 120 expressed in a common coordinate system. In such an example, a dot product exceeding a 15 degree threshold may indicate that the voxel 1010 does not constitute the ground surface. Additionally, the vehicle may cluster voxels that are determined to be locally flat voxels to grow a surface that corresponds to the ground surface.

[0066] At 1022, the vehicle may determine voxels associated with the object and identify a bounding box 1032. In some examples, the vehicle receives or determines an indication of the ground surface or a plane and / or voxels corresponding to the ground surface and removes a subset of the voxels associated with the ground surface. After this removal operation, the voxels remaining in the voxel space may represent objects within the planned path of the vehicle. As shown at 124, a view representation of a voxel space 1008' may correspond to the voxel space illustrated at 1008. In some examples, the voxel space 1008' includes lidar data 1026 representing a first object and lidar data 1028 representing a second object within the path of the vehicle. In some examples, at 1022, the vehicle may cluster the data points to determine that the lidar data is associated with an object represented by a bounding box 1030, and to determine that the lidar data 1028 is associated with an object represented by a bounding box 1032. Additional details of such ground plane detection and object segmentation are detailed in U.S. Patent Application Serial No. 15 / 622,905, incorporated by reference above.

[0067] 11 is a block diagram of an example system 1100 for implementing the techniques described herein, according to an embodiment of the disclosure. In some examples, the system 1100 may include one or more features, components, and / or functionality of the embodiments described herein with reference to FIGS. 1-9. In some embodiments, the system 1100 may include a vehicle 1102. The vehicle 1102 may include a vehicle computing device 1104, one or more sensor systems 1106, one or more communication connections 1108, and one or more driving systems 1110.

[0068] The vehicle computing device 1104 may include one or more processors 1112 and a computer-readable medium 1114 communicatively coupled to the one or more processors 1112. In the illustrated example, the vehicle 1102 is an autonomous vehicle, but the vehicle 1102 can be any other type of vehicle or any other system (e.g., a robotic system, a camera-enabled smartphone, etc.). In the illustrated example, the computer-readable medium 1114 of the vehicle computing device 1104 stores an object detection component 1116, an object classification component 1118, a planning component 1120, one or more system controllers 1122, as well as sensor data 1124 and various thresholds 1126. While shown in FIG. 11 for illustrative purposes as residing within computer-readable medium 1114, it is contemplated that the object detection component 1116, object classification component 1118, planning component 1120, one or more system controllers 1122, as well as the sensor data 1124 and thresholds 1126 may additionally or alternatively be accessible from the vehicle 1102 (e.g., may be stored on or otherwise accessible by computer-readable media remote from the vehicle 1102).

[0069] In at least one example, the object detection component 1116 may be configured to perform techniques such as object detection, filtering, segmentation, pattern detection, white space detection, pixel correlation, feature mapping, etc., to identify objects from sensor data and locate the objects with respect to a map or scene of the physical environment. In some cases, the object detection component 1116 may utilize machine learning models and / or neural networks, which may generate any number of learned inferences or heads.

[0070] The object classification component 1118 may be configured to estimate current characteristics or states and / or predict future characteristics or states of an object (e.g., a vehicle, pedestrian, animal, etc.), including posture, speed, trajectory, velocity, yaw, yaw rate, roll, roll rate, pitch, pitch rate, position, acceleration, or other characteristics (e.g., stiffness, rigidity, depth, volume, etc.), based at least in part on the sensor data and the output of the object detection component 1116.

[0071] The planning system 1120 may determine a path for the vehicle to follow to traverse the physical environment. For example, the planning system 1120 may determine various routes and trajectories and various levels of detail. For example, the planning system 1120 may determine a route to travel around objects within the vehicle's planned path based on various criteria such as the semantic class of the object, the object distance from the ground surface, the object height, the object width, etc. In some cases, the route may include a sequence of waypoints to travel around an object or between two physical locations.

[0072] In at least one example, the vehicle computing device 1104 may include one or more system controllers 1122 that may be configured to control guidance, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 1102. These system controllers 1122 may communicate with and / or control corresponding systems of the driving system 1110 and / or other components of the vehicle 1102.

[0073] In at least one example, the sensor system 1106 can include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, time of flight, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), and one or more time of flight (ToF) sensors, etc. The sensor system 1106 can include multiple instances of each of these or other types of sensors. For example, the lidar sensors can include individual lidar sensors located at corners, front, rear, sides, and / or top of the vehicle 1102. As another example, the camera sensors can include multiple cameras located at various locations on the exterior and / or interior of the vehicle 1102. The sensor system 1106 can provide input to the vehicle computing device 1104. Additionally or alternatively, the sensor system 1106 may transmit sensor data to one or more computing devices over one or more networks 1128 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0074] The vehicle 1102 may also include one or more communication connections 1108 that enable communication between the vehicle 1102 and one or more other local or remote computing devices. For example, the communication connections 1108 may facilitate communication with other local computing devices on the vehicle 1102 and / or with the driving system 1110. The communication connections 1108 may also enable the vehicle 1102 to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). The communication connections 1108 may also enable the vehicle 1102 to communicate with remote teleoperated computing devices or other remote services.

[0075] The communications connection(s) 1108 may include physical and / or logical interfaces for connecting the vehicle computing device 1104 to another computing device (e.g., computing device 1130) and / or to a network, such as network 1128. For example, the communications connection(s) 1108 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing devices.

[0076] In at least one example, the vehicle 1102 can include one or more driving systems 1110. In some examples, the vehicle 1102 can have a single driving system 1110. In at least one example, when the vehicle 1102 has multiple driving systems 1110, the individual driving systems 1110 can be located at opposite ends (e.g., the front and the rear, etc.) of the vehicle 1102. In at least one example, the driving system 1110 can include one or more sensor systems 1106 for detecting a state of the driving system 1110 and / or a state surrounding the vehicle 1102, as discussed above. By way of example and not limitation, the sensor system 1106 may include one or more wheel encoders (e.g., rotational encoders) for sensing the rotation of the wheels of the driving system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the driving system, cameras or other image sensors, ultrasonic sensors, lidar sensors, radar sensors, etc. for acoustically detecting objects around the driving system. Some sensors, such as the wheel encoders, may be unique to the driving system 1110. In some cases, the sensor system 1106 on the driving system 1110 may overlap or supplement the corresponding system of the vehicle 1102.

[0077] In at least one example, the components discussed herein may process the sensor data 1124 as described above and may transmit their respective outputs over one or more networks 1128 to one or more computing devices 1130. In at least one example, the components discussed herein may transmit their respective outputs to one or more computing devices 1130 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0078] In some examples, the vehicle 1102 can transmit sensor data over the network 1128 to one or more computing devices 1130. In some examples, the vehicle 1102 can transmit raw sensor data 1124 to the computing device 1130. In other examples, the vehicle 1102 can transmit processed sensor data 1124 and / or a representation of the sensor data (e.g., object perception tracks) to the computing device 1130. In some examples, the vehicle 1102 can transmit the sensor data 1124 to the computing device 1130 at a particular frequency, after a pre-determined period of time, in near real-time, etc. In some cases, the vehicle 1102 can transmit the sensor data (raw or processed) to the computing device 1130 as one or more log files.

[0079] The computing device 1130 may include a processor 1132 and a computer-readable medium 1134 that stores a modeling component 1136, a machine learning component 1138, and sensor data 1140 and thresholds 1142. For example, the modeling component 1136 and machine learning component 1138 may use the sensor data 1140 received from one or more vehicles 1102 to generate thresholds 1142 (such as the passing distance thresholds of FIGS. 1-7 for different speeds and / or classes of objects). For example, the modeling component 1136 and machine learning component 1138 may label data representing the object with one or more measured parameters or characteristics of the object in the image. The modeling component 1136 and machine learning component 1138 may then use the labeled data to train a machine learning model that can be used to predict a motion state or predict the current or future speed, trajectory, and / or any other characteristic of the object based on the pose of the object depicted in the image data.

[0080] As described herein, an exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to generate an output. Each layer of a neural network can include another neural network, or can include any number of layers (whether convolutional or not). As can be understood in the context of the present disclosure, a neural network can utilize machine learning, which can refer to a broad class of such algorithms in which an output is generated based on learned parameters.

[0081] Although discussed in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, machine learning algorithms include, but are not limited to, regression algorithms (e.g., least-squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute value shrinkage and selection operator (LASSO), elastic net, least-angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative binarization tree 3 (ID3), chi-squared automated interaction detection (CHAID), decision strain, conditional decision tree), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average-one dependence estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-median, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, The algorithms may include: backpropagation, Hopfield networks, radial basis function networks (RBFNs), deep learning algorithms (e.g., deep Boltzmann machines (DBMs), deep belief networks (DBNs), convolutional neural networks (CNNs), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., bootstrap aggregation (bagging), AdaBoost, stacked generalization (blending), gradient boosting machines (GBMs), gradient boosted regression trees (GBRTs), random forests), support vector machines (SVMs), supervised learning, unsupervised learning, semi-supervised learning, etc. Additional examples of architectures include neural networks, such as ResNet50, ResNet101, VGG, DenseNet, and PointNet.

[0082] The processor 1112 of the vehicle 1102 and the processor 1132 of the computing device 1130 may be any suitable processor capable of processing data and executing instructions to perform operations as described herein. By way of example and not limitation, the processor 1112, 1132 may include one or more central processing units (CPUs), graphic processing units (GPUs), or any other device or portion of a device that processes electronic data and converts the electronic data into registers and / or other electronic data that may be stored in a computer-readable medium. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to execute encoded instructions.

[0083] The computer readable media 1114, 1134 are examples of non-transitory computer readable media. The computer readable media 1114, 1134 can store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods and functions attributed to the various systems described herein. In various implementations, the computer readable media can be implemented using any suitable computer readable media technique, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of computer readable medium capable of storing information. The architecture, systems, and individual elements described herein can include many other logical, programmatic, and physical components, and those shown in the accompanying figures are merely examples relevant to the discussion herein.

[0084] As can be appreciated, the components discussed herein are described as separate for purposes of illustration, however, operations performed by various components may be combined with or performed by any other components.

[0085] 11 is illustrated as a distributed system, it should be noted that in alternative examples, components of the vehicle 1102 can be associated with the computing device 1130 and / or components of the computing device 1130 can be associated with the vehicle 1102. That is, the vehicle 1102 can perform one or more of the functions associated with the computing device 1130, and vice versa. Additionally, aspects of the machine learning component 1138 can be executed on any of the devices discussed in the specification.

[0086] Example clauses A. A vehicle comprising a sensor, one or more processors, and one or more computer readable media storing instructions executable by the one or more processors, which, when executed, cause the vehicle to perform operations including: causing the vehicle to follow a planned path; receiving sensor data from the sensor; determining that an object intersects the planned path based at least in part on the sensor data; determining a surface associated with the ground; determining a minimum distance of the object from the surface and a maximum height of the object from the surface; determining an area surrounding the vehicle based on one or more of the minimum distance or maximum height; and controlling operation of the vehicle based at least in part on the area.

[0087] B. The system of paragraph A, wherein determining a surface associated with the ground includes associating sensor data with a voxel space, determining one or more voxels associated with the ground, and fitting a plane to the one or more voxels.

[0088] C. The system of paragraph BA, wherein the region includes at least a first region and a second region, the first region defining a first distance from the vehicle and the second region defining a second distance from the vehicle, the first distance differing from the second distance, and controlling operation of the vehicle includes applying a first set of operating parameters when one or more of the minimum distance or maximum height projects within the first region and a second set of operating parameters when one or more of the minimum distance or maximum height projects within the second region.

[0089] D. The system of paragraph A, further comprising determining a semantic class of the object based at least in part on the sensor data and determining whether the semantic class meets or exceeds a semantic class criterion, and wherein controlling operation of the vehicle is further based at least in part on the semantic class.

[0090] E. A method including receiving data representing a physical environment from a sensor of an autonomous vehicle; determining, at least in part, based on the data, that at least a portion of a representation of an object is within a planned path of the vehicle; determining a surface associated with the planned path based at least in part on the data; determining a distance associated with the object and the surface; determining a first region associated with the vehicle based at least in part on the distance, the first region intersecting at least a portion of the representation of the object; and controlling operation of the vehicle based at least in part on the first region.

[0091] F. The method of paragraph E, further including the step of determining a semantic class of the object, wherein controlling operation of the vehicle is further based at least in part on the semantic class.

[0092] G. The method of paragraph E, further including the step of determining a second region associated with the vehicle based at least in part on the distance, wherein the second region does not intersect with the object and the second region is contained within the first region, and wherein the step of controlling operation of the vehicle is based at least in part on the second region.

[0093] H. The method of paragraph E, wherein the first area is determined based at least in part on a speed of the vehicle.

[0094] The method of paragraph E, wherein the surface is at least one of a ground surface or a side surface of a corridor associated with the planned path of the vehicle.

[0095] J. The method of paragraph E, wherein controlling operation of the vehicle is further based at least in part on a distance of the first region from the vehicle and a volume of traffic in an adjacent lane.

[0096] K. The method of paragraph E, wherein controlling operation of the vehicle includes adjusting a speed of the vehicle based at least in part on a distance of the first region from the vehicle.

[0097] L. The method of paragraph E, wherein the first area is defined as a predetermined distance from the nearest point along the exterior surface of the vehicle.

[0098] M. The method of paragraph E, wherein the distance is at least one of a maximum height of the object or a minimum distance of the object from the surface.

[0099] N. A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform operations including receiving data from a sensor of an autonomous vehicle representing a physical environment; determining, based at least in part on the data, that at least a portion of a representation of an object is within a planned path of the vehicle; determining a surface associated with the planned path based at least in part on the data; determining distances associated with the object and the surface; determining a plurality of regions associated with the vehicle; determining, based at least in part on the distances, that at least a first region of the plurality of regions intersects with at least a first portion of the representation of the object; and controlling operation of the vehicle based at least in part on the first region.

[0100] O. The non-transitory computer-readable medium of paragraph N, wherein the operation further includes determining, based at least in part on the distance, that at least a second region of the plurality of regions intersects with at least a second portion of the representation of the object, and controlling the operation of the vehicle is based at least in part on the second region.

[0101] P. The non-transitory computer-readable medium of paragraph N, wherein determining a surface associated with the planned path includes associating data representing the physical environment with a voxel space, determining one or more voxels associated with a ground surface of the physical environment, and fitting a plane to the one or more voxels.

[0102] Q. The non-transitory computer-readable medium of paragraph N, wherein the operation further includes determining a semantic class of the object, and wherein controlling the operation of the vehicle is based at least in part on the semantic class.

[0103] R. The non-transitory computer-readable medium of paragraph Q, wherein the semantic classes are at least one of a pedestrian, a wall or structure, foliage, a rock, a plant, a vehicle, a vehicle door, debris or clutter, a motorcycle, or a traffic light or cone.

[0104] S. The non-transitory computer-readable medium of paragraph N, wherein the distances include a maximum height of the object from the surface and a minimum distance of the object from the surface, and determining that at least a first region of the plurality of regions intersects with at least a first portion of the representation of the object includes at least one of the maximum height being less than a first threshold associated with a bottom surface of the vehicle or the minimum distance being greater than a second threshold associated with a top surface of the vehicle.

[0105] T. The non-transitory computer-readable medium of paragraph N, wherein the representation includes a bounding box, and determining one or more of the first region or the second region includes determining a distance between a corner of the bounding box and the surface.

[0106] Although the exemplary provisions described above have been described with respect to one particular implementation, it should be understood in the context of this document that the contents of the exemplary provisions may also be implemented via a method, device, system, computer-readable medium, and / or another implementation. In addition, any of Examples A-T may be implemented alone or in combination with any other one or more of Examples A-T.

[0107] conclusion One or more examples of the techniques described herein have been described, although various modifications, additions, permutations, and equivalents thereof fall within the scope of the techniques described herein.

[0108] In describing the examples, reference has been made to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific examples of the claimed subject matter. It is to be understood that other examples may be used, and that modifications or variations, such as structural changes, may be made. Such examples, modifications, or variations do not necessarily depart from the intended scope of the claimed subject matter. Although steps herein may be presented in a certain order, in some cases the ordering may be changed such that certain inputs are provided at different times or in a different order without changing the functionality of the described systems and methods. The disclosed procedures may also be performed in different orders. In addition, the various calculations herein need not be performed in the order disclosed, and other examples using alternative orders of calculations may be readily implemented. In addition to being reordered, calculations may also be decomposed into sub-calculations having the same results.

Claims

1. receiving data representative of a physical environment from a sensor of the autonomous vehicle; determining, based at least in part on the data, that at least a portion of a representation of an object is within a planned path of the vehicle; determining a surface associated with the planned path based at least in part on the data; determining distances associated with the object and the surface; determining whether the representation of the object intersects with a first region associated with the vehicle based at least in part on the distance, the first region intersecting at least a portion of the representation of the object; determining whether the representation of the object intersects a second region associated with the vehicle based at least in part on the distance, wherein the second region does not intersect with the object and the second region is contained within the first region; controlling operation of the vehicle based at least in part on the second region. The method includes:

2. determining a semantic class of the object; and controlling the operation of the vehicle further comprising: based at least in part on the semantic classes. The method of claim 1.

3. the surface is at least one of a ground surface or a side surface of a corridor associated with the planned path of the vehicle; The method according to claim 1 or 2.

4. and controlling the operation of the vehicle further comprises at least partially based on a distance of the first region from the vehicle and a volume of traffic in an adjacent lane. The method according to any one of claims 1 to 3.

5. controlling the operation of the vehicle includes adjusting a speed of the vehicle based at least in part on a distance of the first region from the vehicle. The method according to any one of claims 1 to 4.

6. the first region is defined as a predetermined distance from an exterior surface of the vehicle, the distance being at least one of a maximum height of the object or a minimum distance of the object from the surface; The method according to any one of claims 1 to 5.

7. comprising coded instructions which, when executed on a computer, perform the method of any one of claims 1 to 6, Computer program.

8. 1. A system comprising: one or more processors; one or more non-transitory computer-readable media storing instructions executable by the one or more processors, the instructions, when executed, receiving data representative of a physical environment from a sensor of the autonomous vehicle; determining, based at least in part on the data, that at least a portion of a representation of an object is within a planned path of the vehicle; determining a surface associated with the planned path based at least in part on the data; and determining distances associated with the object and the surface; determining a plurality of regions associated with the vehicle; determining, based at least in part on the distance, that at least a first region of the plurality of regions intersects with at least a first portion of the representation of the object; controlling operation of the vehicle based at least in part on the first region; and one or more non-transitory computer-readable media that cause the system to perform operations including A system comprising:

9. determining, based at least in part on the distance, that at least a second region of the plurality of regions intersects with at least a second portion of the representation of the object. Further comprising: controlling the operation of the vehicle based at least in part on the second region. The system of claim 8.

10. Determining the surface associated with the planned path comprises: associating the data representing the physical environment with a voxel space; determining one or more voxels associated with a ground surface of the physical environment; fitting a plane to the one or more voxels; and Including, 10. A system according to claim 8 or 9.

11. The operation includes: Determining the semantic class of the object. Further comprising: controlling the operation of the vehicle is based at least in part on the semantic classes. A system according to any one of claims 8 to 10.

12. the semantic classes are at least one of a pedestrian, a wall or structure, foliage, a rock, a plant, a vehicle, a vehicle door, debris or clutter, a bike, or a traffic light or cone; A system according to any one of claims 8 to 11.

13. the distances include a maximum height of the object from the surface and a minimum distance of the object from the surface; Determining that at least the first region of the plurality of regions intersects with at least the first portion of the representation of the object includes at least one of the maximum height being less than a first threshold associated with a bottom surface of the vehicle, or the minimum distance being greater than a second threshold associated with a top surface of the vehicle. A system according to any one of claims 8 to 12.

14. the representation of the object includes a bounding box; determining the at least the first region includes determining a distance between a corner of the bounding box and the surface. The system of claim 13.

Images