Determining the Occupancy Rate of Occlusion Areas

Abstract

Techniques are discussed for controlling a vehicle, such as an autonomous vehicle, based on occlusion areas in an environment. The occlusion area may represent an area where the vehicle's sensors cannot sense a portion of the environment due to obstruction by other objects or sensor limitations. The occlusion area of ​​an object is determined along with one or more visibility areas proximate the occlusion area. Entry and exit areas may be determined based on known traffic directions and / or boundaries of drivable surfaces. Based on a threshold speed, the vehicle may designate a portion of the occlusion area as a pseudo-visible area. Furthermore, if a dynamic object passes through the occlusion area from the entry area, the portion of the occlusion area may be considered pseudo-visible. The pseudo-visibility may also be determined based on the movement of the occlusion area. The vehicle may be controlled to traverse the environment based on the pseudo-visibility.

Description

[Technical field]

[0001] The present disclosure relates to determining occupancy of an occluded area. [Background technology]

[0002] This application claims priority to U.S. Patent Application No. 16 / 289,397, filed February 28, 2019, and entitled “DETERMINING OCCUPANCY OF OCCLUDED REGIONS,” the entire contents of which are incorporated herein by reference.

[0003] Various methods, devices, and systems are utilized by autonomous vehicles to guide such autonomous vehicles through environments that include a variety of static and dynamic objects. For example, autonomous vehicles may utilize route planning methods, devices, and systems to guide the autonomous vehicle through areas congested with other moving vehicles (autonomous or otherwise), moving people, stationary buildings, etc. In some examples, an autonomous vehicle may make decisions while traversing an environment to ensure the safety of passengers and surrounding people and objects. However, conventional techniques for making such decisions may be overly conservative or brake frequently when not necessary, causing discomfort to passengers and uncertainty to those in the vicinity regarding the autonomous vehicle's future actions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent Application Serial No. 16 / 011436 [Patent Document 2] U.S. Patent Application Serial No. 16 / 151607 [Brief description of the drawings]

[0005] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit of a reference number identifies the drawing in which the reference number first appears, and the use of the same reference number in different drawings indicates similar or identical components or functions.

[0006] [Figure 1] FIG. 1 is a pictorial flow diagram of an exemplary process for capturing sensor data, determining occluded regions in an environment and visible regions proximate to the occluded regions, determining an occupancy state of the occluded regions, and controlling a vehicle based at least in part on the occupancy state of the occluded regions in accordance with an embodiment of the present disclosure. [Diagram 2] 1 is a diagram illustrating an environment showing a perspective view of an exemplary occlusion grid with multiple occlusion fields in the environment and a vehicle including an occlusion reasoning component that determines occluded and non-occluded areas of the occlusion grid due to obstacles in accordance with an embodiment of the present disclosure. [Diagram 3] FIG. 2 illustrates identifying a static occlusion region and a dynamic object passing from one side of the static occlusion region to the other, according to an embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates identifying a static occlusion region and a dynamic object exiting the occlusion area according to an embodiment of the present disclosure. [Figure 5A] 11A-11C are diagrams illustrating the identification of moving occlusion regions and pseudo-visible regions of moving occlusion regions according to an embodiment of the present disclosure. [Figure 5B] 5B is a diagram illustrating determining the occupancy status of the occluded region illustrated in FIG. 5A according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a block diagram illustrating an example of a system for implementing the techniques described herein. [Figure 7] FIG. 1 illustrates an example of an exemplary process for acquiring sensor data, determining an occlusion area caused by an object in an environment, determining a visibility area proximate the occlusion area, determining whether a dynamic object is located in the visibility area, and if so, determining whether the dynamic object will enter the occlusion area, and controlling an autonomous vehicle based on the determination, in accordance with an embodiment of the present disclosure. [Figure 8] FIG. 1 illustrates an example process for determining, at a first time, a first occlusion area due to an object and a visibility area proximate to the occlusion area, determining that the visibility area includes an entrance area to the occlusion area, determining, at a second time after the first time, a second occlusion area corresponding to at least a portion of the first visibility area, determining whether a dynamic object is present in that portion, and controlling an autonomous vehicle based on the determination, in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The present disclosure is directed to determining the likelihood that an area not visible to a sensor (e.g., an occluded area) is unoccupied by observing the occluded area over time. For example, the techniques herein can use past knowledge of visible entry areas proximate the occluded area to determine the certainty that no objects are present in the occluded area that would prevent a vehicle from safely traversing a portion of the environment. For example, a sensor on a vehicle (e.g., an autonomous vehicle) can capture sensor data of an environment, which may include objects separate from the vehicle, such as other vehicles, pedestrians, buildings, vegetation, etc. The sensor coverage of the environment may be limited (occluded) based on obstacles in the environment. Initially, the occlusion rate of the occluded area is unknown. By determining and observing the area of ​​entry into the occluded area (e.g., based on the direction of travel), the occlusion inference system can determine the likelihood that a portion of the occluded area is unoccupied by a dynamic object. For example, the occlusion inference system can progressively determine that an occluded area is a "pseudo-visible area" based on observing the area for a period of time and / or by assuming that a dynamic object in the occluded area (if there was a dynamic object) has moved out of the portion of the occluded area. The occlusion inference system provides information about the state of occlusion regions to the planning system, which can be used in generating the vehicle trajectory regarding the likelihood of the absence of dynamic objects in the region.

[0008] The occlusion inference system can be implemented separately for static and moving occlusion regions. In a static occlusion region, the location of the occlusion region remains static or stationary over time. In some cases, a static occlusion region may be caused by a scenario in which both the vehicle (e.g., with sensors) and the occluding object in the environment are stationary. As mentioned above, the occlusion inference system may assume that the occlusion rate of the occlusion region is unknown. However, over time and based on a travel direction, a threshold time, and / or a threshold speed associated with the drivable area, the occlusion inference system may determine that a portion of the occlusion region is likely unoccupied. The travel direction may correspond to a known travel direction on the map, such as the flow of traffic in the drivable area. The threshold time may correspond to a typical time that a vehicle spends in a particular scenario, such as the average time that a vehicle takes to stop at an intersection, the time to accelerate to a speed limit, the time to decelerate from the current speed to a complete stop, etc. The threshold speed may correspond to a minimum speed at which a reasonable dynamic object is expected to move in the environment. For example, the threshold speed may be an average walking speed of a pedestrian (e.g., 1.5 meters / second), which encompasses objects moving faster than a minimum speed, such as cars and bicycles traveling along the drivable area. Additionally, the threshold speed and / or threshold time may be determined based on the size of the occluded area, the distance from the occluded area to the vehicle, map data (e.g., speed limits associated with the drivable area), historical data, etc.

[0009] In a moving occlusion region, the location and / or size of the occlusion region may change over time. In some examples, a moving occlusion region may be due to the movement of the vehicle and / or objects in the environment causing the occlusion region to the vehicle's sensors. In the case of a moving occlusion region, the vehicle may infer that no dynamic objects are present and that areas that were previously visible (but now occluded) are still free of dynamic objects. This inference allows the vehicle to make decisions regarding occupancy of areas of the environment, which can be used to determine how to traverse the environment more quickly and reliably than traditional planning techniques.

[0010] The sensor data captured by the vehicle can include LIDAR data, RADAR data, image data (RGB, intensity, depth, infrared, ultraviolet, etc.), time of flight data, sonar data, etc. In some cases, the sensor data can be provided to a perception system configured to determine the type of object (vehicle, pedestrian, bicycle, animal, parked car, tree, building, etc.) in the environment. Additionally, the perception system can determine motion information of objects in the environment based on the sensor data.

[0011] Sensor data captured by the vehicle representing objects in the environment can be used to generate occlusion regions associated with the objects in the environment relative to the vehicle. For example, the occlusion inference system can determine an occlusion grid associated with occlusion regions associated with the object's position over time. Techniques such as ray casting and / or projection can be used to determine which areas of the environment represented by the occlusion grid are occluded relative to the vehicle's position over time. In some examples, different techniques for determining occlusion can be used based on various sensor modalities. As a non-limiting example, ray casting can be used to determine occlusion for LIDAR sensors, while image projection using a camera matrix can be used to determine occlusion for image sensors. The occlusion grid can be three-dimensional and can represent a prediction of the object in voxels that describe the volume of the object in the environment. Techniques for generating occlusion grids are described, for example, in U.S. Patent Application Publication No. 2013 / 0133994, entitled "Occlusion Aware Planning," filed June 18, 2018, and incorporated herein by reference in its entirety.

[0012] As the vehicle traverses the environment, the occlusion inference system may use sensor data and map data of the surrounding environment to detect objects in the environment and determine corresponding occlusion regions due to such objects. Often, occlusion regions include objects that are relevant to the vehicle's decision processes, such as whether to cross a road, whether to cross a crosswalk, which lane to enter, how fast to accelerate / decelerate, etc. In some examples, an occlusion region may have a visible entrance region adjacent or proximate to the occlusion region, which may be used to make decisions about how to control the vehicle. The occlusion inference system may use map data to determine the entrance region. For example, the map data may indicate drivable areas (e.g., lanes of a road) and / or traffic directions associated therewith, and if there is a visible region proximate to the occlusion region, the vehicle may designate an entrance region to the occlusion region based on an assumption that traffic will enter the occlusion region in a particular direction at that location. Similarly, an area corresponding to an exit of an occlusion region (based at least in part on the direction of travel) may be referred to as an "exit region."

[0013] For static occlusion regions, the occlusion inference system may determine that at least a portion of the occlusion region is likely free of dynamic objects based on the sensor data and a threshold speed and / or threshold time associated with the occlusion region. For example, the threshold speed may be related to a pedestrian's walking pace, such as 1.5 m / s, although of course any suitable threshold speed may be used. In some examples, the threshold time may correspond to the time that the occlusion inference system may observe the occlusion region before determining occupancy of the portion of the occlusion region. Regardless of the threshold speed and / or threshold time used, the occlusion inference system may observe and / or track objects moving faster than the threshold speed passing through the occlusion region and control the vehicle accordingly. In many cases, objects moving slower than the threshold speed may not affect many of the decisions used to control the vehicle, and thus the vehicle may treat such objects as "non-moving." In an example, a cyclist may be traveling fast enough to not take their feet off the pedals, but may be approaching an intersection slowly if they are slower than the threshold speed that the vehicle uses to analyze the presence of unknown objects in the occlusion region. Assuming a cyclist is positioned in an area blocked by a vehicle approaching the intersection, the vehicle's knowledge of the cyclist's presence is highly unlikely to influence its decision to cross the intersection at its current speed, allowing it to proceed safely through the intersection.

[0014] In some examples, the occlusion inference system may not rely on assumptions regarding the minimum speed of the associated dynamic object for moving occlusion areas as it does for static occlusion areas. Instead, the occlusion inference system may rely on knowledge of entrances to occlusion areas when the size or dimensions of the occlusion area change over time while either the vehicle or the occluding object is moving. For example, if a portion of the visible area does not contain a dynamic object at a first time, and no dynamic object enters the entrance area as time passes, the vehicle may assume that the portion of the area that was visible at the first time does not contain a dynamic object at a second later time, even if the previously visible area is now occluded. The portion of the previously visible area may be considered "pseudo-visible," meaning that the vehicle can determine with a relatively high degree of certainty whether the area contains a dynamic object relevant to a decision on behalf of the vehicle. In some examples, the pseudo-visible determination may be based on determining that the probability that the occlusion area does not contain a dynamic object meets or exceeds a threshold, such as, for example, 51% certainty, 75% certainty, 90% certainty, etc. In this way, by determining that even a portion of the area is pseudo-visible, it is possible to execute an instruction earlier than with a technique that assumes that all areas are visible at the same time.

[0015] The techniques discussed herein can improve the capabilities of a computing device in many ways. As described above, an occlusion inference system can classify a portion of an occluded area as pseudo-visible, thereby enabling it to make decisions faster than techniques that require all areas to be visualized and evaluated, and can save processing and computing resources when analyzing the surrounding environment. In some cases, the dynamic nature of the described techniques (e.g., controlling a vehicle based on visible or pseudo-visible areas) allows fewer rules to be enumerated to generate and / or select a trajectory for a vehicle, such as an autonomous vehicle, to traverse an environment. Controlling a vehicle based in part on pseudo-visible portions of an occluded area can improve vehicle safety by allowing the vehicle to adapt and react to its surroundings, particularly when situations arise where no specific rules exist, rather than requiring the vehicle to follow set instructions. Additionally, controlling a vehicle based on pseudo-visible portions of an occluded area can reduce processing resources, since the number of rules enumerated for all driving situations the vehicle encounters may be virtually infinite. Additionally, techniques that specify pseudo-visible areas of occluded areas over time can increase confidence that dynamic objects will not interfere with or collide with the vehicle, which can improve safety outcomes, performance, and / or accuracy. These and other improvements in computer functionality are discussed herein.

[0016] The techniques described herein can be implemented in many ways. Exemplary implementations are provided with reference to the following figures. Although discussed in the context of an autonomous vehicle, the methods, apparatus, and systems described herein can be applied to a variety of systems (e.g., sensor systems or robotic platforms) and are not limited to autonomous vehicles. In another example, the techniques can be utilized in an aviation or nautical context, or in any system that includes objects or entities that may be associated with behaviors unknown to the system. Additionally, the techniques described herein can be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

[0017] FIG. 1 is a pictorial flow diagram of an exemplary process 100 for capturing sensor data, determining occluded areas in an environment and visible areas proximate to the occluded areas, determining an occupancy state of the occluded areas, and controlling a vehicle based at least in part on the occupancy state of the occluded areas, in accordance with an embodiment of the present disclosure.

[0018] At operation 102, the process may include capturing sensor data of the environment. In some examples, the sensor data may be captured by one or more sensors on a vehicle (autonomous vehicle or otherwise). For example, the sensor data may include data captured by one or more of a LIDAR sensor, a RADAR sensor, an image sensor, a time of flight sensor, a sonar sensor, etc.

[0019] The example illustrates image data 104 captured in operation 102. For example, image data 104 may represent an object 106 in an environment. In some examples, the object 106 may represent a vehicle, a pedestrian, an animal, etc. In some examples, the sensor data captured in operation 102 may be used to determine information about the object 106, including, but not limited to, the type of object (e.g., semantic information indicative of a classification of the object, such as a vehicle, a pedestrian, a bicycle, an animal, etc.). In some examples, the operations may include determining the type of object 106, a bounding box associated with the object 106, segment information associated with the object 106, and / or motion information associated with the object and any uncertainty associated therewith as discussed herein. Operation 104 may further include accessing map data representing the environment.

[0020] In operation 108, the process may include determining occlusion regions in the environment and visibility regions proximate to the occlusion regions. As used herein, the terms "occlusion" and "visibility" may represent a probability that an area of ​​the environment is occluded and / or visible to a sensor of the vehicle (e.g., above a threshold). Example 110 illustrates a vehicle 112 approaching an intersection of a drivable area 116 in the environment at a first time T1. The drivable area 116 includes a traffic direction 113 approaching the intersection of the drivable area 116 from the right. In some examples, the vehicle 112 may represent occlusion areas of the environment using an occlusion grid 114, which may be stored in association with map data of the environment or may be dynamically generated based on objects in the environment, a given sensor range, field of view, and / or map data. The occlusion grid 114 may include multiple occlusion fields (e.g., boxes drawn in the occlusion grid 114), which may represent discrete areas of the environment, such as a drivable area. In some examples, the occlusion field can indicate an occlusion status (e.g., whether the location is visible to one or more sensors of the autonomous vehicle) and an occupancy status (e.g., whether the location is occupied by an object, such as a vehicle, a pedestrian, an animal, etc.). The occlusion status and / or occupancy status can be determined using LIDAR data, RADAR data, and / or image data captured by the autonomous vehicle. Additional details for generating the occlusion grid are described, for example, in U.S. Patent Application Publication No. 2013 / 0133994, entitled “Occlusion Aware Planning,” filed June 18, 2018, and incorporated herein by reference in its entirety.

[0021] For purposes of discussion, the vehicle 112 capturing (or utilizing) the sensor data may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for all trips with no driver (or passenger) expected to control the vehicle at any time. In such an example, the vehicle 112 may be non-occupied, as it may be configured to control all functions from starting to stopping, including all parking functions. This is merely an example, and the systems and methods described herein may be incorporated into any ground-based, air-based, or water-based vehicle, including vehicles that must be manually controlled by a driver at all times, to vehicles that are partially or fully autonomously controlled. Additional details related to the vehicle 112 are described throughout this disclosure.

[0022] Objects 106 are also located within the environment, as shown in the illustrated example 110. In some examples, the objects 106 may represent vehicles, pedestrians, bicycles, animals, etc. The environment shown in example 110 may include drivable areas 116 and non-drivable areas 118. In some examples, the drivable areas 116 may include any areas of the environment that the vehicle 112 may (legally) traverse (e.g., roads, driveways, parking lots, etc.). In some examples, the non-drivable areas 118 may represent areas of the environment that the vehicle 112 may not legally or physically traverse (e.g., sidewalks, buildings, parks, etc.).

[0023] In some examples, the occlusion grid 114 can be based at least in part on localizing the vehicle 112 within the environment and accessing map data associated with the environment. For example, the map data can include map elements such as lane elements that indicate areas of the environment that correspond to the drivable area 116. Additional examples of map elements can include buildings or structures, crosswalk elements, bike lane elements, intersection elements, lane division elements, traffic light elements, stop sign elements, stop line elements, yield sign elements, yield line elements, parking lane elements, roadway elements, speed bump elements, red light areas (e.g., virtual crosswalks, etc.), trajectory waypoints (such as known trajectories), passenger pickup points, sign location elements, geofence elements, etc. In some cases, a predicted trajectory associated with the object can be based on the map data.

[0024] As described above, the occlusion grid 114 may represent an occlusion and / or an occupancy state of the occlusion field of the occlusion grid 114. Additionally, in some examples, the occlusion state may further include an "uncertain" state, meaning that it is currently unknown whether the occlusion field is occupied or not based on the available data. The uncertain state may be caused by an object, such as the object 106, blocking a portion of the environment from the view of the sensor of the vehicle 112. In the example 110, the occlusion field of the occlusion grid 114 that is obstructed (occluded) by the object 106 is colored light gray, indicating an occlusion region 120. The occlusion field of the occlusion grid 114 that is occupied is colored black (e.g., at least partially covered by the object 106). The occlusion field of the occlusion grid 114 that is colored white indicates a visibility region 122.

[0025] In this example 110, at least a portion of the visibility region 122 is proximate to the occluded region 120. Although the occluded region 120 and visibility region 122 of example 110 are shown only over drivable region 116 for clarity, the occlusion grid 114 and associated occluded, occupied, and / or visibility regions may extend into non-drivable regions 118 as well. For example, the occlusion grid may extend into non-drivable regions, such as sidewalks, to capture information relevant to vehicular decisions associated with pedestrians, bicycles, vehicles entering and exiting drivable regions via roadways spanning sidewalks, and the like.

[0026] The vehicle 112 may designate the visibility region 122 as an "entrance region" to the occluded region based on knowledge (e.g., obtained from map data) that a dynamic object approaches the occluded region 120 from the right. In some examples, the entrance region of the visibility region 122 may be contiguous with the occluded region 120 such that a first extent of the occluded region 120 corresponds to a second extent of the entrance region. When evaluating a drivable region such as the drivable region 116, the second extent of the entrance region may correspond to the drivable region 116 associated with the occluded region 120. In at least some examples, such an entrance region requires a continuous visible occlusion field (e.g., having at least one adjacent side of the respective occlusion fields in common) from one side of the drivable surface to the other along a boundary between the occluded region and the visibility region, the boundary corresponding to an incoming travel direction. A similar approach may be performed to determine an exit region, with the boundary between the occluded region and the visibility region corresponding to an outgoing direction from the occluded region. Once an entrance region and an exit region are identified on either side of the occluded region, inferences may be performed regarding dynamic objects within the occluded region. In such an example, a dynamic object traveling along the drivable region 116 is likely to enter the blocked region 116 from the entrance region, pass through the blocked region, and exit via the exit region along the drivable region 116.

[0027] In operation 124, the process may include determining an occupancy status of the occluded region by observing the visibility region for a period of time. Example 126 illustrates determining a pseudo visibility region 128 in the environment at a second time T2. For example, the pseudo visibility region 128 may correspond to an occlusion field of the occlusion grid 114 that the vehicle 112 has determined with a certain degree of certainty to be unoccupied by a dynamic object based on observing the visibility region 122 from time T1 to time T2. The determination that the pseudo visibility region 128 is unoccupied may also be based on map data of the environment indicating that the travel direction of the drivable region 116 approaches the occluded region 120 from the right and that the visibility region 122 is an entry region as just described. In some examples, even if there is no dynamic object entering the occluded region 120 from the entry region, the vehicle 112 may continue to determine that the occlusion field of the occlusion grid 114 is over time based on a threshold speed and / or a threshold time along the travel direction associated with the area of ​​the occlusion grid until all of the occlusion field is deemed pseudo-visible. That is, occlusion fields closer to the entrance region may be pseudo-visible before occlusion fields closer to the exit region.

[0028] While depicted in example 126 as generally in the x and y directions for clarity, occlusion region 120 and / or visibility region 122 may represent occlusion information related to a z-direction component, for example, corresponding to the height of object 106. In another example, occlusion information may be associated with a z-direction component corresponding to the topography of the surrounding environment, such as hills and valleys, or the height of an object that is occluded but relevant to trajectory determination, such as a stop signal or train crossing signal.

[0029] In operation 130, the process may include controlling the vehicle based at least in part on the occupancy state of the occluded region. For example, if the occupancy state of the occluded region 120 indicates a high probability that no dynamic objects are present in the occluded region (e.g., by having all columns and / or rows of the occlusion field of the occluded region 120 designated as "pseudo-visible"), the vehicle 112 may traverse the environment based on a trajectory to efficiently and safely reach the destination of the vehicle 112.

[0030] In another example, if the occupancy state of the occluded region 120 indicates that a dynamic object is present in the occluded region (or if there is otherwise insufficient information to determine whether a dynamic object is not present in the occluded region), the vehicle 112 may wait for the object to re-emerge from the occluded region 120 (e.g., sensed by the vehicle's sensors) and / or attempt to move to a position that provides a greater percentage of visibility to the sensors. If the dynamic object re-emerges, the vehicle 112 may respond accordingly, such as waiting for the object to clear the area of ​​its intended trajectory before proceeding to the vehicle's intended destination. In at least some examples, when an object enters an occluded region, inferences regarding the pseudo-visible space may be reevaluated. As a non-limiting example, when an object enters an occluded region, all occlusion fields within the occlusion grid 114 may be marked as occluded and pseudo-visible determinations may be recalculated.

[0031] The occupancy state can be used in a variety of ways to control the vehicle, although other inputs may also contribute to choosing to control the vehicle (e.g., safety systems, user input, etc.). For example, in some examples, the occupancy state can be used as one of many costs associated with controlling the vehicle to traverse an environment. Examples of additional costs used to control the vehicle can include, but are not limited to, one or more of steering costs (e.g., costs based on the steering angle of the vehicle), reference costs (e.g., costs based on the distance of the trajectory from a reference line), acceleration costs (e.g., costs based on the maximum acceleration of the vehicle), braking costs (e.g., costs based on a minimum braking time), and the like.

[0032] Additional details regarding occlusion regions, entrance regions, pseudo-visibility regions, and occupancy states are discussed throughout this disclosure.

[0033] FIG. 2 illustrates an environment showing a perspective view of an exemplary occlusion grid with multiple occlusion fields in the environment and a vehicle including an occlusion reasoning component that determines occluded and non-occluded areas of the occlusion grid due to obstacles in accordance with an embodiment of the present disclosure.

[0034] As shown, the environment 200 may include a vehicle 202 that includes one or more sensor systems 204 that capture data representative of the environment 200 .

[0035] In at least one example, as described above, the vehicle 202 can be associated with a sensor system 204 that can be disposed on the vehicle 202. The sensor system 204 can include light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, ultrasonic transducers, sound navigation and ranging (sonar) sensors, position sensors (e.g., global positioning system (GPS), compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (RGB, infrared, intensity, depth, etc.), wheel encoders, microphones, environmental sensors (temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), time of flight sensors (RF modulated light sources, range gated imagers, direct time-of-flight imagers, etc.), etc. The sensor system 204 can generate sensor data that can be utilized by a vehicle computing device 206 associated with the vehicle 202.

[0036] In at least one example, the vehicle computing device 206 can utilize the sensor data in the occlusion inference component 208. For example, the occlusion inference component 208 can access occlusion data and / or map data to determine the occlusion and occupancy status of occlusion fields in an occlusion grid.

[0037] By way of example, and not limitation, a vehicle 202 in environment 200 is approaching an intersection 210. In some examples, the vehicle 202 may access map data and determine that an occlusion grid 212 is associated with the intersection 210. In some examples, the intersection 210 may represent an intersection where the vehicle 202 is required to yield to one-way traffic entering the intersection.

[0038] The occlusion grid 212 may include multiple occlusion fields that divide portions of the environment 200 into discrete regions. While depicted as uniformly sized regions for purposes of illustration, the occlusion fields need not be so limited and may have non-uniform dimensions (e.g., large or small based at least in part on distance from the vehicle 202, type of drivable area (e.g., city, highway, etc.), etc.). In some examples, the occlusion field 214 may represent an occlusion condition (e.g., an indication of whether the occlusion field 214 is within a sensed region 216 associated with one or more sensors of the vehicle 202). As can be appreciated, the environment 200 may include one or more obstacles 218, 220, and 222 that cause portions of the occlusion grid 212 to fall outside the sensed region 216 of the vehicle 202. Thus, the non-occluded region 224 of the occlusion grid 212 may represent an area of ​​the occlusion grid 212 that can be "seen" by the vehicle 202, i.e., the vehicle 202 may capture sensor data representative of the non-occluded region 224. Similarly, occlusion regions 226 may represent areas of occlusion grid 212 that cannot be "seen" by vehicle 202 (e.g., based on expected sensor range, field of view, corresponding map data, etc.). In some examples, occlusion grid 212 may be generated based on the time and / or information required to perform a maneuver (e.g., crossing a lane considering the estimated speed of an oncoming vehicle resulting in a minimum visibility distance for the oncoming lane). As shown in FIG. 2, the occlusion field of occlusion region 226 is shaded gray to represent the absence of sensor data corresponding to that region, while non-occluded region 224 is unshaded.

[0039] In some examples, when an occlusion field is occupied by an object, the occlusion field may store additional metadata indicative of the identity of the object, as well as data indicative of the path of the object through the occlusion grid. For example, the occlusion velocity component 228 of the occlusion inference component 208 may determine the path of the object through the occlusion grid as well as determine the velocity of the object as it moves along the path through the occlusion grid. Additionally, the occlusion velocity component 228 may compare the velocity of the object to one or more threshold velocities associated with making a decision regarding how to control the vehicle 202. In some examples, the occlusion velocity component 228 may use the threshold velocities in identifying and / or assessing occlusion regions within the environment, such as determining whether a dynamic object is present in an area that is otherwise obstructed to the sensor system 204 of the vehicle 202.

[0040] Further details regarding the identification and assessment of static occlusion regions are provided below in conjunction with Figures 3 and 4, and further details regarding the identification and assessment of moving occlusion regions are provided below in conjunction with Figures 5A and 5B.

[0041] FIG. 3 is an example of identifying a static occlusion region and a dynamic object passing from one side of the static occlusion region to the other, according to an embodiment of the present disclosure.

[0042] The environment 302 includes a vehicle 304 (e.g., an autonomous vehicle), a first static object 306 (e.g., another vehicle that is stationary or parked), and a second dynamic object 308 (e.g., another vehicle moving along a drivable surface 310 in a traffic direction 311). As shown, the environment 302 may represent an initial time (e.g., T1). At the initial time T1, the static object 306 may obfuscate a portion of the environment 302 from sensors of the vehicle 304, causing an occlusion area 312, shown with a light gray shading. The vehicle 304 may be able to sense multiple visibility areas, such as an unoccluded visibility area 314. The traffic direction 311 may be determined using map data of the environment 302 and / or by sensory data of the environment 302 received from sensors of the vehicle 304 (e.g., observing which direction the vehicle is traveling along the drivable surface 310), which is analyzed to determine the traffic direction 311.

[0043] In some examples, the occlusion inference system of the vehicle 304 may use map data and / or data from sensors of the vehicle 304 to generate an environment representation 316 of the environment 302 at an initial time T1. The environment representation 316 includes representations of the vehicle 304, the static objects 306, the dynamic objects 308, and the drivable area 310.

[0044] As described above, the vehicle 304 may generate an occlusion grid 318 that includes multiple occlusion fields that represent discrete areas of the environment 302. In some examples, the occlusion fields can indicate an occlusion state (e.g., indicating whether the location is visible to one or more sensors of the autonomous vehicle, whether the location is occluded to the sensors, whether the location is out of range of the sensors, etc.) and an occlusion state (e.g., indicating whether the location is occupied by an object, such as a vehicle, a pedestrian, an animal, etc.). The occlusion grid 318 can be generated from map data of the environment, and the occlusion inference system can use data from the sensors of the vehicle 304 to determine the state of the occlusion fields.

[0045] For example, the occlusion inference system of the vehicle 304 may use map data of the environment to generate an occlusion field of an occlusion grid 318 corresponding to the drivable area 310. The occlusion inference system of the vehicle 304 may also use one or more sensors to identify a static object 306 and designate the occlusion field in which the static object 306 is detected as "occupied" (e.g., indicated by a black-shaded occlusion field). The occlusion inference system may also determine that the static object 306 is blocking one or more sensors from seeing the occlusion area 312. Based on this determination, the occlusion inference system may assign an indication of the occlusion field contained in the occlusion area 312 as "occluded" and / or "uncertain" because it cannot say with certainty whether the occlusion field contained in the occlusion area 312 is occupied or not.

[0046] The occlusion inference system of the vehicle 304 may also use the map data to determine a travel direction associated with the drivable region 310. In the example 300, the travel direction is moving from right to left as indicated in the environment 302 by the dynamic object 308. Based on the travel direction associated with the drivable region 310, the occlusion inference system of the vehicle may designate at least a portion of the visibility region 314 as an entrance region to the occluded region 312. In some examples, an entrance region must, by definition, span all possible entrances to the occluded region. The spanning may be determined by determining a portion of a visible, continuously connected occlusion field (e.g., an occlusion field having at least one side in common with another adjacent occlusion field) that connects a first side of the drivable surface to another side along the boundary of the occluded region 312 of the entrance region based on the travel direction. By spanning all possible entrances to the occluded region, the entrance region provides the vehicle with a higher certainty as to whether a dynamic object is present in the occluded region (e.g., there is no occlusion field that may occlude a dynamic object entering the occluded region 312 along the travel direction of traffic). By designating an entry region, the vehicle 304 can determine with a relatively high degree of certainty that the dynamic object will only enter the occluded region 312 via the visibility region 314 and will continue in the same normal direction through the occluded region 312. The occlusion inference system of the vehicle 304 may then provide this information to a planning system of the vehicle 304, which may be used to make decisions to control the vehicle 304. Similar techniques may be applied to determine exit regions (e.g., regions that straddle a second boundary of the occluded region 312 on the exit side, with respect to the traffic direction through the occluded region 312). In at least some examples, if an entry region and / or exit region is not identified, an alternative control scheme may be employed to operate the vehicle taking into account the known occlusion.

[0047] Environment 320 corresponds to environment 302 at a second time T2, after the initial time T1. As shown, both vehicle 304 and static object 306 remain stationary from the initial time T1 to the second time T2, and thus occlusion region 312 remains the same, i.e., "static." Dynamic object 308 progresses along a trajectory to traverse drivable region 310 between the initial time T1 and the second time T2.

[0048] Further, in some examples, the occlusion inference system of the vehicle 304 may use the map data and / or data from sensors of the vehicle 304 to generate an environment representation 322 of the environment 320 at the second time T2. The environment representation 322 includes representations of the vehicle 304, the static objects 306, the dynamic objects 308, and the drivable area 310.

[0049] The occlusion grid 318 is also represented in the environment representation 322. As time progresses from an initial time T1 to a second time T2, the occlusion inference system of the vehicle 304 may designate occlusion fields previously designated as uncertain as "pseudo-visible." For example, at the initial time T1, the sensor of the vehicle 304 may detect a dynamic object 308 in an entrance region of the visibility region 314. Then, at a second time T2, the sensor of the vehicle 304 may detect the dynamic object 308 reappearing on the opposite side of the occlusion region 312. Based on these observations of static occlusion regions and entrance regions associated with the static occlusion region, the vehicle 304 may assume with a relatively high degree of certainty that no dynamic object is present in the lane of the drivable region 310 occupied by the dynamic object 308 in the occlusion region 312. Such designation may be made, for example, by feature matching or by the use of embeddings to determine that an object that entered a dynamic region is the same as an object that exited. In at least some examples, the dynamics and / or kinematics of the observed entities may be used to determine the pseudo-visible occlusion field. In such an example, if the vehicle enters at a speed of 15 m / s, the occlusion field of the queue in which the vehicle is traveling may be designated as pseudo-visible (based on the length of the field) in sequence from right to left (or traffic direction) at a speed of 15 m / s. For example, the occlusion inference system may assume that the dynamic object 308 passes through the occlusion region 312 without contacting another object present in the occlusion region 312. The occlusion inference system of the vehicle 304 may therefore designate the dynamic field of the occlusion grid 318 through which the dynamic object 308 passed as a pseudo-visible region 324 (shown as a dark grey block) and use this information to control the vehicle 304 accordingly. In at least other examples, a dynamic object entering an occlusion region may "reset" the state of the occlusion field to "uncertain" and may mark the occlusion field as a pseudo-visible region using a process similar to the following:

[0050] In some examples, the occlusion inference system of the vehicle 304 may use a minimum threshold speed and / or a threshold time to designate the occlusion fields of the occlusion grid 318 as pseudo-visible, or alternatively. The threshold speed may correspond to the minimum speed at which a reasonable dynamic object is expected to travel in the environment. For example, the center lane of the drivable area 310 has no dynamic objects passing from the visible area 314 to the occlusion area 312 to use as a reference for pseudo-visibility. In some cases, there may be dynamic objects in the occlusion area that are still relevant to making decisions to control the vehicle 304 and move at a slower speed than the dynamic object 308. To account for such objects, the vehicle may change the designation of the occlusion fields in the occlusion area 312 starting from the location of the entrance area from indeterminate to pseudo-visible over time, using a minimum threshold speed, such as the average speed of a walking pedestrian (e.g., 1.5 m / s). In at least some examples, a different minimum threshold speed may be defined (e.g., increased or decreased based on the distance to the vehicle 304, etc.). Continuing to observe the visibility region 314 from the initial time T1 to the second time T2, the occlusion inference system may change the designation of the occlusion fields from indeterminate to pseudo-visible, from the entry region to the occlusion region 312, unless a dynamic object enters the occlusion region or until all of the occlusion fields of the occlusion region 312 are pseudo-visible. As a non-limiting example, if each occlusion field has a length of 1.5 m, a dynamic object moving at a minimum threshold speed of 1.5 m / s would take 1 s to pass through each occlusion field. In such an example, an occlusion field would be marked as "pseudo-visible" every 1 s, row by row, starting with the right-most occlusion field, through the occlusion region 312, and then to the exit region. Once all occlusion fields are marked as pseudo-visible (or otherwise visualized and described), the vehicle 304 may plan a route across the intersection (including sending an alert to the human driver (e.g., in an advanced driver assistance system) to indicate that it is likely safe to proceed).

[0051] As discussed above, setting the threshold speed to the average speed of a pedestrian (e.g., 1.5 m / s) allows the vehicle 304 sensors to detect dynamic objects moving at a fast pace through the occlusion area in the forward direction. In many cases, objects moving slower than the threshold speed do not affect many of the decisions used to control the vehicle, so the vehicle can treat such objects as "not moving." However, if the vehicle 304 detects a dynamic object moving slower than the threshold speed, the vehicle 304 may alter its trajectory accordingly. Additionally, although a threshold speed of 1.5 m / s is described in various examples herein, any suitable speed may be used as the minimum threshold speed.

[0052] FIG. 4 is an example of identifying a static occlusion region and a dynamic object exiting the occlusion area according to an embodiment of the present disclosure.

[0053] The environment 402 includes a vehicle 404 (e.g., an autonomous vehicle), a first static object 406 (e.g., another vehicle that is stationary or parked), and a second dynamic object 408 (e.g., a pedestrian that has just exited the static object 406 into the drivable area 410; also denoted as "P"). Similar to the drivable area 310 of FIG. 3, the drivable area 410 is associated with a right-to-left travel direction 411. The travel direction 411 may be determined using map data of the environment 402 and / or by sensory data of the environment 402 received from a sensor of the vehicle 404 (e.g., observing which direction the vehicle is traveling along the drivable surface 410), for example, that is analyzed to determine the travel direction 411.

[0054] As shown, the environment 402 may represent an initial time (e.g., T1). At the initial time T1, a static object 406 may obfuscate a portion of the environment 402 from sensors of the vehicle 404, causing an occlusion area 412, shown with light gray shading. The vehicle 404 may be able to sense multiple visibility areas, such as an unoccluded visibility area 414. At the initial time T1, a dynamic object 408 is located in an occlusion area, and thus, the sensors of the vehicle 404 have not yet detected the location or presence of the dynamic object 408. In other words, the vehicle 404 does not see the dynamic object 408 at the initial time T1.

[0055] In some examples, the occlusion inference system of the vehicle 404 may use map data and / or data received from sensors of the vehicle 404 to generate an environment representation 416 of the environment 402 at an initial time T1. The environment representation 416 includes a representation of the vehicle 404, static objects 406, and drivable area 410. Dynamic objects 408 are also included in the environment representation 416 for illustrative purposes, but may not be included in the environment representation generated by the vehicle 404 because they are located in occlusion areas 412 and cannot be sensed by sensors of the vehicle 404.

[0056] 3, the vehicle 404 may generate an occlusion grid 418 that includes multiple occlusion fields that represent discrete areas of the environment 402. The occlusion grid 418 can be generated from map data of the environment, and an occlusion inference system can determine the state of the occlusion fields using data from sensors in the vehicle 404.

[0057] For example, the occlusion inference system of the vehicle 404 may use map data of the environment to generate an occlusion field of occlusion grid 418 that corresponds to drivable area 410. The occlusion inference system of the vehicle 404 may also use one or more sensors to identify a static object 406 and determine that the static object 406 is blocking the view of the one or more sensors from occlusion area 412. Based on this determination, the occlusion inference system may assign an indication of the occlusion field as "uncertain" because it cannot say with certainty whether the occlusion field contained in occlusion area 412 is occupied or not.

[0058] The occlusion reasoning system of the vehicle 404 may also use the map data to determine a travel direction associated with the drivable region 410. In the example 400, the travel direction is moving from right to left. Based on the travel direction associated with the drivable region 410, the occlusion reasoning system of the vehicle can designate the visible region 414 as an entrance region to the occluded region 412.

[0059] The environment 420 corresponds to the environment 402 at a second time T2 after the initial time T1. As shown, both the vehicle 404 and the static object 406 remain stationary from the initial time T1 to the second time T2, and therefore the occlusion area 412 remains the same. Because both the vehicle 404 and the static object 406 (e.g., the object causing the occlusion area 412) remain stationary from the initial time T1 to the second time T2, the occlusion area of ​​the example 400 may be considered a static occlusion area. The dynamic object 408 progresses in a trajectory from behind the static object 406 at the initial time T1 across the drivable area 410 to appear in front of the static object 406 at the second time T2, and is visible to the sensor of the vehicle 404.

[0060] Further, in some examples, the occlusion inference system of the vehicle 404 may use the map data and / or data from sensors of the vehicle 404 to generate an environment representation 422 of the environment 420 at the second time T2. The environment representation 422 includes representations of the vehicle 404, the static objects 406, the dynamic objects 408, and the drivable area 410.

[0061] The occlusion grid 418 is also represented in the environment representation 422. As time progresses from the initial time T1 to the second time T2, the occlusion inference system of the vehicle 404 can designate the occlusion fields previously designated as indeterminate as “pseudo-visible” based on knowledge of the static occlusion regions 412. For example, the occlusion inference system of the vehicle 404 may use a minimum threshold speed, such as the 1.5 m / s speed described above, to designate the occlusion fields of the occlusion grid 418 as pseudo-visible. Using the visibility region 414 and the identified entry regions to the occlusion region 412, the occlusion inference system can begin to change the designation of the occlusion field from indeterminate to pseudo-visible region 424 from right to left. While the occlusion inference system is generating the pseudo-visible region 424 from the initial time T1 to the second time T2, a dynamic object 408 appears from the occlusion region 412 into an area visible to the sensor of the vehicle 404. The occlusion inference system of the vehicle 404 may continue to change the occlusion field designation from uncertain to pseudo-visible based on a minimum threshold speed unless a dynamic object enters the occlusion region or until all occlusion fields within the occlusion region 412 become pseudo-visible.

[0062] By using a minimum threshold speed to change the designation of occlusion fields from indeterminate to pseudo-visible, the occlusion inference system can continue to update these fields over time despite the appearance of previously unseen dynamic objects 408. Additionally, the use of a stationary doorman allows the vehicle's 404 planning system to make decisions about when to release the vehicle 404 through an intersection without slowly creeping through the intersection until all areas of the environment are visible, and then eventually releasing the vehicle through all the way through the intersection. Instead, the vehicle 404 mimics the natural behavior of a human driver by remaining stationary until the occlusion area 412 becomes pseudo-visible, thereby allowing passengers inside the vehicle and humans outside the vehicle to more accurately predict and react to the autonomous vehicle's actions.

[0063] Figure 5A is an example of identifying moving occlusion regions and pseudo-visible regions of moving occlusion regions according to an embodiment of the present disclosure, and Figure 5B is an example of an occlusion grid corresponding to the example shown in Figure 5A.

[0064] The environment 502 includes a vehicle 504 (e.g., an autonomous vehicle) and an object 506 (e.g., another vehicle that is stopped or parked). The vehicle 504 is moving along a trajectory approaching an intersection of a drivable area 508. As shown, the environment 502 may represent an initial time (e.g., T1). At the initial time T1, the object 506 may obfuscate a portion of the environment 502 from a sensor of the vehicle 504, causing an occlusion region 510, shown with a light gray shading and having a left boundary 513(1) and a right boundary 513(2). The vehicle 504 may be able to sense multiple visibility regions, such as an unoccluded visibility region 512.

[0065] The environment 514 is configured to change at a second time T 2A The vehicle 504 and the object 506 at the initial time T 2AAs time passes, vehicle 504 has crossed a portion of drivable region 508 and is near an intersection. Thus, as vehicle 504 moves relative to object 506, occlusion region 510 changes shape and size to become occlusion region 516(A), shown with a dashed line corresponding to the location of occlusion region 510. Occlusion region 510 is shown in environment 514 with left boundary 513(1) and right boundary 513(2). Occlusion region 516(A) is also shown in environment 514 with left boundary 515(1) and right boundary 515(2). Additionally, viewable region 512 changes shape and size to become visible at a second time T 2A In this case, the visible area is 518(A).

[0066] To provide further illustration of the change in occluded and visible areas from an initial time T1 to a second time T2, the environment 520 is 2A A second time T 2BThe environment 520 includes a vehicle 504 and an object 506 at a location in the vicinity of the vehicle 504. The environment 520 also includes a pseudo-visibility area 522(A) corresponding to the difference in size or dimensions between the visibility area 512 and the visibility area 518(A) as the vehicle 504 moves from the initial time T1 to the second time T2A / T2B. The pseudo-visibility area 522(A) is formed by a right boundary 513(2) of the occluded area 510 and a right boundary 515(2) of the occluded area 516(A). In some examples, the vehicle's occlusion inference system can utilize map data and, as described herein, determine a travel direction for the drivable area 508 and designate the visibility area 512 as an entrance area to the occluded area 510 based on the known travel direction. In at least some examples, such designation may be dependent on a travel direction associated with the drivable area. As a non-limiting example, a new occlusion area that occludes a portion of the environment ahead of occlusion region 516(A) (e.g., to the left of occlusion region 510 in this case relative to the direction of travel) may be designated as occluded and not "pseudo-visible." In this example, because at least one of the vehicle 504 and the object 506 creating occlusion region 510 is moving (in this case, vehicle 504), the vehicle's occlusion inference system may further designate occlusion regions 512, 518(A), and / or 518(B) as moving occlusion regions. The occlusion inference system of vehicle 504 may provide information about the moving occlusion regions to a planning system of vehicle 504 for use in controlling vehicle 504, such as whether to stop at an intersection, how long to stop at the intersection, and how fast to accelerate through the intersection.

[0067] In some examples, a portion of the occlusion region 516(A) corresponding to the occlusion region 510 may be considered as a static occlusion region for purposes of determining a pseudo-visible region. That is, the occlusion inference system may designate a portion of the occlusion region 516(B) as a pseudo-visible region 522(B) based on a threshold speed and / or time period between T1 and T2 in addition to the pseudo-visible region 522(A) illustrated in the environment 520. In this example, the pseudo-visible region 522(B) is formed by the boundary 513(2) of the occlusion region 510 and a boundary 523 that progresses in the traffic direction over time from an entrance region of the moving occlusion region. Thus, techniques for evaluating static and moving occlusion regions may be combined to rapidly determine a pseudo-visible region based on captured data. In at least some examples, the determination of a pseudo-visible occlusion region may proceed from the left-most occlusion region (e.g., along the traffic direction) that has a designation of visible or pseudo-visible.

[0068] FIG. 5B is an example 524 of determining occupancy status of the occluded region illustrated in FIG. 5A according to an embodiment of the present disclosure.

[0069] In some examples, the occlusion inference system of the vehicle 504 of FIG. 5A may use map data and / or data from sensors of the vehicle 504 to generate an environment representation 526 of the environment 502 of FIG. 5A at an initial time T1. The environment representation 526 includes a representation of the vehicle 504, the objects 506, and the drivable area 508. Similar to the occlusion grid described above, the vehicle 504 may generate an occlusion grid 528 that includes multiple occlusion fields that represent discrete areas of the environment 502. The occlusion grid 528 can be generated from map data of the environment, and the occlusion inference system can use data from sensors of the vehicle 504 to determine the state of the occlusion fields.

[0070] For example, the occlusion inference system of the vehicle 504 may use map data of the environment to generate occlusion fields of an occlusion grid 528 that correspond to the drivable area 508. The occlusion inference system of the vehicle 504 may also use one or more sensors to identify an object 506 and determine that the object 506 is blocking the one or more sensors from viewing the occluded area 510. The occlusion fields occupied by the object 506 may be colored black to indicate that these fields are occupied by the object 506. Based on this determination, the occlusion inference system may assign the indication of the occlusion fields contained in the occluded area 510 as "occluded" and / or "uncertain" (e.g., with respect to occluded and / or occupied status) because it cannot say with certainty whether the occlusion fields contained in the occluded area 510 are occupied or not.

[0071] In example 524, the vehicle 504 may attempt to cross an intersection. In this manner, the vehicle 504 may determine that an oncoming lane is relevant for determining visibility for safely crossing the intersection in the drivable region 508. The occlusion inference system of the vehicle 504 may also use the map data to determine a direction of travel associated with the drivable region 508. In examples 500 and 524, the direction of travel is moving from right to left. Based on the direction of travel associated with the drivable region 508, the occlusion inference system of the vehicle may designate the visible region 512 as an entrance region to the occluded region 510. In at least some examples, such an entrance region requires a continuous occlusion field (e.g., having at least one adjacent side of each occlusion field in common) from one side of the drivable region 508 to the other side along a boundary between the occluded region and the visible region, the boundary corresponding to the incoming direction of travel. A similar approach may be performed to determine an exit region, except that the boundary between the occluded region and the visible region corresponds to an outgoing direction of travel from the occluded region. If an entry region and an exit region have been identified on either side of the occluded region 510, inference can be made regarding dynamic objects within the occluded region 510. In such an example, a dynamic object traveling along the drivable region 508 is likely to enter, pass through, and exit the occluded region 510 from the entry region along the drivable region 508 via the exit region.

[0072] Further, in some examples, the occlusion inference system of the vehicle 504 may use map data and / or data from sensors of the vehicle 504 to determine the second time T 2A 5A in a virtual reality environment 514. The virtual reality environment 530 may include a representation of the vehicle 504, the objects 506, and the drivable area 508.

[0073] The occlusion grid 528 is also represented in the environment representation 530. The occlusion region 510 is also represented by a left boundary 513(1) and a right boundary 513(2). 2AOnce time has elapsed, the occlusion inference system of the vehicle 504 may designate the occlusion field previously designated as uncertain as “pseudo-visible” based on knowledge of the moving occlusion region. In some examples, the designation of the previously uncertain occlusion field as pseudo-visible may depend on the entrance region and / or the exit region being present from time T1 to time T2, even though the entrance region and the exit region have moved. If the vehicle 504 is no longer able to identify the entrance region and, in at least some examples, the exit region between time T1 and time T2, the vehicle 504 may use other techniques to perform traversal of the environment. In some examples, the occlusion field corresponding to the moving occlusion region may only transition from occluded to pseudo-visible relative to the direction of travel, e.g., from boundary 513(2) toward boundary 515(2).

[0074] For example, time T 2B As shown in the environmental representation 532 of FIG. 1, the object is visible at an initial time T1, but is not visible at time T 2A The occlusion fields of the occlusion grid 528 that were occluded at the initial time T1 are included in the pseudo visibility region 534. To specify the pseudo visibility region 534, the occlusion reasoning system of the vehicle 504 determines whether or not a dynamic object is present in the previously specified entry region at the initial time T1 and whether or not a dynamic object is present in the entry region at the initial time T1 and the entry region at the second time T2. 2A The method leverages knowledge that no dynamic objects entered the entrance region until the vehicle 504 was in the vehicle's path ...

[0075] By specifying the pseudo-visible area 534 as the vehicle approaches an intersection in the drivable area 508, the planning system of the vehicle 504 can make decisions about how to cross the intersection faster and / or with more confidence than conventional techniques that require the vehicle to reach the intersection and slowly move forward until all of the area is visible before crossing the intersection.

[0076] Figure 6 shows a block diagram of an example system 600 for implementing the techniques described herein. In at least one example, the system 600 can include a vehicle 602, which can correspond to the vehicle 112 of Figure 1, the vehicle 202 of Figure 2, the vehicle 304 of Figure 3, the vehicle 404 of Figure 4, and / or the vehicle 504 of Figures 5A and 5B.

[0077] The vehicle 602 may include a vehicle computing device 604, one or more sensor systems 606, one or more emitters 608, one or more communication connections 610, at least one direct connection 612, and one or more drive systems 614.

[0078] The vehicle computing device 604 can include one or more processors 616 and a memory 618 communicatively coupled to the one or more processors 616. In the illustrated example, the vehicle 602 is an autonomous vehicle, however, the vehicle 602 may be any other type of vehicle or robotic platform. In the illustrated example, the memory 618 of the vehicle computing device 604 stores a localization component 620, a perception component 622, one or more maps 624, one or more system controllers 626, an occlusion reasoning component 628, an object trajectory component 630, an occlusion occupancy prediction component 632, a planning component 634, and a trajectory selection component 636. While depicted in FIG. 6 as residing in memory 618 for purposes of explanation, it is contemplated that the localization component 620, the perception component 622, the one or more maps 624, the one or more system controllers 626, the occlusion reasoning component 628, the object trajectory component 630, the occlusion occupancy prediction component 632, the planning component 634, and the trajectory selection component 636 may additionally or alternatively be accessible to the vehicle 602 (e.g., stored on memory separate from the vehicle 602 or otherwise accessible to the vehicle 602).

[0079] In at least one example, the localization component 620 can include functionality to receive data from the sensor system 606 to determine the position and / or attitude of the vehicle 602 (e.g., one or more of x-, y-, z-position, roll, pitch, or yaw). For example, the localization component 620 can include and / or request / receive a map of the environment and continuously determine the position and / or orientation of the autonomous vehicle within the map. In some examples, the localization component 620 can receive image data, LIDAR data, RADAR data, time of flight data, IMU data, GPS data, wheel encoder data, etc., utilizing SLAM (simultaneous localization and mapping), CLAMS (calibration, localization and mapping simultaneously), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc., to accurately determine the position of the autonomous vehicle. In some examples, the localization component 620 can provide data to various components of the vehicle 602 to determine an initial position of the autonomous vehicle to generate a trajectory and / or to determine to retrieve map data from memory, including an occlusion grid, as described herein.

[0080] In some examples, the perception component 622 may include functionality to perform object detection, segmentation, and / or classification. In some examples, the perception component 622 may provide processed sensor data indicative of the presence of an entity in proximity to the vehicle 602 and / or a classification of the entity as a type of entity (e.g., automobile, pedestrian, bicycle, animal, building, tree, road surface, curb, sidewalk, stop light, stop sign, unknown, etc.). In additional or alternative examples, the perception component 622 may provide processed sensor data indicative of one or more characteristics associated with the detected entity (e.g., tracked object) and / or the environment in which the entity is located. In some examples, the characteristics associated with the entity may include, but are not limited to, x-position (global and / or local position), y-position (global and / or local position), z-position (global and / or local position), orientation (e.g., roll, pitch, yaw), type of entity (e.g., classification), velocity of the entity, acceleration of the entity, range (size) of the entity, etc. Characteristics related to the environment include, but are not limited to, the presence of other entities in the environment, the state of other entities in the environment, time of day, day of the week, season, weather, darkness / light indications, etc.

[0081] The memory 618 may further include one or more maps 624 that may be used by the vehicle 602 to navigate within the environment. For purposes of this discussion, a map may be any number of data structures modeled in two, three, or N dimensions that may provide information about the environment, such as topology (such as intersections), streets, mountain ranges, roads, terrain, and the general environment, but is not limited to such. In some examples, the map may include, but is not limited to, the following: texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and / or intensities), reflectance information (e.g., specular reflectance information, retroreflectance information, BRDF information, BSSRDF information, etc.). In one example, the map may include a three-dimensional mesh of the environment. In some examples, the map can be stored in a tiled format, with each tile of the map representing a discrete portion of the environment, and can be loaded into the working memory as needed. In at least one example, the one or more maps 624 can include at least one map (e.g., an image and / or a mesh). In some examples, the vehicle 602 can be controlled based at least in part on the map 624. That is, the map 624 can be used in association with the localization component 620, the perception component 622, the occlusion reasoning component 628, and / or the planning component 634 to determine a position of the vehicle 602, identify objects in the environment, and / or generate a path and / or trajectory to navigate through the environment.

[0082] In some examples, one or more maps 624 may be stored on a remote computing device (such as computing device 640) accessible via network 638. In some examples, multiple maps 624 may be stored, for example, based on characteristics (e.g., type of entity, time of day, day of the week, season of the year, etc.). Storing multiple maps 624 may have similar memory requirements but may improve the speed at which data in the maps can be accessed.

[0083] In some examples, the one or more maps 624 may store occlusion grids associated with individual locations within the environment. For example, as the vehicle 602 traverses the environment and a map representing an area proximate to the vehicle 602 is loaded into memory, one or more occlusion grids associated with the locations may also be loaded into memory. In some examples, the occlusion grids may be dynamically generated based on the map data.

[0084] Typically, the occlusion inference component 628 can determine occlusion areas of the environment due to one or various objects in the environment based on data provided by the sensor system 606. The occlusion inference component may correspond to the occlusion inference component 208 of FIG. 2 in some examples. In some examples, the occlusion inference component 628 can provide occlusion information generated by the occlusion inference component 628 to the planning component 634 to determine when and / or how to control the vehicle 602 to traverse the environment. As described herein, the occlusion inference component 628 can receive LIDAR data, image data, map data, etc. to determine occlusion-related information in the environment. An example of generating and / or using an occlusion grid is described in U.S. Patent Application Publication No. 2013 / 0133994, filed on June 18, 2018. U.S. Patent Application Publication No. 2013 / 0133994 is incorporated by reference in its entirety herein.

[0085] The object trajectory component 630 included in the occlusion inference component 628 determines trajectories of objects in the environment, for example, in real time. The trajectories can be based on the current location of the object as well as evaluating the object over time to determine the object's possible locations based on the object's classification, location, velocity, acceleration, sensor uncertainty, etc. The object trajectory component 630 may be configured to evaluate trajectories of various objects substantially simultaneously. For example, the object trajectory component 630 may evaluate one or more trajectories of an object that causes a corresponding occlusion in the environment, and / or one or more trajectories of a dynamic object that may pass through an occlusion area in the environment, etc. An example of generating one or more trajectories is described in U.S. Patent Application Publication No. 2013 / 0133994, filed on October 4, 2018. U.S. Patent Application Publication No. 2013 / 0133994 is incorporated by reference in its entirety herein.

[0086] The occlusion occupancy prediction component 632 can determine the likelihood that a dynamic object is not present in an occluded area of ​​the environment according to any one or more of the examples provided herein. For example, the occlusion occupancy prediction component 632 can project the occlusion region onto a map 624 of the environment surrounding the vehicle to determine which "known" portions of the environment are occluded by the object. In some examples, the occlusion occupancy prediction component 632 can use ray casting and / or projection techniques to determine the occlusion region associated with the object in addition to information about the particular sensor modality (e.g., range, field of view, etc.). Additionally, the occlusion occupancy prediction component 632 can determine entry and / or exit regions to the occlusion region based on the map 624 and determine whether the occlusion region corresponds to a static occlusion region (e.g., both the vehicle and the occluding object are stationary) or a moving occlusion region (e.g., one or both of the vehicle and the occluding object are moving). In some examples, the occlusion occupancy prediction component 632 determines whether a dynamic object is located within an occlusion area based on observations of the environment (e.g., by the sensor system 606) and provides this information to the planning component 634 for use in controlling the vehicle 602.

[0087] Typically, the planning component 634 can determine a path for the vehicle 602 to follow to traverse an environment. For example, the planning component 634 can determine various paths and trajectories, and various levels of detail. For example, the planning component 634 can determine a path to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this discussion, the path can be a sequence of waypoints for traveling between the two locations. As non-limiting examples, the waypoints include streets, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the planning component 634 can generate instructions for guiding the autonomous vehicle along at least a portion of a path from the first location to the second location. In at least one example, the planning component 634 can determine how to guide the autonomous vehicle from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instructions can be a trajectory, or a portion of a trajectory. In some examples, the multiple trajectories can be generated substantially simultaneously (e.g., within technical tolerances) in accordance with a receding horizon technique, and one of the multiple trajectories is selected for vehicle 602 to navigate.

[0088] In some examples, the planning component 634 can generate one or more trajectories for the vehicle 602 based at least in part on knowledge of occlusion areas in the surrounding environment and / or whether the dynamic object is likely not within an occlusion area, as described herein. In some examples, the planning component 634 can evaluate one or more trajectories for the vehicle 602 using temporal logic, such as linear temporal logic and / or signal temporal logic.

[0089] In at least one example, the vehicle computing device 604 can include one or more system controllers 626 that can be configured to control the steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 602. These system controllers 626 can communicate with and / or control corresponding systems of the drive system 614 and / or other components of the vehicle 602.

[0090] As can be appreciated, the components described herein (e.g., localization component 620, perception component 622, one or more maps 624, one or more system controllers 626, occlusion reasoning component 628, object trajectory component 630, occlusion occupancy prediction component 632, planning component 634, and trajectory selection component 636) are described separately for purposes of illustration. However, operations performed by the various components may be combined or performed by other components. As an example, object positioning functions may be performed by the perception component 622 (e.g., rather than the object trajectory component 630) to reduce the amount of data transferred by the system.

[0091] In at least one example, the sensor system 606 can include LIDAR sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (RGB, infrared, intensity, depth, time of flight, etc.), microphones, wheel encoders, environmental sensors (temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), and the like. The sensor system 606 can include various examples of each of these or other types of sensors. For example, the LIDAR sensors can include individual LIDAR sensors located at corners, front, rear, sides, and / or top of the vehicle 602. As another example, the camera sensors can include various cameras located at various locations on the exterior and / or interior of the vehicle 602. The sensor system 606 can provide input to the vehicle computing device 604. Additionally or alternatively, the sensor system 606 may transmit sensor data via one or more networks 638 to one or more computing devices at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0092] The vehicle 602 may also include one or more emitters 608 for emitting light and / or sound, as described above. The emitters 608 in this example include internal audio and video emitters for communicating with passengers of the vehicle 602. By way of example and not limitation, the internal emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The emitters 608 in this example also include external emitters. By way of example and not limitation, the external emitters in this example include lighting (e.g., indicator lighting, signs, light arrays, etc.) for announcing direction of travel or other indicators of the vehicle's operation, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audio communication with pedestrians or other nearby vehicles, one or more of which may include acoustic beam steering technology.

[0093] Vehicle 602 may also include one or more communication connections 610 that enable communication between vehicle 602 and one or more other local or remote computing devices. For example, communication connections 610 may facilitate communication with other local computing devices on vehicle 602 and / or drive system 614. Communication connections 610 may also provide for the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Communication connections 610 may also enable vehicle 602 to communicate with remote teleoperated computing devices or other remote services.

[0094] The communications connection 610 may include physical and / or logical interfaces for connecting the vehicle computing device 604 to another computing device or network, such as network 638. For example, the communications connection 610 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standards, short-range radio frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing devices.

[0095] In at least one example, the vehicle 602 can include one or more drive systems 614. In some examples, the vehicle 602 can have a single drive system 614. In at least one example, when the vehicle 602 has multiple drive systems 614, the individual drive systems 614 can be located at opposite ends of the vehicle 602 (e.g., front and rear, etc.). In at least one example, the drive system 614 can include one or more sensor systems for detecting a state of the drive system 614 and / or a state surrounding the vehicle 602. By way of example and not limitation, the sensor systems can include one or more wheel encoders (e.g., rotary encoders) that sense the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) that measure the orientation and acceleration of the drive module, cameras or other image sensors, ultrasonic sensors that acoustically detect objects surrounding the drive system, LIDAR sensors, RADAR sensors, etc. Some sensors, such as wheel encoders, can be unique to the drive system 614. In some cases, sensor systems on drive system 614 may overlap or complement corresponding systems on vehicle 602 (e.g., sensor system 606).

[0096] The drive system 614 can include many vehicle systems, such as a high voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use in other vehicle systems, a steering system including a steering motor and a steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the perimeter of the vehicle), and one or more systems (e.g., cooling systems, safety systems, on-board charging systems, electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, the drive system 614 can include a drive system controller that can receive and pre-process data from the sensor systems and control the operation of various vehicle systems. In some examples, the drive system controller can include one or more processors and a memory communicatively coupled to the one or more processors. The memory can store one or more components for performing various functions of the drive system 614. Additionally, drive system 614 also includes one or more communications connections that allow each drive system to communicate with one or more other local or remote computing devices.

[0097] In at least one example, the direct connection 612 can provide a physical interface for coupling one or more drive systems 614 with the body of the vehicle 602. For example, the direct connection 612 can enable the transfer of energy, fluid, air, data, etc. between the drive systems 614 and the vehicle. In some examples, the direct connection 612 can also releasably secure the drive systems 614 to the body of the vehicle 602.

[0098] In at least one example, the localization component 620, the perception component 622, the one or more maps 624, the one or more system controllers 626, the occlusion inference component 628, the object trajectory component 630, the occlusion occupancy prediction component 632, the planning component 634, and the trajectory selection component 636 can process the sensor data as described above and can transmit their respective outputs over one or more networks 638 to one or more computing devices 640. In at least one example, the localization component 620, the perception component 622, the one or more maps 624, the one or more system controllers 626, the occlusion inference component 628, the object trajectory component 630, the occlusion occupancy prediction component 632, the planning component 634, and the trajectory selection component 636 can transmit their respective outputs to the one or more computing devices 640 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0099] In some examples, the vehicle 602 may transmit sensor data over the network 638 to one or more computing devices 640. In some examples, the vehicle 602 may transmit raw sensor data to the computing device 640. In other examples, the vehicle 602 may transmit processed sensor data and / or representations of the sensor data to the computing device 640. In some examples, the vehicle 602 may transmit sensor data at a particular frequency, after a predetermined period of time, in near real-time, etc. In some examples, the vehicle 602 may transmit sensor data (raw or processed) to the computing device 640 as one or more log files.

[0100] The computing device 640 may include a processor 642 and a memory 644 that stores a training component 646 .

[0101] In some examples, the training component 646 can include functionality to train one or more models to determine object positions and / or trajectories, determine occlusion areas, determine moving and / or static doormen, determine dynamic object behavior when in an occlusion area (e.g., based on a particular location of an occlusion area), etc. In some examples, the training component 646 can communicate information generated by the one or more models to the vehicle computing device 604 to modify how to control the vehicle 602 in response to different situations.

[0102] For example, some or all aspects of the components discussed herein may include any model, algorithm, and / or machine learning algorithm. For example, in some examples, the components of memory 644 (and memory 618 described above) may be implemented as neural networks. In some examples, the training component 646 may utilize a neural network to generate and / or execute one or more models for improving various aspects of occlusion reasoning for use in trajectory planning.

[0103] As described herein, an exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to generate an output. Each layer of a neural network may comprise another neural network, and may comprise any number of layers (convolutional or not). As can be understood in the context of this disclosure, a neural network may utilize machine learning, which may refer to a broad class of such algorithms in which an output is generated based on learned parameters.

[0104] Although discussed in the context of neural networks, any type of machine learning compatible with the present disclosure may be used. For example, machine learning algorithms may include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally weighted scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative binary tree 3 (ID3), chi-squared automated interaction detection (CH2), etc.). AID), decision stumps, conditional decision trees), Bayesian algorithms (e.g., Naïve Bayes, Gaussian Naïve Bayes, Multinomial Naïve Bayes, Average Ordinary Dependent Classifier (AODE), Bayesian Belief Networks (BNN), Bayesian Networks), clustering algorithms (e.g., k-means, k-medians, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning Algorithms (e.g., Perceptron, Backpropagation, Hopfield Networks, Radial Basis Function Network (RBFN), etc.), deep learning algorithms (Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN), Convolutional Neural Networks (CNN), Stacked Auto-Encoders, etc.), dimensionality reduction algorithms (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA), etc.), ensemble algorithms (e.g.,These may include, but are not limited to, Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, Stacked Generalization (Blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees (GBRT), Random Forest, SVM (Support Vector Machine), supervised learning, unsupervised learning, semi-supervised learning, etc.

[0105] Additional examples of architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, and PointNet.

[0106] The processor 616 of the vehicle 602 and the processor 642 of the computing device 640 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processors 616 and 642 may comprise one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), or other devices or portions of devices that process electronic data and convert the electronic data into registers and / or other electronic data that can be stored in memory. In some examples, integrated circuits (such as ASICs), gate arrays (such as FPGAs), and other hardware devices may also be considered processors so long as they are configured to implement coded instructions.

[0107] The memories 618 and 644 are exemplary non-transitory computer-readable media. The memories 618 and 644 can store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functions attributed to the various systems described herein. In various implementations, the memories can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or other types of memory capable of storing information. The architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, but those shown in the accompanying drawings are merely examples relevant to the discussion herein.

[0108] 6 is illustrated as a distributed system, it should be noted that in alternative examples, components of the vehicle 602 may be associated with the computing device 640 and / or components of the computing device 640 may be associated with the vehicle 602. That is, the vehicle 602 may perform one or more of the functions associated with the computing device 640 and vice versa. Additionally, aspects of the occlusion reasoning component 628 and / or the planning component 634 may be performed on any of the devices discussed herein.

[0109] FIG. 7 illustrates an example process 700 for capturing sensor data, determining an occlusion region caused by an object in an environment, determining a visibility region proximate the occlusion region, determining whether a dynamic object is located in the visibility region, and if so, determining whether the dynamic object enters the occlusion region, and controlling an autonomous vehicle based on the determination, in accordance with an embodiment of the present disclosure. For example, some or all of process 700 may be performed by one or more components of FIG. 6 as described herein. For example, some or all of process 700 may be performed by vehicle computing device 604. Additionally, any of the operations described in example process 700 may be performed in parallel in a different order than depicted in process 700, any of the operations of process 700 shown may be omitted, and / or process may be combined with any of the actions

[0110] In operation 702, the process may include controlling the autonomous vehicle to traverse an environment. For example, operation 702 may include controlling the autonomous vehicle to follow one or more trajectories between waypoints to reach a destination.

[0111] At operation 704, the process may include capturing sensor data of the environment using sensors on the autonomous vehicle. In some examples, operation 704 may include capturing RIDAR data, image data, RADAR data, time of flight data, etc. of the environment.

[0112] In operation 706, the process may include determining occlusion regions due to objects in the environment. For example, an occlusion grid may be applied to map data of the environment surrounding the vehicle. Sensor data may then be used to detect objects in the environment and applied to determine an occlusion state of the occlusion fields of the occlusion grid at the location of the objects. Occlusion fields that are obfuscated from the sensors by objects may be classified as indeterminate because the vehicle may not know whether such fields are occupied or unoccupied by other objects in the environment.

[0113] In operation 708, the process may include determining visibility regions adjacent to occlusion regions of the environment. For example, an occlusion grid may be used to determine which occlusion fields are both unoccluded and unoccupied to determine that one or more areas of the environment are visible. Map data may also be used to determine traffic directions (e.g., vehicular traffic, pedestrian traffic, bicycle traffic, etc.) that pass through the visibility region and assign portions of the visibility region as entrance regions to the occlusion region based on the traffic directions. As described herein, in some cases, the extent of the visibility region and the extent of the occlusion region must correspond. That is, an entrance region may be defined as a visibility region that spans all entrances to an occlusion region (e.g., determining that one or more occlusion fields share at least a common side from a first side of the drivable region to an opposing side of the drivable region perpendicular to the traffic direction). In some examples, a visibility region that does not span the width of an occlusion region may not be considered an entrance region, and operations described herein may not be used to determine a pseudo visibility region. In at least some examples, a similar approach may be applied to determine an exit region (e.g., the boundary of the side of the occlusion region that is furthest and / or opposite from the entrance region along the traffic direction). In at least some examples, if an exit region is not identified, the operations described herein may not be used to determine the pseudo-visibility region.

[0114] Operation 710 may include determining whether a dynamic object is located in the visibility region. In some examples, the dynamic object may be detected by a sensor of the vehicle, as described above. If the dynamic object is determined to be located in the visibility region (and the occlusion region is within a threshold distance of the vehicle), then at 712 the process may include determining whether the dynamic object enters the occlusion region. For example, the vehicle may maintain a current trajectory (e.g., maintain a current speed, maintain a current acceleration / deceleration rate, maintain a stopped position, etc.) from an initial time that the dynamic object is detected until a second time after the initial time that the dynamic object enters the occlusion region. In at least some examples, additional information about the vehicle, such as, but not limited to, features, embeddings, etc., may be stored.

[0115] At operation 714, if it is determined that the dynamic object has entered an occlusion region, the method may include controlling the autonomous vehicle to wait a threshold time before the dynamic object re-emerges. In some examples, the threshold time may be based on a minimum threshold speed associated with the number of occlusion regions and / or occlusion fields. The minimum threshold speed may be based, for example, on a speed at which the dynamic object is observed traveling in the visibility region. Controlling the vehicle to wait the threshold time may include maintaining the current trajectory of the vehicle, as described above. If the dynamic object re-emerges, at least a portion of the area of ​​the occlusion grid that was previously undetermined may be considered a pseudo-visibility area based on an estimate of where the dynamic object passed through the occlusion region. Upon a determination that the dynamic object has re-emerged from the occlusion region, the vehicle may be controlled to continue traversing the environment. Additionally, if the dynamic object does not re-emerge from the occlusion region, the trajectory of the vehicle may be maintained and / or modified accordingly, such as creeping along the drivable region to gain additional visibility area of ​​the environment. In at least some examples, the observation of the dynamic object entering an occlusion region may cause a state associated with all occlusion fields within the occlusion region to be “undetermined.” In such an example, flow may proceed to 718.

[0116] If, at operation 716, it is determined that the dynamic object will not enter an occluded region, the process may include controlling the autonomous vehicle based on other information. For example, the autonomous vehicle may be controlled in response to instructions from a planning system to avoid objects (e.g., dynamic objects) along a path to reach a destination of the vehicle, creeping to gain additional visibility of the environment via sensors of the vehicle, etc. In some cases, if it is determined that the dynamic object will not enter an occluded region, the vehicle may be controlled as described with reference to operation 718.

[0117] In operation 718, if it is determined that there is no dynamic object located in the visibility region, the method may include controlling the autonomous vehicle to wait a threshold time to determine that the occlusion region is pseudo-visible. Similar to the discussion above, controlling the autonomous vehicle to wait may include maintaining a current trajectory (e.g., maintaining a current speed, maintaining a current acceleration / deceleration rate, maintaining a stopped position, etc.) for the threshold time. In this case, the threshold time may be based on an average pedestrian speed of 1.5 m / s, but any suitable threshold speed may be used. Once the threshold time has elapsed, the vehicle may change the designation of the uncertain occlusion fields of the occlusion grid to pseudo-visible, starting with the entry region of the occlusion grid to the occlusion region. If it is determined that all (or a sufficient number) of the occlusion fields of the occlusion grid have been made pseudo-visible without a dynamic object emerging from the occlusion region, the vehicle may be controlled to continue the route to the destination. Also, if a dynamic object emerges from the occlusion region, control may be performed such as to avoid the dynamic object.

[0118] FIG. 8 illustrates an example process 800 for determining a first occlusion region due to an object and a visibility region proximate the occlusion region at a first time, determining that the visibility region includes an entrance region to (and / or an exit region from) the occlusion region, determining a second occlusion region corresponding to at least a portion of the first visibility region at a second time after the first time, determining whether the portion is free of a dynamic object, and controlling an autonomous vehicle based on the determination, according to an embodiment of the present disclosure. For example, some or all of the process 800 may be performed by one or more components of FIG. 8 as described herein. For example, some or all of the process 800 may be performed by the vehicle computing device 604. Additionally, any of the operations described in the example process 800 may be performed in parallel, in a different order than that shown in the process 800, any of the operations of the illustrated process 800 may be omitted, and / or may be combined with any of the operations of the process 700.

[0119] In operation 802, the process may include determining a first occlusion region caused by an object in the environment at a first time and a visibility region proximate to the occlusion region. In at least some examples, such occlusion region may be based at least in part on the action to be performed (e.g., crossing an intersection), the expected travel speed and safety margin, geolocation, etc. For example, an occlusion grid may be applied to map data of the vehicle's surroundings. Sensor data may also be used to detect objects in the environment and determine the occlusion state of the occlusion field of the occlusion grid at the object's location. Occlusion fields obscured from the sensor by objects may be classified as occluded and / or uncertain, while occlusion fields that are unoccupied and unobfuscated may be considered visible.

[0120] At operation 804, the process may include determining that the visibility region includes an entrance region to (and / or an exit region from) the occluded region based at least in part on map data of the environment. In some examples, the entrance region may be based on determining a traffic direction (e.g., vehicular traffic, pedestrian traffic, bicycle traffic, etc.) relative to the occluded region.

[0121] In operation 806, the process may include determining a second occlusion region corresponding to at least a portion of the first visibility region at a second time after the first time. In some examples, operation 806 is based on a concept of moving occlusion regions, where one or both of the vehicle and the object causing the occlusion region are moving. As time passes (e.g., from the first time to the second time) and the vehicle and / or object moves through the environment, the area and / or dimensions of the occlusion region may also change. In at least some examples, additional information regarding the previously visible region may be determined or otherwise stored, such as on which side of the previously visible region the previously visible region was in relation to the traffic direction. Operation 806 may also include determining whether an entrance region and / or an exit region exists in the second occlusion region. If an entrance region and / or an exit region does not exist in the second occlusion region, the second occlusion region may not be used.

[0122] At operation 808, the process can include determining whether a dynamic object is not present in the portion of the first visibility region that is occluded at the second time. For example, an entrance area can be observed from a first time to a second time, and a determination can be made whether a dynamic object has entered the portion based on knowledge of the traffic direction. In some cases, at the first time, the portion may include a dynamic object.

[0123] If no dynamic object is determined to be present in the portion, operation 810 can include determining that at least a portion of the second occlusion region is pseudo-visible. For example, based on knowledge that a dynamic object is most likely to enter the occlusion region in the travel direction through the entrance region, operation 810 can assume that no dynamic object is present in the portion if no dynamic object is present in the entrance region from the first time to the second time.

[0124] In operation 812, the process may include controlling the autonomous vehicle based on at least the portion being pseudo-visible. Operation 812 may also include determining a state of the occlusion field other than that analyzed for pseudo-visibility to control the vehicle. In some examples, determining that at least a portion of the occlusion area is unoccupied may allow the autonomous vehicle to perform an action, such as continuing along a trajectory, more quickly than requiring all of the area to be visible and clear to the vehicle's sensors before performing such an action.

[0125] If it is determined that the portion does not include a dynamic object, then in operation 814, the process may include controlling the autonomous vehicle based on the portion being occupied by a dynamic object. Operation 814 may also include determining a state of the occlusion field other than that analyzed for pseudo-visibility to control the vehicle. For example, the vehicle may maintain its current trajectory (e.g., maintain a current speed, maintain a current acceleration / deceleration rate, maintain a stopped position, etc.) until the dynamic object reappears, or take other measures to safely traverse the environment with the knowledge that the dynamic object is not visible by the vehicle's sensors.

[0126] Examples A. A system comprising: one or more processors; and one or more non-transitory computer readable media storing instructions executable by the one or more processors that, when executed, cause the system to perform operations comprising: receiving sensor data of an environment captured by a sensor on a vehicle; detecting an object in the environment based at least in part on the sensor data; determining an occlusion region in the environment associated with the object based at least in part on the object and the sensor data; determining a visibility region of the environment adjacent to the occlusion region based at least in part on the sensor data; determining that the visibility region represents an entrance region to the occlusion region based at least in part on map data representing the environment; and controlling the vehicle based at least in part on a probability that a dynamic object is not present in at least a portion of the occlusion region being equal to or greater than a threshold based at least in part on the sensor data and a threshold speed associated with the occlusion region, and determining that a dynamic object is not present in at least a portion of the occlusion region.

[0127] B. The system of paragraph A, wherein a size of the portion of the occlusion region in which no dynamic object is determined to be present is based at least in part on a time period associated with capturing the sensor data.

[0128] C. The system of paragraph A or B, further comprising: the occlusion area is a first occlusion area at a first time; and the operation is determining a second occlusion area associated with a second time after the first time, the second occlusion area corresponding to at least a portion of the visible area; and determining that a probability that a dynamic object is not present in at least a portion of the second occlusion area is equal to or greater than a threshold value is based on determining that a dynamic object is not present in at least a portion of the first occlusion area; and controlling the vehicle is further based on determining that a probability that a dynamic object is not present in the second occlusion area is equal to or greater than a threshold value.

[0129] D. The system of paragraph C, wherein the second occlusion area is based at least in part on a first movement of the vehicle between the first time and the second time, or the second occlusion area is based at least in part on a second movement of the object between the first time and the second time.

[0130] E. Any of the systems of paragraphs A through E, wherein a first extent of the blocked region corresponds to a second extent of the entrance region, and the second extent of the entrance region corresponds to a drivable area associated with the blocked region, such that each portion of the blocked region is associated with a corresponding each portion of the entrance region.

[0131] F. The system of paragraph E, wherein the operations further include determining a drivable area and a heading associated with the drivable area based at least in part on map data representative of the environment.

[0132] G. A method comprising: capturing first sensor data with a sensor; determining a drivable area of ​​the environment based at least in part on the first sensor data; determining an occluded area of ​​the environment associated with the drivable area; determining a visible area of ​​the environment, where the visible area is proximate to the occluded area and where the visible area corresponds to an entrance area to the occluded area; determining a probability that at least a portion of the occluded area is absent or equal to a threshold based at least in part on capturing second sensor data over a period of time; and controlling a vehicle based at least in part on the probability that at least a portion of the occluded area is absent or equal to a threshold.

[0133] H. The method of paragraph G, comprising: determining that the visibility area is a first visibility area and that an object is visible in the entrance area at a first time; determining that at least a portion of the object is present in the occluded area at a second time after the first time; and determining that the object is present in the second visibility area at a third time after the second time, wherein the second visibility area is an exit area from the occluded area; and determining that a probability that a dynamic object is not present in at least a portion of the occluded area is equal to or greater than a threshold based on determining that the object is present in the exit area.

[0134] I. The method of paragraph H, further comprising determining that a probability that no dynamic object is present in the additional portion of the occlusion region is greater than or equal to a threshold based at least in part on a threshold velocity or threshold time associated with the occlusion region.

[0135] J. Any of the methods of paragraphs G through I, further comprising determining a traffic direction, wherein the entrance area corresponds to the traffic direction entering the blocked area and the width of the traffic lanes of the drivable area.

[0136] K. The method of paragraph J, wherein determining the occlusion area comprises identifying an object in the environment that occludes at least a portion of map data associated with the environment, the method further comprising: using the map data to discretize the occlusion area into one or more fields corresponding to drivable areas; and determining, for the one or more fields, that a probability that no dynamic object is present in at least a portion of the occlusion area is equal to or greater than a threshold.

[0137] L. Any of the methods of paragraphs G to K, wherein one or more of a vehicle or occluding object is moving and at least a portion of the occluded area corresponds to an area that is included in the visible area at a first time and is included in the occluded area at a second time.

[0138] M. Any of the methods of paragraphs G to L, wherein determining that the probability that at least a portion of the occlusion region is free of a dynamic object is greater than or equal to a threshold value is based at least in part on a threshold velocity or threshold time associated with the occlusion region.

[0139] N. A non-transitory computer-readable medium having instructions stored thereon that, when executed, cause one or more processors to perform operations including capturing first sensor data with a sensor; determining a drivable area of ​​the environment based at least in part on the first sensor data; determining an occluded area of ​​the environment associated with the drivable area; determining a visible area of ​​the environment, where the visible area is proximate to the occluded area and where the visible area corresponds to an entrance area to the occluded area; determining a probability that at least a portion of the occluded area is free of dynamic objects based at least in part on capturing second sensor data over a period of time; and controlling a vehicle based at least in part on the probability.

[0140] O. The non-transitory computer-readable medium of paragraph N, wherein the visibility area is a first visibility area, the operations comprising: determining that an object is visible in the entrance area at a first time; determining that at least a portion of the object is present in the occlusion area at a second time after the first time; and determining that the object is present in the second visibility area at a third time after the second time, wherein the second visibility area is an exit area from the occlusion area; and based on determining that the object is present in the exit area, determining a probability that at least a portion of the occlusion area is free of the dynamic object, corresponding to a path of the object through the occlusion area.

[0141] P. The non-transient computer-readable medium of paragraph N or paragraph O, wherein the occluded and visible regions are discretized into one or more occluded fields, and the entrance region comprises a continuous occluded field common to at least one adjacent side of each occluded field from one side to the opposite side along a boundary between the occluded and visible regions, the boundary corresponding to a direction of travel.

[0142] Q. The non-transitory computer-readable medium of any of paragraphs N to P, wherein one or more of the vehicles or occluding objects are moving and at least a portion of the occluded region corresponds to an area that is in the visible region at a first time and that is in the occluded region at a second time.

[0143] R. The non-transitory computer-readable medium of paragraph N, wherein determining the probability that a dynamic object is not present in at least a portion of the occlusion region is based at least in part on a threshold speed or threshold time associated with the occlusion region and corresponds to a direction of travel through the occlusion region.

[0144] S. The non-transitory computer readable medium of any of paragraphs N through R, wherein the sensor data comprises one or more of LIDAR sensor data, RADAR sensor data, image sensor data, or time of flight sensor data.

[0145] T. The non-transitory computer-readable medium of any of paragraphs N to S, wherein the operations further include localizing to a map based on the sensor data, determining an area to be considered an occlusion area based on the map, and discretizing the area into a plurality of occlusion fields.

[0146] Although the example sections above are described with respect to one particular implementation, it should be understood that in the context of this specification, the contents of the example sections may also be implemented via methods, apparatus, systems, computer-readable media, and / or other implementations.

[0147] conclusion Having described one or more examples of the technology described herein, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0148] In the description of the examples, reference is made to the accompanying drawings, which form a part of this specification, illustrating specific examples of the claimed subject matter. It is understood that other examples can be used and that modifications or substitutions, such as structural changes, can be made. Such examples, modifications or substitutions do not necessarily depart from the intended scope of the claimed subject matter. Although the procedures herein may be presented in a certain order, in some cases the order can be changed such that certain inputs are provided at different times or in a different order without changing the functionality of the described systems and methods. It is also possible to perform the disclosed procedures in a different order. Furthermore, the various calculations described herein do not have to be performed in the order disclosed, and other examples using alternative orders of calculations can be easily implemented. In addition to changing the order, the calculations can also be decomposed into sub-calculations with the same results.

Claims

1. 1. A system comprising: one or more processors; one or more non-transitory computer-readable media storing instructions executable by the one or more processors; The instructions, when executed, cause the system to perform an action; The operation includes: Capturing first sensor data with a sensor; determining a drivable area of ​​an environment based at least in part on the first sensor data; and determining an occluded area in the environment associated with the drivable area; determining a visibility region of the environment, the visibility region being proximate to the occluded region; determining that the visibility region corresponds to an entrance region to the occluded region; determining, based at least in part on capturing second sensor data over a period of time, that a probability that at least a first portion of the occlusion region is free of a dynamic object is equal to or exceeds a threshold; determining that an object is visible within the entrance area at a first time during the time period; controlling the vehicle based at least in part on the probability that the dynamic object is absent in at least the first portion of the occlusion region being equal to or greater than the threshold or the object being visible in the entrance region; A system equipped with.

2. the visibility region is a first visibility region, and the action is determining that at least a portion of the object is within the occlusion region at a second time in the period after the first time; determining that the object is present in a second visibility region at a third time in the period after the second time, the second visibility region being an exit region from the occlusion region; and determining, based on determining that the object is present within the exit region, that the probability that the at least the portion of the dynamic object is absent from the occlusion region is equal to or greater than the threshold; The system of claim 1 further comprising:

3. 3. The system of claim 1 or 2, wherein the probability is a first probability, and the action further comprises determining that a second probability that the dynamic object is not present in a second portion of the occluded area that is different from the first portion of the occluded area is equal to or greater than the threshold based at least in part on a threshold speed or a threshold time associated with the occluded area.

4. 4. The system of claim 1, wherein the operation further comprises determining a direction of travel, and the entrance area corresponds to the direction of travel entering the blocked area and a width of a traffic lane of the drivable area.

5. determining the occlusion region comprises identifying an object within the environment that occludes at least a portion of map data associated with the environment; The operation, using the map data to discretize the occluded region into one or more fields corresponding to the drivable region; determining, for the one or more fields, that the probability that the dynamic object is not present in at least the first portion of the occlusion region is greater than or equal to the threshold; The system of claim 1 , further comprising:

6. 6. The system of claim 1, wherein one or more of the vehicles or occluding objects are moving and the at least the first portion of the occluded region corresponds to an area comprised in the visibility region at a first time and comprised in the occluded region at a second time.

7. 1. A method comprising: Capturing first sensor data with a sensor; determining a drivable area of ​​an environment based at least in part on the first sensor data; and determining a first occlusion region at a first time in the environment associated with the drivable region; determining a visibility region of the environment, the visibility region being proximate to the first occlusion region; determining that the visibility region corresponds to an entrance region to the first occluded region; determining a probability that at least a portion of the first occlusion region is free of dynamic objects based at least in part on capturing second sensor data over a period of time; controlling the vehicle based at least in part on said probability; determining a second occlusion area associated with a second time after the first time, the second occlusion area corresponding to at least a portion of the visibility area; determining, based at least in part on determining that the dynamic object is absent from the at least the portion of the first occlusion region, that a probability that the dynamic object is absent from the at least the portion of the second occlusion region is equal to or exceeds a threshold; Equipped with and controlling the vehicle is further based on determining that the probability that the dynamic object is not present in the second occlusion region is greater than or equal to the threshold. method.

8. 8. The method of claim 7, wherein determining the probability that the dynamic object is not present in at least the portion of the occlusion region is based at least in part on a threshold speed or a threshold time associated with the occlusion region and corresponds to a direction of travel through the occlusion region.

9. the second occlusion area is based at least in part on a first movement of the vehicle between the first time and the second time. Or, the second occlusion region is based at least in part on a second movement of the dynamic object between the first time and the second time. The method of claim 7, wherein the at least one of

10. 10. The method of claim 7, wherein the occlusion region and the visibility region are discretized in one or more occlusion fields, and the entrance region comprises a continuous occlusion field having a side adjacent to at least one of the common respective occlusion fields from one side to the other along a boundary between the occlusion region and the visibility region, the boundary corresponding to an incoming travel direction.

11. a first extent of the occlusion region corresponds to a second extent of the entrance region such that individual portions of the occlusion region are associated with corresponding individual portions of the entrance region; 11. The method of claim 7, wherein the second extent of the entrance region corresponds to an area of ​​the drivable region associated with the occluded region.

12. the first sensor data and the second sensor data comprise one or more of LIDAR sensor data, RADAR sensor data, image sensor data, or time of flight sensor data; 12. The method of claim 7, 10, or 11, wherein a size of the portion of the occlusion region in which the dynamic object is determined to be absent is based at least in part on the time period associated with capturing the second sensor data.

13. localizing a map based on the first sensor data; determining, based on said map, that a region is to be considered as said occluded region; discretizing the region into a number of occlusion fields; 13. The method of any one of claims 7, 10 to 12, further comprising:

14. A non-transitory computer readable medium storing a computer program comprising coded instructions which, when executed on a computer, implements any of the methods of claims 8 to 13.

Images