Determining occupancy using unobstructed sensor radiation

Abstract

Techniques for determining occupancy using unobstructed sensor radiation. For example, a vehicle may receive sensor data from one or more sensors. The sensor data may represent at least a location for a point in the environment. Using the sensor data, the vehicle may determine areas in the environment that are obscured by an object (e.g., locations where the object is located). Additionally, the vehicle may use the sensor data to determine areas in the environment that are not obscured by an object (e.g., locations where there is no object). In some examples, the vehicle determines the unobstructed area to include an area that is between the vehicle and an identified object. This is because sensor radiation from the sensor(s) has passed through the area just described and reflected off an object located at a distance from the vehicle. The vehicle may then generate a map that is indicative of at least the obscured and unobstructed areas in the environment.

Description

[Background technology]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This PCT international application claims priority to U.S. patent application Ser. No. 17 / 405,826, filed August 18, 2021, and entitled "DETERMINING OCCUPANCY USING UNOBSTRUCTED SENSOR EMISSIONS," and U.S. patent application Ser. No. 17 / 405,865, filed August 18, 2021, and entitled "DETERMINING OBJECT CHARACTERISTICS USING UNOBSTRUCTED SENSOR EMISSIONS," the entire contents of which are incorporated herein by reference.

[0002] Background technology An autonomous vehicle may be configured to navigate along a path from a starting location to a destination location. For example, when providing a ride to a passenger, the autonomous vehicle may pick up the passenger at the starting location and drop off the passenger at the destination location. While navigating, it is important for the autonomous vehicle to determine the locations of objects, such as other vehicles. For example, the autonomous vehicle may analyze sensor data to determine that another vehicle is located along the path of the autonomous vehicle. With the other vehicle located along the path, the autonomous vehicle may take one or more actions to safely avoid the other vehicle. [Brief description of the drawings]

[0003] [Figure 1A] FIG. 13 is a pictorial flow diagram of an exemplary process for determining occupancy using unobstructed sensor data. [Figure 1B] FIG. 13 is a pictorial flow diagram of an exemplary process for determining occupancy using unobstructed sensor data. [Figure 2A] 1 is a pictorial flow diagram of a first exemplary process for determining object characteristics using unobstructed sensor data. [Figure 2B] FIG. 11 is a pictorial flow diagram of a second exemplary process for determining object characteristics using unobstructed sensor data. [Diagram 3] 1 illustrates an example of generating an occupancy heatmap using unobstructed sensor radiation. [Figure 4] 1 illustrates a block diagram of an example system for implementing the techniques described herein. [Diagram 5] 1 illustrates a flow diagram of an example process for determining occupancy using unobstructed sensor data. [Figure 6] 1 illustrates a flow diagram of an exemplary process for determining object characteristics using unobstructed sensor data. [Figure 7] 1 illustrates a flow diagram of another example process for determining object characteristics using unobstructed sensor data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] As discussed above, an autonomous vehicle may be configured to navigate along a path from a starting location to a destination location. For example, when providing a ride to a passenger, the autonomous vehicle may pick up the passenger at the starting location and drop off the passenger at the destination location. During navigation, it is important for the autonomous vehicle to determine the location of an object, such as, for example, another vehicle. For example, the autonomous vehicle may analyze sensor data to determine that another vehicle is located along the path of the autonomous vehicle. As the other vehicle is located along the path, the autonomous vehicle may take one or more actions to safely avoid the other vehicle. In some situations, the one or more actions may include navigating to an area that is not obstructed by any other objects.

[0005] As mentioned above, the present application relates to techniques for using unobstructed sensor radiation to determine the occupancy of an environment in which a vehicle is operating. For example, a vehicle may receive sensor data, e.g., lidar data, that represents an environment in which the vehicle is navigating. The vehicle may then use the sensor data to determine a distance to an object located in the environment. Using the distance, the vehicle may determine (1) areas of the environment that are obstructed by the object (in some examples, referred to as "obstructed areas"), (2) areas of the environment that are not obstructed by the object (in some examples, referred to as "unobstructed areas"), and / or characteristics associated with the object. For example, using the distance, the vehicle may determine that an area of ​​the environment located between the vehicle and the object is not obstructed by any other object. The vehicle may then generate a heat map (and / or other type of map) that represents the unobstructed areas in the environment and the unobstructed areas in the heat map. In some examples, the heat map represents the likelihood that an area is obstructed or unobstructed. Using the heat map, the vehicle may determine one or more actions regarding how to navigate through the environment.

[0006] In some examples, the vehicle includes one or more sensors from which the vehicle receives sensor data representative of the environment in which the vehicle is located. Additionally or alternatively, the vehicle may receive sensor data from one or more sensors located remotely from the vehicle (e.g., over-vehicle sensors, traffic cameras, sensors located at traffic signals, or sensors located on other objects in the environment, etc.). The one or more sensors may include, without limitation, lidar sensor(s), radar sensor(s), camera(s), and / or any other type of sensor. In some examples, such as when the sensor data includes lidar data, the sensor data may represent at least the location (e.g., x-coordinate, y-coordinate, z-coordinate, spacing, etc.) of a point in the environment. Additionally or alternatively, in some examples, the vehicle may analyze the sensor data using one or more techniques to determine the location of the point in the environment. In the above example, the vehicle may use the sensor data to determine an occupancy associated with a location in the environment.

[0007] For example, the vehicle may analyze the sensor data to group points of different objects based on object classification. For example, the vehicle may determine that a first group of points, as represented by the sensor data, is associated with a first object, such as, for example, another vehicle. Additionally, the vehicle may determine that a second group of points, as represented by the sensor data, is associated with a second object, such as, for example, a pedestrian. The vehicle may then use the groups of points to determine a location of the object in the environment. For example, the vehicle may use a location associated with the first group of points to determine an area of ​​the environment in which the first object is located. Additionally, the vehicle may use a location associated with the second group of points to determine an area of ​​the environment in which the second object is located. Each of the just mentioned areas may be determined to be an occluded area in the environment. An occluded area of ​​the environment, as described herein, may include an area in which an object is located such that the vehicle is not able to navigate through the area.

[0008] Additionally, the vehicle may use sensor data (e.g., locations of points in the environment) to verify areas of the environment that are not occluded by any objects. An unoccluded area of ​​the environment, as described herein, may include an area where no objects are located and / or an area where the vehicle can navigate (based on object classification, described in more detail below). In some examples, the vehicle determines that an area located between the vehicle and an identified object comprises an unoccluded area in the environment. In some examples, upon identifying an unoccluded area, the vehicle may determine an unoccluded area in three-dimensional space, such that various height(s) within the area are unoccluded. For example, if the vehicle identifies an object, such as a road sign, that is located 5 meters above the roadway with no other objects located directly below the road sign, the unoccluded area may include the area directly below the road sign.

[0009] Upon determining a region in three-dimensional space as described herein, the region may be defined in voxel space. For example, a vehicle may generate sensor data as it moves through an environment and associate the sensor data with a voxel space. The voxel space may represent a volume of space in the environment. For example, the voxel space represents a volume that is 100m x 100m x 100m. Examples of defining regions in three-dimensional space are described in application Ser. No. 16 / 420,090, entitled "Multiresolution Voxel Space," filed May 22, 2019, the entire contents of which are incorporated herein by reference.

[0010] For an example relating to identifying an unobstructed region, such as when the sensor data includes lidar data, the lidar sensor may emit a pulse of light (also referred to as a "sensor emission") that reflects off an object in the environment and back to the lidar sensor. The lidar sensor then analyzes the reflected light to determine the location of the point where the light was reflected. In some examples, the lidar sensor determines the location of the point using one or more techniques, such as, for example, the time it takes for the light to return to the lidar sensor. As described above, because a pulse of light may reflect off an object and back to the light sensor, the vehicle may infer that there is no other object(s) between the lidar sensor emitter (e.g., located on the autonomous vehicle) and the object. In other words, the vehicle may determine that a separation between the vehicle and a point includes an unobstructed region as a result of the vehicle receiving a point from an object behind the unobstructed region. Additionally, in some examples, the vehicle may use an angle at which the point is located relative to the lidar sensor to make a determination in three-dimensional space.

[0011] In some examples, when performing the just described process of determining occupancy of various locations in the environment, the vehicle may first assume locations of potential objects in the environment. In some examples, the vehicle may first assume potential objects located at various locations around the vehicle. Additionally or alternatively, in some examples, the vehicle may first assume potential objects located on a driving surface around the vehicle. For example, if the vehicle is parked in a parking lot, the vehicle may assume that a first potential vehicle may be located at a first interval on a first side of the vehicle, a second potential vehicle may be located at a second interval on a second side of the vehicle, a third potential vehicle may be located at a third interval on a third side of the vehicle, and a fourth potential vehicle may be located at a fourth interval on a fourth side of the vehicle.

[0012] Additionally, in some examples, the vehicle may assume additional characteristics associated with the object. For example, the vehicle may use the type of the environment to assume the type of object located in the environment. For example, the vehicle may assume that other objects located in a first type of environment (e.g., drivable surfaces such as roadways, driveways, and / or the like) include a first type of object(s) (e.g., other vehicle(s), bikes, etc.), and that other objects located in a second type of environment (e.g., non-drivable surfaces such as sidewalks, yards, etc.) include a second type of object(s) (e.g., pedestrians, bikes, road signs, etc.), and / or others. In this way, the vehicle may better assume the location of the actual object in the environment.

[0013] For another example, the vehicle may assume an orientation of other objects in the environment. For example, if the vehicle assumes the location of another vehicle along the roadway, the vehicle may assume the orientation of the other vehicle along the roadway. In some examples, the vehicle assumes an orientation based on the roadway, such that the other vehicle is oriented in the direction of travel along the roadway. In any of the above examples just mentioned, the vehicle may then perform the above processing to verify whether an actual object is located at one or more of the assumed locations.

[0014] For example, using the example above in which the vehicle assumes that the first potential vehicle is located a first distance from the first side of the vehicle, the vehicle may analyze the sensor data and, based on the analysis, determine that a point(s) emitted to the first side of the vehicle is reflected off of an object(s) located a fifth distance from the vehicle. The vehicle may then determine that the fifth distance is greater than the first distance. As above, the vehicle may determine that an area located on the first side of the vehicle between the vehicle and the fifth distance is not obstructed by an object. Because of this, the vehicle may determine that the hypothesis that the first potential vehicle is located a first distance to the first side of the vehicle was incorrect.

[0015] For another example, using the example above in which the vehicle assumes that the second potential vehicle is located a second distance from the second side of the vehicle, the vehicle may analyze the sensor data and determine based on the analysis that the point(s) emitted to the second side of the vehicle are reflected off of the object(s) located a sixth distance from the vehicle. The vehicle may then determine that the sixth distance is the same as or similar to the second distance (e.g., within a threshold distance). As above, the vehicle may determine that the hypothesis that the second potential vehicle is located the second distance to the second side of the vehicle was accurate. In other words, first, the vehicle may assume characteristics (e.g., location, type, orientation, etc.) of objects in the environment and then use the sensor data to determine whether the hypothesis is accurate (the object is actually located at the location, the object includes a type, the object includes an orientation, etc.) or inaccurate (the object is not actually located at the location, the object does not include a type, the object does not include an orientation, etc.).

[0016] In some examples, the vehicle may use the sensor data to determine characteristics associated with an object identified in the environment. For example, the sensor data may represent both a first point associated with the object (e.g., sensor radiation(s) reflected off the object) and a second point associated with another object(s) located further from the vehicle than the object, such as when the object located proximate to the vehicle includes a conductive object (smoke, fog, exhaust, etc.). As described above, the first point may indicate that the object is located approximately a first distance from the vehicle, while the second point may indicate that there is no object located between the vehicle and the second point at a second distance that is further than the first distance. To this end, the vehicle may use the first point and the second point to determine a classification of the object.

[0017] In some examples, the vehicle makes a determination based on a percentage of the first point compared to a percentage of the second point. For example, a first type of object, such as fog, may reflect a first percentage of the sensor radiation, and a second type of object, such as exhaust, may reflect a second percentage of the sensor radiation. As described above, based on the percentage of the first point and the percentage of the second point, the vehicle may determine whether the object is a first type of object or a second type of object. Additionally or alternatively, in some examples, the vehicle may make a determination of the type of object based on the location of the first point compared to the location of the second point. For example, if the object includes another vehicle, sensor radiation that strikes the trunk of the other vehicle may reflect back to the sensor(s) on the other vehicle, while sensor radiation that strikes a window of the other vehicle may travel through the window and reflect off another object located on the other side of the other vehicle. As described above, the vehicle may be able to analyze the sensor data to determine that the object has the shape of another vehicle (e.g., the first point represents at least the trunk of the other vehicle, while the second point represents a window of the other vehicle).

[0018] Additionally, in some examples, the vehicle may determine an orientation of the object using at least the sensor data. For example, the sensor data may again represent a first point associated with an object located at a first distance from the vehicle and a second point associated with object(s) located at a second, farther distance(s) from the vehicle. As described above, the vehicle may analyze the sensor data to determine a dimension(s) of the object, such as, for example, the length, width, and / or height of the object. Using the dimension(s), the vehicle may then determine an orientation of the object. For example, if the vehicle determines that the other object is another vehicle, the vehicle may determine an orientation of the other vehicle based on the length and / or width of the other vehicle as determined using the sensor data. For example, a vehicle may determine that when the width is a first width, the other vehicle has a first orientation (e.g., the other vehicle is oriented so that the front or rear of the other vehicle faces toward the vehicle), and when the width is a second, greater width, the other vehicle has a second orientation (e.g., the other vehicle is oriented so that a side of the other vehicle faces toward the vehicle).

[0019] In some examples, the vehicle may use one or more components (e.g., models, algorithms, and / or machine learning algorithms) when performing the processing described herein. For example, a first component may be trained and configured to analyze data to identify sensor radiation that includes a path that does not reflect off of the object but passes close to the object. Without limitation, data input to the first component may include sensor data representing a point(s) associated with the object (e.g., a point that reflected off of the object), data representing the location of the sensor that output the sensor radiation (e.g., the x-, y-, and z-coordinates of the sensor in the environment, the location of the sensor on the vehicle, etc.), information about the rotation of the sensor (e.g., the speed of rotation if the sensor includes a lidar sensor), sensor data representing a point(s) that reflected further away from the vehicle than the object (e.g., a point(s) that reflected off an object(s) that is further away from the vehicle than the object), and / or the like. Output from the first model may then include data representing the sensor radiation that did not reflect off of the object but passes close to the object. In some examples, the output data may further represent the proximity of the sensor radiation to the object.

[0020] For another example, the second component may be trained and configured to analyze data to determine the type of object. Without limitation, the data may include sensor data representing a first point(s) reflected on the object, sensor data representing a second point including the sensor radiation that passed through the object, data representing a percentage of the sensor radiation reflected on the object, data representing a percentage of the sensor radiation that passed through the object, and / or the like. The second component may then analyze the input data and, based on the analysis, output data representing the type of object and / or the location of the object using one or more of the processes described herein. For example, the second component may analyze the percentage of the sensor radiation reflected on the object and the percentage of the sensor radiation that passed through the object using one or more of the processes described above to determine the type of object. The second component may then output data representing the type of object and / or an indicator, such as a bounding box, representing the location of the object in the environment.

[0021] For another example, the third component may be trained and configured to analyze data to determine the orientation of the object. Without limitation, the data may include sensor data representing points associated with the object, data representing sensor radiation that has passed close to the object without being reflected with respect to the object (e.g., may be determined using the first component), sensor data representing point(s) that have reflected further away from the vehicle than the object (e.g., point(s) that have reflected off object(s) that are further away from the vehicle than the object), and / or the like. The third component may then analyze the input data and output data representing the orientation of the object based on the analysis. For example, the third component may analyze the points associated with the object along with the sensor radiation that has passed close to the object without being reflected with respect to the object using one or more of the processes described above to determine the orientation of the object. The third component may then output data representing the orientation of the object. In some examples, the output data may include an indicator, such as a bounding box, that represents the object in the orientation determined by the third component.

[0022] Further, for another example, the fourth component may be trained and configured to analyze data to verify whether a region of the environment is occluded by a hypothetical object and / or is unoccluded. Without being limited thereto, the data may include data representing the location of the hypothetical object in the region, sensor data representing a point reflected with respect to a real object in the region (if such point exists), data representing a sensor radiation that passed through the region without being reflected with respect to a real object (if such point exists), sensor data representing a point(s) reflected with respect to an object(s) that is further from the vehicle than the region and includes a direction substantially toward the region (which may be associated with the sensor radiation that passed through the region), and / or the like. The fourth component may then analyze the input data and, based on the analysis, output data representing whether the region is actually occluded by a real object (e.g., whether at least one of the hypothetical objects is located in the region) or is unoccluded (e.g., whether there are no hypothetical objects located in the region). For example, the fourth component may use one or more of the processes described herein to analyze points reflected off real objects in the region, as well as sensor radiation that passes through the region without being reflected off an object, to determine whether the region is occluded or unoccluded. The fourth component may then output data representing whether the region is occluded, an indicator (e.g., a bounding box) that represents the location of real objects in the region if the region is occluded, or whether the region is unoccluded. While what has just been described is merely one example set of components that a vehicle may use to perform the processes described herein, in other examples, a vehicle may use additional and / or alternative components that perform one or more of the processes described herein.

[0023] In some examples, the vehicle may generate a map that represents the occupancy of the environment. For example, the map may represent the locations of occluded areas in the environment and / or the locations of unoccluded areas in the environment. Additionally, in some examples, the map may further indicate characteristics associated with the objects, such as, for example, a classification of the object (e.g., object type), an orientation of the object, a size of the object, and / or the like. The vehicle may then use the map when navigating around the environment. For example, the vehicle may use the map such that the vehicle does not collide with identified objects and / or such that the vehicle navigates only through unoccluded areas.

[0024] By performing the processing described herein, the vehicle can both determine the location of objects in the environment and use the sensor data to verify the location of unobstructed regions in the environment. As described herein, the vehicle can use the unobstructed portion of the sensor radiation to verify the location of the unobstructed regions. For example, the vehicle can verify that the region(s) between the vehicle and the point(s) represented by the sensor data include unobstructed region(s) with respect to the environment. By verifying the unobstructed regions in the environment, the vehicle may generate a map that more accurately depicts the environment in which the vehicle is navigating. What has just been described may increase the safety of the vehicle while navigating, as the vehicle is better able to avoid collisions with other objects.

[0025] For example, in some instances, the vehicle first assumes locations where objects may be located in the environment. The vehicle then uses sensor data to verify whether actual objects are at each of the locations. Thus, the vehicle may also verify locations in the environment that are not obstructed by actual objects. In other words, the vehicle may assume that an object is at a location before the vehicle verifies that the object is not actually at the location. What has just been described may increase the overall safety of the vehicle when navigating around the environment.

[0026] A sensor, as described herein, may emit sensor radiation (e.g., light pulses, etc.) in various directions within an environment. The just-mentioned sensor radiation may then travel through the environment until it contacts an object within the environment, where the sensor radiation is then reflected back towards a sensor on the vehicle. For example, a sensor radiation emitted by a lidar sensor may travel through the environment until it is reflected off of another vehicle and then back to the lidar sensor. The vehicle and / or sensor may then analyze the reflected sensor radiation to determine information associated with the sensor radiation. Without being limited thereto, the information associated with the sensor radiation may include a location of the point of reflection (e.g., x position (global location), y position (global location), z position (global location)), a direction associated with the sensor radiation (e.g., x angle, y angle, z angle), a confidence level associated with the sensor radiation, a classification associated with the object from which the sensor radiation was reflected (e.g., object type), and / or any other type of information. In some examples, the vehicle and / or sensor may generate the information using at least the direction associated with the sensor radiation and the time it took for the sensor radiation to be reflected off of the object and back to the sensor.

[0027] Additionally, as described herein, a first sensor radiation may be within proximity of a second sensor radiation (and / or pass within proximity near a point associated with the second sensor radiation) based on a first direction associated with the first sensor radiation being within a threshold angle relative to a second direction associated with the second sensor radiation. In some examples, such as, for example, when the vehicle is analyzing sensor data in a two-dimensional space, the threshold angle may include, without limitation, 0.1 degrees, 0.5 degrees, 1 degree, 2 degrees, and / or any other angle. Additionally, in some examples, such as, for example, when the vehicle is analyzing sensor data in a three-dimensional space, the threshold angle may include, without limitation, 0.1 degrees in a given direction(s), 0.5 degrees in a given direction(s), 1 degree in a given direction(s), 2 degrees in a given direction(s), and / or the like. For example, the vehicle may determine that a first sensor radiation is close to a second sensor radiation when the first direction is within 1 degree in the x direction, within 1 degree in the y direction, and within 1 degree in the z direction. While the example just described includes the same threshold angle in each direction, in other examples, one or more of the directions may include unique threshold angles.

[0028] The techniques described herein may be implemented in a number of ways. Exemplary implementations are provided below with reference to the following drawings. Although described in the context of an autonomous vehicle, the methods, apparatus, and systems described herein may be applied to a variety of systems (e.g., sensor systems or robotic platforms) and are not limited to autonomous vehicles. In another example, the techniques may be utilized in aviation or nautical situations, or in any system that evaluates the spacing between reference points in an environment (e.g., in systems that use route-relative planning). In addition, the techniques described herein may be used with real data (e.g., captured using a sensor(s)), simulated data (e.g., generated by a simulator), or any combination of the two.

[0029] 1A and 1B are pictorial flow diagrams of an example process 100 for determining occupancy using unobstructed sensor data. At operation 102, process 100 may include determining potential characteristics for potential objects located within the environment. For example, example 104 illustrates that a vehicle 106 may determine (e.g., assume) potential characteristics 108(1)-(6) (also referred to as "potential characteristics 108") of potential objects located within the environment. In the example of FIG. 1, potential characteristics 108 include at least a location and an orientation of the potential objects. For example, latent characteristic 108(1) represents a first location and a first orientation for a potential object, latent characteristic 108(2) represents a second location and a second orientation for the potential object, latent characteristic 108(3) represents a third location and a third orientation for the potential object, latent characteristic 108(4) represents a fourth location and a fourth orientation for the potential object, latent characteristic 108(5) represents a fifth location and a fifth orientation for the potential object, and latent characteristic 108(6) represents a sixth location and a sixth orientation for the potential object. While the example of Figures 1A and 1B illustrates only six latent characteristics 108, in other examples, the vehicle 106 may determine any number of latent characteristics 108 for any number of potential objects in the environment.

[0030] In some examples, the vehicle 106 may further assume additional characteristic(s) associated with the potential object. For example, the vehicle 106 may assume at least an object type associated with the object. When determining the object type, the vehicle 106 may use the type of environment to determine the object type. For example, if the type of environment for which the vehicle 106 is determining potential characteristics 108 includes a drivable surface, such as a highway, the vehicle 106 may assume that the potential object includes other vehicles.

[0031] At operation 110, the process 100 may include receiving sensor data representative of a location of a point in the environment. For example, example 112 illustrates a vehicle 106 receiving sensor data 114 representative of the environment. In the example of FIG. 1A and FIG. 1B, the vehicle 106 may obtain the sensor data by using a sensor(s) that emits a sensor radiation (represented by a line of dashes) that may include a light pulse, a first portion of the sensor radiation being reflected off of an object 116 (including another vehicle in the example of FIG. 1A and FIG. 1B) and returning toward the sensor(s) of the vehicle 106. Additionally, a second portion of the sensor radiation may reflect off of other object(s) located in the environment and return toward the sensor(s) and / or may not be reflected toward the sensor(s) (e.g., if the second portion of the sensor radiation does not contact an object for a threshold interval). As described above, the sensor data may represent at least the location of the point from which the sensor radiation was reflected. For example, sensor data representing a first portion of the sensor radiation may be associated with a point on the object 116, while sensor data representing a second portion of the sensor radiation may be associated with a point on another object(s).

[0032] At operation 118, process 100 may include determining a validation associated with the potential characteristic based at least in part on the point. For example, example 120 illustrates vehicle 106 using sensor data to determine whether potential characteristic 108 is validated, not validated, or not determinable. In the example of FIGS. 1A and 1B, vehicle 106 may determine that potential characteristics 108(1), 108(2), and 108(4) are not validated. In some examples, vehicle 106 makes a determination based on a distance associated with the point represented by the sensor data. For example, vehicle 106 may determine that a region of the environment between vehicle 106 and the point represented by the sensor data includes an unoccluded region such that there is no object(s) located within the region.

[0033] As discussed above, and as illustrated by the example of FIGS. 1A and 1B, the vehicle 106 may determine that the object may not be located at the first location associated with potential characteristic 108(1) or the second location associated with potential characteristic 108(2). This may be because the sensor radiation traveled through the first location associated with potential characteristic 108(1) and through the second location associated with potential characteristic 108(2) without contacting any object. Additionally, the vehicle 106 may determine that the object may be located at a fourth location associated with potential characteristic 108(4), however, the object may not include a fourth orientation. This may be because the first portion of the sensor radiation reflected off of the object 116 did not travel through the fourth location associated with potential characteristic 108(4). As discussed above, the vehicle 106 may determine that the object may be located at the fourth location. However, the second portion of the sensor radiation traveled through portions of the area of ​​the fourth location, and as discussed above, the object may not be located in those portions. As noted above, an object located at a fourth location cannot include a fourth orientation.

[0034] Additionally, the vehicle 106 may determine that the potential characteristic 108(3) is verified. In some examples, the vehicle 106 makes the just-mentioned determination based on the first portion of the sensor radiation reflecting off the object 116 and not traveling through the third location associated with the potential characteristic 108(3), as described above. Additionally, the vehicle 106 may determine that the areas to the sides of the third location (e.g., above and below the potential characteristic 108(3) in the example of FIGS. 1A-1B) are not obstructed based on the second portion of the sensor radiation. As described above, the vehicle 106 may determine that the object 116 includes an orientation that is at least similar (e.g., within a threshold frequency limit) to the third orientation. Due to the just-mentioned determination, the vehicle 106 may verify that the object 116 includes the potential characteristic 108(3).

[0035] Further, the vehicle 106 may determine that the potential characteristics 108(5) and 108(6) are not capable of being verified or not verified. In some examples, the vehicle 106 makes the just-mentioned determination based on the sensor radiation not reaching and / or passing through the region of the environment associated with the potential characteristics 108(5) and 108(6), which in the example of FIGS. 1A and 1B is because the object 116 is blocking the sensor radiation from passing through the region. In some examples, based on the just-mentioned determination, the vehicle 106 may assume that the object(s) are located in and / or include the orientation associated with the potential characteristics 108(5) and 108(6). For example, because latent characteristics 108(5) and 108(6) are associated with the roadway along which vehicle 106 is navigating, vehicle 106 may hypothesize that another vehicle is located at the location, including an orientation associated with latent characteristics 108(5) and 108(6). Vehicle 106 may then take one or more actions based on the hypotheses just stated.

[0036] At operation 122, process 100 may include updating the latent trait to represent the object being verified. For example, example 124 illustrates that vehicle 106 may update latent trait 108(3) to represent object 116. In the example of FIGS. 1A and 1B, updating latent trait 108(3) may include rotating latent trait 108(3) to match the actual orientation of object 116 represented by rotated 128 bounding box 126. However, in other examples, updating latent trait 108(3) may additionally and / or alternatively include changing one or more other traits associated with latent trait 108(3), such as, for example, the location of latent trait 108(3) and / or the type of object hypothesized to be at a third location associated with latent trait 108(3). The vehicle 108 may then take one or more actions based on the updated and verified potential characteristics 128 and the potential characteristics 108(5) and 108(6) that could not have been verified or not verified.

[0037] For example, vehicle 106 may navigate to avoid colliding with object 126. For example, vehicle 106 may refrain from navigating through a third location associated with potential characteristic 108(3) until object 126 is no longer located at the third location. Additionally, vehicle 106 may refrain from navigating through a fifth location associated with potential characteristic 108(5) and / or a sixth location associated with characteristic 108(6) until vehicle 106 can verify that the fifth location and / or the sixth location are unobstructed.

[0038] FIG. 2A is a pictorial flow diagram of a first exemplary process 200 for determining a characteristic associated with an object using unobstructed sensor data. At operation 202, the process 200 may include receiving sensor data representative of a location of a point associated with an object in an environment. For example, example 204 illustrates a vehicle 106 receiving sensor data 206 representative of an environment. In the example of FIG. 2A, the vehicle 106 may obtain the sensor data by using a sensor(s) that emits a sensor radiation (represented by a line of dashes) that may include a light pulse, a first portion of the sensor radiation is reflected off an object 208 (including an exhaust in the example of FIG. 2A) and returns toward the sensor(s) of the vehicle 106. Additionally, a second portion of the sensor radiation is reflected off an object 210 (including another vehicle in the example of FIG. 2A) and returns toward the sensor(s) of the vehicle 106. As described above, the sensor data may represent at least a location of the point to which the sensor radiation is reflected. For example, the sensor data representing a first portion of the sensor radiation may be associated with a point on the object 208 , while the sensor data representing a second portion of the sensor radiation may be associated with a point on the object 210 .

[0039] At operation 212, process 200 may include detecting an unobstructed region in the environment based at least in part on the sensor data. For example, example 214 illustrates vehicle 106 using sensor data to determine an unobstructed region in the environment. In some examples, vehicle 106 uses sensor rays 216(1)-(4) (also referred to as “sensor rays 216”) associated with the sensor data to determine the unobstructed region. For example, vehicle 106 may determine that a first unobstructed region includes a first distance from vehicle 106 along first sensor ray 216(1). This is because first sensor ray 216(1) reflects off object 208 and back to sensor(s) of vehicle 106. As noted above, vehicle 106 may assume that no other object is located between vehicle 106 and the point associated with first sensor ray 216(1). Additionally, vehicle 106 may determine that the second unobstructed region includes a second distance from vehicle 106 along second sensor radiation 216(2). This is because second sensor radiation 216(2) reflects off of object 210 and back to the sensor(s) of vehicle 106. As described above, vehicle 106 may assume that no other objects are located between vehicle 106 and the point associated with second sensor radiation 216(2).

[0040] Additionally, the vehicle 106 may determine that the third unobstructed region includes a third interval from the vehicle 106 along the third sensor radiation 216(3). This is because the third sensor radiation 216(3) reflects off of the object 210 and back to the sensor(s) of the vehicle 106. As stated above, the vehicle 106 may assume that no other objects are located between the vehicle 106 and the point associated with the third sensor radiation 216(3). Finally, the vehicle 106 may determine that the fourth unobstructed region includes a fourth interval from the vehicle 106 along the fourth sensor radiation 216(4). This is because the fourth sensor radiation 216(4) reflects off of the object 208 and back to the sensor(s) of the vehicle 106. As stated above, the vehicle 106 may assume that no other objects are located between the vehicle 106 and the point associated with the fourth sensor radiation 216(4). Although the example of FIG. 2A illustrates the vehicle 106 as using only four sensor rays 216 to identify unobstructed areas within the environment, in other examples, the vehicle 106 may use any number of sensor rays.

[0041] At 218, process 200 may include determining a characteristic associated with the object based at least in part on the unobstructed region. For example, example 220 illustrates that vehicle 106 may use the unobstructed region to determine a characteristic associated with at least object 208. As shown, based on a first portion of the sensor data represented by sensor radiation 216(2)-(3), vehicle 106 may determine that object 208 is located in the region within the environment because sensor radiation 216(2)-(3) reflect off object 208 and return toward vehicle 106. However, based on a second portion of the sensor data represented by sensor radiation 216(1) and 216(4), vehicle 106 may initially determine that the region includes an unobstructed region because sensor radiation 216(1) and 216(4) passed through object 208 and reflected off object 210. As described above, vehicle 106 may use the sensor data to determine a classification (e.g., object type) associated with object 208.

[0042] In some examples, the vehicle 106 makes a determination based on a percentage of the sensor radiation 216 reflected relative to the object 208 compared to a percentage of the sensor radiation 216 that passed through the object 208. For example, a first type of object, such as fog, may reflect a first percentage of the sensor radiation 216 output by the sensor(s), while a second type of object, such as exhaust, may reflect a second percentage of the sensor radiation 216. Additionally or alternatively, in some examples, the vehicle 106 makes a determination based on a location of the sensor radiation 216 reflected relative to the object 208 and a location of the sensor radiation 216 that passed through the object 208. In the example of FIG. 2A, the vehicle 106 may determine that the object 208 includes exhaust.

[0043] In some examples, the vehicle 106 may determine a characteristic associated with the object 208 using one or more components (e.g., one or more models). For example, the vehicle 106 may input data to the component(s) representing at least the distance to points associated with the sensor radiation 216(2)-(3), the direction of the sensor radiation 216(2)-(3), the distance to points associated with the sensor radiation 216(1) and 216(4), the direction of the sensor radiation 216(1) and 216(4), the percentage of the sensor radiation 216(1) and 216(4) that passed through the object 208, the percentage of the sensor radiation 208(2)-(3) that was reflected with respect to the object 208, and / or the like. The component(s) may then analyze the data to determine the characteristic using one or more of the processes described herein. Additionally, the vehicle 106 may receive data representing the characteristic from the component(s).

[0044] At operation 222, process 200 may include having the vehicle navigate based at least in part on the characteristic. For example, example 224 illustrates that the vehicle 106 may determine a path to navigate based on the characteristic. For example, as shown by the example of FIG. 2A, the vehicle 106 may determine to continue along a path that passes through the object 208 because the object 208 includes an exhaust.

[0045] FIG. 2B is a pictorial flow diagram of a second example process 226 for determining a characteristic associated with an object using unobstructed sensor data. At operation 228, the process 226 may include receiving sensor data representative of a location of a point associated with an object in an environment. For example, example 230 illustrates a vehicle 106 receiving sensor data 232 representative of an environment. In the example of FIG. 2B, the vehicle 106 may obtain the sensor data by using a sensor(s) that emits a sensor radiation (represented by a line of dashes) that may include a light pulse, a first portion of the sensor radiation being reflected off of an object 234 (including another vehicle in the example of FIG. 2B) and returning toward the sensor(s) of the vehicle 106. Additionally, a second portion of the sensor radiation may be reflected off of an additional object (not illustrated in the example of FIG. 2B for clarity reasons) and returning toward the sensor(s) of the vehicle 106 and / or may not return to the vehicle 106. As described above, the sensor data may represent at least a location of the point from which the sensor radiation was reflected. For example, the sensor data representing a first portion of the sensor radiation may be associated with a point on the object 234 .

[0046] At operation 236, process 226 may include detecting unobstructed regions in the environment based at least in part on the sensor data. For example, example 238 illustrates vehicle 106 using sensor data to determine unobstructed regions in the environment. In some examples, vehicle 106 uses sensor rays 240(1)-(4) (also referred to as “sensor rays 240”) associated with the sensor data to determine the unobstructed regions. For example, vehicle 106 may determine that a first unobstructed region includes a first distance from vehicle 106 along first sensor ray 240(1). This is because first sensor ray 240(1) reflects off additional objects (not illustrated) and back to sensor(s) of vehicle 106. As noted above, vehicle 106 may assume that no other objects are located between vehicle 106 and the point associated with first sensor ray 240(1). Additionally, vehicle 106 may determine that the second unobstructed region includes a second distance from vehicle 106 along second sensor radiation 240(2). This is because second sensor radiation 240(2) reflects off of object 234 and back to the sensor(s) of vehicle 106. As described above, vehicle 106 may assume that no other objects are located between vehicle 106 and the point associated with second sensor radiation 240(2).

[0047] Additionally, the vehicle 106 may determine that the third unobstructed region includes a third interval from the vehicle 106 along the third sensor radiation 240(3). This is because the third sensor radiation 240(3) reflects off of the object 234 and back to the sensor(s) of the vehicle 106. As noted above, the vehicle 106 may assume that no other objects are located between the vehicle 106 and the point associated with the third sensor radiation 240(3). Finally, the vehicle 106 may determine that the fourth unobstructed region includes a fourth interval from the vehicle 106 along the fourth sensor radiation 240(4). This is because the fourth sensor radiation 240(4) reflects off of an additional object (not illustrated) and back to the sensor(s) of the vehicle 106. As noted above, the vehicle 106 may assume that no other objects are located between the vehicle 106 and the point associated with the fourth sensor radiation 240(4). Although the example of FIG. 2B illustrates the vehicle 106 as using only four sensor rays 240 to identify unobstructed areas within the environment, in other examples, the vehicle 106 may use any number of sensor rays.

[0048] At 242, process 226 may include determining a characteristic associated with the object based at least in part on the unobstructed region. For example, example 224 illustrates that vehicle 106 may use the unobstructed region to determine a characteristic associated with at least object 234. As shown, based on a first portion of the sensor data represented by sensor radiation 240(2)-(3), vehicle 106 may determine that object 234 is located in a region within the environment (e.g., a first characteristic associated with object 234) as sensor radiation 240(2)-(3) reflects off of object 234 and returns toward vehicle 106. Additionally, based on a second portion of the sensor data represented by sensor radiation 240(1) and 216(4), vehicle 106 may determine that the region located proximate to object 234 includes unobstructed regions 246(1)-(2). The vehicle 106 may then use the unobstructed regions 246(1)-(2) to determine an orientation (e.g., a second characteristic) associated with the object 234.

[0049] For example, the vehicle 106 may first determine that the object 234 includes another vehicle. The vehicle 106 may then use the unobstructed regions 246(1)-(2) to determine a width associated with the other vehicle 236. Using the width of the other vehicle, the vehicle 106 may determine an orientation. In some examples, the vehicle 106 may make the just-mentioned determination because the other vehicle has a first width when the other vehicle includes a first orientation, a second width when the other vehicle includes a second orientation, a third width when the other vehicle includes a third orientation, and / or the like. As noted above, the vehicle 106 may use the width to determine an orientation. While the example of FIG. 2B illustrates only using the unobstructed regions 246(1)-(2) to determine two characteristics (e.g., location and orientation) associated with the object 234, in other examples, the vehicle 106 may determine additional and / or alternative characteristics.

[0050] In some examples, the vehicle 106 may determine characteristics associated with the object 208 using one or more components (e.g., one or more models). For example, the vehicle 106 may input data to the component(s) representing at least the distance to points associated with sensor radiation 240(2)-(3), the direction of sensor radiation 240(2)-(3), the distance to points associated with sensor radiation 240(1) and 240(4), the direction associated with sensor radiation 240(1) and 240(4), the difference between the distance associated with sensor radiation 240(2)-(3) and the distance associated with sensor radiation 240(1) and 240(4), the region 246(1)-(2) of the environment that is unobstructed based on the sensor data (as described herein), and / or the like. The component(s) may then analyze the data to determine the characteristics using one or more of the processes described herein. Additionally, the vehicle 106 may receive data representing the characteristics from the component(s).

[0051] At operation 248, the process 226 may include having the vehicle navigate based at least in part on the characteristic. For example, example 250 illustrates that the vehicle 106 may determine a path to navigate based on the characteristic. For example, as shown by the example of FIG. 2, the vehicle 106 may determine to continue along a path through the object 234 because the object 234 includes an orientation along a roadway.

[0052] FIG. 3 illustrates an example of generating an occupancy heat map using unobstructed sensor emissions. As shown by the example of FIG. 3, a vehicle 106 may be navigating around an environment 302 that includes various objects 304(1)-(6) (also referred to as “objects 304”). While navigating, the vehicle 106 may be receiving sensor data representative of the objects in the environment 302. In some examples, as illustrated by the example of FIG. 3, the vehicle 106 may obtain the sensor data by emitting sensor emissions (e.g., light pulses) that reflect off the objects 304 and back to a sensor(s) of the vehicle 106, the sensor emissions being represented by a line of dashes. The sensor data, as described herein, may represent the location of a point in the environment 302. For example, a portion of the sensor data may represent the location of a point associated with object 304(5). Additionally, another portion of the sensor data may represent the location of a point associated with object 304(6).

[0053] The vehicle 106 may then analyze the sensor data to determine occupancy associated with the environment 302. For example, the vehicle 106 may analyze the sensor data to first determine areas in the environment 302 that are obstructed by the object 304. The vehicle 106 may further analyze the sensor data to determine areas in the environment 302 that are not obstructed by the object in the environment 302. As described herein, the vehicle 106 may determine the unobstructed areas using a distance associated with a point represented by the sensor data. For example, using the sensor radiation, the vehicle 106 may determine that the unobstructed area includes an area in the environment 302 that is between the vehicle 106 and a location associated with a reflection point of the sensor radiation. For example, as shown in Example 3, the vehicle 106 may determine that the unobstructed area includes an area between the vehicle 106 and the object 304(6) based on the sensor radiation that reflects off of the object 304(6) and returns to the vehicle 106. Additionally, vehicle 106 may determine that the unobstructed region includes the region between vehicle 106 and object 304(5) based on sensor radiation reflected off object 304(5) and returned to vehicle 106.

[0054] The vehicle 106 may then generate a heatmap 306 that represents the occupancy of the environment 302. In some examples, the heatmap 306 represents the likelihood that an area is occluded or unoccluded. The heatmap 306, as described herein, may represent a discretized region of the environment 302 proximate to the vehicle 106. For example, the heatmap 306 may represent a 64×64 grid (or a J×K sized grid) that represents a 100 meter×100 meter region around the vehicle 106. Of course, in other examples, the heatmap 306 may represent a region of any size and may represent any number of discrete portions of a region. That is, the heatmap 306 may represent the environment at any level of resolution. In some cases, portions of the heatmap 306 may be referred to as cells of the heatmap 306. Each cell may include a predicted probability that represents the probability that an object is in the area represented by the cell. For example, as illustrated in the example of FIG. 3, the black cells may represent areas in the environment 302 that are occluded by the object 304. Additionally, the gray cells 308(1)-(7) may represent areas in the environment 302 where the vehicle 106 has verified unobstructed areas. Additionally, the white cells may represent areas of the environment 302 where the vehicle 106 cannot be determined as occluded or unobstructed. This may be because the aforementioned area of ​​the environment 302 is obstructed by an object 304 such that the sensor radiation cannot advance into the area.

[0055] 4 illustrates a block diagram of an example system 400 for implementing the techniques described herein in accordance with aspects of the disclosure. In at least one example, the system 400 may include a vehicle 106. The vehicle 106 may include a vehicle computing device 402, one or more sensor system(s) 404, one or more emitters 406, one or more communication connections 408, at least one direct connection 410, and one or more drive systems 412.

[0056] The vehicle computing device 402 may include one or more processors 414 and a memory 416 communicatively coupled to the processor(s) 414. In the illustrated example, the vehicle 106 is an autonomous vehicle. However, the vehicle 106 may be any other type of vehicle (e.g., a manually driven vehicle, a semi-autonomous vehicle, etc.) or any other system having at least an image capture device. In the illustrated example, the memory 416 of the vehicle computing device 402 stores a localization component 418, a perception component 420, a planning component 422, an unobstructed area component 424, a characteristic component(s) 426, one or more system controllers 428, and one or more maps 430. Although depicted in FIG. 4 as residing in memory 416 for illustrative purposes, it is contemplated that the localization component 418, the perception component 420, the planning component 422, the unobstructed area component 424, the characteristic component(s) 426, the system controller(s) 428, and / or the map(s) 430 may additionally or alternatively be accessible to the vehicle 106 (e.g., stored in memory remote from the vehicle 106 or otherwise accessible by memory remote from the vehicle 302).

[0057] In at least one example, the localization component 418 can include functionality to receive sensor data 432 from the sensor system(s) 404 and determine a position and / or orientation of the vehicle 106 (e.g., one or more of an x ​​position, a y position, a z position, a roll, a pitch, or a yaw). For example, the localization component 418 can include and / or request / receive a map of the environment and can continuously determine the location and / or orientation of the vehicle 106 within the map. In some cases, the localization component 418 can utilize simultaneous localization and mapping (SLAM), calibration, localization and mapping, simultaneously (CLAMS), relative SLAM, bundle adjustment, non-linear least squares optimization, or the like to receive image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, and the like to accurately determine the location of the autonomous vehicle 106. In some cases, the localization component 418 can provide data to various components of the vehicle 106 to determine an initial position of the vehicle 106 in order to generate candidate trajectories, as described herein.

[0058] In some cases, the perception component 420 may include functionality for object detection, segmentation, and / or classification. In some cases, the perception system 420 may provide processed sensor data 432 indicative of the presence of an object approaching the vehicle 106 and / or the classification of the object as an object type (e.g., car, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional and / or alternative examples, the perception component 420 may provide processed sensor data 432 indicative of one or more characteristics associated with the detected object and / or with the environment in which the object is located. In some cases, the characteristics associated with the object may include, but are not limited to, x position (global position), y position (global position), z position (global position), orientation (e.g., roll, pitch, yaw), object type (e.g., classification), object speed, object acceleration, object range (magnitude), etc. Characteristics associated with an environment may include, but are not limited to, the presence of other objects in the environment, the state of other objects in the environment, time of day, day of the week, season, weather conditions, darkness / light indications, etc.

[0059] In general, the planning component 422 can determine a path for the vehicle 106 to follow to navigate through an environment. For example, the planning component 422 can determine various routes and trajectories and various levels of detail. For example, the planning component 422 can determine a route to navigate from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this discussion, the route can be a sequence of waypoints to navigate between the two locations. By way of non-limiting example, the waypoints include streets, intersections, GPS (Global Positioning System) coordinates, and the like. Additionally, the planning component 422 can generate instructions to guide the vehicle 106 along at least a portion of the route from the first location to the second location. In at least one example, the planning system 422 can determine how to guide the vehicle 106 from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some cases, the instructions can be a trajectory, or a portion of a trajectory. In some cases, multiple trajectories can be generated substantially simultaneously (e.g., within technical tolerances) according to a receding horizon technique, and one of the multiple trajectories is selected for the navigating vehicle 106.

[0060] In at least one example, the planning component 422 can determine a pickup location associated with the location. As used herein, a pickup location can be a particular location (e.g., a parking space, a loading zone, a portion of a surface, etc.) within a threshold interval of a location (e.g., an address or location associated with a ride request) where the vehicle 106 can stop to pick up a passenger. In at least one example, the planning component 422 can determine the pickup location based at least in part on determining a user identity (e.g., determined via image recognition, as described herein, or received as an indication from a user device). Arrival at a pickup location, arrival at a destination location, a passenger entering the vehicle, and receiving a "start ride" command are additional examples of events that may be used for event-based data logging.

[0061] In general, the unobstructed area component 424 may be configured to perform one or more of the processes described herein to determine areas in the environment that are not obscured by objects. For example, the unobstructed area component 424 may be configured to analyze the sensor data 432 to determine the location of a point represented by the sensor data 432. The unobstructed area component 424 may then be configured to identify unobstructed areas that include areas that are between the vehicle 106 and the location of the point in the environment.

[0062] In general, the characteristic component(s) 426 may be configured to analyze the data to determine one or more characteristics associated with the object. For example, such as, but not limited to, when the characteristic component 426 is configured to determine a type of object, the data may include a first distance to a first point in the environment associated with the object, a first direction of a first sensor radiation associated with the first point, a second distance to a second point in the environment associated with a second sensor radiation that passed through the object, a second direction of the second sensor radiation, a percentage of the sensor radiation that passed through the object, a percentage of the sensor radiation that was reflected with respect to the object, and / or the like. Thus, the characteristic component 426 may analyze the data using one or more of the processes described herein to analyze the data to determine a type of object.

[0063] For another example, such as, but not limited to, when the characteristics component 426 is configured to determine an orientation of the object, the data may include a distance to a first point associated with the object, a direction of a first sensor radiation associated with the first point, a distance to a second point associated with a second sensor radiation that passed close to the first point, a direction associated with the second sensor radiation, a difference between the second distance and the first distance, an area of ​​unobstructed environment (as described herein) based on the sensor data, and / or the like. Thus, the characteristics component 426 may analyze the data using one or more of the processes described herein to analyze the data to determine the orientation of the object.

[0064] In some examples, the computing device may use one or more techniques to train the just-mentioned characteristic component(s) 426. For example, the computing device may input data to the characteristic component(s) 426 along with known outcomes associated with the data to train the characteristic component(s) 426. In other words, the computing device may train the just-mentioned classification component(s) 426 to use unobstructed sensor emissions to determine the characteristics of an object.

[0065] In at least one example, the computing device 402 may include system controller(s) 428 that may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 106. The just mentioned system controller(s) 428 may communicate with and / or control corresponding systems of the drive system(s) 412 and / or other components of the vehicle 106.

[0066] The memory 416 may further include map(s) 430 that may be used by the vehicle 106 to navigate within the environment. For purposes of discussion, the map may be any number of modeled data structures in 2-, 3-, or N-dimensions capable of providing information about the environment, such as, but not limited to, topology (e.g., intersections, etc.), streets, mountain ranges, roads, terrain, and the environment in general. In some cases, but not limited to, the map may include texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), and the like), intensity information (e.g., lidar information, radar information, and the like), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and / or intensities), reflectance information (e.g., specular information, retroreflectivity information, BRDF information, BSSRDF information, and the like). In one example, the map may include a 3-dimensional mesh of the environment. In some cases, the map may be stored in a tiled format, with individual tiles of the map representing discrete portions of the environment, and may be loaded into the working memory as needed. In at least one example, the map(s) 430 may include at least one map (e.g., an image and / or a mesh). In some examples, the vehicle 106 may be controlled based at least in part on the map(s) 430. That is, the map(s) 430 may be used in conjunction with the localization component 418, the perception component 420, and / or the planning component 422 to determine a location of the vehicle 106, identify entities in the environment, and / or generate a route and / or trajectory for navigating within the environment.

[0067] In some cases, some or all aspects of the components described herein may include any model, algorithm, and / or machine learning algorithm. For example, in some cases, the components in memory 416 may be implemented as a neural network. As described herein, a typical neural network is a biologically inspired algorithm that generates an output through input data through a series of connected layers. Furthermore, each layer in a neural network may include another neural network, or may include any number of layers (whether convolutional or not). As may be understood in the context of the present disclosure, a neural network may utilize machine learning, which may refer to a broad class of algorithms described above in which an output is generated at least in part based on a learning parameter.

[0068] Although described in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, machine learning algorithms can include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatter plot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic nets, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), etc.), and any other algorithms that may be used. ), Iterative Dichotomy 3 (ID3), Chi-Square Automatic Interaction Detection (CHAID), Decision Cut, Conditional Decision Tree), Bayesian algorithms (e.g., Naïve Bayes, Gaussian Naïve Bayes, Multinomial Naïve Bayes, Average One Dependence Estimators (AODE), Bayesian Belief Networks (BNN), Bayesian Networks), Clustering algorithms (e.g., k-means, k-medians, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning algorithms ( For example, perceptrons, backpropagation, Hopfield networks, radial basis function networks (RBFNs), deep learning algorithms (e.g., deep Boltzmann machines (DBMs), deep belief networks (DBNs), convolutional neural networks (CNNs), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (blending), gradient boosting machines (GBMs), gradient boosted regression trees (GBRTs), random forests), support vector machines (SVMs), supervised learning, unsupervised learning, semi-supervised learning, etc.Additional examples of architectures include neural networks such as, for example, ResNet40, ResNet101, VGG, DenseNet, PointNet, and the like.

[0069] As noted above, in at least one example, the sensor system(s) 404 may include lidar sensors, radar sensors, sonar sensors, location sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, time-of-flight (TOF), etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), and the like. The sensor system(s) 404 may include multiple instances of each of the just mentioned or other types of sensors. For example, the lidar sensors may include individual lidar sensors located at the corners, front, rear, sides, and / or top of the vehicle 106. As another example, the camera sensors may include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle 106. The sensor system(s) 404 may provide input to the vehicle computing device 402. Additionally or alternatively, the sensor system(s) 404 may transmit the sensor data 432 via one or more network(s) 434 to a computing device(s) 436 at a specific frequency, at predetermined intervals, when one or more conditions occur, in near real-time, etc.

[0070] Additionally, the vehicle 106 may include emitter(s) 406 for emitting light and / or sound, as described above. The emitter(s) 406 in the present example include interior audio and visual emitters that communicate with passengers of the vehicle 106. By way of example and not limitation, the interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. Additionally, the emitter(s) 406 in the present example may also include exterior emitters. By way of example and not limitation, the external emitters in the examples just described may include one or more lights that signal direction of travel or other indication regarding the operation of the vehicle (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) that audibly communicate with pedestrians or other nearby vehicles, including acoustic beam steering technology.

[0071] Additionally, the vehicle 106 may also include communication connection(s) 408 that enable communication between the vehicle 106 and one or more other local or remote computing device(s). For example, the communication connection(s) 408 may facilitate communication of the vehicle 106 with other local computing device(s) and / or with the drive system(s) 412. Additionally, the communication connection(s) 408 may also enable the vehicle 106 to communicate with other nearby computer device(s) (e.g., other nearby vehicles, traffic signals, etc.). Additionally, the communication connection(s) 408 may enable the vehicle 106 to communicate with remote teleoperated computing devices or other remote services.

[0072] The communication connection(s) 408 may include physical and / or logical interfaces for connecting the vehicle computing device(s) 402 to another computing device or to a network, such as, for example, network(s) 434. For example, the communication connection(s) 408 may enable Wi-Fi-based communications, such as, for example, frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as, for example, Bluetooth, cellular communications (e.g., 2G, 2G, 4G, 4G LTE, 4G, etc.), or via any suitable wired or wireless communications protocol that enables each computing device to interface with other computing device(s).

[0073] In at least one example, the vehicle 106 may include one or more drive systems 412. In some cases, the vehicle 106 may have a single drive system 412. In at least one example, if the vehicle 106 has multiple drive systems 412, the individual drive systems 412 may be located at opposing ends (e.g., front and rear, etc.) of the vehicle 106. In at least one example, the drive system(s) 412 may include one or more sensor systems that detect conditions of the drive system(s) 412 and / or conditions surrounding the vehicle 106. By way of example and not limitation, the sensor system(s) 404 may include one or more wheel encoders (e.g., rotary encoders) that sense the rotation of the wheels of the drive system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) that measure the orientation and acceleration of the drive system(s), cameras or other image sensors, ultrasonic sensors that acoustically detect entities surrounding the drive system(s), lidar sensors, radar sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive system(s) 412. In some cases, the sensor system(s) 404 of the drive system(s) 412 may overlap or supplement a corresponding system (e.g., sensor system(s) 404) of the vehicle 106.

[0074] The drive system(s) 412 can include many vehicle systems, including a high voltage battery, a motor to propel the vehicle 106, an inverter to convert direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and steering rack (which may be electrically powered), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system to distribute braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the exterior surroundings of the vehicle), and one or more other systems (e.g., a cooling system, a safety system, an on-board charging system, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). In addition, the drive system(s) 412 can receive and preprocess sensor data 432 from the sensor system(s) 404 and can include a drive system controller to control the operation of various vehicle systems. In some cases, the drive system controller can include one or more processors and a memory communicatively connected to the one or more processors. The memory may store instructions that perform various functionality of the drive system(s) 412. Additionally, the drive system(s) 412 also include one or more communication connection(s) that enable each drive system to communicate with one or more other local or remote computing device(s).

[0075] In at least one example, the direct connection 410 can provide a physical interface coupling one or more drive system(s) 412 with the body of the vehicle 106. For example, the direct connection 410 can allow for the transfer of energy, fluid, air, data, etc. between the drive system(s) 412 and the vehicle 106. In some cases, the direct connection 410 can also releasably secure the drive system(s) 412 to the body of the vehicle 106.

[0076] As further illustrated in FIG. 4, the computing device(s) 436 may include a processor(s) 438, a communication connection(s) 440, and a memory 442. The processor(s) 414 of the vehicle 106 and / or the processor(s) 438 of the computing device(s) 436 (and / or other processor(s) described herein) may be any suitable processor capable of processing data and executing instructions to perform operations as described herein. By way of example and not limitation, the processor(s) 414 and the processor(s) 438 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts the electronic data into registers and / or other electronic data that may be stored in memory. In some cases, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement the encoded instructions.

[0077] Memory 416 and memory 442 (and / or other memories described herein) are examples of non-transitory computer-readable media. Memory 416 and memory 442 can store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods described herein and the functions attributed to the various systems. In various implementations, the memory can be implemented using any suitable memory technology, for example, SRAM (Static RAM), SDRAM (Synchronous DRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architecture, systems, and individual elements described herein can include many other logical, programmatic, and physical components, and those shown in the accompanying drawings are merely examples relevant to the description of this specification.

[0078] 4 is illustrated as a distributed system, it should be noted that in alternative examples, the components of the computing device(s) 436 may be associated with the vehicle 106. That is, the vehicle 106 may perform one or more of the functions associated with the computing device(s) 436 and / or the computing device(s) 436 may perform one or more of the functions associated with the vehicle 106. For example, the computing device(s) 436 may include the unobstructed area component 424 and / or the characteristic component(s) 426. The computing device(s) 436 may then use the unobstructed area component 424 and / or the characteristic component(s) 426 to perform one or more of the processes described herein.

[0079] 5 and 6 are diagrams illustrating an exemplary process according to the present disclosure. The process just described is illustrated as a logical flow graph, where each operation represents a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media, which, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be omitted or combined in any order and / or in parallel to implement the process.

[0080] 5 illustrates a flow diagram of an example process for determining occupancy using unobstructed sensor data. At operation 502, process 500 includes determining an area within the environment in which a potential object is hypothesized to be located, the area may be associated with a first interval. For example, the vehicle 106 may hypothesize that the potential object may be located in the area within the environment. In some examples, the vehicle 106 may further hypothesize one or more additional characteristics associated with the potential object, such as, for example, an orientation of the potential object. Additionally, in some examples, the vehicle 106 may hypothesize that additional potential object(s) may be located in additional area(s) within the environment and / or may include additional positions (e.g., orientations) in the area.

[0081] At operation 504, the process 500 may include receiving sensor data from one or more sensors. For example, the vehicle 106 may receive the sensor data from one or more sensor(s) of the vehicle 106. In some examples, the sensor data may include lidar data obtained from a lidar sensor(s) of the vehicle 106. In the above examples, the sensor data may represent at least a location of a point in the environment, where the lidar sensor(s) determine the location(s) of the point using sensor radiation (e.g., light pulses). Additionally or alternatively, in some examples, the sensor data may include different types of data, such as image data obtained from a camera(s) of the vehicle 106. Additionally or alternatively, the sensor data may be received from one or more other sensors of the vehicle 106 and / or from one or more sensors located remotely from the vehicle.

[0082] At operation 506, the process 500 may include determining a second distance to the point in the environment based at least in part on the sensor data. For example, the vehicle 106 may analyze the sensor data to determine the second distance to the point in the environment. In some examples, the vehicle 106 may further analyze the sensor data to determine a classification of an object associated with the point (e.g., an object from which the sensor radiation associated with the point was reflected). Additionally, in some examples, the vehicle 106 may analyze the sensor data to determine that a direction of the sensor radiation associated with the point is substantially directed toward a region in the environment. In other words, the vehicle 106 may determine whether the sensor radiation contacted the potential object if the potential object was located within the region of the environment based on the direction.

[0083] At operation 508, the process 500 may include determining whether the second interval is greater than the first interval. For example, the vehicle 106 may compare the second interval to the first interval to determine whether the second interval is greater than the first interval. If at operation 508 it is determined that the second interval is not greater than the first interval, then at operation 510 the process 500 may include determining that the region is occluded. For example, if the vehicle 106 determines that the second interval is not greater than the first interval, the vehicle 106 may determine that the region is occluded by an object. This may be because the sensor radiation is associated with a point reflecting off an object located within the region and returning toward the sensor.

[0084] However, if at operation 508 it is determined that the second interval is greater than the first interval, then at operation 512 the process 500 may include determining that the region is unobstructed. For example, if the vehicle 106 determines that the second interval is greater than the first interval, the vehicle 106 may determine that the region is unobstructed by an object. This may be because the sensor radiation was associated with a point that passed through the region without being reflected off of an object. As described above, the vehicle 106 may determine that no object is within the region.

[0085] FIG. 6 illustrates a flow diagram of a first example process 600 for determining a characteristic of an object using unobstructed sensor data. At operation 602, the process 600 may include receiving sensor data from one or more sensors. For example, the vehicle 106 may receive sensor data from a sensor(s) of the vehicle 106. In some examples, the sensor data may include lidar data from a lidar sensor(s) of the vehicle 106. In the above examples, the sensor(s) of the vehicle may obtain the sensor data by emitting sensor radiation (e.g., light pulses) that reflect off objects in the environment and return to the sensor(s). Additionally or alternatively, in some examples, the sensor data may include different types of data, such as image data obtained by a camera(s) of the vehicle 106. Additionally or alternatively, the sensor data may be received from one or more other sensors of the vehicle 106 and / or from one or more sensors located far away from the vehicle.

[0086] At operation 604, process 600 may include determining a first distance to a first point associated with the first object based at least in part on the sensor data. For example, vehicle 106 may analyze the sensor data to determine the first distance to the first point. In some examples, such as when the sensor data includes lidar data, the sensor data may represent the first distance to the first point. Additionally or alternatively, in some examples, such as when the sensor data includes image data, vehicle 106 may analyze the sensor data to determine the first distance to the first point (e.g., determine the first distance to the first object).

[0087] At operation 606, process 600 may include determining a second distance to a second point associated with the second object based at least in part on the sensor data. For example, vehicle 106 may analyze the sensor data to determine the second distance to the second point. In some examples, such as when the sensor data includes lidar data, the sensor data may represent the second distance to the second point. Additionally or alternatively, in some examples, such as when the sensor data includes image data, vehicle 106 may analyze the sensor data to determine the second distance to the second point (e.g., determine the second distance to the second object).

[0088] At operation 608, the process 600 may include determining whether the second point is associated with a sensor radiation that passed through the first object. For example, the vehicle 106 may determine whether the sensor radiation associated with the second point passed through the first object. In some examples, the vehicle 106 makes a determination based on a direction associated with the second point. For example, based on the location of the second point within the environment, the vehicle 106 may determine whether the direction of the sensor radiation was such that the sensor radiation passed through the first object (e.g., whether the sensor radiation was directed substantially toward the first object).

[0089] If at operation 608 it is determined that the second point is associated with a sensor radiation that did not pass through the first object, then at operation 610 the process 600 may include determining a characteristic of the first object using the first point. For example, if the vehicle 106 determines that the sensor radiation did not pass through the first object, the vehicle 106 may determine a characteristic of the first object using the sensor data associated with the first point but not the sensor data associated with the second point. Without limitation, the characteristic may include a classification of the object (e.g., type of object), an orientation of the object, a location of the object, and / or a genus.

[0090] However, if at operation 608 it is determined that the second point is associated with the sensor radiation that passed through the first object, then at operation 612 the process 600 may include determining a characteristic of the first object using the first point and the second point. For example, if the vehicle 106 determines that the sensor radiation passed through the first object, the vehicle 106 may determine a characteristic of the first object using the sensor data associated with the first point and the sensor data associated with the second point. For example, the vehicle 106 may determine that the object includes a particular classification, such as fog, exhaust, and / or the like, that allows some sensor radiation to pass while reflecting other sensor radiation.

[0091] FIG. 7 illustrates a flow diagram of a second example process 700 for determining a characteristic of an object using unobstructed sensor data. At operation 702, the process 700 may include receiving sensor data from one or more sensors. For example, the vehicle 106 may receive sensor data from a sensor(s) of the vehicle 106. In some examples, the sensor data may include lidar data from a lidar sensor(s) of the vehicle 106. In the above examples, the sensor(s) of the vehicle may obtain the sensor data by emitting sensor radiation (e.g., light pulses) that reflect off objects in the environment and return to the sensor(s). Additionally or alternatively, in some examples, the sensor data may include different types of data, such as image data obtained by a camera(s) of the vehicle 106. Additionally or alternatively, the sensor data may be received from one or more other sensors of the vehicle 106 and / or from one or more sensors located far away from the vehicle.

[0092] At operation 704, process 700 may include determining a first distance to a first point associated with the first object based at least in part on the sensor data. For example, the vehicle 106 may analyze the sensor data to determine the first distance to the first point. In some examples, such as when the sensor data includes lidar data, the sensor data may represent the first distance to the first point. Additionally or alternatively, in some examples, such as when the sensor data includes image data, the vehicle 106 may analyze the sensor data to determine the first distance to the first point (e.g., determine the first distance to the first object).

[0093] At operation 706, process 700 may include determining a second distance to a second point associated with the second object based at least in part on the sensor data. For example, the vehicle 106 may analyze the sensor data to determine the second distance to the second point. In some examples, such as when the sensor data includes lidar data, the sensor data may represent the second distance to the second point. Additionally or alternatively, in some examples, such as when the sensor data includes image data, the vehicle 106 may analyze the sensor data to determine the second distance to the second point (e.g., determine the second distance to the second object).

[0094] At operation 708, the process 600 may include determining whether the second point is associated with a sensor radiation that passed close to the first object. For example, the vehicle 106 may determine whether the sensor radiation associated with the second point passed close to the first object. In some examples, the vehicle 106 makes the determination based on a direction associated with the second point. For example, in some examples, the vehicle 106 may determine that the sensor radiation passed close to the first object when the direction associated with the sensor radiation is within a threshold angle with respect to a direction associated with the first object (and / or a direction associated with additional sensor radiation for the first point).

[0095] If at operation 708 it is determined that the second point is associated with a sensor emission that did not approach and pass the first object, then at operation 710 the process 700 may include determining a characteristic of the first object using the first point. For example, if the vehicle 106 determines that the sensor emission did not approach the first object, the vehicle 106 may determine a characteristic of the first object using the sensor data associated with the first point but not the sensor data associated with the second point. Without limitation, the characteristic may include a classification of the object (e.g., type of object), an orientation of the object, a location of the object, and / or a genus.

[0096] However, if at operation 708 it is determined that the second point is associated with a sensor radiation that passed close to the first object, then at operation 712 process 700 may include using the sensor radiation to determine an unobstructed region. For example, the vehicle 106 may use one or more of the processes described herein to determine the unobstructed region. The unobstructed region may be located close to the first object because the sensor radiation passed close to the first object.

[0097] At operation 714, process 700 may include determining a characteristic of the first object using the first point and the unobstructed region. For example, the vehicle 106 may determine a characteristic of the first object using sensor data associated with the first point and the unobstructed region. For example, the vehicle 106 may determine that the object includes a particular orientation based on a region proximate to the first object that includes the unobstructed region.

[0098] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

[0099] The components described herein represent instructions that may be stored on any type of computer-readable medium and that may be implemented in software and / or hardware. All of the methods and processes described above may be embodied and fully automated in software code components and / or computer-executable instructions executed by one or more computers or processors, hardware, or a combination. Alternatively, some or all of the methods may be embodied in specialized computer hardware.

[0100] Unless specifically stated otherwise, conditional words such as "may," "could," or "could be," among others, are understood within the context that one example may include a certain feature, element, and / or step while another example does not. Thus, in general, the above conditional words are not intended to imply that a certain feature, element, and / or step is required in any respect for one or more examples, or that one or more examples necessarily includes logic for determining whether a certain feature, element, and / or step is included or should be performed in any particular example, with or without user input or prompting.

[0101] Unless specifically stated otherwise, conjunctive terms, such as the phrase "at least one of X, Y, or Z," should be understood to indicate that the item, term, etc. may be either X, Y, or Z, or any combination, including a plurality of each element. "a" means singular and plural, unless expressly described as singular.

[0102] Any routine descriptions, elements, or blocks in the flow diagrams described herein and / or depicted in the accompanying drawings should be understood as potentially representing modules, segments, or portions of code that include one or more computer-executable instructions for implementing a particular logical function or element in the routine. Alternative implementations are included within the scope of the examples described herein in which elements or functions may be performed substantially simultaneously, in reverse order, with additional operations, or deleted from those shown or described, including omitting operations, depending on the functionality involved as understood by one of ordinary skill in the art.

[0103] Numerous variations and modifications may be made to the examples described above, and elements should be understood to be among the other acceptable examples. All such modifications and modifications are intended to be included herein within the scope of this disclosure and protected by the following claims.

[0104] Example clauses A: A system including one or more processors; and one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations including: determining a first region within the environment in which a first plurality of hypothetical objects may be located; determining a second region within the environment in which a second plurality of hypothetical objects may be located; receiving sensor data from one or more sensors of the vehicle, where the sensor data is determined based on at least one sensor radiation that passes through the first region and reflects off a real object in the second region; determining, at least in part, based on the sensor data, that the first region is an unobstructed region that does not include at least one of the first plurality of hypothetical objects; determining, at least in part, based on the sensor data, an orientation and location of a real object located in the second region; and controlling the vehicle based at least in part on the second region including the real object.

[0105] B: The system described in paragraph A, wherein determining an orientation associated with the real object includes at least determining, based at least in part on the sensor data, that an emission path associated with the point passes close to the real object without reflecting off the real object, and determining an orientation associated with the real object based at least in part on the emission path that passes close to the real object without reflecting off the real object.

[0106] C: The system of either paragraph A or paragraph B, wherein the operations further include determining a third region within the environment in which a third plurality of hypothetical objects may be located, the third region being located farther from the vehicle than the second region, determining, based at least in part on the sensor data, that at least one sensor radiation does not pass through the third region, and determining, based at least in part on the at least one sensor radiation not passing through the region, that one or more of the third plurality of hypothetical objects may still be located in the third region, and controlling the vehicle is further based at least in part on determining that one or more of the third plurality of hypothetical objects may still be located in the third region.

[0107] D: The system of any one of paragraphs A-C, wherein the operations further include determining a type of environment associated with the first area, determining a type of hypothetical object that may be located in the first area based at least in part on the type of environment, and determining a first plurality of hypothetical objects based at least in part on the type of hypothetical object that may be located in the first area.

[0108] E: A method including: determining a plurality of hypothetical objects that may be present at a location within an environment; receiving sensor data from one or more sensors of a vehicle; determining, at least in part, based on the sensor data, that at least some of the plurality of hypothetical objects are not located at the location; and controlling the vehicle based, at least in part, on determining that at least some of the plurality of hypothetical objects are not located at the location.

[0109] F: The method of paragraph E, further including determining a hypothetical orientation of a hypothetical object from a plurality of hypothetical objects, determining an actual orientation of the real object based at least in part on the sensor data, and updating the location to include an additional location with respect to the environment based at least in part on the actual orientation, wherein controlling the vehicle is based at least in part on the additional location.

[0110] G: The method of either paragraph E or paragraph F, wherein determining that at least some of the plurality of hypothetical objects are not located at the location includes at least determining that a real object is located at the location based at least in part on the sensor data, and the method further includes determining, at least in part on the sensor data, that a radiation path associated with the point passes close to the real object without reflecting off the real object, and determining an orientation of the real object based at least in part on the radiation path that passes close to the real object without reflecting off the real object.

[0111] H: The method of any one of paragraphs E-G, further including determining a type of environment associated with the location and determining a type of hypothetical object that may be located at the location based at least in part on the type of environment, wherein determining a number of hypothetical objects that may be present at the location within the environment is based at least in part on the type of hypothetical object.

[0112] I: The method of any one of paragraphs E-H, further comprising: determining a type of environment associated with the location; determining a type of hypothetical object that may be located at the location based at least in part on the type of environment; determining a first position associated with a first hypothetical object from the plurality of hypothetical objects based at least in part on the type of hypothetical object; and determining a second position associated with a second hypothetical object from the plurality of hypothetical objects based at least in part on the type of hypothetical object.

[0113] J: The method of any one of paragraphs E-I, wherein determining that at least some of the plurality of hypothetical objects are not located at the location includes at least determining, based at least in part on the sensor data, that at least one sensor radiation passes through the location without reflecting off a real object, and determining, based at least in part on the sensor radiation passing through the location without reflecting off a real object, that at least some of the plurality of hypothetical objects are not located at the location.

[0114] K: The method of any one of paragraphs E-J, further including: determining additional locations of real objects in the environment based at least in part on the sensor data; determining that the additional locations are along the path of the vehicle; and determining that the locations are also along the path of the vehicle and are occluded by the real object, wherein determining a number of hypothetical objects that may be present at locations in the environment is based at least in part on the locations being along the path of the vehicle and are occluded by the real object.

[0115] L: The method of any one of paragraphs E-L, further including: determining an additional number of hypothetical objects that may be present at additional locations within the environment; determining, based at least in part on the sensor data, that at least one sensor radiation passes through the additional locations and reflects off a real object at the locations; and determining that the additional locations are not obstructed by the additional number of hypothetical objects.

[0116] M. The method of paragraph L, further comprising generating a map indicating at least which locations in the environment are occluded by real objects and which additional locations in the environment are unoccluded.

[0117] N: The method of any one of paragraphs E-M, including determining an additional plurality of hypothetical objects that may be present at additional locations within the environment, the additional locations being farther from the vehicle than the location, determining, based at least in part on the sensor data, that at least one sensor radiation does not pass through the location and to the additional location, and determining, based at least in part on the at least one sensor radiation not passing through the location and to the additional location, that one or more of the additional plurality of hypothetical objects may still be located at the additional location, wherein controlling the vehicle is further based at least in part on determining that one or more of the additional plurality of hypothetical objects may still be located at the additional location.

[0118] O: The method of any one of paragraphs E-N, wherein receiving sensor data from one or more sensors of the vehicle includes receiving lidar data from one or more lidar sensors of the vehicle.

[0119] P: One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including determining a plurality of hypothetical objects that may be present at a location within the environment; receiving sensor data from one or more sensors of the vehicle; determining, at least in part, based on the sensor data, that at least some of the plurality of hypothetical objects are not located at the location; and controlling the vehicle based, at least in part, on determining that at least some of the plurality of hypothetical objects are not located at the location.

[0120] Q: One or more non-transitory computer-readable media described in paragraph P, wherein the operations further include determining a hypothetical orientation of a hypothetical object from a plurality of hypothetical objects, determining an actual orientation of the real object based at least in part on the sensor data, and updating the location to include an additional location with respect to the environment based at least in part on the actual orientation, and controlling the vehicle is based at least in part on the additional location.

[0121] R: One or more non-transitory computer-readable media described in either paragraph P or paragraph Q, wherein the operations further include determining a type of environment associated with the location and determining a type of hypothetical object that may be located at the location based at least in part on the type of environment, and determining a number of hypothetical objects that may be present at the location within the environment is based at least in part on the type of hypothetical object.

[0122] S: One or more non-transitory computer-readable media described in any one of paragraphs P-R, wherein the operations further include determining a type of environment associated with the location, determining a type of hypothetical object that may be located at the location based at least in part on the type of environment, determining a first position associated with a first hypothetical object from the plurality of hypothetical objects based at least in part on the type of hypothetical object, and determining a second position associated with a second hypothetical object from the plurality of hypothetical objects based at least in part on the type of hypothetical object.

[0123] T: The non-transitory computer-readable medium of any one of paragraphs P-S, wherein determining that at least some of the plurality of hypothetical objects are not located at the location includes at least determining, based at least in part on the sensor data, that at least one sensor radiation passes through the location without reflecting off a real object, and determining, based at least in part on the sensor radiation passing through the location without reflecting off a real object, that at least some of the plurality of hypothetical objects are not located at the location.

[0124] U: a system including one or more processors; and one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations including: receiving sensor data of an environment, the sensor data associated with receiving reflections of radiation in the environment at the sensor, the spacing being determined based on the reflections of the radiation; determining, based at least in part on the sensor data, that a first point corresponds to an object in the environment and that a second point has a corresponding radiation path that passes proximate to the first point, the second point being at a greater spacing away from the sensor than the first point; determining a characteristic associated with the object based on the first point and the corresponding radiation path of the second point that passes proximate to the first point; and controlling the vehicle based at least in part on the characteristic associated with the object.

[0125] V: The system described in paragraph U, wherein determining a characteristic associated with the object includes at least inputting first data into the model, the first data representing at least a first point corresponding to the object and a second point having a corresponding radiation path that passes proximate to the first point without reflecting off the object, and receiving second data from the model representing the characteristic.

[0126] W: The system of either paragraph U or paragraph V, wherein the operations further include determining that a region located in proximity to the object is unobstructed based at least in part on the second point having a corresponding radiation path that passes in proximity to the first point, and determining a characteristic associated with the object determines an orientation associated with the object based at least in part on the first point and the region being unobstructed.

[0127] X: The system of any one of paragraphs U-W, wherein the operations further include: determining an area within the environment in which the object is located based at least in part on the first point; determining that the corresponding radiation path passes through the area without reflecting off the object based at least in part on the second point having a corresponding radiation path that passes in close proximity to the first point; and determining an object type or orientation associated with the object based at least in part on the corresponding radiation path that passes through the area without reflecting off the object.

[0128] Y: A method including receiving sensor data from one or more sensors of a vehicle; determining, based at least in part on the sensor data, that a first point is associated with an object; determining, based at least in part on the sensor data, that a second point has a radiation path that passes proximate to the object without reflecting off the object; determining a characteristic associated with the object based at least in part on the radiation path of the second point that passes proximate to the object without reflecting off the object; and navigating the vehicle based at least in part on the characteristic.

[0129] Z: The method of paragraph Y, further including: determining that the second point is associated with an additional object; and determining that the radiation path did not reflect off the object based at least in part on the second point being associated with the additional object.

[0130] AA: The method of any of paragraphs Y or Z, further including determining a first interval associated with the first point and determining a second interval associated with the second point, wherein determining a characteristic associated with the object is further based at least in part on the first interval and the second interval.

[0131] AB: The method of any one of paragraphs Y-AA, further including: determining an area within the environment in which the object is located based at least in part on the first point; determining that a radiation path of a second point passes through the area without reflecting off the object; and determining that a first interval associated with the second point is greater than a second interval associated with the first point, wherein determining a characteristic associated with the object includes determining a type associated with the object based at least in part on the radiation path passing through the area and the first interval being greater than the second interval.

[0132] AC: The method of any one of paragraphs Y-AB, further including determining a proportion of the sensor radiation that passes through the unobstructed object based at least in part on the sensor data, and the radiation path of the second point being associated with the proportion of the sensor radiation, and determining a characteristic associated with the object includes determining a type associated with the object based at least in part on the proportion of the sensor radiation.

[0133] AD: The method of any one of paragraphs Y-AC, wherein determining a characteristic associated with the object includes determining an orientation associated with the object based at least in part on a radiation path of a first point and a second point that passes close to the object without being reflected off the object.

[0134] AE: The method of any one of paragraphs Y-AD, wherein determining a characteristic associated with the object includes at least inputting first data into a model, the first data representing at least a first point associated with the object and a radiation path of a second point passing close to the object without reflecting off the object, and receiving second data from the model representing a characteristic associated with the object.

[0135] AF: The method of any one of paragraphs Y-AE, further including determining a percentage of the sensor radiation that passes through the unobstructed object based at least in part on the sensor data, and the radiation path of the second point being associated with the percentage of the sensor radiation, wherein determining a characteristic associated with the object includes at least inputting the first data into a model, the first data representing a percentage of the sensor radiation that passed through the unobstructed object, and receiving second data from the model, the second data representing the characteristic associated with the object.

[0136] AG: The method of any one of paragraphs Y-AF, wherein determining that the second point has a radiation path that passes in close proximity to the object includes at least inputting first data into a model, the first data representing at least a location associated with a sensor that generated the sensor data representing the second point and a location of the object within the environment, and receiving second data from the model indicating that the radiation path of the second point passed in close proximity to the object without reflecting off the object.

[0137] AH: The method of any one of paragraphs Y-AG, wherein determining that the second point has a radiation path that passes proximate to the object without reflecting off the object includes at least determining a first angle associated with the additional sensor radiation of the first point based at least in part on the sensor data, determining a second angle associated with the sensor radiation of the second point based at least in part on the sensor data, determining that the first angle is within a threshold for the second angle, and determining that the second point has reflected off the additional object.

[0138] AI: One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause one or more processors to perform operations including receiving sensor data from one or more sensors of the vehicle; determining, at least in part, that a first point is associated with the object based at least in part on the sensor data; determining, at least in part, that a second point has a radiation path that passes proximate to the object without reflecting off the object based at least in part on the sensor data; determining a characteristic associated with the object based at least in part on the radiation path of the second point that passes proximate to the object without reflecting off the object; and navigating the vehicle based at least in part on the characteristic.

[0139] AJ: The one or more non-transitory computer-readable media of paragraph AI, wherein the operations further include determining that the second point is associated with an additional object and determining that the radiation path did not reflect off the object based on the second point being associated with the additional object.

[0140] AK: The one or more non-transitory computer-readable media of any of paragraphs AI or AJ, wherein the operations further include determining an area within the environment in which the object is located based at least in part on the first point and determining that a radiation path of the second point passes through the area without reflecting off the object, and determining a characteristic associated with the object includes determining a type associated with the object based at least in part on the radiation path that passes through the area without reflecting off the object.

[0141] AL: One or more non-transitory computer-readable media of any one of paragraphs AI-AK, further including determining a percentage of sensor radiation that passes through the unobstructed object based at least in part on the sensor data, and the radiation path of the second point being associated with the percentage of sensor radiation, and determining a characteristic associated with the object includes determining a type associated with the object based at least in part on the percentage of sensor radiation.

[0142] AM: One or more non-transitory computer-readable media of any one of paragraphs AI-AL, wherein determining a characteristic associated with the object includes determining an orientation associated with the object based at least in part on a radiation path of a first point and a second point that passes proximate to the object without reflecting off the object.

[0143] AN: One or more non-transitory computer-readable media described in any one of paragraphs AI-AM, wherein determining a characteristic associated with the object includes at least inputting first data into a model, the first data representing at least a first point associated with the object and a radiation path of a second point passing in close proximity to the object without reflecting off the object, and receiving second data from the model representing a characteristic associated with the object.

Claims

1. determining a plurality of hypothetical objects that may be present at a location within the environment; receiving sensor data from one or more sensors of a vehicle; determining, based at least in part on the sensor data, that at least some of the plurality of hypothetical objects are not at the location; controlling the vehicle based at least in part on determining that the at least some of the plurality of hypothetical objects are not at the location; and A method comprising:

2. determining a hypothetical orientation of a hypothetical object from the plurality of hypothetical objects; determining an actual orientation of a real object based at least in part on the sensor data; and updating the location to include additional locations with respect to the environment based at least in part on the actual orientation; and Furthermore, The method of claim 1 , wherein controlling the vehicle is based at least in part on the additional location.

3. determining that at least some of the plurality of hypothetical objects are not at the location includes at least determining, based at least in part on the sensor data, that a real object is located at the location; The method comprises: determining, based at least in part on the sensor data, that a radiation path associated with a point passes proximate to the real object without reflecting off the real object; determining an orientation of the real object based at least in part on the radiation paths passing proximate to the real object without being reflected off the real object; The method of claim 1 further comprising:

4. determining a type of the environment associated with the location; determining a type of hypothetical object that may be located at the location based at least in part on the type of the environment; and Furthermore, The method of claim 1 , wherein determining the plurality of hypothetical objects that may be present at the location in the environment is based at least in part on the type of hypothetical object.

5. determining a type of the environment associated with the location; determining a type of hypothetical object that may be located at the location based at least in part on the type of the environment; determining a first location associated with a first hypothetical object from the plurality of hypothetical objects based at least in part on the type of the hypothetical object; determining a second location associated with a second hypothetical object from the plurality of hypothetical objects based at least in part on the type of the hypothetical object; and The method of claim 1 further comprising:

6. Determining that at least some of the plurality of hypothetical objects are not at the location includes: determining, based at least in part on the sensor data, that at least one sensor radiation passes through the location without reflecting off a real object; determining that at least some of the plurality of hypothetical objects are not at the location based at least in part on the sensor radiation passing through the location without being reflected off the real object; 2. The method of claim 1, comprising at least:

7. determining additional locations of real objects in the environment based at least in part on the sensor data; and determining the additional locations along the path of the vehicle; determining that the location is also along the path of the vehicle and that the location is occluded by the real object; Furthermore, 2. The method of claim 1, wherein determining the plurality of hypothetical objects that may be present at the location in the environment is based at least in part on the location being also along the path of the vehicle and the location being occluded by the real object.

8. determining additional hypothetical objects that may be present at additional locations within the environment; determining, based at least in part on the sensor data, that at least one sensor radiation passes through the additional location and reflects off a real object at the location; determining that the additional locations are not occluded by the additional plurality of hypothetical objects; The method of claim 1 further comprising:

9. the location in the environment is occluded by the real object; and The additional location within the environment is unobstructed 10. The method of claim 8, further comprising generating a map indicative of at least:

10. determining additional hypothetical objects that may be present at additional locations within the environment, the additional locations being farther from the vehicle than the location; and determining, based at least in part on the sensor data, that at least one sensor emission does not pass through the location and does not pass through the additional location; determining, based at least in part on the at least one sensor radiation not passing through the location and not passing through the additional location, that one or more of the additional plurality of hypothetical objects may still be located at the additional location; Furthermore, 10. The method of claim 1, wherein controlling the vehicle is further based at least in part on determining that the one or more of the additional plurality of hypothetical objects may still be located at the additional location.

11. The method of claim 1 , wherein receiving the sensor data from the one or more sensors of the vehicle comprises receiving lidar data from one or more lidar sensors of the vehicle.

12. determining, based at least in part on the sensor data, that a first point is associated with an object; determining, based at least in part on the sensor data, that a second point has a radiation path that passes proximate to the object without reflecting off the object; determining a characteristic associated with the object based at least in part on the radiation path of the second point passing proximate to the object without being reflected by the object; causing the vehicle to navigate based at least in part on the characteristic; and The method of claim 1 further comprising:

13. Determining the property associated with the object comprises: inputting first data into a model, the first data comprising: the first point associated with the object; the radiation path of the second point passes close to the object without being reflected by the object; and receiving second data from the model representing the property associated with the object; 13. The method of claim 12, comprising at least:

14. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 13.

15. 1. A system comprising: one or more processors; one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, configure the system to perform the method of any one of claims 1 to 13; A system comprising:

Images