Height estimation using sensor data

Abstract

Techniques for estimating the size of objects in an environment are discussed herein. For example, a sensor, such as a lidar sensor, can capture three-dimensional data of the environment. The sensor data can be associated with a two-dimensional representation. The ground can be removed from the sensor data, and clustering techniques can be used to cluster the remaining sensor data provided in the two-dimensional representation to determine the objects represented therein. The height of the sensor object can be represented as a first height based on the range of sensor data associated with the object and a second height based on the beam spread aspect of the sensor data and / or sensor data associated with additional objects. Thus, the minimum and / or maximum height of the object can be determined in a robust manner. Such height ranges can be used to control an autonomous vehicle.

Description

[Technical Field]

[0001] This application relates to techniques for estimating the size of objects in an environment. [Background technology]

[0002] (CROSS-REFERENCE TO RELATED APPLICATIONS) This patent application claims priority to U.S. Patent Application No. 16 / 698,055, filed November 27, 2019. Application No. 16 / 698,055 is incorporated herein by reference in its entirety.

[0003] A vehicle can use sensors to capture sensor data and detect objects in an environment. Accurate determination of object characteristics can, for example, assist an autonomous vehicle in traversing an environment. [Brief explanation of the drawings]

[0004] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. Use of the same reference number in different drawings indicates similar or identical components or features.

[0005] [Figure 1] FIG. 1 is a schematic flow diagram of an exemplary process for estimating the height of an object. [Figure 2] FIG. 1 is a diagram of an exemplary process for clustering data to determine object data. [Figure 3A] FIG. 1 illustrates a scenario for estimating height associated with sensor data. [Figure 3B] FIG. 1 illustrates a scenario for estimating height associated with sensor data. [Figure 3C] FIG. 1 illustrates a scenario for estimating height associated with sensor data. [Figure 3D] FIG. 1 illustrates a scenario for estimating height associated with sensor data. [Figure 4] FIG. 1 is a block diagram of an example system for implementing the techniques described herein. [Figure 5] 1 is an exemplary process for estimating a height associated with an object based on sensor data. [Figure 6] 1 is an exemplary process for determining an object height based on upper and lower bounds associated with the object data. DETAILED DESCRIPTION OF THE INVENTION

[0006] Techniques for estimating the size of objects in an environment are discussed herein. For example, a sensor, such as a lidar sensor, can capture three-dimensional data of the environment. The three-dimensional sensor data can be correlated to a two-dimensional data representation for further processing. For example, the lidar data can be represented as an azimuth angle. of Bottles and channel The ground may be removed from the two-dimensional data representation, and the sensor data may be clustered using a clustering technique to determine the object represented in the sensor data. A first height of the object may be estimated or otherwise determined based on the vertical range of the sensor data associated with the object. A second height of the object may be estimated or otherwise determined based on a first vertical distance from the top of the object to a ray associated with the upper lidar beam, or a second vertical distance from the bottom of the object to a ray associated with the lower lidar beam or the ground. Thus, the minimum and / or maximum height of the object may be determined in a robust manner.

[0007] In some examples, the techniques discussed herein may be implemented in a vehicle that includes a primary system for controlling the vehicle and a secondary system that operates on the vehicle to verify the operation of the primary system and control the vehicle to avoid collisions. For example, the techniques may be implemented in the secondary system to detect objects in an environment and verify that the system accurately detects objects of minimum and / or maximum height.

[0008] The techniques discussed herein may involve capturing or receiving environmental sensor data, such as lidar data from a lidar sensor. To reduce the amount of data for processing, the techniques may include capturing or receiving environmental sensor data associated with a drivable area. road and road The method may include receiving or otherwise determining a representation of the ground surface represented in the image. road Data outside of or associated with the ground may be removed or discarded, thereby resulting in a data set that includes non-ground lidar returns.

[0009] In some examples, the lidar data may be associated with a two-dimensional data representation, either before or after removing the ground plane. The two-dimensional data representation (also referred to as a two-dimensional representation or 2D space) may include an azimuth angle of The first dimension represents the bin, and channel For example, a rotating lidar may include scanning the environment 360 degrees. The first dimension of the 2D space may be the azimuth angle, which may include 5 degrees, 8 degrees, 10 degrees, 18 degrees, 36 degrees, etc. of The second dimension of the 2D space can be discretized into bins (also called bins). The channel Which channel can be discretized into

[0010] The data represented in 2D space can be clustered to determine the objects represented in the data. The technique involves selecting non-terrestrial data that are closest to the location associated with the lidar sensor. point That is, each data pointcan be related to the distance or range between an object in the environment and a position associated with the lidar sensor (such as the origin). In some examples, channel The data in the range can be ordered within the data structure by range for efficient data retrieval.

[0011] In some examples, the clustering technique, as discussed herein, channel and then at a second time after the first time. channel It can be applied across

[0012] road Nearest non-ground data in point After determining the closest point (e.g., a first point), the technique can include determining the distance to the next closest point in 2D space (e.g., a second point). For example, the distance can be determined as a three-dimensional distance in the environment. If the distance is less than a threshold, the first point and the second point can be clustered to represent the same object. Clustering can involve determining additional data about the first point. point The clustering can then be repeated for each additional data point. point For example, the distance between a second point and a third point can be determined, and if the distance is less than a threshold distance, the third point can be associated with an object. Data associated with the same object can be referred to as object data.

[0013] same channel After clustering is performed on the data in , the technique can be performed on other data in 2D space. For example, the technique can be performed on data in the same or different azimuth angles. of bins, and / or different channel Further aspects of clustering are discussed throughout this disclosure.

[0014] The techniques further include estimating or otherwise determining a height (and / or height range) associated with the object data. In some examples, a first height (e.g., a lower limit of a height range) can be determined based on a vertical distance associated with the object data. For example, the vertical distance between the beams can be based at least in part on a range between the object data and a location or origin associated with the lidar sensor. That is, the vertical spread of the lidar beam can be a function of the distance between the object data and the lidar sensor. For example, at a first distance (e.g., 5 meters), the vertical distance between the lidar beams can be a first vertical distance (e.g., 3 cm). At a second distance (e.g., 10 meters), the vertical distance between the lidar beams can be a second vertical distance (e.g., 6 cm). Thus, a first height of the object can be determined based at least in part on the number of beams associated with the object. In some examples, the height of the object can be based on a three-dimensional distance determined based on the x-, y-, and z-coordinates of the data.

[0015] However, because the vertical resolution of a lidar sensor may be somewhat limited, the captured object data may not accurately represent the height of an object in the environment. Therefore, the present techniques may include estimating or otherwise determining a second height (e.g., an upper limit of a height range) based on a distance between the object data and a higher or lower beam of the sensor data. For example, the second height of the object may be estimated or otherwise determined based on a first vertical distance from the top of the object to a ray associated with the upper lidar beam, or a second vertical distance from the bottom of the object to a ray associated with the lower lidar beam or the ground. Additional aspects of estimating or otherwise determining the height of an object are discussed throughout this disclosure.

[0016] In some examples, the vehicle's perception system may filter objects based on the object's estimated size. In some examples, the vehicle may be controlled based on the size of the object exceeding a threshold. For example, objects below a threshold may be filtered or ignored for planning purposes (or other actions). In another example, objects above a threshold may be considered for planning purposes (or other actions). Example actions for controlling the vehicle may include, but are not limited to, a safety stop, an emergency stop, determining a trajectory to avoid the object, determining a trajectory to avoid the object's predicted location, etc. Additional actions based on the object's height may include classification actions, predictive actions, etc.

[0017] The height estimation techniques described herein can improve the functionality of computing devices by providing a robust method for estimating or otherwise determining the height of objects in an environment. For example, height data can enable subsequent processes, such as classification, tracking, prediction, path planning, and trajectory generation, to be performed more accurately and may require less processing power and / or less memory. For example, in some examples, faster and / or more accurate segmentation (e.g., using the clustering techniques discussed herein) can be used to generate trajectories for autonomous vehicles, which can improve the safety of occupants of the autonomous vehicles. In some examples, the techniques can be used in a secondary computing system of a primary / secondary computing system architecture, allowing for a redundant system to significantly improve overall reliability and safety outcomes. Furthermore, the techniques discussed herein can be incorporated into systems that can be verified for safety. These and other improvements to computer functionality are discussed herein.

[0018] The technology described herein can be implemented in many ways. Exemplary implementations are provided below with reference to the following drawings. While discussed in the context of autonomous vehicles, the methods, apparatus, and systems described herein can be applied to a variety of systems (e.g., sensor systems, or robotic platforms) and are not limited to autonomous vehicles. In one example, similar technology may be utilized in driver-controlled vehicles where such systems may provide indications of whether it is safe to perform various actions. In another example, the technology may be utilized in manufacturing assembly lines. In the context of , aerial survey In the context of quantity , or In the context of the sea Additionally, the techniques described herein can be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

[0019] FIG. 1 is a schematic flow diagram of an exemplary process 100 for estimating the height of an object.

[0020] The operation 102 may include receiving sensor data of an environment. For example, the operation 102 may include receiving lidar data representing the environment 104. The environment 104 may include a ground surface 106 and an object 108. In some examples, the operation 102 may include receiving sensor data from multiple sensors (e.g., multiple lidar sensors) and fusing or otherwise combining the data into a single point cloud (e.g., a calibrated and aligned data set). In some examples, the operation 102 may be performed by the autonomous vehicle as the autonomous vehicle traverses the environment 104.

[0021] Operation 110 may include removing ground data within environment 104. For example, ground data (e.g., representing ground surface 106) may be removed to reduce the amount of data and / or increase the accuracy of subsequent operations. Additionally, operation 110 may include removing a ground surface associated with the trajectory of the autonomous vehicle. road For example, operation 110 may include identifying a vehicle based at least in part on a width and / or length of the vehicle, a current speed of the vehicle and / or a speed associated with the trajectory, etc. road can be determined.

[0022] Regarding the removal of ground data, operation 110 road The method may include classifying a subset of the sensor data (associated with a roadway and / or other environmental surface) as belonging to either a ground classification or an object classification. The ground classification may identify that the sensor data is associated with a roadway and / or other environmental surface, while the object classification may include any other object that is not the ground. In some examples, the techniques described herein may simply classify the sensor data as either the ground or an object without further identifying which type of object.

[0023] Operation 110 may include techniques that may improve (e.g., reduce) false negative and / or false positive rates associated with classifying sensor data as ground or an object. In some examples, the techniques may include estimating one or more lines and / or surfaces to model the roadway surface profile (e.g., roadway elevation profile), particularly for complex roadways that may include one or more gradient (e.g., roadway surface inclination) changes. These ground fitting techniques may include determining a spline (e.g., one or more polynomials, Bézier curves) associated with the ground based at least in part on determining the number of control points and knots of the spline based at least in part on the number or percentage of sensor data that include valid returns. In some examples, when sensor data is sparse for a portion of the environment, the determined spline may be inaccurate. The techniques may additionally or alternatively weight the sensor data before generating the spline (e.g., performing least squares regression to filter out the lowest and / or highest (altitude) sensor data). point, by weighting the spline toward the control point height (e.g., ordinate) associated with one or more control points based at least in part on the sensor data as a post-fitting operation, and / or modifying the spacing of the knots as a post-fitting operation. In some examples, determining the spline may include determining the control point and / or knot values ​​based at least in part on a regression algorithm (e.g., least squares). Additional examples of identifying ground data can be found, for example, in U.S. Patent Application No. 16 / 588,717, entitled "Complex Ground Profile Estimation," filed September 30, 2019, which is incorporated by reference in its entirety.

[0024] Operation 114 may include clustering the data to determine the objects. Example 116 illustrates clustering the data. point For example, two-dimensional data representation 118 may include a first dimension representing bins 120, 122, 124, and 126. Such bins 120-126 may be clustered according to azimuth angles, as discussed herein. of Additionally, the two-dimensional data representation 118 may include a second dimension representing channels 130, 132, 134, and 136. In some examples, the channels 130-136 may represent bins, as discussed herein. channel In some examples, the number of channels may be based at least in part on number of beams or channels associated with the lidar sensor.

[0025] In some examples, operation 102 or 114 may include associating sensor data with a two-dimensional data representation 118. For example, operation may include associating three-dimensional data with a two-dimensional data representation 118.

[0026] As shown, the two-dimensional data representation 118 is pointIncludes 138, 140, 142, 144, 146, 148, 150, and 152.

[0027] A clustering operation (e.g., operation 114) is used to classify the data point Object data 154 may be determined that represents objects 108 in the environment 104. In general, clustering involves clustering the data in the two-dimensional data representation 118. point For example, the distance between the data point Since the distance between 138 and 140 is less than the threshold distance, point 138 and 140 are included in the object data 154. Furthermore, data point 144 is data point 142 (data associated with object data 154 point ) and data point 144 exceeds the threshold distance and is therefore not included as object data 154. Additional details of the clustering data are discussed in connection with FIG. 2, as well as throughout this disclosure.

[0028] Operation 156 may include estimating (or otherwise determining) the height of the object. Example 158 illustrates determining a first height 160 associated with object data 154 and a second height 162 associated with object data 154. Example 158 represents a side profile of the data represented in example 116 to better illustrate range or depth information between lidar data captured by lidar sensor 164. Example 158 represents additional data (to illustrate various data operations as discussed herein) point Includes 166 and 168.

[0029] The first height 160 can be determined based on the vertical extent of the data associated with the object data 154. For example, the first height can be determined based on the vertical extent of the data associated with the object data 154. pointIn some examples, the first height may be determined based on the vertical heights associated with data 138, 146, and 148. point (e.g., data point 138, 140, 142, 146, and 148) (e.g., the lowest data point 148 and the highest data point 146) or all of the above.

[0030] The second height 162 is the height above the top of the object (e.g., the data point 146) from point (e.g., data point 166), or the bottom of the object (e.g., the data point 148) from point (e.g., data point 152) or a second vertical distance to a ray associated with a lower lidar beam associated with the ground.

[0031] In some examples, the first height and / or second height may be determined for each bin associated with the object, while in some cases the first height and / or second height may be determined for the entire object. That is, a first and second height may be determined for object data associated with bin 120, a first and second height may be determined for object data associated with bin 122, and a first and second height may be determined for object data associated with bin 124. In some cases, the first and second heights may be determined for the entire object data 154 (e.g., all of the data point 138, 140, 142, 146, and 148, or minimum and maximum data associated with the object data. point (For example, each data point148 and 146).

[0032] Additional aspects of determining the height and / or height range associated with object data 154 are discussed in FIGS. 3A-3D, as well as throughout this disclosure.

[0033] 2 is a diagram of an example process 200 for clustering data to determine object data. Process 200 includes examples 202, 204, and 206 that illustrate various processing states.

[0034] Example 202 shows sensor data represented in a two-dimensional data representation 118. In some examples, clustering techniques may be used to identify data that are closest to the location associated with the sensor capturing the data. point For example, the data point 138 is data point 138 (e.g., lidar sensor 164) point In some cases, the data point Determining that 138 is the closest point is point to the object 208. In some examples, the two-dimensional data representation 118 may sort the data values ​​by range, where the data closest to the lidar sensor point Selecting may include accessing a first data value in an array of data values.

[0035] In some examples, when a region (also referred to as a data cell or cell) of the two-dimensional data representation is determined to be associated with an object (e.g., object 208), the data associated with that region is point can be associated with the object 208. In this case, the regions associated with the bins 122 and channels 134 are point Since it contains 138 and 140, both data point may be associated with the object 208.

[0036] In some cases, the data in the data cells point is the anchor point or initial point (e.g., data point 138). For example, distance 210 can be used to cluster the data. point 138 and Data point 140. If the distance 210 is less than the threshold distance, the data point 140 may be associated with object 208. In example 202, the thick black line associated with the data cell associated with bin 122 and channel 134 indicates that the data cell is associated with object 208.

[0037] Example 204 shows the data in channel 134. point For example, clustering refers to an operation performed on data. point 138 and Data point 142. In this example, the distance 212 is less than the distance threshold, so the data point 142 may be associated with an object 214. As mentioned above, in some instances, a single data element within a data cell may be associated with an object 214. point If the is associated with an object, the data associated with the data cell point Some or all of may be associated with the same object.

[0038] Then, the distance 216 is point 142 and Data point 144. In this example, the distance 216 exceeds the threshold distance, so the data point 144 is not associated with the object 214.

[0039] As discussed herein, distances 212, 216, etc., are used to measure the data in the environment. pointThe distance can be determined as a Euclidean distance based on the three-dimensional position of the data. In some cases, distances associated with individual directions can be considered. For example, point The x-distance between the data points can be considered with respect to a first distance threshold. point The y-distance between the data can be considered with respect to a distance threshold. point The z-distance between can be considered with respect to a distance threshold.

[0040] In some instances, the action may involve some or all of the data point The operations may include determining the distance between pairs, and are not limited to the distances explicitly shown in FIG. 2. For example, the operations may include determining the distance between pairs of data point 138 and Data point 144 and evaluating the distance relative to a threshold value.

[0041] Example 206 illustrates additional operations based on data associated with the remaining channels 130, 132, and 136. For example, distance 218 may be calculated based on the data point 140 and Data point In this example, the distance 218 is less than the threshold distance, so the data point 148 can be associated with an object 220. Similarly, data point The distance 222 between 138 and 146 may be below the threshold distance, and therefore the data point 146 (and / or data point 146) may not be associated with the object 220.

[0042] data point 142 and Data point 150 meets or exceeds the distance threshold, so in this example, point 150 is not associated with object 220. Similarly, in this example, data point 148 and Data point152 meets or exceeds the distance threshold, so the data point 152 is not associated with the object 220.

[0043] In some examples, distances 210, 212, 216, 218, 222, 224, and / or 226 can be determined based on the three-dimensional position of the data associated with two-dimensional data representation 118 (e.g., based on x-, y-, and z-coordinates).

[0044] In some examples, the operations may include verifying that the object data 154 corresponds to a minimum width and / or height (and / or that the object data spans a threshold number of bins or channels). In some examples, if the dimensionality of the object data is below a threshold, the object data may be filtered or otherwise ignored, and the technique may be repeated for the remaining subset of data. In some examples, the filtered object data may be recombined with other object data based on a subsequent clustering operation.

[0045] In some examples, an object (e.g., object 220) may be created at an initial point (e.g., data point 138), the process 200 calculates the remaining data in the two-dimensional data representation 118. point That is, a new initial point can be iterated based on the remaining data point (e.g., data not associated with object 220 point ), and the process can be repeated to determine any additional objects in the two-dimensional data representation 118.

[0046] 3A-3D illustrate various scenarios for estimating height associated with sensor data.

[0047] FIG. 3A illustrates an example 300 similar to the scenario described above with respect to example 158 of FIG.

[0048] Example 300 shows data representing object data 154. point 138, 146, and 148. Example 158 represents a side profile of the data represented by examples 116 and 206 to better illustrate range or depth information between the lidar data captured by lidar sensor 164. Example 300 represents a side profile of the data represented by examples 116 and 206 (to illustrate various data operations as discussed herein). point Includes 166 and 168.

[0049] The first height 160 is the data point In some examples, the first height can be determined based on the vertical extents of the data 138, 146, and 148 associated with the object data 154. point (e.g., data point 138, 140, 142, 146, and 148) (e.g., the lowest data point 148 and the highest data point 146) or all of the above.

[0050] The second height 162 is the height above the top of the object (e.g., the data point 146) from point (e.g., data point 166), or the bottom of the object (e.g., the data point 148) from point (e.g., data point 152) or a second vertical distance to a ray associated with a lower lidar beam associated with the ground.

[0051] FIG. 3B shows data representing object data 154. point 3 illustrates a second example scenario 302 for estimating the height associated with a .

[0052] The first height 160 may be determined in the same manner as described above in example 300 .

[0053] Example 302 shows the data point Data, not 152 point 304. In this example 302, the data point The range associated with 304 is smaller than the range associated with some or all of the object data 154, so the data point 304 (and data point The ray associated with 304 may not be used to estimate the second height associated with the object data 154. In some examples, additional channels may be iterated to determine the lowest channel in which data exists at a range further than the range associated with the object data. In this example 302, the second height 306 is determined by the data point 168. That is, the operation of determining the second height (e.g., the estimate of the maximum height) is based at least in part on the ray associated with the data point The method may include determining whether a range associated with the object data 154 exceeds a range associated with the object data 154.

[0054] FIG. 3C shows data representing object data 154. point 3 shows a third example scenario 308 for estimating the height associated with a .

[0055] The first height 160 may be determined in the same manner as described above in example 300 .

[0056] Example 308 shows the data shown in Example 302. point (for) data pointIn this example 308, the upper limit associated with the second height 310 is the data of the object data 154 that represents the highest point associated with the object data 154. point 146. In some examples, depending on the implementation, the second height determined in example 308 can be the same as second height 306. That is, second height 310 is determined based at least in part on data point is associated with a lidar beam, can be based at least in part on the vertical distance between the top of the object data 154 and the next lidar beam.

[0057] FIG. 3D shows data representing object data 154. point 3 shows a fourth example scenario 312 for estimating the height associated with a

[0058] The first height 160 may be determined in the same manner as described above in example 300 .

[0059] Example 312 shows the data illustrated in Example 308. point (for) data point Data, not 168 point 314. In this example 312, the data point The ranges associated with 304 and 314 are smaller than the ranges associated with some or all of the object data 154, so the data point 304 and 314 (and data point The rays associated with 304 and 314 may not be used to estimate the second height associated with the object data 154. However, in this example 312, the non-ground data point is represented below the object data 154 (its range is further than the object data 154). Therefore, the second height 316 is the data representing the ground. point The height above 318 can be determined at least in part based on a known height above 318 .

[0060] As can be appreciated, the techniques from examples 300, 302, 308, and / or 312 can be used in any combination to determine first and / or second heights associated with objects in an environment based on the configuration of the data presented.

[0061] 4 is a block diagram of an example system 400 for implementing the techniques described herein. In at least one example, the system 400 can include a vehicle 402. In the illustrated example system 400, the vehicle 402 is an autonomous vehicle, although the vehicle 402 can be any other type of vehicle.

[0062] Vehicle 402 may be a driverless vehicle, such as an autonomous vehicle configured to operate in accordance with a Level 5 classification issued by the U.S. Department of Transportation's National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for an entire journey without the expectation that a driver (or passenger) will be in control of the vehicle at any time. In such an example, vehicle 402 may be configured to control all functions from the start to the completion of a journey, including all parking functions, and may therefore not include a driver and / or controls for driving vehicle 402, such as a steering wheel, accelerator pedal, and / or brake pedal. This is merely an example, and the systems and methods described herein may be incorporated into any ground, air, or water vehicle, ranging from vehicles that must be manually controlled by a driver at all times to vehicles that are partially or fully autonomously controlled.

[0063] The vehicle 402 may include one or more first computing devices 404, one or more sensor systems 406, one or more emitters 408, one or more communication connections 410 (also referred to as communication devices and / or modems), at least one direct connection 412 (e.g., for physically coupling with the vehicle 402 to exchange data and / or provide power), and one or more drive systems 414. The one or more sensor systems 406 may be configured to capture sensor data associated with an environment.

[0064] The sensor system 406 may include time-of-flight sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement unit (IMU), accelerometer, magnetometer, gyroscope, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, intensity, depth, etc.), microphone sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, etc. The sensor system 406 may include multiple instances of each of these or other types of sensors. For example, the time-of-flight sensors may include individual time-of-flight sensors positioned at the corners, front, rear, sides, and / or top of the vehicle 402. As another example, the camera sensors may include multiple cameras positioned at various locations on the exterior and / or interior of the vehicle 402. The sensor system 406 may provide input to the first computing device 404.

[0065] Vehicle 402 may also include emitters 408 for emitting light and / or sound. Emitters 408 in this example include interior audio and visual emitters to communicate with occupants of vehicle 402. By way of example, and not limitation, interior emitters may include speakers, lights, signs, display screens, touchscreens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.). Emitters 408 in this example may also include exterior emitters. By way of example, and not limitation, exterior emitters in this example include lights for signaling direction of travel or other indicators of vehicle operation (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which may include acoustic beam steering technology.

[0066] Vehicle 402 may also include communications connection 410 that enables communication between vehicle 402 and one or more other local or remote computing devices (e.g., remote teleoperated computing devices) or remote services. For example, communications connection 410 may facilitate communication with other local computing devices on vehicle 402 and / or drive system 414. Communications connection 410 may also enable vehicle 402 to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.).

[0067] The communication connection 410 may include physical and / or logical interfaces for connecting the first computing device 404 to another computing device or to one or more external networks 416 (e.g., the Internet). For example, the communication connection 410 may include Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standard, Bluetooth (registered trademark)This may enable short-range radio frequencies such as, cellular communications (e.g., 2G, 3G, 4G, 4GLTE, 5G, etc.), satellite communications, dedicated short-range communications (DSRC), or any suitable wired or wireless communications protocol that allows each computing device to interface with other computing devices.

[0068] In at least one example, the vehicle 402 may include a drive system 414. In some examples, the vehicle 402 may have a single drive system 414. In at least one example, if the vehicle 402 has multiple drive systems 414, the individual drive systems 414 may be located at opposite ends of the vehicle 402 (e.g., the front and rear, etc.). In at least one example, the drive system 414 may include a sensor system 406 for detecting conditions surrounding the drive system 414 and / or the vehicle 402. By way of example, and not limitation, the sensor system 406 may include one or more wheel encoders (e.g., rotary encoders) to sense the rotation of the wheels of the drive system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) to measure the orientation and acceleration of the drive system, cameras or other imaging sensors, ultrasonic sensors, lidar sensors, radar sensors, etc. to acoustically detect objects in the environment surrounding the drive system. Some sensors, such as the wheel encoders, may be specific to the drive system 414. In some cases, the sensor system 406 on the drive system 414 may overlap or supplement a corresponding system (e.g., sensor system 406) of the vehicle 402.

[0069] The drive system 414 can include many vehicle systems, including a high-voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for braking force distribution to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as headlights / taillights that illuminate the perimeter of the vehicle), and one or more other systems (e.g., a cooling system, safety systems, other electrical components such as an on-board charging system, a DC / DC converter, a high-voltage junction, high-voltage cables, a charging system, a charge port, etc.). The drive system 414 can also receive and preprocess data from the sensor system 406 and can include a drive system controller for controlling the operation of various vehicle systems. In some examples, the drive system controller can include one or more processors and memory communicatively coupled to the one or more processors. The memory can store one or more components for performing various functions of the drive system 414. Additionally, drive system 414 also includes one or more communication connections that enable each drive system to communicate with one or more other local or remote computing devices.

[0070] The vehicle 402 may include one or more second computing devices 418 to provide redundancy, error checking, and / or validation of decisions and / or commands determined by the first computing device 404.

[0071] By way of example, the first computing device 404 may be considered a primary system, and the second computing device 418 may be considered a secondary system. The primary system may generally perform processing that controls how the vehicle operates within an environment. The primary system may implement various artificial intelligence (AI) techniques, such as machine learning, to understand the environment around the vehicle and / or direct the vehicle to move within the environment. For example, the primary system may implement AI techniques to position the vehicle, detect objects around the vehicle, segment sensor data, determine object classifications, predict object tracks, generate vehicle trajectories, etc. In an example, the primary system processes data from multiple types of sensors on the vehicle, such as light detection and ranging (lidar) sensors, radar sensors, image sensors, depth sensors (time-of-flight, structured light, etc.), etc.

[0072] The secondary system may verify the operation of the primary system and may take over control of the vehicle from the primary system when the primary system experiences problems. The secondary system may implement probabilistic techniques based on the positioning, velocity, acceleration, etc. of the vehicle and / or objects surrounding the vehicle. For example, the secondary system may implement one or more probabilistic techniques for independently localizing the vehicle (e.g., in the local environment), detecting objects around the vehicle, segmenting sensor data, identifying object classifications, predicting object tracks, generating vehicle trajectories, etc. In examples, the secondary system processes data from several sensors, such as a subset of the sensor data processed by the primary system. To illustrate, the primary system may process lidar data, radar data, image data, depth data, etc., and the secondary system may process only lidar data and / or radar data (and / or time-of-flight data). However, in other examples, the secondary system may process sensor data from any number of sensors, such as data from each of the sensors, data from the same number of sensors as the primary system, etc.

[0073] Additional examples of vehicle architectures including primary and secondary computing systems can be found, for example, in U.S. Patent Application No. 16 / 189,726, entitled "Perception Collision Avoidance," filed November 13, 2018, the entire contents of which are incorporated herein by reference.

[0074] The first computing device 404 may include one or more processors 420 and a memory 422 communicatively coupled to the one or more processors 420. In the illustrated example, the memory 422 of the first computing device 404 stores a localization component 424, a perception component 426, a prediction component 428, a planning component 430, a map component 432, and one or more system controllers 434. While depicted as residing in the memory 422 for illustrative purposes, it is contemplated that the positioning component 424, the perception component 426, the prediction component 428, the planning component 430, the map component 432, and the one or more system controllers 434 may additionally or alternatively be accessible to the first computing device 404 (e.g., stored in a different component of the vehicle 402) and / or may be accessible to the vehicle 402 (e.g., stored remotely).

[0075] In the memory 422 of the first computing device 404, the positioning component 424 may include functionality for receiving data from the sensor system 406 to determine the position of the vehicle 402. For example, the localization component 424 may include and / or request / receive a three-dimensional map of the environment and continuously determine the position of the autonomous vehicle within the map. In some examples, the localization component 424 may use simultaneous localization and mapping (SLAM) or calibration, localization, and mapping (CLAMS) to receive time-of-flight data, image data, lidar data, radar data, sonar data, IMU data, GPS data, wheel encoder data, or any combination thereof to accurately determine the position of the autonomous vehicle. In some examples, the localization component 424 may provide data to various components of the vehicle 402 to determine an initial position for generating a trajectory, as described herein.

[0076] The perception component 426 may include functionality for performing object detection, segmentation, and / or classification. In some examples, the perception component 426 may detect objects in proximity to the vehicle 402. Object The presence and / or Object as types (e.g., cars, pedestrians, bicyclists, buildings, trees, road surfaces, curbs, sidewalks, unknowns, etc.) Object In an additional or alternative example, the perception component 426 may provide processed sensor data indicative of a classification of the detected Object and / or Object In some examples, the sensor data may provide processed sensor data indicative of one or more characteristics associated with the environment in which the sensor is located. Object The properties associated with are x position (global position), y position (global position), z position (global position), direction, ObjectType (e.g. classification), Object The speed of Object Characteristics associated with an environment can include, but are not limited to, the extent (size) of another environment. Object the presence of other Object These may include, but are not limited to, status, time of day, day of the week, season, weather conditions, dark / light indication, etc.

[0077] As described above, the perception component 426 can use perception algorithms to determine perception-based bounding boxes associated with objects in the environment based on sensor data. For example, the perception component 426 can receive image data and classify the image data to determine that an object is represented in the image data. Then, using a detection algorithm, the perception component 426 can generate a two-dimensional bounding box and / or a perception-based three-dimensional bounding box associated with the object. The perception component 426 can further generate a three-dimensional bounding box associated with the object. As described above, the three-dimensional bounding box can provide additional information, such as a position, orientation, pose, and / or size (e.g., length, width, height, etc.) associated with the object.

[0078] The perception component 426 may include functionality for storing the sensory data generated by the perception component 426. In some examples, the perception component 426 may determine a track corresponding to an object classified as an object type. By way of example only, the perception component 426 using the sensor system 406 may capture one or more images of an environment. The sensor system 406 may capture an image of the environment including an object, such as a pedestrian. The pedestrian may be at a first location at time T and at a second location at time T+t (e.g., movement during a span of time t after time T). In other words, the pedestrian may move from a first location to a second location during this time period. Such movement may be logged, for example, as stored sensory data associated with the object.

[0079] The stored sensory data, in some examples, may include fused sensory data captured by the vehicle 402. The fused sensory data may include a fusion or other combination of sensor data from the sensor system 406, such as image sensors, lidar sensors, radar sensors, time-of-flight sensors, sonar sensors, global positioning system sensors, interior sensors, and / or any combination thereof. The stored sensory data may additionally or alternatively include classification data including a semantic classification of an object represented in the sensor data (e.g., a pedestrian, a vehicle, a building, a road surface, etc.). The stored sensory data may additionally or alternatively include track data (position, orientation, sensor features, etc.) corresponding to the movement of an object classified as a dynamic object through the environment. The track data may include multiple tracks of multiple different objects over time. This track data may be mined to identify images of particular types of objects (e.g., pedestrians, animals, etc.) when the object is stationary (e.g., stationary) or moving (e.g., walking, running, etc.). In this example, the computing device determines a track corresponding to a pedestrian.

[0080] The prediction component 428 can generate one or more probability maps that represent predicted probabilities of possible locations of one or more objects in the environment. For example, the prediction component 428 can generate one or more probability maps for vehicles, pedestrians, animals, etc. within a threshold distance from the vehicle 402. In some examples, the prediction component 428 can measure the tracks of the objects and generate discretized predicted probability maps, heat maps, probability distributions, discretized probability distributions, and / or trajectories for the objects based on the observed and predicted behavior. In some examples, the one or more probability maps can represent the intent of one or more objects in the environment.

[0081] The planning component 430 can determine a path for the vehicle 402 to take to traverse an environment. For example, the planning component 430 can determine various routes and paths and various levels of detail. In some examples, the planning component 430 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this description, a route is a series of steps to travel between two locations. Route location information As a non-limiting example, Route location information The route map may include roads, intersections, global positioning system (GPS) coordinates, etc. Additionally, the planning component 430 may generate instructions for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 430 may generate instructions for guiding the autonomous vehicle along a series of Route location information The first Route location information From a series of Route location information The second Route location informationThe instructions can determine how to guide the vehicle to a destination. In some examples, the instructions can be a path or a portion of a path. In some examples, multiple paths can be generated substantially simultaneously (e.g., within technical tolerances) according to receding horizon techniques. A single path of the multiple paths in the antibody data horizon that has the highest confidence level can be selected for maneuvering the vehicle.

[0082] In other examples, the planning component 430 can alternatively or additionally use data from the perception component 426 and / or the prediction component 428 to determine a path for the vehicle 402 to take to traverse an environment. For example, the planning component 430 can receive data from the perception component 426 and / or the prediction component 428 regarding objects associated with the environment. Using this data, the planning component 430 can determine a path to travel from a first location (e.g., a current location) to a second location (e.g., a target location) to avoid the objects in the environment. In at least some examples, such a planning component 430 can determine that no such collision-free path exists and then provide a path that leads the vehicle 402 to a safe stop that avoids all collisions and / or otherwise mitigates damage.

[0083] The memory 422 may further include one or more maps 432 that can be used by the vehicle 402 to navigate within an environment. For purposes of this discussion, maps may include, but are not limited to, ,intersection, The map can be any number of data structures modeled in 2-, 3-, or N-dimensions that can provide information about the environment, such as streets, mountains, roads, terrain, and the general environment. In some examples, the map can include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information, etc.), spatial information (e.g., image data projected onto a mesh, etc.), and the like. , piecesThe map(s) 432 may include at least one map (e.g., an image and / or a mesh). In some examples, the map(s) 432 may include at least one map (e.g., an image and / or a mesh). In some examples, the vehicle 402 may be controlled based at least in part on the map 432. That is, the map 432 may be used in conjunction with the localization component 424, the perception component 426, the prediction component 428, and / or the planning component 430 to determine the location of the vehicle 402, identify objects in the environment, generate predicted probabilities associated with the objects and / or the vehicle 402, and / or generate a path and / or trajectory to navigate through the environment.

[0084] In some examples, one or more maps 432 may be stored on a remote computing device (such as computing device 448) accessible via network 416. In some examples, multiple maps 432 may be stored, for example, based on a characteristic (e.g., type of entity, time of day, day of the week, season of the year, etc.). Storing multiple maps 432 may have similar memory requirements but may increase the speed at which data in the maps can be accessed.

[0085] In at least one example, first computing device 404 can include one or more system controllers 434, which can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 402. These system controllers 434 can communicate with and / or control corresponding systems of drive system 414 and / or other components of vehicle 402, which can be configured to operate according to the path provided by planning component 430.

[0086] Second computing device 418 may include one or more processors 436 and memory 438 that include components for verifying and / or controlling aspects of vehicle 402, as discussed herein. In at least one example, one or more processors 436 may be similar to processor 420, and memory 438 may be similar to memory 422. However, in some examples, processor 436 and memory 438 may comprise different hardware than processor 420 and memory 422 for additional redundancy.

[0087] In some examples, the memory 438 can comprise a localization component 440 , a perception / prediction component 442 , a planning component 444 , and one or more system controllers 446 .

[0088] In some examples, the localization component 440 may receive sensor data from the sensors 406 to determine one or more of the position and / or orientation (along with the pose) of the autonomous vehicle 402, where the position and / or orientation may be relative to a point and / or object in the environment in which the autonomous vehicle 402 is located. In examples, the orientation may include an indication of the yaw, roll, and / or pitch of the autonomous vehicle 402 relative to a reference surface and / or relative to a point and / or object. In examples, the localization component 440 may perform less processing (e.g., higher-level localization) than the localization component 424 of the first computing device 404. For example, rather than determining the pose of the autonomous vehicle 402 relative to a map, the localization component 440 may only determine the pose of the autonomous vehicle 402 relative to objects and / or surfaces detected in the autonomous vehicle 402's surroundings (e.g., local positions rather than global positions). Such position and / or orientation may be determined, for example, using some or all of the sensor data using probabilistic filtering techniques such as a Bayesian filter (Kalman filter, extended Kalman filter, unscented Kalman filter, etc.).

[0089] In some examples, the perception / prediction component 442 may include functionality to detect, identify, classify, and / or track objects represented in the sensor data. For example, the perception / prediction component 442 may perform clustering operations and operations as discussed herein to estimate or determine heights associated with the objects.

[0090] In some examples, the perception / prediction component 442 may include an M-estimator but may lack an object classifier, such as, for example, a neural network, a decision tree, and / or the like, for classifying objects. In additional or alternative examples, the perception / prediction component 442 may comprise any type of ML model configured to disambiguate object classification. In contrast, the perception component 426 may comprise a pipeline of hardware and / or software components that may include one or more machine learning models, a Bayesian filter (e.g., a Kalman filter), a graphics processing unit (GPU), and / or the like. In some examples, the perception data determined by the perception / prediction component 442 (and / or 426) may include object detection (e.g., identification of sensor data associated with objects in the environment surrounding the autonomous vehicle), object classification (e.g., identification of an object type associated with a detected object), object track (e.g., past, current, and / or predicted object position, velocity, acceleration, and / or orientation), and / or the like.

[0091] The perception / prediction component 442 may also process the input data to determine one or more predicted trajectories for the object. For example, based on the object's current position and the object's velocity over a period of several seconds, the perception / prediction component 442 may predict the path the object will travel over the next several seconds. In some examples, such predicted paths may include using linear assumptions of position, orientation, velocity, and / or motion given orientation. In other examples, such predicted paths may include more complex analysis.

[0092] In some examples, the planning component 444 may include functionality to receive a trajectory from the planning component 430 to verify that the trajectory is collision-free and / or within a safety margin. In some examples, the planning component 444 may generate a safe stopping trajectory (e.g., a trajectory that stops the vehicle 402 with a "comfortable" deceleration (e.g., less than maximum deceleration)), and in some examples, the planning component 444 may generate an emergency stopping trajectory (e.g., maximum deceleration with or without steering input).

[0093] In some examples, the system controller 446 may include functionality to control safety-critical components of the vehicle (e.g., steering, brakes, motors, etc.) In this manner, the second computing device 418 may provide redundant and / or additional hardware and software layers for vehicle safety.

[0094] Vehicle 402 can be connected to computing device 448 via network 416 and can include one or more processors 450 and memory 452 communicatively coupled to the one or more processors 450. In at least one example, one or more processors 450 can be similar to processor 420, and memory 452 can be similar to memory 422. In the example shown, memory 452 of computing device 448 stores component 454, which can correspond to any of the components discussed herein.

[0095] Processors 420, 436, and / or 450 may be any suitable processor capable of processing data and executing instructions to perform operations as described herein. By way of example, and without limitation, processors 420, 436, and / or 450 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts it into other electronic data that can be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement encoded instructions.

[0096] Memory 422, 438, and / or 452 are examples of non-transitory computer-readable media. Memory 422, 438, and / or 452 may store an operating system and one or more software applications, instructions, programs, and / or data that implement the methods and functionality attributed to the various systems described herein. In various implementations, memory 422, 438, and / or 452 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash-type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which those shown in the accompanying drawings are merely examples relevant to the description herein.

[0097] In some examples, some or all aspects of the components described herein may include any model, algorithm, and / or machine learning algorithm. For example, in some examples, the components in memory 422, 438, and / or 452 may be implemented as neural networks. In some examples, the components in memory 422, 438, and / or 452 may not include machine learning algorithms to reduce complexity and be validated and / or certified for safety.

[0098] As described herein, an exemplary neural network is a biologically inspired algorithm in which input data is passed through successively connected layers to produce an output. Each layer in a neural network can include another neural network, or can include any number of layers (convolutional or not). As can be understood in the context of this disclosure, neural networks can utilize machine learning, which can refer to such a broad class of algorithms in which output is generated based on learned parameters.

[0099] Although described in the context of neural networks, any type of machine learning may be used consistent with this disclosure. For example, machine learning algorithms include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression spline (MARS), locally estimated scatterplot smoothing (LOESS), instance-based algorithms (e.g., ridge regression, least absolute value shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression tree (CART), iterative dichotomy 3 (ID3), chi-squared automated interaction detection (CHAID), decision stump, conditional decision tree), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average-one dependence estimator (AODE), Bayesian confidence network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, error The learning algorithms may include, but are not limited to, backpropagation, Hopfield networks, radial basis function networks (RBFNs), deep learning algorithms (e.g., deep Boltzmann machines (DBMs), deep confidence networks (DBNs), convolutional neural networks (CNNs), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), adaboost, hierarchical generalization (blending), gradient boosting machines (GBMs), gradient boosted regression trees (GBRTs), random forests), support vector machines (SVMs), supervised learning, unsupervised learning, semi-supervised learning, etc.

[0100] Further example architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, and PointNet.

[0101] Thus, the techniques discussed herein provide a robust implementation for determining height data of objects in an environment to enable safe operation of vehicles, such as autonomous vehicles.

[0102] 5 and 6 illustrate exemplary processes according to examples of the present disclosure. These processes are illustrated as logical flow graphs, with each operation representing a sequence of actions that may be implemented in hardware, software, or a combination thereof. In a software context, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited actions. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0103] 5 is an example process 500 for estimating a height associated with an object based on sensor data. For example, as described herein, some or all of process 500 may be performed by one or more components in FIG. 4. For example, some or all of process 500 may be performed by second computing device 418.

[0104] In operation 502, the process may include receiving data captured by sensors of the vehicle. In some examples, operation 502 may include receiving or capturing lidar data from one or more lidar sensors of the autonomous vehicle as the autonomous vehicle traverses an environment.

[0105] In operation 504, the process can include associating the data with a two-dimensional space. For example, operation 504 can include associating the three-dimensional data with a two-dimensional data representation. The two-dimensional data representation can include an azimuth angle. of The first dimension may represent a bin, and the second dimension may represent a lidar beam or channel. In some examples, operation 504 filters the data to remove data associated with the ground and / or associated with a trajectory followed by the autonomous vehicle. road This may include removing data outside of the

[0106] At operation 506, the process may include determining object data representing the object based at least in part on the data clustering. As discussed herein, the data clustering may be based at least in part on determining that the distance of the points is less than a threshold distance. Additional aspects of data clustering are described in FIGS. 1 and 2 and throughout this disclosure.

[0107] At operation 508, the process may include determining a height associated with the object. As discussed herein, operation 508 may include determining a first height and a second height associated with the object, which in some examples may represent a minimum height (or lower limit) associated with the object and a maximum height (or upper limit) associated with the object, as discussed herein.

[0108] At operation 510, the process may include determining a first height based at least in part on the extent of the object data. For example, the first height may be based at least in part on a vertical distance associated with the object data. Additional aspects of determining the first height are described in Figures 1 and 3A-3D and throughout this disclosure.

[0109] At operation 512, the process may include determining a second height based at least in part on a distance between 1) a first portion within the first channel and 2) a second location within the second channel or a third location associated with the ground. Additional aspects of determining the second height are described in Figures 1, 3A-3D, and 6, as well as throughout this disclosure.

[0110] In operation 514, the process determines whether the vehicle is moving based at least in part on the height associated with the object. How it works In some examples, controlling the vehicle may include generating a trajectory, executing a safety stop trajectory, executing an emergency stop trajectory, etc. In some examples, the operation may include determining whether the height meets or exceeds a threshold height for subsequent classification, filtering, etc.

[0111] 6 is an example process 600 for determining the height of an object based on upper and lower bounds associated with the object data. For example, as described herein, some or all of process 600 may be performed by one or more components in FIG. 4. For example, some or all of process 600 may be performed by second computing device 418.

[0112] In some examples, process 600 may represent the operations performed in operation 512 of Figure 5 .

[0113] In general, operations 604, 606, 608, and 610 may correspond to estimating or otherwise determining an upper limit (or upper range) associated with the height of the object, and operations 612, 614, 616, and 618 may correspond to estimating or otherwise determining a lower limit (or lower range) associated with the height of the object.

[0114] At operation 602, the process may include receiving object data. For example, the object data may be received after being associated with a two-dimensional data representation and / or clustered based on the techniques discussed herein.

[0115] At operation 604, the process may include determining whether data exists above the object data. For example, data in any channel above the object data may correspond to another object (e.g., the data may be associated with a second object different from the object data). If the data is in the channel above the object data, the process continues to operation 606. In some examples, operation 604 may be limited to data within the same bin above the object data. That is, if the object data is not associated with a bin, operation 604 may not consider such data in operation 604.

[0116] At operation 606, the process may include determining whether the range of the data in the channel (e.g., above the object data) is greater than the range associated with the object data. If the range of the data in the channel is greater than the range associated with the object data, the process proceeds to operation 608, which may include using a ray associated with the channel as an upper limit for the height of the object. In some examples, operation 608 may include determining a vertical distance between the object data (e.g., the highest point associated with the object data), a ray associated with data above the object data, and a position associated with the lidar sensor.

[0117] If there is no data in the channel above the object (e.g., "No" at operation 604), or if the range of the data in the channel above the object data is smaller than the range of the object data (e.g., "No" at operation 606), the process proceeds to operation 610.

[0118] At operation 610, the process may include using the top of the object as an upper limit (or upper range) for the object's height. In some examples, operation 610 may include using a location associated with the top of the object as an upper limit (or upper range) for the object's height, which may be based at least in part on a channel directly above the top of the object.

[0119] At operation 612, the process may include determining whether data exists below the object data. For example, data in any channel below the object data may correspond to another object (e.g., the data may be associated with a second object different from the object data). If the data is in a channel below the object data, the process proceeds to operation 614. In some examples, operation 612 may be limited to data within the same bin below the object data. That is, if the object data is not associated with a bin, operation 612 may not consider such data in operation 612.

[0120] At operation 614, the process may include determining whether the range of the data in the channel (e.g., below the object data) is greater than the range associated with the object data. If the range of the data in the channel is greater than the range associated with the object data, the process proceeds to operation 616, which may include using the ray associated with the channel as a lower bound on the height of the object. In some examples, operation 616 may include determining the vertical distance between the object data (e.g., the lowest point associated with the object data), the ray associated with the data below the object data, and a position associated with the lidar sensor.

[0121] If there is no data in the channel below the object (e.g., "No" at operation 612), or if the range of the data in the channel below the object data is smaller than the range of the object data (e.g., "No" at operation 614), the process proceeds to operation 618.

[0122] In operation 618, the process may include using the ground plane as a lower limit (or range) for the height of the object.

[0123] At operation 620, the process may include determining a second height based at least in part on the upper and lower bounds as determined in operations 604-618. Thus, process 600 illustrates a robust technique for estimating or otherwise determining the height of an object. (Illustrative clause)

[0124] A: one or more processors; and when executed, the one or more processors may receive lidar data captured by a lidar sensor of the autonomous vehicle; associate the lidar data with a two-dimensional data representation; receive an indication of a ground surface represented in the two-dimensional data representation; remove a first subset of the lidar data associated with the ground surface from the two-dimensional data representation to determine a second subset of data; determine object data in the second subset of data representing the object based at least in part on the data clustering; determine a first height associated with the object based at least in part on a vertical extent of the object data; and, based on the data clustering at least in part on the vertical extent of the object data, determine a first height associated with the object. point and a height associated with a second channel of the two-dimensional data representation; and controlling an autonomous vehicle based at least in part on at least one of the first height or the second height associated with the object.

[0125] B: The vertical distance is the first distance, and the data clustering is the first data of the lidar data. point and the second data of the lidar data point and determining the object data based at least in part on the second distance.

[0126] C: The system of paragraph A or B, wherein the object is a first object, the first data associated with the first channel represents a first portion of the object data, the second data associated with the second channel represents a second object different from the first object, the vertical distance represents a distance between a ray associated with the first data and the second data, and the ray is based at least in part on an origin associated with the second object and the lidar sensor.

[0127] D: The system of any of paragraphs A-C, wherein the vertical distance is a first distance and the second height is based at least in part on a second distance between the object and a position associated with the lidar sensor.

[0128] E: The first dimension of the two-dimensional data representation is the azimuth angle associated with the lidar data. of The second dimension of the two-dimensional data representation represents the bins associated with the lidar data. channel The system of any of paragraphs A-D,

[0129] F: A method comprising: receiving data captured by a sensor of a vehicle; associating the data with a two-dimensional data representation; determining object data representing an object based at least in part on the data clustering; determining as a height associated with the object at least one of a first height associated with the object based at least in part on a range of the object data, or a second height associated with the object based at least in part on a distance between a first position in a first channel of the two-dimensional data representation and a second position in a second channel of the two-dimensional data representation; and controlling the vehicle based at least in part on the height associated with the object.

[0130] G: The distance is the first distance, and the data clustering is the first data of the data point and the second data pointand determining the object data based at least in part on the distance.

[0131] H: The method of paragraph F or G, wherein the object is a first object, the first data associated with the first channel represents a first portion of the object data, the second data associated with the second channel represents a second object different from the first object, the distance represents a perpendicular distance between a ray associated with the first data and the second data, and the ray is based at least in part on a position associated with the second object and the sensor.

[0132] I: The method of paragraph H, wherein the distance is a first distance, and the method further includes determining a second distance between the first data and a location associated with the sensor, determining a third distance between the second data and a location associated with the sensor, determining that the second distance is less than the third distance, and determining a second height based at least in part on the second distance being less than the third distance.

[0133] J: The method of any of paragraphs FI, further including receiving an indication of a ground plane represented in the data; and removing a subset of the data from the two-dimensional data representation based at least in part on the ground plane.

[0134] K: The method of any of paragraphs FJ, wherein the first height represents a minimum height associated with the object and the second height represents a maximum height associated with the object.

[0135] L: The method of any of paragraphs FK, wherein the distance is a first distance, the data is captured by a lidar sensor, the second height is based at least in part on a vertical distance between a first beam of the lidar sensor and a second beam of the lidar sensor, and the vertical distance is based at least in part on a second distance between the object and a position associated with the lidar sensor.

[0136] M: The method of any of paragraphs FL, further including determining a second height based at least in part on the position of the object above the ground.

[0137] N: The data is lidar data captured by a lidar sensor, and the first dimension of the two-dimensional data representation is an azimuth angle associated with the lidar data. of The second dimension of the two-dimensional data representation represents the bins associated with the lidar data. channel The method of any of paragraphs FM, representing

[0138] O: A non-transitory computer-readable medium storing instructions executable by a processor, which, when executed, cause the processor to perform operations including receiving data captured by a sensor of the vehicle; associating the data with a two-dimensional data representation; determining object data representing the object based at least in part on the data clustering; determining as a height associated with the object at least one of: a first height associated with the object based at least in part on a range of the object data; or a second height associated with the object based at least in part on a perpendicular distance between a first position in a first channel of the two-dimensional data representation and a light ray in a second channel of the two-dimensional data representation, wherein the light ray is based at least in part on the sensor; and controlling the vehicle based at least in part on the height associated with the object.

[0139] P: The distance is the first distance, and the data clustering is the first data of the data point and the second data point and determining the object data based at least in part on the distance.

[0140] Q: The non-transitory computer-readable medium of paragraph O or P, wherein the object is a first object, the first data associated with the first channel represents a first portion of the object data, the second data associated with the second channel represents a second object different from the first object, the vertical distance represents a distance between a ray associated with the first data and the second data, and the ray is based at least in part on a position associated with the second object and the sensor.

[0141] R: The non-transitory computer-readable medium of any of paragraphs OQ, wherein the data is captured by a lidar sensor, and the second height is based at least in part on a vertical distance between a first beam of the lidar sensor and a second beam of the lidar sensor, and the vertical distance is based at least in part on a distance between the object and a position associated with the lidar sensor.

[0142] S: The non-transitory computer-readable medium of any of paragraphs OR, determining a second height based at least in part on a position of the object above the ground.

[0143] T: The data is lidar data captured by a lidar sensor, and the first dimension of the two-dimensional data representation is the azimuth angle associated with the lidar data. of The second dimension of the two-dimensional data representation represents the bins associated with the lidar data. channel 10. The non-transitory computer-readable medium of any of paragraphs OS, representing:

[0144] While the example sections described above are described with respect to one particular implementation, it should be understood that in the context of this specification, the content of the example sections can also be implemented via a method, device, system, computer-readable medium, and / or another implementation. Furthermore, any of the example ATs can be implemented alone or in combination with one or more other example ATs. (Conclusion)

[0145] While one or more examples of the technology described herein have been described, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0146] In describing examples, reference is made to the accompanying drawings, which form a part hereof, that illustrate specific examples of the claimed subject matter. It is understood that other examples may be used and that modifications or variations, such as structural changes, may be made. Such examples, modifications, or variations do not necessarily constitute a departure from the intended scope of the claimed subject matter. While the steps described herein may be presented in a particular order, in some cases the ordering may be changed such that certain inputs are provided at different times or in a different order without changing the functionality of the described systems and methods. The disclosed procedures may also be performed in a different order. Furthermore, the various calculations herein need not be performed in the order disclosed, and other examples using alternative orders of calculations may be readily implemented. In addition to being reordered, calculations may also be decomposed into partial calculations with the same result.

Claims

1. 1. A method for estimating the size of an object in an environment, comprising: receiving sensor data of an environment captured by a three-dimensional sensor of the vehicle; representing the sensor data in a two-dimensional data representation; performing clustering on the sensor data to determine object data representing an object, wherein performing clustering on the sensor data includes classifying first data points included in the object data and second data points not included in the object data, and the object data is represented by the first data points determined to be included in the object data; as the determined height of the object, a first height representing the minimum height of the object by determining the difference between a first vertical height of the highest data point and a second vertical height of the lowest data point among the data points included in the object data; or a second height representing a maximum height of the object by determining the distance between a position indicating an upper limit of the height of the object and a position indicating a lower limit of the height of the object within a channel included in the two-dimensional data representation; determining at least one of: determining a classification of the object based on the determined height, and controlling operation of the vehicle based on the object and the classification of the object; A method comprising:

2. Classifying a first data point included in the object data and a second data point not included in the object data includes: determining a three-dimensional Euclidean distance between an initial point included in the sensor data and the object data and one or more data points other than the initial point among the data points other than the initial point included in the sensor data, and comparing the three-dimensional Euclidean distance with a threshold distance to determine that the one or more data points other than the initial point are included in the object data if the three-dimensional Euclidean distance is less than the threshold distance; The method of claim 1.

3. The distance is a vertical distance from the highest data point to a ray of light emitted from the vehicle on which the three-dimensional sensor is mounted to a data point of object data having a range greater than the range of the object data located at a position higher than the position of the highest data point; a perpendicular distance from the lowest data point to the ray for a data point of object data having an extent greater than the extent of the object data located below the location of the lowest data point; the vertical distance between the highest data point and the lowest data point; Including, 3. The method according to claim 1 or 2.

4. The distance between the positions of the highest data point and the lowest data point and the position of the three-dimensional sensor is 4. The method of claim 3, wherein the second height is calculated by adding together the vertical distance from the highest data point to the ray for a data point of object data that is located higher than the position of the highest data point and lower than the position of the lowest data point and has a range greater than the range of the object data, the vertical distance from the lowest data point to the ray for a data point of object data that is located lower than the position of the lowest data point and has a range greater than the range of the object data, and the vertical distance between the highest data point and the lowest data point, if the second height is less than the distance between the position of the three-dimensional sensor and a data point of object data that is located higher than the position of the highest data point and lower than the position of the lowest data point and has a range greater than the range of the object data.

5. receiving a representation of the ground surface represented in the sensor data; removing a subset of the sensor data of the ground surface from the two-dimensional data representation; 5. The method of claim 1, further comprising:

6. the sensor data is captured by a lidar sensor; 5. The method of claim 3 or 4, wherein the light beam is a lidar beam.

7. A method according to any one of claims 1 to 6, wherein the position indicating the lower limit of the height of the object is the ground.

8. the sensor data is lidar data from a lidar sensor; 8. The method of claim 1, wherein the two-dimensional data representation includes a first dimension representing azimuth bins of the lidar data and a second dimension representing channels of the lidar data, wherein the azimuth bins refer to angles in azimuth separated by a fixed angle along the azimuth direction of the lidar beam, and the channels refer to widths of distance separated by a fixed distance along the radial direction of the lidar beam.

9. A computer program comprising coded instructions which, when executed on a computer, implements the method of any one of claims 1 to 8.

10. 1. A system for estimating the size of an object in an environment, comprising: one or more processors; one or more computer-readable storage media storing instructions executable by the one or more processors, the instructions, when executed, causing the system to: receiving environmental sensor data from a three-dimensional sensor on the vehicle; representing the sensor data in a two-dimensional data representation; performing clustering on the sensor data to determine object data representing an object, wherein performing clustering on the sensor data includes classifying first data points included in the object data and second data points not included in the object data, and the object data is represented by the first data points determined to be included in the object data; as the determined height of the object, a first height representing the minimum height of the object by determining the difference between a first vertical height of the highest data point and a second vertical height of the lowest data point among the data points included in the object data; or a second height representing the maximum height of the object by determining the vertical distance between a position vertically above the highest data point and a position where a light ray emitted from the vehicle on which the three-dimensional sensor is mounted intersects, and a position vertically below the lowest data point and a position where the light ray intersects; determining at least one of: determining a classification of the object based on the determined height, and controlling operation of the vehicle based on the object and the classification of the object; and A system comprising:

11. Classifying a first data point included in the object data and a second data point not included in the object data includes: determining a three-dimensional Euclidean distance between an initial point included in the sensor data and the object data and one or more data points other than the initial point among the data points other than the initial point included in the sensor data, and comparing the three-dimensional Euclidean distance with a threshold distance to determine that the one or more data points other than the initial point are included in the object data if the three-dimensional Euclidean distance is less than the threshold distance; The system of claim 10.

12. The vertical distance is: the perpendicular distance from the highest data point to the ray for a data point of object data having an extent greater than the extent of the object data located higher than the location of the highest data point; a perpendicular distance from the lowest data point to the ray for a data point of object data having an extent greater than the extent of the object data located below the location of the lowest data point; the vertical distance between the highest data point and the lowest data point; Including, 12. The system according to claim 10 or 11.

13. the sensor data is captured by a lidar sensor; The system of claim 10 , wherein the light beam is a lidar beam.

14. the sensor data is lidar data from a lidar sensor; 14. The system of claim 10, wherein the two-dimensional data representation includes a first dimension representing azimuth bins of the lidar data and a second dimension representing channels of the lidar data, wherein the azimuth bins refer to angles in azimuth separated by a fixed angle along the azimuth direction of the lidar beam, and the channels refer to widths of distance separated by a fixed distance along the radial direction of the lidar beam.

Images