Pedestrians with objects

Abstract

Detection of pedestrians with objects is described (e.g., strollers, pushing carts, opening doors, carrying umbrellas, etc.). In an example, a perception component of a vehicle can receive sensor data from sensors coupled to the vehicle. The perception component can determine observations associated with the sensor data via a model, where the observations include a first object (e.g., a pedestrian). The perception component can determine whether the first object is associated with a second object (e.g., a pedestrian object), where the first object and the second object are associated in a composite object (e.g., a pedestrian / pedestrian-object system). The perception component can provide a representation of the first object or a representation of the composite object to at least one of a prediction component or a planning component of the vehicle for control of the vehicle.

Description

[Technical field]

[0001] The present invention relates to the recognition of pedestrians with objects. [Background technology]

[0002] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. patent application Ser. No. 16 / 726,042, filed Dec. 23, 2019, entitled “Pedestrian with Objects,” and U.S. patent application Ser. No. 16 / 726,097, filed Dec. 23, 2019, entitled “Pedestrian Object Detection Training,” the entire contents of both of which are incorporated herein by reference.

[0003] An autonomous vehicle may use sensors to obtain data of the environment. To efficiently navigate the environment, the autonomous vehicle may use sensor data to detect objects in the environment and avoid collisions. For example, a perception system may enable the autonomous vehicle to identify objects in the environment so that the autonomous vehicle may plan a safe route through the environment. In an example, segmentation techniques may be used to associate sensor data with objects. Safe operation of an autonomous vehicle depends at least in part on information made available by the perception system in detecting, classifying, and predicting the movement of objects detected in the sensor data. Thus, inaccurate and / or incomplete information provided by the detection, classification, and / or prediction regarding such objects may reduce the safety with which the autonomous vehicle can operate. [Brief description of the drawings]

[0004] The detailed description will be described with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. Use of the same reference number in different drawings indicates similar or identical components or functionality.

[0005] [Figure 1] FIG. 2 illustrates an example of an environment in which pedestrian objects can be detected as described herein. [Diagram 2] FIG. 2 illustrates an example process for training a model for pedestrian object detection as described herein. [Diagram 3] FIG. 2 illustrates an example process for detecting a pedestrian object and detecting motion associated therewith, as described herein. [Figure 4] FIG. 1 is a block diagram illustrating an example of a system for implementing the techniques described herein. [Diagram 5] FIG. 2 illustrates an example process for training a model for pedestrian object detection as described herein. [Figure 6] FIG. 2 illustrates an example process for detecting pedestrian objects as described herein. [Figure 7] 4A-4C illustrate an example process for determining whether a pedestrian object is associated with a pedestrian as described herein. [Figure 8] 4A-4C illustrate an example process for determining whether a pedestrian object is associated with a pedestrian as described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] [Detailed Description] Techniques described herein relate to training models to detect objects associated with pedestrians and use of the models by vehicles, such as autonomous vehicles. In examples, such models can be used to detect objects associated with pedestrians in the vehicle's environment, and based on detection of such objects, the vehicle can determine how to handle such pedestrians and associated objects. In examples, a pedestrian can be associated with an object, such as a suitcase, a stroller, a cart, an umbrella, or other object. Such objects can cause augmentations in the pedestrian's posture and / or position, but the movement of the object can be controlled by the pedestrian and, in some examples, the pedestrian's movement can be constrained (e.g., a pedestrian may not be able to move fast when carrying a suitcase, may not be able to move up and down curbs with a stroller, or may be constrained to lateral acceleration). As such, a pedestrian can be larger (e.g., take up more space) and / or move differently than a pedestrian without the object. The techniques described herein may enable detection of objects associated with pedestrians, e.g., via models trained to detect objects and / or models trained to determine how pedestrians and associated objects move, thereby allowing a computing device onboard an autonomous vehicle to determine, e.g., how to control such an autonomous vehicle.

[0007] In an example, a model can be trained to detect the presence of an object associated with a pedestrian. For purposes of this discussion, such an object can be referred to as a "pedestrian object." A pedestrian object is an object attached to a pedestrian that (1) changes the shape of the pedestrian (e.g., from the shape of the pedestrian without the object) and (2) has a motion trajectory that is controlled by the pedestrian. Non-limiting examples of pedestrian objects include strollers, suitcases, umbrellas, large boxes, children (e.g., being carried), motorcycles, carts, wheelchairs, etc. An object such as an animal on a leash cannot be considered a pedestrian object because the motion trajectory for the animal is not controlled by the pedestrian (e.g., the animal can walk in any trajectory it wishes within the constraints of the leash). Similarly, a child holding hands with an adult pedestrian cannot be considered a pedestrian object.

[0008] In an example, the model can be a neural network using a "top-down segmentation" algorithm, as described in U.S. Patent Application Serial No. 15 / 963,833, filed April 26, 2018, the entire contents of which are incorporated herein by reference. Such a model can be trained using machine learning techniques to detect pedestrian objects associated with pedestrians. In at least one example, an indication such as a boundary region can be associated with pedestrians, pedestrian objects, and combinations of pedestrians and pedestrian objects (referred to herein as pedestrian / pedestrian object systems). That is, pedestrians and pedestrian objects can each be components of a composite object made up of pedestrians and pedestrian objects. In at least one example, the boundary region can be provided to a prediction component and / or a planning component, as described below, to generate a trajectory for an autonomous vehicle passing through an environment using the trajectory. That is, the boundary region can be provided to downstream components of the autonomous vehicle to control the autonomous vehicle. In some examples, data associated with the detected object can be provided to other segmentation algorithms and / or classification algorithms for further processing. These and other operations may be part of the techniques described herein.

[0009] Existing classification techniques may properly classify objects such as cars, pedestrians, and bicycles. However, existing classification techniques may misclassify or may not be trained to classify rare objects that are not commonly seen while driving. However, such rare objects are important to how a driving vehicle perceives the state of the environment in which it travels and determines how to navigate through such an environment. One example of such a rare object is a pedestrian with an object. By enabling the techniques described herein to detect such pedestrian objects, the techniques described herein maintain and / or improve the accuracy of object determination and detection in the environment, thereby improving the functionality of the computing device. As can be appreciated, maintaining and / or improving the accuracy of object determination and detection can improve safety outcomes, particularly in the context of vehicles and autonomous vehicles. Thus, the techniques described herein can improve not only safety outcomes, but also the functionality of the computing device.

[0010] Although the techniques described above are directed to the detection of pedestrians and associated objects (e.g., pedestrian objects), the techniques described herein may be applied to any number of types of objects and associated objects that are attached to each object (e.g., each of the other objects controls the movement of the associated object) and / or change the shape of each of the other objects. For example, the techniques described herein may be applied to the detection of an open door on a vehicle, a child carrier attached to a bicycle, etc. That is, the techniques described herein should not be construed as limited to the detection of pedestrians and pedestrian objects.

[0011] The techniques described herein can be implemented in several ways. Examples of implementations are provided below with reference to the subsequent figures. Although examples of implementations are discussed in the context of autonomous vehicles, the methods, apparatus, and components described herein can be applied to a variety of components (e.g., sensor components or robotic platforms) and are not limited to autonomous vehicles. In one example, the techniques described herein can be utilized in driver-controlled vehicles that can provide an indication to the vehicle driver of whether it is safe for such components to perform various maneuvers. In another example, the techniques can be utilized in aviation or nautical applications, or in any component that includes objects and entities that can be associated with actions that are unknown to the component. That is, in some examples, the techniques described herein can be applicable to weather data, financial data, and the like. Furthermore, the techniques described herein can be used with real data (e.g., acquired using sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

[0012] FIG. 1 illustrates an example of an environment 100 in which pedestrian objects may be detected. FIG. 1 is provided as an overview of the techniques described herein. FIGs. 2 and 3 provide additional details related to training models to detect pedestrian objects (e.g., FIG. 2) and to use such models (e.g., FIG. 3). Additional details related to the systems described herein are described below with reference to FIG. 4.

[0013] In at least one example, the vehicle 102 may be traveling within the environment 100. In at least one example, the vehicle 102 may be an autonomous vehicle configured to operate according to classification level 5 issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for an entire trip with a driver (or passenger) who is not expected to control the vehicle at any time. In such an example, it may not be met because the vehicle 102 may be configured to control all functions from starting to stopping, including all parking functions. This is merely an example, and the components and methods described herein may be incorporated into any vehicle on land, air, or water, including vehicles ranging from vehicles that require manual control by a driver at all times to vehicles that are partially or fully autonomously controlled. That is, in the example described, the vehicle 102 is an autonomous vehicle, but the vehicle 102 may be any other type of vehicle.

[0014] In at least one example, the vehicle 102 can be coupled to a vehicle computing device, described below with reference to FIG. 4, which can include one or more components for controlling the vehicle 102. In at least one example, the perception component can perform object detection, segmentation, and / or classification based at least in part on sensor data received from a sensor component of the vehicle 102. In at least one example, the sensor component can include a lidar sensor, a radar sensor, an ultrasonic transducer, a sonar sensor, a position sensor (e.g., a global positioning component (GPS), a compass, etc.), an inertial sensor (e.g., an inertial measurement unit, an accelerometer, a magnetometer, a gyroscope, etc.), a camera (e.g., a trichromat, an infrared, an intensity, a depth, etc.), a wheel encoder, a microphone, an environmental sensor (e.g., a temperature sensor, a humidity sensor, a light sensor, a pressure sensor, etc.), a time-of-flight (ToF) sensor, etc. Such sensor data may include, but is not limited to, lidar data, radar data, ultrasonic transducer data, sonar data, position data (e.g., global positioning component (GPS), compass, etc.), inertial data (e.g., inertial measurement unit data, accelerometer data, magnetometer data, gyroscope data, etc.), camera data (e.g., trichromatic data, infrared data, intensity data, depth data, etc.), wheel encoder data, microphone data, environmental sensor data (e.g., temperature sensor data, humidity sensor data, light sensor data, pressure sensor data, etc.), ToF sensors, etc.

[0015] In at least one example, the recognition component can receive the sensor data and can utilize one or more machine-trained models to perform object detection, segmentation, and / or classification for objects identified in the sensor data. In some examples, the recognition component can associate a bounding region (or other indicia) with the identified object and can associate a confidence score related to the classification of the identified object with the identified object. In some examples, when rendered via a display, objects can be colored based on their recognized class.

[0016] By way of example, lidar data can be acquired by one or more lidar sensor components in the vehicle 102, and the perception component can relate the acquired lidar data to a voxel space. The voxel space can represent the lidar data in a grid of voxels in three-dimensional space. With the lidar data represented in the voxel space, the perception component can output a mask associated with each object detected in the environment by using a machine-trained model. In some examples, the mask can be represented in a plan view, or in a top-down perspective view (e.g., top-down segmentation). That is, as noted above, in at least one example, the machine-trained model can be a neural network, such as a "top-down segmentation" algorithm, as described in U.S. Patent Application Serial No. 15 / 963,833, filed April 26, 2018, the contents of which are incorporated herein by reference in their entirety. Additional details related to segmentation are described in U.S. Patent Application Serial No. 15 / 820,245, filed November 21, 2017, the contents of which are incorporated herein by reference in their entirety. In some examples, the mask may be represented in a stereoscopic view, or in a side perspective view. In at least one example, as described above, a machine training model may be trained to output a mask having dimensions slightly smaller than the ground truth object.

[0017] While the techniques described above are directed to detection, segmentation, and / or classification of objects in lidar data, the techniques described herein may be applicable to additional or alternative types of sensor data, including, but not limited to, image data, radar data, acoustic sensor data, thermal sensor data, pressure sensor data, etc., which may not be within the domain of an autonomous vehicle.

[0018] In at least one example, a perception component coupled to the vehicle 102 can detect a pedestrian 104 in the environment 100. In such an example, the perception component can associate a boundary region 106 (or other indicia) with the pedestrian 104. In at least one example, the pedestrian 104 can be associated with a pedestrian object 108 (e.g., a stroller) that can be used to detect the pedestrian using techniques described herein. That is, the techniques described herein relate in part to detecting the pedestrian object 108 by training a machine training model. In such an example, based at least in part on a determination that the pedestrian object 108 is present in the environment 100, the perception component can associate a boundary region 110 (or other indicia) with the pedestrian object 108. That is, the machine training model can detect boundary regions associated with each of the pedestrian 104, the pedestrian object 108, and the detected object. In at least one example, the machine training model can associate boundary region 112 (or other representation) with boundary region 106 and boundary region 110 such that boundary region 112 encompasses boundary region 106 and boundary region 110. That is, boundary region 112 can be represented as a composite object of a pedestrian with objects, with pedestrian 104 and pedestrian object 108 each associated with a separate boundary region that is within boundary region 112.

[0019] In at least one example, the sensor data and associated boundary regions (106, 110, and 112) may be provided to a prediction and / or planning component coupled to the vehicle 102, as described below with reference to FIGURE 4, to generate a trajectory for the vehicle 102 through the environment 100 using the trajectory. That is, the boundary regions (106, 110, and 112) may be provided to downstream components of the vehicle 102 to control the vehicle 102. Additional details are provided below.

[0020] FIG. 2 illustrates an example of a process 200 for training a model for pedestrian object detection, as described herein.

[0021] As discussed above, the vehicle may be coupled to one or more sensor components capable of generating sensor data related to the vehicle's environment. An example of such sensor data may include lidar data. By way of example, the vehicle may be coupled to lidar sensors located at the corners, front, rear, sides, and / or top of the vehicle. The lidar sensors may generate lidar data 202 related to the vehicle's environment. In at least one example, the one or more vehicles may transmit to one or more computing devices coupled to a training component. The lidar data 202 may be used for training models, as described below. As discussed above, the vehicle may be coupled to additional or alternative sensor components, each of which may generate its own sensor data.

[0022] In some examples, the lidar data 202 can be lidar data processed by a perception component onboard the vehicle 102, or the lidar data 202 can be raw lidar data that can be processed by one or more computing devices coupled to a training component. In at least one example, a model (which can be provided to the vehicle for use in the vehicle) can segment and classify objects by analyzing the raw lidar data. In an example, the raw lidar data can be associated with a voxel space that can represent the lidar data in a grid of voxels in three-dimensional space. With the lidar data represented in voxel space, a machine training model can analyze the lidar data (represented in voxel space) and output a mask that is associated with individual objects detected in the environment. In some examples, the mask can be represented in a plan view or a top-down perspective view (e.g., "top-down segmentation"). Lidar data that has been segmented and classified such that individual objects are associated with a mask can be referred to as "processed lidar data." In some examples, the lidar data can be associated with a representation other than individual pixels or voxels, as described herein.

[0023] In FIG. 2, a portion of lidar data 204 (e.g., processed lidar data) is illustrated in a top-down perspective view. As illustrated, some voxels associated with the portion of lidar data 204 are associated with a first mask (white fill) and some voxels associated with the portion of lidar data 204 are associated with a second mask (black fill). The coverage with the first mask (white fill) and the coverage with the second mask (black fill) can be associated with different classifications for the object. For example, the coverage with the first mask (white fill) can be associated with a classification of a pedestrian and the coverage with the second mask (black fill) can be associated with a classification of an object.

[0024] In at least one example, lidar data 202 (e.g., processed lidar data) can be provided to annotation component 206. In at least one example, annotation component 206 can receive or otherwise determine annotated data based at least in part on lidar data 202. In at least one example, annotation component 206 can receive annotated data 208 including labels associated with pedestrian object systems (e.g., pedestrians and composite objects of pedestrian objects) and labels associated with pedestrians. In at least one example, the "labels" can be bounding areas, such as bounding boxes (or other representations) associated with each object or composite object. For example, as illustrated in FIG. 2 , a first boundary region 210 is associated with a portion of the lidar data 204 related to a pedestrian and pedestrian object system (e.g., voxels that are a combination of a first mask (white fill) and a second mask (black fill)), and a second boundary region 212 is associated with a portion of the lidar data 204 related to the pedestrian (e.g., masked voxels associated with the white fill). In at least one example, the annotation component 206 can analyze the annotated data 208 using a machine-trained model. In such an example, the annotation component 206 can apply a geometric algorithm, an example of which is described below, to determine a portion of the annotated data 208 associated with the pedestrian object by subtracting at least a portion of the annotated data 208 associated with the first boundary region 210 from a portion of the annotated data 208 associated with the second boundary region 212.

[0025] In at least one example, the annotation component 206 can take in two bounding regions (e.g., a first bounding region 210 and a second bounding region 212) and calculate a major (e.g., longest) axis of the bounding region associated with the composite object (e.g., the first bounding region 210). The annotation component 206 can then determine the location of the pedestrian object by utilizing heuristics. For example, the annotation component 206 can assume that along the major axis, the pedestrian object is located between the second bounding region 212 associated with the pedestrian and the farthest edge of the first bounding region 210. The annotation component 206 can then construct a boundary for the pedestrian object based on the area between the farthest edge of the first bounding region 210 (and associated corners) and the second bounding region 212 associated with the pedestrian. The annotation component 206 can then associate a third bounding region 214 to the portion of the lidar data 204 related to the pedestrian object.

[0026] In additional or alternative examples, the annotation component 206 can analyze the lidar data 204 to determine portions of a pedestrian object, for example, based at least in part on an occupancy state.

[0027] In at least one example, the annotation component 206 can associate a third boundary region 214 with a portion of the lidar data 204 that is associated with a pedestrian object. That is, the annotation component 206 can annotate the portion of the lidar data 204 to indicate that voxels associated with the portion of the lidar data 204 are associated with a pedestrian object. As a result, the annotation component 206 can annotate the portion of the lidar data 204 to include a first boundary region 210 that is associated with the pedestrian and a third boundary region 214 that is associated with the pedestrian object. Thus, the resulting output 216 can consist of lidar data associated with annotations that label the pedestrian and the associated pedestrian object (e.g., boundary region 210 and boundary region 214).

[0028] In at least one example, training input data 218 can be provided to training component 220. Training input data 218 can comprise output from annotation component 206, such as result output 216, as described above with reference to FIG. 2. Furthermore. In some examples, training input data 218 can be annotated via additional or alternative annotation processes other than those described above. For example, in some examples, annotation component 206 can receive annotated data including annotations (e.g., boundary regions) associated with pedestrians (e.g., their associated lidar data) and annotations (e.g., boundary regions) associated with pedestrian objects (e.g., their associated lidar data). Such data can constitute training input data 218. Although a single result output 216 is illustrated, training input data 218 can constitute aggregate data associated with various pedestrians and pedestrian objects.

[0029] In some examples, the training component 220 can receive the training input data 218 and can apply a loss weight mask to the training input data 218 such that voxels associated with pedestrian objects are weighted more than voxels associated with pedestrians and / or other objects. That is, in at least one example, a multiplicative mask can be applied to the training input data 218 to emphasize regions of the lidar data 202 associated with pedestrian objects. With reference to FIG. 2, the training component 220 can associate voxels corresponding to pedestrian objects (e.g., those associated with boundary region 214) with a greater weight than voxels corresponding to pedestrians (e.g., those associated with boundary region 210). In at least one example, this loss weight mask can enable the machine training model to learn novel classifications associated with pedestrian objects, even though pedestrian objects are rarely detected in a vehicular environment (e.g., compared to vehicles, pedestrians, bicyclists, and other objects). In some examples, pedestrian objects may be associated with different semantic classes (e.g., stroller, suitcase, umbrella, large box, children (e.g., being carried), motorcycle, cart, wheelchair, etc.). However, training component 220 may fuse pedestrian objects associated with different semantic classes into a single classification for pedestrian objects when training model 222.

[0030] In at least one example, the training component 220 can output a classification associated with the pedestrian object by training a model 222 based at least in part on the training input data 218. In at least one example, the training component 220 can train the data model using machine learning techniques. For example, in at least one example, a neural network can be trained using the training input data 218. As an example, the training component 220 can input the training input data 218 to the model 222. The model 222 can determine an output associated with a representation of the pedestrian object. The training component 220 can determine a difference by comparing the output to the annotated data (e.g., the training input data 218). In at least one example, the training component 220 can modify one or more parameters for the model based at least in part on the difference.

[0031] As described above, the resulting model 222 can output classifications associated with pedestrian objects in addition to other objects including, but not limited to, pedestrians, vehicles, bicycles, etc. That is, the training component 220 can train the model 222 for additional output heads corresponding to pedestrian objects.

[0032] In at least one example, model 222 can be provided to one or more vehicles 224, which can be configured to be controlled based at least in part on the output of model 222. Additional details related to the use of model 222 are described below with reference to FIG.

[0033] FIG. 3 illustrates an example of a process 300 for determining a pedestrian object and its associated motion, as described herein.

[0034] As discussed above with reference to FIG. 2, a vehicle may be coupled to one or more sensor components capable of generating sensor data related to the vehicle's environment. One example of such sensor data may include lidar data 302, which may be related to the vehicle's environment. In at least one example, a perception component of the vehicle may receive the lidar data 302 (e.g., from a lidar sensor associated therewith), an example portion of this data (e.g., portion of lidar data 304) is illustrated in FIG. 3. As discussed above, a vehicle may be coupled to additional or alternative sensor components, each of which may generate their own sensor data.

[0035] In at least one example, the lidar data 302 can be input to the model 222 trained above with reference to FIG. 2. In at least one example, the model 222 can analyze the lidar data 302 and output one or more observations 306. In at least one example, the observations can be separate adjacent segments in space that are classified (e.g., by the model 222) into a particular class (e.g., associated with an object). In at least one example, one of the observations 306 can be associated with a pedestrian. An enlarged view of a portion of the lidar data 304 is illustrated. The enlarged view 308 illustrates a plurality of voxels 310 associated with a first mask (white-filled) indicating that the plurality of voxels 310 are associated with a classification of a pedestrian. The pedestrian can be associated with a boundary region 312 (or other representation).

[0036] In at least one example, each of the observations 306 can be provided to a tracking component 314, such that the tracking component 314 can utilize an indication for each of the observations 306 to track a corresponding object over time. That is, the tracking component 314 can iterate through each of the observations 306 (and associated objects) identified in the vehicle's environment. Each observation and associated object can be associated with an indication 316(A) through 316(N) (collectively, indications 316).

[0037] Indications 316 may be associated with different objects (e.g., objects A through N) in the vehicle's environment, but may each be associated with states 318(A) through 318(N) (collectively, states 318). Each state may indicate whether a pedestrian object is associated with the pedestrian and, if so, information associated with the pedestrian object. In at least one example, if a pedestrian is associated with a pedestrian object, the state may include information as described below. However, if a pedestrian is not associated with a pedestrian object or if the observation is not associated with a pedestrian (and therefore not associated with a pedestrian object), the state may be empty or invalid.

[0038] In at least one example, the state can be associated with a first indication as to whether the pedestrian is associated with a pedestrian object. The first indication, which can be a preliminary indication, can be based on the output of model 222. That is, as described above, model 222 can detect a pedestrian object. If a pedestrian object is detected in proximity to the pedestrian, the state can be associated with a first (preliminary) indication as to such.

[0039] In at least one example, the state can include a score indicating a likelihood that the pedestrian is associated with the pedestrian object. In some examples, the score can be determined at least in part based on an analysis of one or more portions of received (e.g., lidar) data over time and modifying the score based on whether an instance of the pedestrian object is detected in close proximity (e.g., using a threshold distance) to the pedestrian. That is, the score can be persistent over time such that the score can provide an accurate signal regarding the presence (or absence) of the pedestrian object (e.g., when compared to a single output of the model 222 at a particular moment in time). In at least one example, if the score is equal to or greater than the threshold, the tracking component 314 can validate that the pedestrian object is associated with the pedestrian by associating a second indication (e.g., an indication of confidence) with the state. In some examples, such a second indication can be a Boolean indication (e.g., true or false). Additional details related to determining the score are described below with reference to FIGS. 7 and 8.

[0040] In some examples, an object may be detected by the model 222 but may not be classified as a pedestrian object. In such examples, the tracking component 314 may determine that the object is a pedestrian object before associating a second indication (e.g., a confidence indication) with the state by analyzing one or more additional criteria. In at least one example, the tracking component 314 may consider (1) an area intersection ratio, (2) whether the detected object overlaps with the detected pedestrian, and (3) whether the object is classified as a pedestrian. In at least one example, the area intersection ratio may represent the extent to which a best guess associated with the pedestrian object overlaps with a current detection for the object. If the area intersection ratio is equal to or greater than a threshold, the tracking component 314 may associate a confidence indication with the state (as long as the other two criteria are met). In at least one example, if the detected object overlaps with the detected pedestrian, the tracking component 314 may cease associating a confidence indication with the state. That is, the tracking component 314 can associate a confidence indication (e.g., an indication of which state is associated) with a state so long as the detected object does not overlap with a detected pedestrian (so long as the other two criteria are met). Further, the tracking component 314 can associate a confidence indication with a state (so long as the other two criteria are met) so long as the object is not classified as a pedestrian. That is, in at least one example, if the detected object is classified as a pedestrian, the tracking component 314 can cease associating a confidence indication with a state. In some examples, if a state is associated with a confidence indication indicating that the object is associated with a pedestrian (and thus is a pedestrian object), such classification can invalidate the original classification output by the model 222.

[0041] In at least one example, a state can be associated with one or more poses for the pedestrian object, which can include an output position (e.g., based on model 222) and / or an estimated position (e.g., which can include the output position but occupy more space than the output position). In some examples, the estimated position can be a regression average of a current observation and one or more past observations, which can be determined by use of an exponentially smoothed moving average calculated for the pose of the pedestrian. In some examples, an estimated position cannot be determined (e.g., if a noise metric associated with model 222 is below a threshold). Additional details related to determining states and associated positions are described below with reference to FIG. 6.

[0042] In some examples, the tracking component 314 can associate a kinematic model with the state. Such a kinematic model (e.g., associated with a pedestrian model) can be based at least in part on a kinematic model associated with a pedestrian. In at least one example, the tracking component 314 can determine a kinematic model for a pedestrian object based on characteristics of the pedestrian object (e.g., mass of the pedestrian object, density of the pedestrian object, semantic attributes of the pedestrian object, etc.). In at least one example, building a kinematic model can be useful for propagating past observations forward to determine overlaps with current observations (e.g., of the observations 306), to associate past observations and current observations if they overlap. That is, in some examples, the tracking component 314 can utilize the kinematic model to associate objects (e.g., pedestrians and pedestrian objects) in new sensor data (e.g., lidar data) received over time (e.g., to track the object over time).

[0043] In at least one example, if the state indicates that the pedestrian is associated with a pedestrian object, the tracking component 314 can associate a boundary region (or other representation) with the portion of the lidar data 304 that corresponds to the pedestrian object. In FIG. 3, the boundary region 320 is shown associated with a number of voxels 322 associated with a second mask (e.g., representing the object). In at least one example, based at least in part on the detection of the pedestrian and the pedestrian object, the tracking component 313 can associate a boundary region 324 with the pedestrian and the pedestrian object. That is, the tracking component 314 can output a boundary region 324 that encompasses the boundary region 312 and the boundary region 320. The boundary region 324 can represent a composite object (e.g., a pedestrian / pedestrian object system). In at least one example, the representation of the object (e.g., boundary regions 312, 320, and 324) and associated state 326 can be provided to one or more other vehicle components 328, which can include a prediction component and / or a planning component. Additional details relating to the forecasting and / or planning components are described below with reference to FIG.

[0044] FIG. 4 is a block diagram illustrating an example of a system 400 for performing the techniques as described herein. In at least one example, the vehicle 402 can correspond to the vehicle 102 described above with reference to FIG. 1, but can include one or more vehicle computing devices 404, one or more sensor components 406, one or more emitters 408, one or more communication connections 410, at least one direct connection 412, and one or more drive systems 414. As mentioned above, the vehicle 402 can be an autonomous vehicle configured to operate according to classification level 5 issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for an entire trip with a driver (or passenger) who is not expected to control the vehicle at any time. In such an example, it can be that the vehicle 402 is configured to control all functions from start to stop, including all parking functions, and therefore not met. This is merely an example, and the systems and methods described herein can be incorporated into any vehicle on land, air, or water, including vehicles ranging from those that require manual control by a driver at all times to those that are partially or fully autonomously controlled. That is, in the example described, the vehicle 402 is an autonomous vehicle, but the vehicle 402 can be any other type of vehicle. Although only a single vehicle 402 is illustrated in FIG. 4, in practical applications, the example system 400 can include multiple vehicles, and in some examples, the multiple vehicles can comprise a fleet of vehicles.

[0045] The vehicle computing device 404 may include a processor 416 and a memory 418 communicatively coupled to the processor 416. In the illustrated example, the memory 418 of the vehicle computing device 404 stores a localization component 420, a perception component 422, a prediction component 424, a planning component 426, and one or more system controllers 428. Additionally, the memory 418 may include a storage device 430, which stores maps, models, past outputs, and the like. As discussed above, a map may be any number of data structures capable of providing information about an environment, such as, but not limited to, topology (such as junctions, lanes, merge zones, and the like), streets, mountains, roads, terrain, and the general environment. The map may relate to a real environment or a simulated environment. The model may include a machine learning model, as described below. In some examples, the storage device 430 may store past outputs.

[0046] In at least one example, the localization component 420 can determine an attitude (position or orientation) of the vehicle 402 with respect to local and / or global maps based at least in part on sensor data received from the sensor component 406 and / or map data associated with the map (e.g., of multiple maps). In at least one example, the localization component 420 can include or be coupled with a calibration component that enables substantially simultaneous execution of calibration (determination of various intrinsic and extrinsic parameters associated with any one or more of the sensor components 406), localization, and map generation operations.

[0047] In at least one example, the recognition component 422 can perform object detection, segmentation, and / or classification based at least in part on the sensor data received from the sensor component 406. In at least one example, the recognition component 422 can receive raw sensor data (e.g., from the sensor component 406). In at least one example, the recognition component 422 can receive the sensor data and utilize one or more processing algorithms to perform object detection, segmentation, and / or classification for objects identified in the sensor data. In some examples, the recognition component 422 can associate a bounding region (or other instance segmentation) with the identified object and can associate a confidence score with the identified object's classification. In some examples, when rendered via a display, the object can be colored based on its recognized class. While the techniques described herein are described with respect to lidar data, the recognition component 422 can also perform other processing such as: A similar process can be performed for one or more sensor modalities including, but not limited to, image data, lidar data, etc.

[0048] In at least one example, the recognition component 422 can include a tracking component 432. The tracking component 432 can correspond to the tracking component 314 described above with reference to FIG. 3. In at least one example, the tracking component 432 can receive observations related to the sensor data. Each observation can be associated with an object detected in the environment of the vehicle 402. In at least one example, the tracking component 432 can determine whether such an observation is associated with another object (e.g., a pedestrian object) and can determine a representation of the object that can be provided to other components coupled to the vehicle computing device 404.

[0049] The prediction component 424 can receive sensor data from the sensor component 406, map data associated with a map (e.g., of multiple maps that can be present in storage 430), and / or perception data (e.g., processed sensor data) output from the perception component 422, and can output predictions associated with one or more objects within the environment of the vehicle 402. In at least one example, the planning component 426 can determine an output to control the vehicle based at least in part on the sensor data received from the sensor component 406, the map data, and / or any decisions made by other components related to the vehicle 402.

[0050] Additional details regarding usable localization, recognition, prediction, and / or planning components can be found in U.S. Patent No. 9,612,123, issued April 4, 2017, and U.S. Patent No. 10,353,390, issued July 16, 2019, the entire contents of both of which are incorporated herein by reference. In some instances (e.g., when the vehicle 402 is not an autonomous vehicle), one or more of the above-mentioned components can be omitted from the vehicle 402. Although the components described above are illustrated as "on board" the vehicle 402, in other implementations, the components can be located remotely from and / or accessible to the vehicle 402. Furthermore, although components are described above as "components," such components can be comprised of one or more components for performing the operations attributable to each of the components and can be part of a system.

[0051] In at least one example, the localization component 420, the recognition component 422, the prediction component 424, and / or the planning component 426 can process the sensor data as described above and transmit their respective outputs over the network 434 to the computing device 436. In at least one example, the localization component 420, the recognition component 422, the prediction component 424, and / or the planning component 426 can transmit their respective outputs to the computing device 436 at a particular frequency, such as in near real-time, after a predetermined period of time has elapsed.

[0052] In at least one example, vehicle computing device 404 can include one or more system controllers 428 that can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems associated with vehicle 402. These system controllers 428 can communicate with and / or control drive system 414 and / or corresponding systems in other systems associated with vehicle 402.

[0053] In at least one example, the sensor component 406 can include a lidar sensor, a radar sensor, an ultrasonic transducer, a sonar sensor, a position sensor (e.g., GPS, compass, etc.), an inertial sensor (e.g., inertial measurement unit, accelerometer, magnetometer, gyroscope, etc.), a camera (e.g., trichromatic, infrared, intensity, depth, etc.), a wheel encoder, an acoustic sensor, an environmental sensor (e.g., temperature sensor, humidity sensor, light sensor, pressure sensor, etc.), a ToF sensor, etc. The sensor component 406 can provide input to the vehicle computing device 404. In some examples, the sensor component 406 can perform pre-processing on at least some of the sensor data before transmitting the sensor data to the vehicle computing device 404. In at least one example, the sensor component 406 can transmit the sensor data over the network 434 to the computing device 436 at a particular frequency, in near real-time, after a predetermined period of time has elapsed, etc.

[0054] Vehicle 402 may also include one or more emitters 408 for emitting light and / or sound, as described above. Emitters 408 in this example include interior acoustic and visual emitters for communicating with occupants of vehicle 402. By way of example and not limitation, interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt retractors, seat adjustments, head rest adjustments, etc.), and the like. Emitters 408 in this example also include exterior emitters. By way of example and not limitation, emitters 408 in this example include optical emitters (e.g., indicator lights, signs, light arrays, etc.) for visually communicating with pedestrians, other drivers, other nearby vehicles, etc., and one or more acoustic emitters (e.g., speakers, speaker arrays, horns, etc.) for audio-visually communicating with pedestrians, other drivers, other nearby vehicles, etc. In at least one example, emitters 408 may be positioned in various locations on the exterior and / or interior of vehicle 402 .

[0055] The vehicle 402 may also include communication connections 410 that enable communication between the vehicle 402 and local or remote computing devices. For example, the communication connections 410 may facilitate communication to other local computing devices in the vehicle 402 and / or to the drive system 414. The communication connections 410 may also enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). The communication connections 410 may also enable the vehicle 402 to communicate with remote teleoperated computing devices or other remote services.

[0056] The communication connection 410 may include a physical and / or logical interface for connecting the vehicle computing device 404 to another computing device or to a network, such as the network 434. For example, the communication connection 410 may enable Wi-Fi based communications, such as over frequencies defined by the IEEE 802.11 communications standard, short-range radio frequencies such as BLUETOOTH, or any suitable wired or wireless communication protocol that allows each computing device to interface with other computing devices.

[0057] The direct connection 412 may directly connect the drive system 414 to other systems associated with the vehicle 402 .

[0058] In at least one example, the vehicle 402 can include a drive system 414. In some examples, the vehicle 402 can have a single drive system 414. In at least one example, when the vehicle 402 has multiple drive systems 414, each drive system 414 can be located at either end of the vehicle 402 (e.g., at the front or the rear, etc.). In at least one example, the drive system 414 can include sensor components to detect the state of the drive system 414 and / or the surroundings of the vehicle 402. By way of example and not limitation, the sensor components can include wheel encoders (e.g., rotary encoders) for detecting the rotation of the wheels on the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the position and acceleration of the drive module, cameras or other imaging sensors, ultrasonic sensors for acoustically detecting objects in the surroundings of the drive module, lidar sensors, radar sensors, etc. Some sensors, such as wheel encoders, can be specific to the drive system 414. In some cases, sensor components of the drive system 414 may be duplicated or supplemental to corresponding systems (eg, sensor component 406) of the vehicle 402.

[0059] The drive system 414 can include numerous vehicle systems, including a high voltage battery, a motor to propel the vehicle 402, an inverter to convert direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system to distribute braking forces to mitigate loss of traction and maintain control, an HVAC (heating, ventilation, and air conditioning) system, lighting (e.g., lighting such as headlights / taillights that illuminate the exterior surroundings of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrically powered components such as DC-DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, the drive system 414 can include a drive module controller that can receive and pre-process data from sensor components and control the operation of various vehicle systems. In some examples, the drive module controller can include a processor and a memory communicatively coupled to the processor. The memory can store one or more modules to implement various functionality for the drive system 414. Additionally, the drive system 414 also includes communication connections that allow each drive module to communicate with other local or remote computing devices.

[0060] 4, the vehicle computing device 404, the sensor component 406, the emitter 408, and the communication connections 410 are shown onboard the vehicle 402. However, in some examples, the vehicle computing device 404, the sensor component 406, the emitter 408, and the communication connections 410 can be implemented outside of the actual vehicle (i.e., without being onboard the vehicle 402).

[0061] As described above, the vehicle 402 can transmit sensor data over a network to the computing device 436. In some examples, the vehicle 402 can transmit raw sensor data to the computing device 436. In other examples, the vehicle 402 can transmit processed sensor data and / or representations of the sensor data (data output from the localization component 420, the recognition component 422, the prediction component 424, and / or the planning component 426) to the computing device 436.

[0062] The computing device 436 can receive sensor data (raw or processed) from the vehicle 402 and / or other data collection devices, as well as data from one or more third party sources and / or systems. In at least one example, the computing device 436 can include a processor 438 and a memory 440 communicatively coupled to the processor 438. In the illustrated example, the memory 440 of the computing device 436 stores a training component 442, which can include an annotation component 444, a map store 446 (e.g., storage of one or more maps), a training data store 448 (e.g., storage of training data accessible to the training component 442), and a model store 450 (e.g., models output by the training component 442). In some examples, one or more systems and / or storage repositories can be coupled to the vehicle 402 or other computing devices associated with the system 400 without or in addition to being coupled to the memory 440 of the computing device 436.

[0063] In at least one example, the training component 442 can train a data model, which can be used for various operations as described herein. The training component 442 can correspond to the training component 220 described above with reference to Figure 2, and the annotation component 444 can correspond to the annotation component 206 described above with reference to Figure 2. Details associated with each of the components are described above with reference to Figure 2.

[0064] 2, the training component 442 can train a model based at least in part on the training input data (e.g., received from the annotation component 444) to output classifications associated with pedestrian objects. In at least one example, the training component 442 can train the data model using machine learning techniques. For example, machine learning algorithms for machine learning models include regression algorithms (e.g., ordinary least squares regression (OLSR)), linear regression, logistic regression, stepwise multiple regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and model selection (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative dichotomy (ID3), automated detection of interactions based on chi-squared tests (CHAID), decision strains, conditional decision trees), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, naive Bayes based on multinomial distribution, average-order-of-dependence estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms, and so on. The algorithms may include, but are not limited to, algorithms (e.g., K-means, K-median, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., recognition, backpropagation, Hopfield networks, radial basis function networks (RBFN)), deep learning algorithms (e.g., deep Boltzmann machines (DBM), other deep belief networks (DBN), artificial neural networks (ANN), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), support vector machines (SVM), supervised learning, unsupervised learning, semi-supervised learning, etc.The resulting data model may be stored in model storage 450 and / or storage 430 in the vehicle 402 and may be accessed in near real-time by one or more components on the vehicle computing device 404 .

[0065] The processor 416 of the vehicle 402 and the processor 438 of the computing device 436 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processors 416 and 438 may comprise one or more central processing units (CPUs), graphic processing units (GPUs), or any other device or portion of a device that processes electronic data, thereby converting that electronic data into other electronic data that can be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors, so long as they are configured to implement coded instructions.

[0066] Memories 418 and 440 are examples of non-transitory computer-readable media. Memories 418 and 440 can store an operating system and one or more software applications, instructions, programs, and / or data implementing the methods and functions attributed to the various systems described herein. In various implementations, memories can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic random access memory (SDRAM), non-volatile flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, and those shown in the accompanying drawings are merely examples of those described herein.

[0067] 4 is illustrated as a distributed system, in some examples components of the vehicle 402 may be associated with the computing device 436 and / or components of the computing device 436 may be associated with the vehicle 402. That is, the vehicle 402 may perform one or more functions associated with the computing device 436, and vice versa.

[0068] Figures 5-8 are flow charts illustrating examples of processes that may include techniques as described herein. The processes illustrated in Figures 5-8 are described with respect to the system 400 shown in Figure 4 for convenience and example understanding. However, the processes illustrated in Figures 5-8 are not limited to being performed using the system 400. Moreover, the system 400 described herein is not limited to performing the processes illustrated in Figures 5-8.

[0069] Processes 500-800 are illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by a processor, perform the referenced instructions. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks may be combined in any order and / or in parallel to implement a process. In some combinations, one or more blocks in a process may be omitted entirely. Furthermore, processes 500-800 may be combined, in whole or in part, with each other or with other processes.

[0070] FIG. 5 illustrates an example of a process 500 for training a model for pedestrian object detection, as described herein.

[0071] Block 502 illustrates receiving sensor data. As discussed above, the vehicle 402 can be coupled to one or more sensor components 406 that can generate sensor data related to the environment of the vehicle 402. An example of such sensor data can include lidar data. In at least one example, the one or more vehicles can transmit the lidar data to the computing device 436 via the network 434. As discussed above, in some examples, the lidar data can be lidar data processed by the perception component 422 on board the vehicle 402 and / or the lidar data can be raw sensor data that can be processed by the computing device 436. In at least one example, a model (that can be provided to the vehicle for use in the vehicle and / or that can be used by the computing device 436) can analyze the raw lidar data to perform object segmentation and classification. In an example, the raw lidar data can be associated with a voxel space, which can represent the lidar data as a grid of voxels in three-dimensional space. With the lidar data represented in voxel space, the machine training model can analyze the lidar data (represented in voxel space) to output a mask that is associated with individual objects detected in the environment. In some examples, the mask can be represented in a plan view or a top-down perspective view (e.g., "top-down segmentation"). The lidar data that has undergone segmentation and classification can be referred to as "processed lidar data" because the individual objects are associated with masks. In some examples, the lidar data can be associated with individual pixels or representations other than voxels, as described herein.

[0072] Block 504 illustrates receiving annotated data comprising a representation of a pedestrian with an object and a representation of the pedestrian. In at least one example, the lidar data (e.g., processed lidar data) can be provided to an annotation component 444 coupled to the training component 442. In at least one example, the annotation component 444 can receive or otherwise determine the annotated data based at least in part on the lidar data. In at least one example, the annotation component 444 can receive annotated data including a label associated with a pedestrian object system (e.g., a composite object of a pedestrian and a pedestrian object) and a label associated with the pedestrian. In at least one example, the "label" can be a bounding area (or other representation) associated with each object or composite object.

[0073] Block 506 illustrates a determination regarding the representation of a pedestrian object. In at least one example, the annotation component 444 can analyze the annotated data using a machine-trained model. In such an example, the annotation component 444 can apply a geometric algorithm to determine a portion of the annotated data associated with the pedestrian object by subtracting at least a portion of the annotated data associated with the first bounding region from a portion of the annotated data associated with the second bounding region.

[0074] In at least one example, the annotation component 444 can associate a third boundary region with a portion of the lidar data associated with a pedestrian object. That is, the annotation component 444 can annotate the portion of the lidar data to indicate which of the voxels associated with the portion of the lidar data are associated with the pedestrian object. As a result, the annotation component 444 can annotate the portion of the lidar data to include a first boundary region associated with the pedestrian and a third boundary region associated with the pedestrian object. Thus, the resulting output can consist of the lidar data associated with annotations that label the pedestrian and the associated pedestrian object.

[0075] As described above, additional or alternative annotation processes may be utilized to determine which annotations, indications, or other labels are associated with pedestrians and which annotations, indications, or other labels are associated with pedestrian objects.

[0076] Block 508 illustrates inputting representations of pedestrians and representations of pedestrian objects into the model to determine outputs related to the pedestrian objects. In at least one example, training input data can be provided to training component 442. The training input data can comprise output from annotation component 444, which can comprise lidar data that is labeled with boundary regions or other representations associated with pedestrians and pedestrian objects.

[0077] In some examples, the training component 442 can receive training input data and can apply a loss weight mask to the training input data such that voxels associated with pedestrian objects are weighted more than voxels associated with pedestrians and / or other objects. That is, in at least one example, a multiplicative mask can be applied to the training input data to emphasize regions of the lidar data associated with pedestrian objects. In at least one example, this loss weight mask can enable the machine training model to learn novel classifications associated with pedestrian objects even though pedestrian objects are rarely detected in the vehicular environment (as compared to vehicles, pedestrians, cyclists, and other objects). In some examples, pedestrian objects can be associated with various semantic classes (e.g., strollers, suitcases, umbrellas, large boxes, children, motorcycles, carts, wheelchairs, etc.). However, the training component 442 can fuse objects associated with various semantic classes into a single classification for pedestrian objects when training the model.

[0078] In at least one example, the training component 442 can train a model based at least in part on the training input data to output a classification associated with the pedestrian object. In at least one example, the training component 442 can train the data model using machine learning techniques. For example, in at least one example, a neural network can be trained using the training input data. As an example, the training component 442 can input the training input data to the model. The model can determine an output associated with a representation of the pedestrian object. The training component 442 can determine a difference by comparing the output to the annotated data (e.g., the training input data). In at least one example, the training component 442 can modify one or more parameters of the model based at least in part on the difference.

[0079] As described above, the resulting model can add other objects, including but not limited to pedestrians, vehicles, bicycles, etc., to output classifications associated with pedestrian objects. That is, training component 442 can train a model for additional output heads corresponding to pedestrian objects.

[0080] In some examples, process 500 may receive new sensor data and update the model based at least in part on the newly received sensor data by returning to block 502.

[0081] Block 510 illustrates transmission of the model to a vehicle configured to be controlled by the output of the model. In at least one example, the model can be provided to one or more vehicles, which can be configured to be controlled based at least in part on the output of the model. For example, the model can be transmitted from the computing device 436 to the vehicle 402 (e.g., via the network 434) and stored on a storage device 430 coupled to the vehicle 402. In this manner, the perception component 422 can access the model to analyze incoming lidar data (e.g., from the sensor component 406) in near real-time for control of the vehicle 402. The model can be provided to one or more additional vehicles for similar use.

[0082] While Figure 5 illustrates detecting a novel class, i.e., pedestrian objects, by detecting pedestrians and pedestrian objects to train a model, the techniques described herein may be applied to any number of types of objects and associated objects that are attached to each object (e.g., each of the other objects controls the movement of the associated object) and change the shape of each of the other objects. For example, the techniques described herein may also be applicable to detecting an open door on a vehicle, a child carrier attached to a bicycle, etc. That is, the techniques described herein should not be construed as limited to detecting pedestrians and pedestrian objects.

[0083] Moreover, while the techniques described herein are directed to analyzing lidar data and training models for pedestrian object detection, the techniques described herein may be applicable to additional or alternative types of sensor data, including, but not limited to, image data, sonar data, radar data, etc.

[0084] FIG. 6 illustrates an example of a process 600 for detecting pedestrian objects as described herein.

[0085] Block 602 illustrates receiving sensor data associated with the vehicle. As discussed above, the vehicle 402 may be coupled to one or more sensor components 406 capable of generating sensor data associated with the vehicle's environment. One example of such sensor data may include LIDAR data that may be associated with the vehicle's environment. In at least one example, the perception component 422 of the vehicle 402 may receive the LIDAR data (e.g., from a LIDAR sensor associated therewith). As discussed above, the vehicle may be coupled to additional or alternative sensor components, each of which may generate its own sensor data.

[0086] Block 604 illustrates a determination of whether a pedestrian is detected. In at least one example, the lidar data can be input to a model trained above with reference to FIG. 5. In at least one example, the model can analyze the lidar data and output one or more observations. In at least one example, the observations can be separate contiguous segments of space that have been classified (e.g., by the model) into a particular class (e.g., associated with an object). In at least one example, one of the observations can be associated with a pedestrian. In at least one example, the lidar data associated with the pedestrian can be associated with a boundary region (or other indication). If a pedestrian is not detected in the lidar data, process 600 can return to block 602.

[0087] Block 606 illustrates association of a first boundary region to a pedestrian, where the pedestrian is associated with an indication of the pedestrian. In at least one example, based at least in part on the detected pedestrian, the recognition component 422 can associate a boundary region, or other indication, with a portion of the lidar data corresponding to the pedestrian. Further, each of the observations can be provided to the tracking component 432, whereby the tracking component 432 can associate an indication of each of the observations with each of the observations to track the corresponding object over time. That is, the tracking component 432 can iterate through each of the observations (and associated objects) identified in the environment of the vehicle 402. Each observation and associated object can be associated with an indication.

[0088] Block 608 can associate a state with the representation of a pedestrian. In at least one example, the state can indicate whether a pedestrian object is associated with the pedestrian and, if so, information associated with the pedestrian object. However, if the pedestrian is not associated with the pedestrian object or the observation is not associated with the pedestrian (and therefore not associated with the pedestrian object), the state can be empty or invalid.

[0089] Block 610 illustrates a determination of whether a pedestrian is associated with a pedestrian object. In at least one example, the model can detect the pedestrian object, as described above. If the pedestrian object is within a threshold distance of the pedestrian, the tracking component 432 can determine that the pedestrian is associated with the pedestrian object. The tracking component 432 in at least one example can associate a first indication that the pedestrian is associated with the pedestrian object with the state. The first indication can be a preliminary indication, but can be based on the output of the model. That is, as described above, the model can detect the pedestrian object. If a pedestrian object is detected in proximity to the pedestrian, the state can be associated with such first (preliminary) indication.

[0090] In at least one example, the state can further include a score indicating a likelihood that the pedestrian is associated with the pedestrian object. In some examples, the score can be determined based at least in part on an analysis of one or more portions of the sensor data (e.g., lidar data) received over time and modifying the score based on whether an instance of the pedestrian object is detected in proximity to the pedestrian (e.g., at a threshold distance of the pedestrian). That is, the score can be persistent over time such that the score can provide an accurate signal regarding the presence (or absence) of the pedestrian object (e.g., when compared to a single output for the model at a particular moment in time). In at least one example, if the score is equal to or greater than the threshold, the tracking component 432 can corroborate that the pedestrian object is associated with the pedestrian by associating a second indication (e.g., a confidence score) with the state. In some examples, such a second indication can be a Boolean indication (e.g., true or false). Additional details related to the determination of the score are described below with reference to FIGS. 7 and 8.

[0091] In some examples, an object may be detected by the model but may not be classified as a pedestrian object. In such examples, the tracking component 432 may determine that the object is a pedestrian object before associating a second indication (e.g., a confidence indication) with the state by analyzing one or more additional criteria. In at least one example, the tracking component 432 may consider (1) an area intersection ratio, (2) whether the detected object overlaps with the detected pedestrian, and (3) whether the object is classified as a pedestrian. In at least one example, the area intersection ratio may indicate the extent to which a best guess associated with the pedestrian object overlaps with a current detection for the object. If the area intersection ratio is equal to or greater than a threshold, the tracking component 432 may associate a confidence indication with the state (as long as the other two descriptions are met). In at least one example, if the detected object overlaps with the detected pedestrian, the tracking component 432 may cease associating a confidence indication with the state. That is, the tracking component 432 can associate a confidence indication with a state (as long as the other two criteria are met) so long as the detected object does not overlap with a detected pedestrian (e.g., an indication that the state is associated with). Moreover, the tracking component 432 can associate a confidence indication with a state (as long as the other two criteria are met) so long as the object is not classified as a pedestrian. That is, in at least one example, if the detected object is classified as a pedestrian, the tracking component 432 can cease associating a confidence indication with a state. In some examples, when a state is associated with a confidence indication indicating that the object is associated with a pedestrian (and thus is a pedestrian object), such a classification can invalidate the original classification output by the model.

[0092] In some examples, the tracking component 432 can associate a kinematic model with the state. Such a kinematic model (e.g., associated with a pedestrian object) can be based at least in part on a kinematic model associated with a pedestrian. In at least one example, the tracking component 432 can determine a kinematic model for a pedestrian object based on characteristics of the pedestrian object (e.g., mass of the pedestrian object, density of the pedestrian object, semantic attributes of the pedestrian object, etc.). In at least one example, building a kinematic model can be useful for propagating past observations forward to determine overlaps with current observations (e.g., of the observations 306), for correlating observations when past and current observations overlap. That is, in some examples, the tracking component 314 can utilize the kinematic model to associate objects (e.g., pedestrians and pedestrian objects) in received sensor data over time (e.g., to track the objects over time).

[0093] Further, in at least one example, the state can be associated with one or more poses for the pedestrian object, which can include an output position (e.g., based on the model) and / or an estimated position (e.g., which can include the output position but take up more space than the output position). In some examples, the estimated position can be a regression average of a current observation and one or more past observations, and can be determined using an exponentially smoothed moving average calculated in relation to the poses of the pedestrian. In some examples, the estimated position cannot be determined (e.g., if a noise metric associated with the model is below a threshold).

[0094] Block 612 illustrates the association of the second boundary region to the pedestrian object, and the association of the third boundary region to the first boundary region (e.g., associated with the pedestrian) and the second boundary region (e.g., associated with the pedestrian object). In at least one example, if the tracking component 432 determines that the state is associated with an indication of confidence, the tracking component 432 can determine that the pedestrian is associated with the pedestrian object. In at least one example, if the pedestrian is associated with the pedestrian object (the state is associated with such an indication), the tracking component 432 can associate the second boundary region to the pedestrian object and the third boundary region to the first boundary region (e.g., associated with the pedestrian) and the second boundary region (e.g., associated with the pedestrian object). That is, in at least one example, if the state indicates that the pedestrian is associated with the pedestrian object, the tracking component 432 can associate a boundary region (or other indication) with a portion of the lidar data corresponding to the pedestrian object. In at least one example, based at least in part on the pedestrian and pedestrian object, the tracking component 432 can associate a boundary region with the pedestrian and pedestrian object. That is, the tracking component 432 can output a boundary region that includes a first boundary region (e.g., associated with the pedestrian) and a second boundary region (e.g., associated with the pedestrian object). The boundary region can represent a composite object (e.g., a pedestrian / pedestrian object system).

[0095] Block 614 illustrates providing the boundary region to the prediction component and / or the planning component to control the vehicle. In at least one example, based at least in part on the tracking component 432 associating the first, second, and third boundary regions with the lidar data, the tracking component 432 can transmit an indication of the pedestrian / pedestrian object system to one or more vehicle components, which may include the prediction component 424 and the planning component 426. In at least one example, the boundary region can be associated with state information, which may include one or more poses for the pedestrian object. As described above, the position information (associated with the pose) may include an output position (e.g., based on the model) and / or an estimated position (e.g., which may include the output position but occupy more space than the output position). In some examples, the estimated position may be a regression average of the current observation and one or more past observations and may be determined using an exponentially smoothed moving average calculated in association with the pose of the pedestrian. In some examples, the estimated position may not be determined (e.g., if a noise metric associated with the model is below a threshold).

[0096] If a pedestrian object is not detected, the tracking component 432 can send lidar data associated with a boundary region corresponding to the pedestrian (e.g., a first boundary region) to the prediction component 424 and / or the planning component 426.

[0097] As described above, the prediction component 424 and / or the planning component 426 can determine a trajectory for controlling the vehicle 402 .

[0098] 7 illustrates an example of a process 700 for determining whether a pedestrian object is associated with a pedestrian, as described herein. As noted above, in some examples, the model (trained as described above) can output an indication that the pedestrian is associated with a pedestrian object. In at least one example, a score described below can be used as confirmation that the object is a pedestrian object to ensure accurate decision-making by the vehicle computing device 404.

[0099] Block 702 illustrates receiving sensor data associated with a portion of the sensor data. As described above, the vehicle 402 can be coupled to one or more sensor components 406 that can generate sensor data related to the vehicle's environment. An example of the sensor data can include lidar data that can be associated with the vehicle's environment. In at least one example, the perception component 422 of the vehicle 402 can receive the lidar data (e.g., from a lidar sensor associated therewith). As described above, the vehicle can be coupled to additional or alternative sensor components that can generate their own sensor data. In at least one example, the sensor data can be received over time. For example, in some examples, the portion of the lidar data can be associated with a fixed time or rotation associated with the acquisition of the lidar data. Such portions of the sensor data received over time can be provided to the perception component 422 in near real-time.

[0100] Block 704 illustrates a determination of whether an object is detected within a threshold distance of a pedestrian. As described above with reference to blocks 604 and 606 in FIG. 6, in at least one example, for example, lidar data can be an input to the model trained above with reference to FIG. 5. In at least one example, the model can analyze the lidar data and output one or more observations. In at least one example, the observations can be separate adjacent segments in space that have been classified (e.g., by the model) into a particular class (e.g., associated with an object). In at least one example, one of the observations can be associated with a pedestrian. In at least one example, the lidar data associated with the pedestrian can be associated with a boundary region (or other indication). In at least one example, each of the observations can be provided to a tracking component 432 coupled to the recognition component 422, whereby the tracking component 432 can associate an indication of each of the observations with each of the observations to track the corresponding object over time. That is, the tracking component 432 can iterate through each of the observations identified in the environment of the vehicle 402. Each observation and associated object can be associated with a representation.

[0101] In at least one example, the tracking component 432 can determine whether an object is detected within a pedestrian threshold distance by analyzing a portion of the lidar data. Based at least in part on a determination that an object is detected within a pedestrian threshold distance, the tracking component 432 can increase a score associated with a state associated with an indication of a pedestrian, as illustrated in block 706. In at least one example, a state associated with an indication of a pedestrian can start with a score of 0.0. Each time an object is detected within the pedestrian threshold distance, the tracking component 432 can increase the score by a unit amount.

[0102] In at least one example, if an object is not detected and / or is detected but not within the pedestrian threshold distance, the tracking component 432 can cease increasing the score associated with the state, as illustrated in block 708. In some examples, if an object is not detected and / or is detected but not within the pedestrian threshold distance, the tracking component 432 can decrease the score (e.g., after an indication of confidence regarding the pedestrian object has been associated with the state, as described above). In some examples, if an object is not detected and / or is detected but not within the pedestrian threshold distance, the tracking component 432 can leave the score unchanged.

[0103] Block 710 illustrates a determination of whether the score is greater than or equal to a first threshold. In at least one example, the tracking component 432 can determine whether the score is greater than or equal to a first threshold. The first threshold can correspond to a score by which the tracking component 432 can be confident that the object is a pedestrian object. As such, based at least in part on a determination that the score is greater than or equal to the first threshold, as illustrated in block 712, the tracking component 432 can determine whether a state associated with the pedestrian indication is associated with a confidence indication that the object is a pedestrian object. That is, if the state is already associated with a confidence indication, then the score associated with the state has already exceeded the first threshold due to the pedestrian object being present in past portions of the received (e.g., lidar) data over time. However, if the state is not associated with a confidence indication, then the score has not previously exceeded the first threshold. If the state is not associated with an indication that the object is a pedestrian object, the tracking component can associate a confidence indication that the object is a pedestrian object with the state, as illustrated in block 714. In some examples, the tracking component 432 may refrain from associating an indication of confidence that the object is a pedestrian object until the tracking component 432 determines whether the object satisfies one or more additional criteria.

[0104] In some examples, an object may be detected by the model but may not be classified as a pedestrian object. In such examples, the tracking component 432 may determine that the object is a pedestrian object before associating a second indication (e.g., a confidence indication) with the state by analyzing one or more additional criteria. In at least one example, the tracking component 432 may consider (1) an area intersection ratio, (2) whether the detected object overlaps with the detected pedestrian, and (3) whether the object is classified as a pedestrian. In at least one example, the area intersection ratio may represent the degree to which a best guess associated with the pedestrian object overlaps with a current detection for the object. If the area intersection ratio is equal to or greater than a threshold, the tracking component 432 may associate a confidence indication with the state (as long as the other two criteria are met). In at least one example, if the detected object overlaps with the detected pedestrian, the tracking component 432 may cease associating a confidence indication with the state. That is, the tracking component 432 can associate a confidence indication with a state (as long as the other two criteria are met) as long as the detected object does not overlap with a detected pedestrian (e.g., the indication with which the state is associated). Further, the tracking component 432 can associate a confidence indication with a state (as long as the other two criteria are met) as long as the object is not classified as a pedestrian. That is, in at least one example, if the detected object is classified as a pedestrian, the tracking component 432 can cease to associate a confidence indication with a state. In some examples, when a state is associated with a confidence indication indicating that the object is associated with a pedestrian (and thus is a pedestrian object), such a classification can override the original classification output by the model.

[0105] In additional or alternative examples, such limitations need not apply for the association of the two. As a non-limiting example, similar techniques and model constructions may be used to model and track two (or more) pedestrians traveling together (e.g., holding hands). In such examples, the limitation that the other object is not a pedestrian may be relaxed. Furthermore, in such examples, notwithstanding the fact that the pedestrians are themselves detected and tracked, given an indication that the two (or more) are linked using the techniques described herein, more accurate state propagation for multiple entries may be implemented by taking into account better kinematic models.

[0106] In at least one example, if the score is not greater than or equal to the first threshold (at block 710 ), the process may return to block 702 .

[0107] Block 716 illustrates determining whether the score is equal to the maximum score. In at least one example, the tracking component 432 can determine whether the score is equal to the maximum score. If the score is equal to the maximum score, the tracking component 432 can determine whether the observations are still being received, as illustrated at block 718. That is, the tracking component 432 can determine whether the observations are still being received (e.g., based at least in part on the lidar data). If the observations are still being received, the process 700 can return to block 702 until the observations are no longer being received. If the observations are no longer being received, the process 700 can transition to block 802 in FIG. 8.

[0108] If the score is not equal to the maximum score, the process may return to block 702 .

[0109] In at least one example, an indication of the confidence that the object is a pedestrian object associated with the pedestrian can override a misclassification (or non-classification) originally output by the model. As described above with reference to FIG. 6, if a pedestrian is determined to be associated with a pedestrian object, the tracking component 432 can associate a first boundary region with the pedestrian, a second boundary region with the pedestrian object, and a third boundary region with the first boundary region and the second boundary region. That is, in at least one example, if the state indicates that a pedestrian is associated with a pedestrian object, the tracking component 432 can associate a boundary region (or other indication) with a portion of the lidar data corresponding to the pedestrian object. In at least one example, based at least in part on the detection of the pedestrian and the pedestrian object, the tracking component 432 can associate a boundary region with the pedestrian and the pedestrian object. That is, the tracking component 432 can output a boundary region and a boundary region encompassing the boundary region. The boundary region can represent a composite object (e.g., a pedestrian / pedestrian object system).

[0110] In at least one example, based at least in part on the tracking component 432 associating the first, second, and third boundary regions with the lidar data, the tracking component 432 can transmit an indication of a representation of the pedestrian / pedestrian object system to one or more vehicle components 326, which may include a prediction component 424 and / or a planning component 426. In at least one example, the boundary regions can be associated with state information, which may include one or more poses for the pedestrian object. As described above, the position information (related to the pose) can include an output position (e.g., based on a model) and / or an estimated position (e.g., which may include the output position, but occupy more space than the output position).

[0111] FIG. 8 illustrates an example of a process 800 for determining whether a pedestrian object is associated with a pedestrian, as described herein.

[0112] Block 802 illustrates determining whether a certain period of time has elapsed. In at least one example, the tracking component 432 can determine whether a certain period of time has elapsed since the last observation was received. If a certain period of time has elapsed, the tracking component 432 can decrease the score, as illustrated in block 804. If a certain period of time has not elapsed, the tracking component 432 can stop decreasing the score, as illustrated in block 806, and the process 800 can return to block 802.

[0113] Block 808 illustrates determining whether the score is greater than or equal to a second threshold. In at least one example, the second threshold can be less than the first threshold. Based at least in part on a determination that the score is greater than or equal to the second threshold, process 800 can cease changing the state associated with the pedestrian indication, as illustrated at block 810, and can determine whether another period of time has passed by returning to block 802. If the score is less than the second threshold, tracking component 432 can disassociate the state from the indication of confidence (of the pedestrian object), as illustrated at block 812. In at least one example, the period of time can be less than the frequency at which a new piece of sensor data is received over time (e.g., lidar data) such that the decay is slower than the rise to ensure that, if a pedestrian object is present, it can be detected and vehicle 402 can respond to it.

[0114] [Example clauses] A. A method comprising: receiving sensor data related to an environment; receiving annotated data based at least in part on the sensor data, the annotated data including a first representation of a pedestrian with an object and a second representation of the pedestrian within the first representation of the pedestrian with the object; determining a third representation of a pedestrian object related to the second representation of the pedestrian based at least in part on the annotated data, the second representation of the pedestrian and the third representation of the pedestrian object constituting training inputs; inputting the training inputs to a model; determining an output by the model including a fourth representation of the pedestrian object; determining a difference between the output and the annotated data; modifying one or more parameters of the model based at least in part on the difference; and transmitting the model to a vehicle configured to be controlled by another output of the model.

[0115] B. The method as clause A refers to the determination regarding the third representation for the pedestrian object including subtracting at least a portion of the annotated data associated with the second representation for the pedestrian from a portion of the annotated data associated with the first representation for the pedestrian with the object.

[0116] C. The method as section B recites that subtracting at least a portion of the annotated data associated with the second representation of the pedestrian from a portion of the annotated data associated with the first representation of the pedestrian with the object is based at least in part on application of a geometric algorithm to the annotated data.

[0117] D. The method as any of claims A to C further comprises applying a loss weight mask to the training input, wherein a portion of the annotated data associated with a third representation of the pedestrian object is weighted more than a portion of the annotated data associated with the second representation of the pedestrian.

[0118] E. The method of any of claims A through D recites that the sensor data comprises lidar data, and that a model is trained through machine learning to segment the lidar data and determine classifications associated with respective segments in the lidar data, at least one of the classifications corresponding to a pedestrian object.

[0119] F. A system comprising one or more processors and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations including: receiving sensor data relating to an environment; determining annotated data based at least in part on the sensor data, the annotated data including a first representation for a first object and a second representation for a second object, the first object and the second object constituting a composite object; inputting training inputs to a model, the training data including the annotated data; determining an output by the model including a third representation for the second object; determining a difference between the output and the annotated data; and modifying one or more parameters for the model based at least in part on the difference.

[0120] G. The system as section F mentions that the annotated data further includes a fourth representation for the composite object, and the operation further includes determining a second representation for the second object based at least in part on a subtraction of a portion of the annotated data associated with the first representation for the first object from a portion of the annotated data associated with the fourth representation for the composite object.

[0121] H. The system as in Section G refers to the subtraction of at least a portion of the annotated data associated with the first representation of the first object from a portion of the annotated data associated with the fourth representation of the composite object being based at least in part on application of a geometric algorithm to the annotated data.

[0122] I. The system as any of claims F through H further includes applying a loss weight mask to the training input, wherein a portion of the annotated data associated with the second representation for the second object is weighted more than a portion of the annotated data associated with the first representation for the first object.

[0123] J. The system as claimed in any of claims F through I further includes transmitting the model to at least one vehicle whose operation is configured to be controlled by another output related to the model.

[0124] K. The system as claimed in any of claims F through J recites that the composite object includes a pedestrian accompanying object, the first object includes a pedestrian, and the second object includes a pedestrian object.

[0125] L. The system as clause K refers to pedestrian objects including a stroller, a suitcase, an umbrella, a large box, a child, a motorcycle, a cart, or a wheelchair.

[0126] M. One or more computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including determining annotated data associated with a first representation of a first object and a second representation of a second object from sensor data related to an environment, where the first object and the second object comprise a composite object; training a model to determine an output related to the second object based at least in part on the annotated data; and transmitting the model to at least one vehicle configured to be controlled by another output related to the model.

[0127] N. The one or more computer-readable media as Section M refers to training the model including inputting the annotated data into the model, determining an output by the model comprising a third representation of the second object, determining a difference between the output and the annotated data, and modifying one or more parameters for the model based at least in part on the difference.

[0128] O. The one or more computer-readable media as clauses M or N recite the operation includes determining a second representation for the second object based at least in part on the first representation for the first object and the third representation for the composite object.

[0129] P. The one or more computer-readable media as clause O recites that the determination regarding the second representation regarding the second object includes subtracting at least a portion of the annotated data associated with the first representation regarding the first object from a portion of the annotated data associated with the third representation regarding the composite object.

[0130] Q. One or more computer-readable media as any of claims M to P further include applying a loss weight mask to the annotated data, wherein a portion of the annotated data associated with a second representation for the second object is weighted more than a portion of the annotated data associated with the first representation for the first object.

[0131] R. One or more computer-readable media as any of claims M through P recite the composite object includes a pedestrian accompanying object, the first object includes the pedestrian, and the second object includes the pedestrian object.

[0132] S. The one or more computer-readable media as in clause R may further comprise a first representation of the first object including a third representation having a first bounding region, a second bounding region within the first bounding region, and a third bounding region within the first bounding region. The second representation is associated with a second object comprising the second representation.

[0133] T. The one or more computer-readable media of any of claims M through S recite that the sensor data comprises lidar data, and that a model is trained through machine learning to segment the lidar data and determine classifications associated with respective segments of the lidar data, at least one classification corresponding to a second object.

[0134] U. A method comprising: receiving lidar data from one or more sensors coupled to the vehicle; determining a pedestrian proximate to the vehicle with a model based at least in part on the lidar data, where the detection of the pedestrian comprises a first boundary region; determining that the pedestrian is associated with a pedestrian object, where the pedestrian object comprises a second boundary region; associating the first boundary region and the second boundary region with a third boundary region, where the third boundary region encompasses the first boundary region and the second boundary region; and providing the first boundary region, the second boundary region, and the third boundary region to at least one predictive or planning component in the vehicle for controlling the vehicle.

[0135] V. The method as in Section U notes that determining that the pedestrian is associated with a pedestrian object includes receiving an output from the model indicative of detection of the pedestrian object.

[0136] W. The method as in clause V recites that the step of determining that a pedestrian is associated with a pedestrian object is based at least in part on a score determined based on analysis of the lidar data over time.

[0137] X. The method as set forth in Section W refers to the score being determined at least in part based on analyzing one or more portions of the lidar data received over time and increasing the score each time a pedestrian object is detected.

[0138] Y. Note that the method as clause W or X further comprises determining that the score is greater than or equal to a threshold and determining that the pedestrian is associated with the pedestrian object based at least in part on a determination that the score is greater than or equal to the threshold.

[0139] Z. A system comprising one or more processors and one or more non-transitory computer readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations, the operations including receiving data from one or more sensors coupled to a vehicle; determining an observation associated with the data, where the observation is associated with a first object; determining whether the first object is associated with a second object, where the first object and the second object constitute a composite object; and providing a representation of the first object or a representation of the composite object to at least one predictive or planning component in the vehicle for controlling the vehicle.

[0140] AA. The system as in clause Z refers to the operation further including associating the kinematics model with the second object based at least in part on the association of the first object with the second object.

[0141] AB. The system as in clause AA further includes determining a semantic classification for the second object and determining a kinematic model associated with the second object based at least in part on the semantic classification.

[0142] AC. The system as in clause AA or AB refers to associating a first object and a second object in new data received over time based at least in part on a kinematic model.

[0143] AD. The system as any of claims Z through AC further includes determining an association of the first object and the second object based at least in part on a score indicating whether the detected object is associated with a classification corresponding to the second object.

[0144] AE. The system as in clause AD notes that the operation further includes analyzing one or more portions of the data over time and increasing a score each time a detected object is within a threshold distance of the first object.

[0145] AF. The system as in clause AD or AE refers to the operation further including determining that the score is greater than or equal to a first threshold and determining that the first object is related to the second object based at least in part on the determination that the score is greater than or equal to the first threshold.

[0146] The system as in clause AG. AF recites that the determination that the first object is related to the second object is further based at least in part on one or more of: (1) an area intersection ratio; (2) an indication of whether the detected object overlaps the first object; or (3) an indication of whether the detected object is classified in the same class as the first object.

[0147] AH. The system as any of claims Z through AG further includes the operation determining at least one position associated with the second object, the at least one position including an output position of the second object or an estimated position of the second object, the output position being output by a model trained to at least detect the first object and the second object, the estimated position including the output position, and associating the at least one position with a representation of the composite object.

[0148] The system as in any of AI. Claims Z to AH recites that the composite object includes a pedestrian with objects, the first object includes a pedestrian, the second object includes a pedestrian object, and the pedestrian object includes a stroller, a suitcase, an umbrella, a large box, a child, a motorcycle, a cart, or a wheelchair.

[0149] AJ. One or more computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including receiving data from one or more sensors associated with a vehicle; determining an observation associated with the data, where the observation is associated with a first object; determining whether the first object is associated with a second object, where the first object and the second object comprise a composite object; and providing a representation of the first object or a representation of the composite object to at least one predictive or planning component in the vehicle for controlling the vehicle.

[0150] AK. The one or more computer-readable media as section AJ refers to the operation further including analyzing one or more portions of the data over time and increasing a score each time a detected object is within a threshold distance of a first object, the score indicating a likelihood that the first object is related to the second object.

[0151] AL. The one or more computer-readable media as clause AK references the operation further including determining that the score is greater than or equal to a threshold and determining that the first object is associated with the second object.

[0152] AM. The one or more computer-readable media as any of claims AJ to AL recites that the operation further includes associating a kinematic model with the second object based at least in part on an association of the first object to the second object, wherein the association of the first object and the second object in new sensor data received over time is based at least in part on the kinematic model.

[0153] AN. The one or more computer-readable media as any of claims AJ to AM further recite that the operations further include determining at least one of an output position of the second object or an estimated position of the second object, where the estimated position is determined based at least in part on the output position and one or more past positions associated with the second object; providing the estimated position along with a representation of the composite object to at least one of the prediction component or the planning component if a noise metric associated with a model trained to detect at least the first object and the second object is equal to or greater than a threshold; and providing the output position along with a representation of the composite object to at least one of the prediction component or the planning component if a noise metric associated with the model is less than the threshold.

[0154] Although the exemplary provisions described above are described with respect to one particular implementation, it should be understood in connection with this specification that the contents of the exemplary provisions can also be implemented through a method, an apparatus, a system, a computer-readable medium, and / or another implementation. Furthermore, any of the examples A-AN can be implemented alone or in combination with one or more of the others in the examples A-AN.

[0155] [Conclusion] One or more examples of the techniques described herein have been described; however, various substitutions, additions, rearrangements, and the like, are included within the scope of the techniques described herein.

[0156] In describing the examples, reference is made to the accompanying drawings which form a part of this specification, which show by way of example specific examples of the claimed subject matter. It should be understood that other examples can be used and that modifications or substitutions, such as structural changes, can be made. Such examples, modifications or substitutions do not necessarily depart from the intended scope of the claimed subject matter. Although steps in this specification may be presented in a particular order, in some cases the order can be changed such that certain inputs are provided at different times or in different orders without changing the functionality of the described systems and methods. The disclosed procedures can also be performed in different orders. Furthermore, the various operations in this specification need not be performed in the order disclosed, and other examples can be easily implemented using alternative orders of operations. In addition to being reordered, operations can also be decomposed into sub-operations which produce the same results.

Claims

1. A system comprising: one or more processors; One or more non-transitory computer-readable media that, when executed by the one or more processors, cause the system to: Receiving data from one or more sensors associated with the vehicle; determining an observation associated with the data, the observation being associated with a first object, the first object including a first boundary region; determining whether the first object is associated with a second object, the second object including a second boundary region; generating a third bounding region based at least in part on the first bounding region and the second bounding region, the third bounding region encompassing the first bounding region and the second bounding region; providing the representation of the first boundary region or the representation of the third boundary region to at least one of a prediction component or a planning component in the vehicle for controlling the vehicle; one or more non-transitory computer-readable media storing instructions for performing operations including: A system comprising:

2. The operation is associating a kinematic model with the second object based at least in part on an association of the first object with the second object; determining a semantic classification for the second object; determining the kinematic model associated with the second object based at least in part on the semantic classification; Further comprising: The association of the first object and the second object in new data received over time is based at least in part on the kinematic model.

2. The system of claim 1 .

3. The system of claim 1 or 2, wherein the operations further include determining an association of the first object and the second object based at least in part on a score indicating whether the detected object is associated with a classification corresponding to the second object.

4. The operation is analyzing one or more portions of the data over time; and increasing said score each time a detected object is within a threshold distance of said first object; The system of claim 3 further comprising:

5. The operation is determining that the score is greater than or equal to a first threshold; (1) determining that the score is greater than or equal to the first threshold; (2) determining that the first object is associated with the second object based at least in part on one or more of: an area intersection ratio; (3) an indication of whether the detected object overlaps the first object; or (4) an indication of whether the detected object is classified in the same class as the first object. The system according to claim 3 or 4, further comprising:

6. The operation is determining at least one location associated with the second object, the at least one location comprising: an output position of the second object, the output position being output by a model trained to detect at least the first object and the second object; or an estimated position of the second object, the estimated position including the output position; and associating said at least one location with a representation of said third bounding region; 6. The system according to claim 1, further comprising:

7. 7. The system of claim 1, wherein the third boundary area includes a pedestrian accompanied by an object, the first object includes a pedestrian, and the second object includes a pedestrian object, the pedestrian object including a stroller, a suitcase, an umbrella, a large box, a child, a motorcycle, a cart, or a wheelchair.

8. receiving data from one or more sensors associated with the vehicle; determining an observation associated with the data, the observation being associated with a first object, the first object including a first boundary region; determining whether the first object is associated with a second object, the second object including a second boundary region; generating a third bounding region based at least in part on the first bounding region and the second bounding region, the third bounding region encompassing the first bounding region and the second bounding region; providing the representation of the first boundary region or the representation of the third boundary region to at least one of a predictive component or a planning component in the vehicle for controlling the vehicle; A method comprising:

9. analyzing one or more portions of the data over time; determining a score based at least in part on said analysis; incrementing a score each time a detected object is within a threshold distance of the first object, indicating a likelihood that the first object is associated with the second object; determining whether the score is greater than or equal to a threshold; determining that the first object is associated with the second object; The method of claim 8 further comprising:

10. The method of claim 9, further comprising determining whether the first object is associated with the second object based at least in part on one or more of: (1) an area intersection ratio; (2) an indication of whether the detected object overlaps the first object; or (3) an indication of whether the detected object is classified into the same class as the first object.

11. 11. The method of claim 8, further comprising the step of associating a kinematic model with the second object based at least in part on an association of the first object with the second object, wherein the association of the first object and the second object in new sensor data received over time is based at least in part on the kinematic model.

12. determining at least one of an output position of the second object or an estimated position of the second object, the estimated position being determined at least in part based on the output position and one or more past positions associated with the second object; if a noise metric associated with a model trained to at least detect the first object and the second object is equal to or greater than a threshold, providing the estimated location along with an indication of the third bounding region to at least one of the prediction component or the planning component; if the noise metric associated with the model is less than the threshold, providing the output location along with an indication of the third boundary region to at least one of the prediction component or the planning component; 12. The method of claim 8, further comprising:

13. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any one of claims 8 to 12.

Images