Attribute-based pedestrian prediction

Attribute-based prediction methods using machine learning models improve the accuracy of pedestrian and vehicle position forecasting, enabling safer and more efficient navigation for autonomous vehicles.

JP7593935B2Active Publication Date: 2024-12-03ZOOX INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2021557327
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-03-25
Filing Date
2020-03-24
Publication Date
2024-12-03
Estimated Expiration
2040-03-24

AI Technical Summary

Technical Problem

Current prediction techniques for the future state of entities in an environment, such as pedestrian movement, often require physically-based modeling and rule-of-the-road simulations, which can be inflexible and less accurate.

Method used

The use of attribute-based prediction methods, utilizing machine learning models like neural networks, to determine the future position of pedestrians or vehicles by analyzing attributes such as location, velocity, and acceleration, allowing for more dynamic and accurate predictions.

Benefits of technology

Enhances the ability of autonomous vehicles to safely navigate environments by providing more precise predictions of pedestrian movements and vehicle positions, reducing the risk of collisions and improving overall navigation efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593935000001
    Figure 0007593935000001
  • Figure 0007593935000002
    Figure 0007593935000002
  • Figure 0007593935000003
    Figure 0007593935000003
Patent Text Reader

Abstract

Techniques are discussed for predicting the location of an object based on attributes of the object and / or attributes of other objects proximate the object. The techniques can predict the location of a pedestrian proximate a crosswalk when crossing or preparing to cross the crosswalk. The techniques can also predict the location of an object as it passes through an environment. Attributes can include information about the object, such as its position, speed, acceleration, classification, orientation, area or distance relative to other objects, bounding box, etc. Attributes can be determined for an object over time such that, when a set of attributes is input into a prediction component (e.g., a machine learning model), the prediction component can output, for example, a predicted location of the object at some point in the future. A vehicle, such as an autonomous vehicle, can be controlled to traverse an environment based on the predicted location.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to attribute-based pedestrian prediction. [Background technology]

[0002] This patent application claims priority to U.S. utility patent application Ser. No. 16 / 363541, filed March 25, 2019, and U.S. utility patent application Ser. No. 16 / 363627, both of which are incorporated herein by reference in their entireties.

[0003] Prediction techniques can be used to determine the future state of an entity within an environment, i.e., predictive techniques can be used to determine how a particular entity is likely to behave in the future. Current predictive techniques often require physics-based modeling or rule-of-the-road simulations to predict the future state of an entity within an environment. [Brief description of the drawings]

[0004] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears, and the use of the same reference number in different drawings indicates similar or identical components or functions.

[0005] [Figure 1] FIG. 1 is a pictorial flow diagram of an example process for acquiring sensor data, determining attributes associated with an object, determining a predicted position based on the attributes, and controlling control of a vehicle based on the predicted position. [Diagram 2] FIG. 11 is a diagram showing examples of object attributes. [Figure 3A] 1 is an exemplary diagram of determining a destination associated with an object in an environment. [Figure 3B] FIG. 1 illustrates another example of determining a destination associated with an object in an environment. [Figure 4] FIG. 13 is a diagram illustrating determining a predicted location of an object based on attributes of the object over time. [Diagram 5] FIG. 13 illustrates updating a reference frame used in determining a predicted position. [Figure 6] FIG. 13 is a pictorial flow diagram illustrating a process of acquiring sensor data, determining that a first object and a second object are within an environment, determining attributes associated with the second object, determining a predicted position based on the attributes and the reference line, and controlling a vehicle based on the predicted position. [Figure 7] FIG. 11 is a diagram showing examples of object attributes. [Figure 8] FIG. 1 illustrates an example of determining a predicted location of a first object based on attributes of a second object over time. [Figure 9] FIG. 1 is a block diagram illustrating an example system for implementing the techniques described herein. [Figure 10] FIG. 1 illustrates a process of acquiring sensor data, determining attributes associated with an object, determining a predicted position based on the attributes, and controlling a vehicle based on the predicted position. [Figure 11] FIG. 1 illustrates a process of acquiring sensor data, determining that a first object and a second object are within an environment, determining attributes associated with the second object, determining a predicted position based on the attributes and a reference line, and controlling a vehicle based on the predicted position. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] The present disclosure is directed to techniques for predicting the location of an object based on attributes of the object and / or attributes of other objects proximate the object. In a first example, the techniques discussed herein can be implemented to predict the location of a pedestrian proximate a pedestrian crossing area in an environment as the pedestrian crosses or prepares to cross the pedestrian crossing area. In a second example, the techniques discussed herein can be implemented to predict the location of an object (e.g., a vehicle) as the vehicle crosses an environment. For example, the predicted location of the vehicle can be based on attributes of the vehicle and attributes of other vehicles proximate the vehicle in the environment. The attributes can comprise information about the object, such as, but not limited to, location, speed, acceleration, bounding box, etc. The attributes, when input to a prediction component (e.g., a machine learning model such as a neural network), are used by the prediction component to predict the location of the object at future times (e.g., times T1, T2, T3, ..., T N ) for the object, so that we can output a prediction (e.g., the predicted position of the object) at time (e.g., time T -M , ..., T -2 , T -1 , T0). A vehicle, such as an autonomous vehicle, can be controlled to traverse an environment based at least in part on the predicted positions of the objects.

[0007] As mentioned above, in a first example, the techniques discussed herein can be implemented to predict a location of a pedestrian proximate a pedestrian crossing area in an environment when the pedestrian crosses the pedestrian crossing area or prepares to cross the pedestrian crossing area. For example, sensor data can be acquired in the environment and objects can be identified and classified as pedestrians. Furthermore, a pedestrian crossing area can be identified in the environment based on map data and / or based on sensor data (e.g., identifying a pedestrian crossing area directly from the sensor data by observing visual indications of a pedestrian crossing area (stripes, crosswalk signs, etc.) or indirectly by detecting a history of pedestrians crossing a road at such locations). At least one destination can be associated with the pedestrian crossing area. For example, if the pedestrian is on a sidewalk proximate to the pedestrian crossing, the destination can represent the opposite side of the road to the pedestrian crossing area. If the pedestrian is on the road (either inside or outside the pedestrian crossing area), the destination can be selected or otherwise determined based on the pedestrian's attributes (e.g., position, speed, acceleration, path, etc.). For multiple crosswalk areas in close proximity to one another, a score associated with the likelihood that a pedestrian will cross a particular crosswalk may be determined based on the pedestrian's attributes (e.g., location, speed, acceleration, path, etc.) The crosswalk area associated with the highest score may be selected or otherwise determined to be the target crosswalk associated with the pedestrian.

[0008] In some examples, such as when running a red light or crossing a road where a pedestrian crossing area is not easily identifiable, a destination associated with the pedestrian can be determined based on many factors. For example, the destination can be determined based at least in part on one or more of a linear extrapolation of the pedestrian's speed, the closest location of a pedestrian crossing area associated with the pedestrian, a gap between parked vehicles, an open door associated with a vehicle, etc. In some examples, sensor data can be obtained about the environment to determine the likelihood that these exemplary candidate destinations are present in the environment. In some examples, a score can be associated with each candidate destination, and the likely destinations can be used in accordance with the techniques discussed herein.

[0009] If a crosswalk area (or other location) is determined to be the pedestrian's destination, the technique can include predicting the location of the pedestrian crossing the crosswalk area over time. In some examples, the object attribute can be a time (e.g., time T -M , ..., T -2 , T -1 , T0), whereby the attributes can be represented in a frame of reference associated with the object at time T0. That is, the position of the object at T0 can be considered as an origin (e.g., coordinate (0,0) in an xy coordinate system), whereby a first axis can be defined by the origin and a destination associated with the pedestrian crossing area. In some examples, other points can be considered as the origin with respect to another frame of reference. As mentioned above, if the pedestrian is on a first side of the road, the destination associated with the pedestrian crossing area can be selected as a point on a second side of the road opposite the first side of the road, although any destination can also be selected. The second axis of the frame of reference can be perpendicular to the first axis and, at least in some examples, lies along a plane that includes the pedestrian crossing area.

[0010] In some examples, attributes of a pedestrian may be determined based on sensor data captured over time, and may include a pedestrian's position at a time (e.g., the position may be expressed in a frame of reference as described above), the pedestrian's speed at that time (e.g., magnitude and / or angle relative to a first axis (or other reference line)), the pedestrian's acceleration at that time, an indication of whether the pedestrian is in a drivable area (e.g., whether the pedestrian is on the sidewalk or on the road), an indication of whether the pedestrian is in a crosswalk area, the state of indicators controlling the area (e.g., whether the intersection is signal controlled and / or whether the crosswalk is signal controlled (e.g., walk / no walk) and / or the state of the signal), vehicular context (e.g., the presence of vehicles in the environment and attributes associated with the vehicles), flux through the crosswalk area over a period of time (e.g., the number of objects (e.g., vehicles) that have passed through the crosswalk area over a period of time), object associations (e.g., whether the pedestrian is in a crosswalk area with multiple pedestrians), and / or attributes of the pedestrian's speed at that time. the first pedestrian is moving in a group of pedestrians (e.g., a first pedestrian is moving in a group of pedestrians (e.g., a second pedestrian is moving in a group of pedestrians (e.g., a first pedestrian is moving in a group of pedestrians (e.g., a second pedestrian is moving in a group of pedestrians (e.g., a third pedestrian is moving in a group of pedestrians (e.g., a fourth pedestrian is moving in a group of pedestrians (e.g., a fourth pedestrian is moving in a group of pedestrians (e.g., a fourth pedestrian is moving in a group of pedestrians (e.g., a fourth pedestrian is moving in a group of pedestrians (e.g., a fifth ...

[0011] The attribute can be a time (e.g., but not limited to, a time T that represents any time before the current time and / or including the current time, such as 0.01 seconds, 0.1 seconds, 1 second, 2 seconds, etc.) -M , ..., T-2 , T -1 , T0 (where M is an integer)) and can be input to a prediction component to determine a predicted position of the pedestrian. In some examples, the prediction component is a machine learning model such as a neural network, a fully connected neural network, a convolutional neural network, a recurrent neural network, etc.

[0012] In some examples, the prediction component can output information associated with future pedestrians. For example, the prediction component can output information associated with future times (e.g., times T1, T2, T3, ..., T4, ..., T5, ... N (where N is an integer). In some examples, the prediction information may comprise a predicted position of the pedestrian at a future time. For example, the predicted position may comprise a distance (e.g., distance s) between an origin (e.g., the position of the pedestrian at T0) and the pedestrian at T1 in a frame of reference, and / or a lateral offset (e y In some examples, the distance s and / or the lateral offset e y may be expressed as a rational number (e.g., 0.1 meters, 1 meter, 1.5 meters, etc.). In some examples, the distance s, and / or the lateral offset may be binned (e.g., input to a binning algorithm) to discretize the original data values ​​into one or many discrete intervals. In some examples, the bins of distance s may be 0-1 meters, 1-2 meters, 3-4 meters, etc., although any regular or irregular interval may be used for such bins.

[0013] In some examples, a vehicle, such as an autonomous vehicle, can be controlled to traverse an environment based at least in part on the predicted positions of pedestrians.

[0014] As mentioned above, in a second example, the techniques discussed herein may be implemented to predict the location of an object (e.g., a vehicle) as the vehicle passes through an environment. For example, sensor data may be acquired within the environment, and an object may be identified and classified as a vehicle. Additionally, a reference line may be identified and associated with the vehicle based on map data (e.g., identifying drivable areas, such as lanes) and / or sensor data (e.g., identifying drivable areas, or lanes, from the sensor data). As can be appreciated, the environment may include any number of objects. For example, an object of interest, or a vehicle of interest (e.g., a vehicle that is the subject of such prediction techniques) is moving through an environment in which there are other vehicles in close proximity to the vehicle of interest. In some examples, the techniques include identifying K objects closest to the object of interest (where K is an integer). For example, the techniques may include identifying the five vehicles or other objects closest to the vehicle of interest, although any number of vehicles or other objects may be identified or otherwise determined. In some examples, the techniques may include identifying objects within a threshold distance to the object of interest. In some examples, a vehicle capturing sensor data is identified as one of the objects proximate to the vehicle of interest. In at least some examples, additional features are used to determine which objects to consider. As non-limiting examples, objects traveling in the opposite direction, objects on the other side of a divided road, objects with certain classifications (e.g., non-vehicle), etc. are ignored when considering the K closest objects.

[0015] In some examples, attributes may be determined for the object of interest and / or other objects proximate to the object of interest. For example, the attributes may include, but are not limited to, one or more of the following: the speed of the object at a point in time, the acceleration of the object at a point in time, the position of the object at a point in time (e.g., in global or local coordinates), the bounding box of the object at a point in time (e.g., representing the range, roll, pitch, and / or yaw of the object), the state of the lights associated with the object at an initial point in time (headlights, brake lights, hazard lights, turn signals, backlights, etc.), the orientation of the wheels of the vehicle, the distance between the object and a map element at a point in time (distance to a stop line, speed bump, yield line, intersection, roadway, etc.), the classification of the object (car, vehicle, animal, truck, bicycle, etc.), features associated with the object (whether the object is changing lanes, whether it is a double-parked vehicle, etc.), lane type (lane direction, parking lane, etc.), road signs (such as those indicating whether passing or lane changing is permitted), etc.

[0016] In some examples, attribute information associated with an object of interest, and / or other objects proximate the object of interest, can be captured over time and input to a prediction component to determine predictive information associated with the object of interest. In some examples, the predictive information can include predicted locations of the object at various time intervals (e.g., times T1, T2, T3, ..., T N (predicted position in ).

[0017] In some examples, the predicted location can be compared to candidate reference lines in the environment to determine a reference line associated with the object of interest. For example, the environment includes two lanes that are eligible (e.g., legal) drivable areas for the vehicle of interest to cross. Furthermore, such drivable areas are associated with representative reference lines (e.g., the center of the lanes or drivable areas). In some examples, the predicted location can be compared to the reference lines to determine a similarity score between the predicted location and the reference line candidates. In some examples, the similarity score can be based at least in part on the distance between the predicted location and the reference line, etc. In some examples, the predicted location can be compared to the candidate reference lines to determine a similarity score between the predicted location and the reference line candidates, etc. In some examples, the predicted location can be compared to the candidate reference lines, etc., based on attributes associated with the object (e.g., the time T -M , T -1 , T0) can be input to a reference line prediction component, which can output a reference line that is likely to be associated with the object. The technique can include receiving, selecting, or otherwise determining a reference line and representing a predicted position for the reference line in the environment. That is, the predicted position can be represented as a distance s along the reference line, which represents the distance between the object's position at time T0 and the predicted position of the object at a future time (e.g., time T1). A lateral offset e y may represent the distance between the reference line and a point that intersects a line perpendicular to the tangent associated with the reference line.

[0018] The prediction techniques can be repeated iteratively or in parallel to determine predicted positions associated with objects in the environment. That is, a first object of interest is associated with a first subset of objects in the environment and a second object of interest is associated with a second subset of objects in the environment. In some examples, the first object of interest is included in the second subset of objects, while the second object of interest is included in the first subset of objects. In this manner, predicted positions can be determined for multiple objects in the environment. In some cases, the predicted positions can be determined substantially simultaneously within technical tolerances.

[0019] In some examples, a vehicle, such as an autonomous vehicle, can be controlled to traverse an environment based at least in part on a predicted position of an object, for example, such a predicted position can be input to a planning component of the vehicle to understand the predicted position of the object in the environment and traverse the environment.

[0020] The techniques discussed herein can improve the functionality of a computing device, such as a computing device of an autonomous vehicle, in many additional ways. In some examples, determining attributes and inputting the attributes into a predictive component, such as a machine learning component, can avoid hard-coded rules that would otherwise represent the environment inflexibly. In some cases, determining predicted positions associated with objects (such as pedestrians and vehicles) in the environment can provide other vehicles and objects with better planning trajectories for safe and comfortable travel through the environment. For example, a predicted position that indicates a potential collision or near collision can provide the autonomous vehicle with the ability to change trajectory (e.g., change lanes, stop, etc.) to safely traverse the environment. Such and other improvements in the functionality of computing devices are discussed herein.

[0021] The techniques described herein can be implemented in many ways. Exemplary implementations are provided with reference to the following figures. Although discussed in the context of an autonomous vehicle, the methods, apparatus, and systems described herein can be applied to a variety of systems (e.g., sensor systems or robotic platforms) and are not limited to autonomous vehicles. In one example, similar techniques are utilized in driver-controlled vehicles where such systems provide indications of whether it is safe to perform various maneuvers. In another example, the techniques can be utilized in the context of a manufacturing assembly line or in the context of aerial surveying. Additionally, the techniques described herein can be used with real data (e.g., captured through the use of sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

[0022] FIG. 1 is a pictorial flow diagram of an example process 100 of capturing sensor data, determining attributes associated with an object, determining a predicted position based on the attributes, and controlling a vehicle based on the predicted position.

[0023] In operation 102, the process may include capturing sensor data of the environment. In some examples, the sensor data may be captured by one or more sensors on the vehicle (autonomous or otherwise). For example, the sensor data may include data captured by a LIDAR sensor, an image sensor, a RADAR sensor, a time of flight sensor, a sonar sensor, etc. In some examples, operation 102 may include determining a classification of the object (e.g., determining that the object is a pedestrian in the environment).

[0024] At operation 104, the process may include determining a destination associated with the object (e.g., a pedestrian). Example 106 shows a vehicle 108 and an object 110 (e.g., a pedestrian) in an environment. In some examples, the vehicle 108 may perform the operations discussed in process 100.

[0025] The operation 104 may include determining attributes of the object 110 to determine a position, a speed, a path, etc. of the object 110. Additionally, the operation 104 may include accessing map data to determine whether a pedestrian crossing area (e.g., the pedestrian crossing area 112) exists in the environment. In some examples, the pedestrian crossing area 112 may represent a perimeter of a pedestrian crossing in the environment. In some examples, the operation 104 may include determining that the object is within a threshold distance (e.g., 5 meters) of a portion of the pedestrian crossing area 112. In some examples, the threshold distance is considered to be the shortest distance from the object to any portion of the pedestrian crossing area. If the object 110 is within the threshold distance of various pedestrian crossing areas in the environment, the operation 104 may include determining a probability, or score, associated with a pedestrian (e.g., the object 110) crossing each pedestrian crossing area and selecting the most likely pedestrian crossing area. In some examples, a destination 114 may be associated with the pedestrian crossing area 112. In some examples, destination 114 may represent any point in the environment associated with crosswalk area 112, but may represent the center or midpoint of the side of crosswalk area 112 opposite the location of object 110. Additional details of determining the destination are discussed in connection with Figures 3A and 3B, as well as throughout this disclosure.

[0026] In operation 116, the process may include determining attributes associated with the object. As illustrated in example 118, the attributes may be determined for the object 110 from various instances in time up to and including the most recent time associated with the attributes (e.g., time T -M , ..., T -2 , T -1, T0) the object 110 can be determined by the object 120 (e.g., at time T -2 ) as object 122 (e.g., at time T -1 ) and object 124 (e.g., at time T0). In some examples, time T0 represents the time when data is input to a prediction component (described below), and time T -1 represents one second before time T0, and -2 represents 2 seconds before time T0. However, -1 , and T -2 It can be understood that time T can represent any time instance and / or period of time. For example, time T -1 represents 0.1 seconds before time T0, and time T -2 represents 0.2 seconds before time T0. In some examples, the attributes determined in operation 116 may include, but are not limited to, information about objects 120, 122, and / or 124. For example, a velocity attribute associated with object 120 may be -2 The velocity attribute associated with the object 122 represents the velocity of the object 120 at time T -1 The velocity attribute associated with the object 124 represents the velocity of the object at time T0. In some examples, some or all of the attributes are expressed in a frame of reference relative to the object 124 (e.g., object 110 at time T0) and the destination 114. In such examples, the velocity attribute associated with the object 124 represents the velocity of the object at time T0. -M There are three unique reference frames associated with the time series (T1 to T0), and each attribute is associated with its particular time frame of reference. Additional details of the attributes are discussed in connection with FIG. 2 as well as throughout this disclosure.

[0027] In operation 126, the process may include determining a predicted location associated with the object based on the attributes. Example 128 illustrates predicted location 130 (e.g., a predicted location of object 110 at time T1, a time after T0). In some examples, operation 126 may be performed at or near time T0, such that predicted location 130 at time T1 may represent a location of object 110 in the future. As can be appreciated, in some examples, operation 126 may include determining predicted locations for multiple times associated with future object 124. For example, operation 126 may determine predicted locations for times T1, T2, T3, ..., T N The method may include determining a predicted position of the object at N, where N is an integer representing a time in the future, e.g., 1 second, 2 seconds, 3 seconds, etc. In some examples, the predicted position is determined by a distance s along the reference line and a lateral offset e from the reference line. y In at least some examples, the distance s and the offset e y , where θ is relative to the relative coordinate system defined at each time step and / or relative to the last determined reference frame. Additional details regarding determining predicted positions are discussed in connection with Figures 4 and 5 and throughout this disclosure.

[0028] In some examples, operations 102, 104, 116, and / or 126 may be performed iteratively or repeatedly (e.g., at each time step, at a frequency of 10 Hz, etc.), although process 100 may be performed at any interval or for any length of time.

[0029] At operation 132, the process may include controlling the vehicle based at least in part on the predicted position. In some examples, operation 132 may include generating a trajectory for the vehicle 108 to follow (e.g., stopping before an intersection and / or before a pedestrian crossing area 112, providing for the pedestrian 110 to cross through the pedestrian crossing area 112 to the destination 114).

[0030] 2 illustrates example attributes 200 of an object. In some examples, the attributes 202 may represent various information about or associated with an object in an environment (e.g., object 110 in FIG. 1). In some examples, the attributes 202 may be determined for one or more time instances associated with the object. For example, object 120 is captured at time T -2 represents object 110 at time T -1 , and object 124 represents object 110 at time T0. -2 , T -1 , and T0, respectively.

[0031] Examples of attributes 202 include, but are not limited to, distance between the object and the road, x-distance (or first distance) to the region, y-distance (or second distance) to the region, distance to destination, speed (magnitude), speed (angle), x-position, y-position, region flux, state of an indicator controlling the region, vehicle context (or generally, object context), object association, etc. In at least some examples, the attributes discussed herein are relative to a relative coordinate system defined at each time step (e.g., associated with each of the objects 120, 122, 124), relative to a last determined reference frame, relative to a reference frame defined with respect to the vehicle 108 (e.g., at various time steps(s)), relative to a global coordinate reference frame, etc.

[0032] Example 204 illustrates various attributes associated with object 124. For example, example 204 illustrates attributes related to crosswalk region 112 and destination 114. In some examples, an x-distance to the region can correspond to distance 206. That is, distance 206 can represent a distance in a first direction (which can be a global or local reference frame) between object 124 and an edge of crosswalk region 112 that is closest to object 124. In some examples, a y-distance to the region can correspond to distance 208. That is, distance 208 can represent a distance in a second direction between object 124 and an edge of crosswalk region 112. In at least some examples, the simplest distance between object 124 and crosswalk region is determined and then decomposed into respective x- and y-components as x-distance and y-distance.

[0033] As shown in example 204, object 124 is at a location on sidewalk region 210 (or generally, non-drivable region 210). In some examples, crosswalk region 112 provides a path across road 212 (or generally, drivable region 212). In some examples, the distance to the road may correspond to distance 214, which may correspond to a minimum or minimum distance between object 124 and the portion of road 212 within crosswalk region 112.

[0034] In some examples, the distance to the destination may correspond to distance 216. As shown, distance 216 represents the distance between object 124 and destination 114.

[0035] As mentioned above, in some examples, the attribute 202 can be represented in a frame of reference. As discussed herein, a frame of reference is defined with respect to the position of the object at each time step, relative to a last frame of reference, a global coordinate system, etc. In some examples, an origin corresponding to the frame of reference can correspond to the position of the object 124. Example 218 illustrates a frame of reference 220 (also referred to as reference frame 220). In some examples, a first axis of the frame of reference 220 is defined by a unit vector from the position of the object 124 in the direction of the destination 114. The first axis is labeled as the x-axis in example 218. In some examples, a second axis is perpendicular to the second axis and can lie in a plane comprising the crosswalk. The second axis is labeled as the y-axis in example 218. In some examples, the first axis can represent a reference line from which a distance s can be determined, while a lateral offset e can be determined from the reference line s. y can be determined for a second direction (eg, the y-axis).

[0036] Example 222 shows a velocity vector 224 associated with object 124 and an angle 226 that represents the angle between the velocity vector 224 and a reference line. In some examples, the reference line may correspond to a first axis of frame of reference 220, although any reference line may be selected or otherwise determined.

[0037] As discussed herein, attributes associated with objects 124, 122, and 120 may be expressed with respect to a frame of reference 220. That is, at time T0, the x- and y-positions of object 124 may be expressed as (0,0) (e.g., object 124 represents the origin of frame of reference 220). Furthermore, with respect to frame of reference 220, the x- and y-positions of object 122 (at time T0) may be expressed as (-x1,-y1), and the x- and y-positions of object 120 (at time T0) may be expressed as (-x2,-y2). In at least some examples, a single coordinate frame is used, while in other examples, a relative coordinate frame is associated with every point and attributes are defined for each relative coordinate frame.

[0038] As mentioned above, the attributes 202 can include area flux. In some examples, the area flux can represent the number of objects passing through the crosswalk area 112 within a period of time. For example, the area flux can correspond to J cars (and / or other objects, such as other pedestrians) that passed through the crosswalk area 112 (or any area) within K seconds (e.g., T -2 5 vehicles during the time between T0 and T1). In some examples, the area flux can represent any time period. Additionally, the area flux can include information such as speed, acceleration, velocity, etc., for those vehicles that passed through the crosswalk area 112 during the time period.

[0039] Additionally, attributes 202 may include indicators controlling the area. In some examples, the indicators controlling the area may correspond to the state of a signal or indicator controlling pedestrian traffic within crosswalk area 112. In some examples, the indicators controlling the area may indicate whether a signal is present, the state of the signal (e.g., green, yellow, red, etc.), and / or the state of the crosswalk indicator (e.g., walk, no walk, unknown, etc.).

[0040] In some examples, the attributes 202 may include a vehicle context that indicates whether a vehicle or other object is in proximity to the object (e.g., 124), and attributes associated with any such vehicle or object. In some examples, the vehicle context may include, but is not limited to, speed, direction, acceleration, bounding box, position (e.g., within the frame of reference 220), distance between the object and the object 124, etc.

[0041] In some examples, the attributes 202 can include object associations. For example, the object associations can indicate whether the object 124 is associated with other objects (e.g., whether the object 124 is in a group of pedestrians). In some examples, the object association attributes 202 can include associated attributes for the associated object.

[0042] The attributes 202 further include, but are not limited to, information associated with acceleration, yaw, pitch, roll, relative velocity, relative acceleration, whether the object is on a road 212, whether the object is on a sidewalk 210, whether the object is in a crosswalk area 112, whether the destination has changed (e.g., whether the object turned back at an intersection), the height of the object, whether the object is a bicycle, and the like.

[0043] The attributes 202 may further include, but are not limited to, pedestrian hand gestures, pedestrian gaze detection, an indication of whether a pedestrian is standing, walking, running, etc., whether other pedestrians are in a crosswalk, pedestrian crosswalk flux (e.g., the number of pedestrians moving through a crosswalk (e.g., across a driveable area) in a period of time), the ratio of a first number of pedestrians on the sidewalk (or non-driveable area) to a second number of pedestrians in the crosswalk area (or driveable area), a variance, confidence, and / or probability associated with each attribute, and the like.

[0044] 3A and 3B show examples of determining destinations associated with objects in an environment. Generally, FIG. 3A illustrates selecting between two pedestrian crossing areas, while FIG. 3B illustrates selecting between two destinations associated with a single pedestrian crossing area.

[0045] FIG. 3A illustrates an example 300 of determining a destination associated with an object in an environment. As discussed above, and generally, FIG. 3A illustrates selecting between two pedestrian crossing areas. Example 302 illustrates a selection between two pedestrian crossing areas at a time T -1 1 illustrates an object 304 corresponding to a pedestrian at time T0 and an object 306 corresponding to a pedestrian at time T0. For example, a vehicle, such as vehicle 108, can capture sensor data of an environment and can determine that a pedestrian is present in the environment.

[0046] Further, based at least in part on the objects 304 and 306, the computing system can determine that the objects 304 and / or 306 are proximate to one or more pedestrian crossing areas within the environment. For example, the computing device can access map data including map elements indicating the locations and extents (e.g., length and width) of such pedestrian crossing areas. Example 302 illustrates the environment as including a first pedestrian crossing area 308 (also referred to as area 308) and a second pedestrian crossing area 310 (also referred to as area 310).

[0047] In some examples, region 308 can be associated with threshold region 312 (also referred to as threshold 312) and region 310 can be associated with threshold region 314 (also referred to as threshold 314). As shown, objects 304 and 306 are within thresholds 312 and 314. Based at least in part on objects 304 and / or 306 being within thresholds 312 and 314, a computing device can determine that objects 304 and / or 306 are associated with regions 308 and 310, respectively.

[0048] In some examples, threshold 312 may represent any region or area associated with region 308. As illustrated, threshold 312 may represent a 5 meter threshold surrounding region 308, although any distance or shape of threshold 312 may be associated with region 308. Similarly, threshold 314 may include any distance or shape associated with region 310.

[0049] In some examples, region 308 can be associated with destination 316. Further, in some examples, region 310 can be associated with destination 318. In some examples, the location of destination 316 and / or 318 is located across a road from object 304 and / or 306. That is, a destination associated with a crosswalk region can be selected based at least in part on the location of a pedestrian relative to the crosswalk region.

[0050] The objects 304 and / or 306 can be associated with attributes as discussed herein, i.e., techniques can include determining the position, velocity, heading, acceleration, etc. of the objects 304 and 306, respectively.

[0051] Additionally, the information depicted in example 302 (e.g., attributes associated with objects 304 and / or 306, the locations of regions 308 and / or 310, the locations of thresholds 312 and / or 314, the locations of destinations 316 and / or 318, etc.) can be input to a destination prediction component 320. In some examples, the destination prediction component 320 can output a score, or probability, that object 306 will cross region 308 and / or region 310. In example 302, two time steps (e.g., T -1、 and T0), any object information over time can be used to determine the destination.

[0052] In some examples, attributes associated with objects 302 and 306 can be input to destination prediction component 320 using one or more frames of reference. For example, to evaluate destination 316, attributes associated with objects 304 and 306 can be input to destination prediction component 320 using a frame of reference based at least in part on destination 316. Further, to evaluate destination 318, attributes associated with objects 304 and 306 can be input to destination prediction component 320 using a frame of reference based at least in part on destination 318.

[0053] In some examples, such as in the case of a pedestrian running a red light or crossing a road where a crosswalk area is not easily identifiable, a destination associated with the pedestrian can be determined based on a number of factors. For example, the destination can be determined based at least in part on one or more of a linear extrapolation of the pedestrian's speed, the closest location of a sidewalk area associated with the pedestrian, a gap between parked vehicles, an open door associated with a vehicle, etc. In some examples, sensor data of the environment can be captured to identify possible destinations within the environment. Additionally, attributes associated with the object can be represented in a frame of reference based at least in part on the determined destination, and the attributes can be input to the destination prediction component 320 for evaluation, as discussed herein.

[0054] Example 322 illustrates output of the destination prediction component 320. For example, based at least in part on attributes of the object 304 and / or 306, the destination prediction component 320 predicts that the object 304 and / or 306 is heading toward the destination 318.

[0055] Figure 3B illustrates another example of determining destinations associated with objects in an environment 324. As discussed above, Figure 3B illustrates selecting between two destinations associated with a single pedestrian crossing area.

[0056] Example 324 shows the time T -1 3B shows an object 326 corresponding to a pedestrian at time T0 and an object 328 corresponding to a pedestrian at time T0. In some examples, because the objects 326 and 328 are on a road 330 (or drivable area 330) (as opposed to being located on a sidewalk 332 (or non-drivable area 332)), the computing device identifies two destinations 334 and 336 associated with an area 338. In some examples, attributes associated with the objects 326 and 328 can be input to the destination prediction component 320 to determine which of the destinations 334 and 336 is most likely (information about the destinations 334 and 336 and the area 338, as well as other information). While this FIG. 3B is depicted as entering and leaving a crosswalk for illustrative purposes, such a crosswalk area is not required. As a non-limiting example, such a destination prediction component 320 would typically determine that the pedestrian is about to run a red light or is otherwise crossing an area that is not a crosswalk area, and output the corresponding destination. In such examples, attributes associated with the area are not determined (because the area does not exist). However, in some instances, a fixed region perpendicular to the road segment and having a certain width is used as the region for determining such parameters.

[0057] As discussed above, in some examples, region 338 is associated with object 326 and / or 328 at a time when object 326 and / or 328 is within a threshold distance of region 338 .

[0058] FIG. 4 illustrates an example 400 of determining a predicted location of an object based on attributes of the object over time.

[0059] Example 402 illustrates the motion of object 120 (e.g., at time T -2 pedestrian at time T -11, the example 402 illustrates objects 120, 122, and 124 associated with a crosswalk area 112, and a destination 114. As discussed herein, objects 120, 122, and 124 may be represented in a frame of reference that has object 124 as the origin (and / or in one or more frames of reference associated with any one or more times). Additionally, example 402 illustrates objects 120, 122, and 124 associated with a crosswalk area 112, and a destination 114.

[0060] Data associated with the instances 402 can be input to a location prediction component 404 that can output a predicted location associated with the objects 120 , 122 , and / or 124 .

[0061] Example 406 illustrates a predicted position based on objects 120, 122, and / or 124. For example, location prediction component 404 can output a predicted position 408, which represents the position of the object at time T1. In some examples, predicted position 408 can include a distance (e.g., s) 410 and a lateral offset 412 (e.g., e) based at least in part on a frame of reference defined by object 124 (e.g., origin) and destination 114. y )

[0062] As shown, the location prediction component 404 can output five predicted positions corresponding to each of times T1, T2, T3, T4, and T5, respectively, although it can be understood that the location prediction component 404 can output any number of predicted positions associated with any future time. In some examples, such additional predicted positions are defined with respect to a global coordinate frame, a local coordinate frame, a relative reference frame associated with a previous predicted point, etc.

[0063] In some examples, the location prediction component 404 may estimate the distance s or the lateral offset e yThe location prediction component 404 may include a function to bin the output values, such as 0.5 m, 1.0 m, 0.9 m, 1.0 m, 0.5 m, 1.0 m, 0.9 ...

[0064] In general, the predicted location shown in example 406 can be referred to as predicted location 414 .

[0065] In some examples, the location prediction component 404 can output a variance, covariance, probability, or certainty associated with each predicted location 414 indicating the certainty that the object 124 will be located at each predicted location at each time.

[0066] FIG. 5 shows an example 500 of updating the frame of reference used in determining a predicted position.

[0067] Example 406 shows the time T A 5 to illustrate the present invention. As shown, objects 120, 122, and 124 are represented in a frame of reference 220 that is defined in part by the location of object 124 and the location of destination 114.

[0068] In some examples, the instance 406 may be updated for the next time step and an updated predicted position may be determined (eg, in operation 502).

[0069] Such an updated example is shown as example 504, which shows an environment corresponding to example 406, but at time T A Time T that occurs after B The object 506 of the example 504 represents a time T0 relative to a frame of reference 508. Similarly, the example 504 represents a time T -1 Additionally, object 512 represents an object at time T -2 Represents an object in.

[0070] In some examples, the object 510 (e.g., time T -1 ) may correspond to object 124 (e.g., an object at time T0 in frame of reference 220). Similarly, object 512 (e.g., an object at time T -2 object at time T -1 For comparison, example 504 shows object 120, whereby object 120 (and / or attributes associated with object 120) are used or object 120 is not used when determining the updated predicted position in example 504.

[0071] As can be appreciated, the frame of reference 508 can be defined by, or at least in part based on, the positions of the object 506 and the destination 114. In this manner, a relative frame of reference can be defined with respect to the most recently determined positions of the destination 114 and the object 124 (e.g., such a coordinate reference frame can change with changes in the objects in the environment).

[0072] Thus, information associated with the instance 504 (including or not including information associated with the object 120) can be input to the location prediction component 404 to determine an updated predicted location 514. As discussed herein, the updated predicted location 514 is based at least in part on the frame of reference 508.

[0073] In some examples, the updated predicted position may be determined at a frequency of 10 Hz, although the predicted position may be determined at any frequency or at any regular or irregular time interval.

[0074] FIG. 6 is a pictorial flow diagram illustrating a process 600 of capturing sensor data, determining that a first object and a second object are within an environment, determining attributes associated with the second object, determining a predicted position based on the attributes and a reference line, and controlling a vehicle based on the predicted position.

[0075] Although discussed in the context of determining attributes of a first and second object to determine a predicted location associated with a first object, in some examples, no attributes are determined for one or more second objects, and the predicted location of the first object can be determined based on the attributes associated with the first object.

[0076] At operation 602, the process may include capturing sensor data of the environment. In some examples, the sensor data may be captured by one or more sensors on a vehicle (autonomous or otherwise). For example, the sensor data may include data captured by a LIDAR sensor, an image sensor, a RADAR sensor, a time of flight sensor, a sonar sensor, etc. In some examples, operation 602 may include determining a classification of the object (e.g., determining that the object is a vehicle in the environment).

[0077] Example 604 illustrates a vehicle 606 capturing sensor data in operation 602. The environment further includes objects 608, 610, 612, 614, 616, and 618. In some examples, object 618 may be referred to as a target object 618 because it is a target (e.g., a subject) of a predictive operation as discussed herein.

[0078] In some examples, vehicle 606 traverses the environment via trajectory 620. As can be understood in the context of FIG. 6, object 608 can be traveling in the same direction as vehicle 606 (e.g., in the same lane as vehicle 606), while in some examples objects 610-618 and target object 618 can travel in the opposite direction (e.g., target object 618 can represent an oncoming vehicle with respect to vehicle 606). Of course, process 600 can be used in any environment and is not limited to the particular objects and / or geometries shown in FIG.

[0079] At operation 622, the process may include determining attributes associated with the target object and objects proximate to the target object. Example 624 shows vehicle 606, objects 606-616, and target object 618. In some examples, operation 622 includes determining attributes associated with the target object without determining attributes of other objects. For example, no such other objects are present in the environment, or such attributes of the other objects are not needed, desired, or required for determining a predicted location of target object 618 in accordance with an implementation of the techniques discussed herein.

[0080] For purposes of illustration, the outline of object 612 is shown as a dotted line, and elements 626, 628, and 630 corresponding to object 612 are represented as dots. -2 In some examples, element 628 represents a position associated with object 612 at time T -1 In some examples, element 630 represents a position associated with object 612 at time T0.

[0081] As further shown, vehicle 606, objects 608 through 616, and target object 618 are associated with elements, although such elements are not labeled in FIG. 6. Such elements are associated with elements at respective times (e.g., time T -2 , T -1 , and T0) and / or may represent positions associated with the vehicle and / or object and / or attributes associated with the object at the respective times, it may be understood in the context of this disclosure.

[0082] In some examples, the attributes determined in operation 622 may represent information about each object. For example, such attributes may include, but are not limited to, the object's position (e.g., global position and / or relative position with respect to any frame of reference), speed, acceleration, bounding box, lighting conditions, lane attributes, reference lines, or offset from a predicted path, etc. Additional details of such attributes are discussed in connection with FIG. 7, as well as throughout this disclosure.

[0083] In some examples, operation 622 may include determining or identifying an object based at least in part on the proximity of the object to the target object. For example, operation 622 may include determining the N closest objects in proximity to target object 618, where N is an integer. Additionally or alternatively, operation 622 may include identifying or selecting an object based on the object being within a threshold distance of target object 618. In at least some examples, such selection excludes certain objects based on one or more characteristics such as, but not limited to, object classification (e.g., considering only vehicles), direction of motion (e.g., considering only objects moving in the same direction), location relative to a map (e.g., considering only vehicles in one or more lanes of a road), etc.

[0084] At operation 632, the process may include determining a predicted position associated with the target object based at least in part on the attribute, the predicted position being relative to a reference line in the environment (which in some examples comprises a centerline of a lane associated with the object). Example 634 illustrates a predicted position 636 associated with the target object 618 in the environment. In some examples, the predicted position 636 may be defined by and / or based at least in part on the reference line 638. That is, the predicted position 636 may be defined by a distance s along the reference line 638 and a lateral offset e from the reference line 638. y and

[0085] In some examples, the reference lines 638 may be based at least in part on map data within the environment. Further, in some examples, the reference lines 638 may correspond to centerlines of lanes of a road or other drivable area.

[0086] In some examples, operation 632 may include receiving a reference line associated with the target object 618, such as from a reference line prediction component. In some examples, the reference line prediction component may comprise a machine learning model trained to output a most likely reference line based at least in part on map data, attributes of objects in the environment, etc. In some examples, the reference line prediction component may be integrated with other machine learning models discussed herein, and in some examples, the reference line prediction component may be a separate component.

[0087] In some examples, operation 632 may include selecting a reference line 638 from a plurality of candidate reference lines. In some examples, the reference line 638 may be selected based at least in part on a similarity score that represents a similarity of the predicted position 636 to the reference line 638. Some examples include the predicted position 636 relative to a predicted path and / or trajectory, previously predicted waypoints, etc. Additional examples of predicted positions, reference lines, and similarity scores are discussed throughout this disclosure in connection with FIG.

[0088] In operation 640, the process may include controlling the vehicle based at least in part on the predicted position. In some examples, operation 640 may include generating a trajectory for the vehicle 608 to follow, or an updated trajectory 642 (e.g., biasing the vehicle 606 away from the predicted position 636 associated with the vehicle 618 if the object 618 crosses closely relative to the expected path of the vehicle 608).

[0089] Figure 7 illustrates example object attributes 700. In some examples, the attributes 702 may represent various information about or associated with objects in an environment (e.g., object 612 of Figure 6 and target object 618, represented by example 604 reproduced in Figure 7).

[0090] In some examples, the attributes 702 can be determined for one or more time instances of the object. -1 , and object 612 at T0. For example, element 626 shows -2 , and element 628 represents object 612 at time T -1 6 represents object 612 at time T0, and element 630 represents object 612 at time T0.

[0091] Further, attributes may be determined for any type and / or number of objects in example 704 and are not limited to object 612. For example, attributes may be determined for elements 706 (e.g., time T -26, which represents the object of interest 618 at time T -1 Attributes may be determined for element 710 (e.g., representing object 618 at time T0) and element 710 (e.g., representing object 618 at time T0). Additionally, attributes may be determined for any number of time instances, T -2 , T -1 , and is not limited to T0.

[0092] Examples of attributes 702 include, but are not limited to, an object's speed, an object's acceleration, an object's x position (e.g., position relative to a global position, local position, and / or other frame of reference), an object's y position (e.g., position relative to a local position, global position, and / or other frame of reference), a bounding box associated with the object (e.g., range (length, width, and / or height), yaw, pitch, roll, etc.), lighting status (e.g., brake lights, blinker lights, hazard lights, headlights, backlights, etc.), wheel direction of the object, map elements (e.g., distance between the object and a stop light, stop sign, speed bump, intersection, yield sign, etc.), object classification (e.g., vehicle, car, truck, bicycle, motorcycle, pedestrian, animal, etc.), object characteristics (e.g., whether or not a lane is being changed, whether the object is a double parked vehicle, etc.), proximity to one or more objects (in any coordinate frame), lane type (lane direction, parking lane, etc.), road signs (such as those indicating whether passing or lane changes are permitted), etc.

[0093] In some examples, object attributes may be determined with respect to a local frame of reference, global coordinates, etc. For example, a frame of reference may be determined to have an origin that corresponds to the position of target object 618 (e.g., object 710) at time T0.

[0094] FIG. 8 illustrates an example 800 of determining a predicted location of a first object based on attributes of a second object over time.

[0095] As shown, information associated with the example 704 of FIG. 7 can be input to a location prediction component 802, which can output a predicted location associated with the target object. For example, the location prediction component 802 can be used to predict the location of the target object at various times (e.g., T -2 , T -1 , and T0) can be input to a location prediction component 802.

[0096] The example 804 illustrates a predicted location 806 associated with an object of interest 618. That is, the location prediction component 802 can receive attribute information associated with the object of interest 618 as well as attribute information associated with objects proximate the object of interest 618 and output a predicted location 806 representing the object of interest 618 in the future.

[0097] Object 808 is -2 6 represents the object of interest 618 at time T -1 6 represents the target object 618 at time T0, and object 812 represents the target object at time T0.

[0098] The location prediction component 802 can determine a predicted location 806 based on the attribute information discussed herein. In some examples, the predicted location can be initially represented in a global coordinate system, with a frame of reference originating from the target object, etc. Additionally, the predicted location can be represented relative to a reference line in the environment.

[0099] In some examples, the environment represents multiple reference lines, such as reference line 814 and reference line 816. As depicted in FIG. 8 for illustrative purposes, reference line 816 corresponds to, for example, a lane change of an object. In some examples, reference line 814 represents a centerline of a first road segment, and reference line 816 represents a centerline of a second road segment (and / or a transition therebetween). In some examples, such as a single lane road, the environment represents a single reference line. However, in some examples, the environment represents multiple reference lines.

[0100] In some examples, the location prediction component 802 can receive as input an indication of a most likely reference line (e.g., 814). In some examples, the location prediction component 802 can determine the likely reference line based at least in part on one or more attributes of the target object 618, of other objects, and / or of the environment, as described herein.

[0101] In some examples, the location prediction component 802 can determine a similarity score 818 that represents a similarity between the predicted location 806 and the reference line 814. Additionally, the location prediction component 802 can determine a similarity score 820 that represents a similarity between the predicted location 806 and the reference line 816. In some examples, the similarity score can be based at least in part on individual or cumulative lateral offsets between the predicted location and each reference line, although other factors can also be used to determine the similarity score.

[0102] In some examples, the location prediction component 802 may determine that the similarity score 818 is lower than the similarity score 820 and, accordingly, select the reference line 814 as a basis for partially defining the predicted location 806. However, in other examples, each potential reference line is input to the location prediction component 802 along with previously calculated attributes such that the location prediction component 802 selects the appropriate reference line and / or trajectory to use as a basis based on machine-learned parameters.

[0103] The predicted location 806 can include predicted locations 822, 824, 826, 828, and / or 830. In some examples, the predicted location 822 can include a first distance s and a first lateral offset (e.g., (s1,e y1 The predicted position 824 may represent a second distance s relative to the reference line 814, and a second lateral offset (e.g., (s2,e y2 The predicted position 826 may represent a third distance s relative to the reference line 814, and a third lateral offset (e.g., (s3,e y3 The predicted position 828 may represent a fourth distance s relative to the reference line 814, and a fourth lateral offset (e.g., (s4,e y4 )) relative to the reference line 814. The predicted position 830 may then represent a fifth distance s and a fifth lateral offset (e.g., (s5,e y5 Of course, the location prediction component 802 can determine fewer or more predicted locations, as discussed herein.

[0104] 9 shows a block diagram of an example system 900 for implementing the techniques described herein. In at least one example, the system 900 may include a vehicle 902 that may correspond to the vehicle 108 of FIG. 1 and the vehicle 606 of FIG.

[0105] An exemplary vehicle 902 may be a driverless vehicle, such as an autonomous vehicle configured to operate in accordance with a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle that can perform all safety-critical functions throughout a trip, with no driver (or passenger) expected to control the vehicle at any time. In such an example, the vehicle 902 may be configured to control all functions from the beginning to the completion of a trip, including all parking functions, and thus does not include a driver and / or controls for operating the vehicle 902, such as a steering wheel, accelerator pedal, and / or brake pedal. This is merely an example, and the systems and methods described herein may be incorporated into any land-based, air-based, or water-based vehicle, including vehicles that must be manually controlled by a driver at all times, to vehicles that are partially or fully autonomously controlled.

[0106] Vehicle 902 may include a vehicle computing device 904, one or more sensor systems 906, one or more emitters 908, one or more communication connections 910, at least one direct connection 912, and one or more drive systems 914.

[0107] The vehicle computing device 904 can include one or more processors 916 and a memory 918 communicatively coupled to the one or more processors 916. In the illustrated example, the vehicle 902 is an autonomous vehicle, although the vehicle 902 may be other types of vehicles or a robotic platform. In the illustrated example, the memory 918 of the vehicle computing device 904 stores a localization component 920, a perception component 922, one or more maps 924, one or more system controllers 926, an attribute component 930, a prediction component 928 comprising a destination prediction component 932, and a location prediction component 934, and a planning component 936. While depicted in FIG. 9 as residing in memory 918 for purposes of explanation, it is contemplated that the localization component 920, the perception component 922, the one or more maps 924, the one or more system controllers 926, the prediction component 928, the attribute component 930, the destination prediction component 932, the position prediction component 934, and the planning component 936 may additionally or alternatively be accessible to the vehicle 902 (e.g., stored in memory separate from the vehicle 902 or otherwise accessible to the vehicle 902).

[0108] In at least one example, the localization component 920 can include functionality to receive data from the sensor system 906 to determine the position and / or orientation of the vehicle 902 (e.g., one or more of an x-position, a y-position, a z-position, a roll, a pitch, or a yaw). For example, the localization component 920 can include and / or request / receive a map of the environment and continuously determine the position and / or orientation of the autonomous vehicle within the map. In some examples, the localization component 920 can receive image data, LIDAR data, RADAR data, time of flight data, IMU data, GPS data, wheel encoder data, etc., to accurately determine the position of the autonomous vehicle using SLAM (simultaneous localization and mapping), CLAMS (simultaneous calibration, localization, and mapping), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc. In some examples, the localization component 920 can provide data to various components of the vehicle 902 to determine an initial position of the autonomous vehicle, to generate a trajectory, and / or to determine that an object is in proximity to one or more pedestrian crossing areas, and / or to identify candidate reference lines, as discussed herein.

[0109] In some examples, and generally, the perception component 922 may include functionality to perform object detection, segmentation, and / or classification. In some examples, the perception component 922 may provide processed sensor data indicative of the presence of an entity in proximity to the vehicle 902 and / or classifying the entity as a type of entity (e.g., automobile, pedestrian, bicycle, animal, building, tree, road surface, curb, sidewalk, stop light, stop sign, unknown, etc.). In additional or alternative examples, the perception component 922 may provide processed sensor data indicative of one or more features associated with the detected entity (e.g., tracked object) and / or the environment in which the entity is located. In some examples, the features associated with the entity may include, but are not limited to, x-position (global and / or local position), y-position (global and / or local position), z-position (global and / or local position), orientation (e.g., roll, pitch, yaw), type of entity (e.g., classification), velocity of the entity, acceleration of the entity, range (size) of the entity, etc. Features associated with an environment may include, but are not limited to, the presence of another entity in the environment, the state of another entity in the environment, time of day, day of the week, season, weather, darkness / light indications, etc.

[0110] The memory 918 may further include one or more maps 924 that may be used by the vehicle 902 to navigate within the environment. For purposes of this discussion, a map may be any number of data structures modeled in two, three, or N dimensions that may provide information about the environment, such as, but not limited to, topology (such as intersections), streets, mountains, roads, terrain, and the environment in general. In some examples, the map may include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and / or intensities), reflectance information (e.g., specular reflectance information, retroreflectance information, BRDF information, BSSRDF information, etc.), and the like. In one example, the map may include a three-dimensional mesh of the environment. In some examples, the maps may be stored in a tiled format, such that individual tiles of the map represent discrete portions of the environment, and may be loaded into the working memory as needed. In at least one example, the one or more maps 924 may include at least one map (e.g., an image and / or a mesh).

[0111] In some examples, vehicle 902 can be controlled based at least in part on map 924. That is, map 924 can be used in conjunction with localization component 920, perception component 922, prediction component 928, and / or planning component 936 to determine a position of vehicle 902, identify objects within the environment, and / or generate a route and / or trajectory for navigating within the environment.

[0112] In some examples, one or more maps 924 may be stored on a remote computing device (such as computing device 940) accessible via network 938. In some examples, multiple maps 924 may be stored, for example, based on characteristics (e.g., type of entity, time of day, day of the week, season of the year, etc.). Storing multiple maps 924 may have similar memory requirements but may improve the speed at which data in the maps can be accessed.

[0113] In at least one example, vehicle computing device 904 can include one or more system controllers 926 that can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 902. These system controllers 926 can communicate with and / or control corresponding systems of drive system 914 and / or other components of vehicle 902.

[0114] In general, the prediction component 928 can include functionality for generating predictive information associated with objects in the environment. In some examples, the prediction component 928 can be implemented to predict the location of a pedestrian proximate a crosswalk area (or area or location associated with pedestrians crossing a road) in the environment when the pedestrian crosses or prepares to cross the crosswalk area. In some examples, the techniques discussed herein can be implemented to predict the location of an object (e.g., a vehicle, a pedestrian, etc.) as the vehicle crosses the environment. In some examples, the prediction component 928 can generate one or more predicted trajectories of the target object based on attributes of such target object and / or other objects proximate the target object.

[0115] The attribute component 930 can include functionality for determining attribute information associated with objects in the environment. In some examples, the attribute component 930 can receive data from the perception component 922 and determine attribute information for objects over time.

[0116] In some examples, attributes of an object (e.g., a pedestrian) can be determined based on sensor data captured over time and can include one or more of the pedestrian's position at a point in time (e.g., the position can be expressed in a frame of reference as described above), the pedestrian's velocity at a point in time (e.g., magnitude and / or angle with respect to a first axis (or other reference line)), and the pedestrian's acceleration at a point in time. The information may include, but is not limited to, the pedestrian's speed (e.g., magnitude and / or angle with respect to a first axis (or other reference line)), the pedestrian's acceleration at that time, an indication of whether the pedestrian is in a drivable area (e.g., whether the pedestrian is on the sidewalk or the road), an indication of whether the pedestrian is in a crosswalk area, an indication of whether the pedestrian is running a red light, the state of a signal controlling the area (e.g., whether the crosswalk is controlled by a signal and / or the state of the signal), vehicle context (e.g., the presence of vehicles in the environment and attributes associated with the vehicles), flux through the crosswalk area over a period of time (e.g., the number of objects (e.g., vehicles and / or pedestrians) passing through the crosswalk area over a period of time), object associations (e.g., whether the pedestrian is moving among a group of pedestrians), the distance to the crosswalk in a first direction (e.g., global x-direction), the distance to the crosswalk in a second direction (e.g., global y-direction), the distance to the road within the crosswalk area (e.g., the shortest distance to the road within the crosswalk area), and the like.

[0117] In some examples, attributes can be determined for the target object (e.g., a vehicle) and / or other objects (e.g., other vehicles) in proximity to the target object. For example, the attributes can include, but are not limited to, the speed of the object at a point in time, the acceleration of the object at a point in time, the position of the object at a point in time (e.g., in global or local coordinates), a bounding box associated with the object at a point in time (e.g., representing the range, roll, pitch, and / or yaw of the object), the lighting conditions associated with the object at a point in time (e.g., headlights, brake lights, hazard lights, turn signals, backlights, etc.), the distance of the object to map elements at a point in time (stop lines, traffic lines, speed bumps, yield lines, intersections, distance to roadways, etc.), the distance of the object to other objects, the classification of the object (car, vehicle, animal, truck, etc.), features associated with the object (whether changing lanes, whether double-parked, etc.), etc.

[0118] In some examples, any combination of attributes of an object may be determined as discussed herein.

[0119] Attributes can be time (e.g., time T -M , ..., T -2 , T -1 , T0 (where M is an integer, and various times represent any time up to the most recent time) and can be input to a destination prediction component 932 and / or a location prediction component 934 to determine predictive information associated with such objects.

[0120] The destination prediction component 932 may include functionality to determine a destination of an object in the environment, as discussed herein. In a pedestrian context, the destination prediction component 932 may determine which crosswalk area applies to the pedestrian based on the pedestrian being within a threshold distance of the crosswalk area, as discussed herein. In at least some examples, such a destination prediction component 932 may determine a point on the opposite sidewalk regardless of the presence of a crosswalk. Additionally, attributes of objects associated with any time period may be input to the destination prediction component 932 to determine a score, probability, and / or margin that the pedestrian is heading toward or associated with a crosswalk area.

[0121] In some examples, the destination prediction component 932 is a machine learning model, such as a neural network, a fully connected neural network, a convolutional neural network, a recurrent neural network, or the like.

[0122] In some examples, the destination prediction component 932 can be trained by reviewing data logs to determine events in which a pedestrian crosses a crosswalk. Such events can be identified, attributes can be determined for the object (e.g., pedestrian) and the environment, and data representing the event can be identified as training data. The training data can be input to a machine learning model, and known results (e.g., ground truth, such as known "future" attributes) can be used to adjust the weights and / or parameters of the machine learning model to minimize error.

[0123] The location prediction component 934 may include functionality for generating or otherwise determining a predicted location associated with an object in the environment. For example, as discussed herein, attribute information may be determined for one or more objects in the environment, including the target object and / or other objects in proximity to the target object. In some examples, attributes associated with the vehicle 902 may be used to determine a predicted location associated with the object in the environment.

[0124] The location prediction component 934 can further include functionality for expressing attribute information in various frames of reference, as discussed herein. In some examples, the location prediction component 934 can use the object's position at time T0 as the origin of the frame of reference, which can be updated for each instance of time.

[0125] In some examples, the location prediction component 934 may include functionality to identify candidate reference lines within the environment (e.g., based on map data), select a reference line (e.g., based on a similarity score), and determine a predicted location for the reference line.

[0126] In some examples, the location prediction component 934 is a machine learning model such as a neural network, a fully connected neural network, a convolutional neural network, a recurrent neural network, or any combination thereof.

[0127] For example, the location prediction component 934 can be trained by reviewing data logs and determining attribute information. Training data representing relevant events (e.g., vehicles traveling a threshold distance from a reference line, pedestrians crossing a crosswalk, pedestrians running red lights, etc.) can be input into a machine learning model where known outcomes (e.g., ground truth, such as known "future" attributes / locations) can be used to adjust the weights and / or parameters of the machine learning model to minimize error.

[0128] In general, the planning component 936 can determine a path for the vehicle 902 to follow to traverse an environment. For example, the planning component 936 can determine various routes and trajectories, and various levels of detail. For example, the planning component 936 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a location of interest). For purposes of this discussion, the route can be a sequence of waypoints for traveling between the two locations. As non-limiting examples, the waypoints include streets, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the planning component 936 can generate instructions for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 936 can determine how to guide the autonomous vehicle from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instructions can be a trajectory, or a portion of a trajectory. In some examples, the multiple trajectories can be generated substantially simultaneously (e.g., within technical tolerances) in accordance with a receding horizon technique, and one of the multiple trajectories is selected for vehicle 902 to navigate.

[0129] In some examples, the planning component 936 can generate one or more trajectories for the vehicle 902 based at least in part on predicted positions associated with objects in the environment. In some examples, the planning component 936 can use temporal logic, such as linear temporal logic and / or signal temporal logic, to evaluate the one or more trajectories for the vehicle 902.

[0130] As can be appreciated, the components discussed herein (e.g., localization component 920, perception component 922, one or more maps 924, one or more system controllers 926, prediction component 928, attribute component 930, destination prediction component 932, location prediction component 934, and planning component 936) are described as separate for purposes of explanation. However, operations performed by the various components may be combined or performed by any other component. Additionally, any component discussed as being implemented in software may be implemented in hardware, and vice versa. Additionally, any functionality implemented in the vehicle 902 may be implemented in the computing device 940, or another component, and vice versa.

[0131] In at least one example, the sensor system 906 can include time of flight sensors, LIDAR sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (RGB, infrared, intensity, depth, etc.), microphones, wheel encoders, environmental sensors (temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), and the like. The sensor system 906 can include multiple instances of each of these or other types of sensors. For example, the time of flight sensors can include individual time of flight sensors located at corners, the front, the rear, the sides, and / or the top of the vehicle 902. As another example, the camera sensors can include multiple cameras located at various locations on the exterior and / or interior of the vehicle 902. The sensor system 906 can provide input to the vehicle computing device 904. Additionally or alternatively, the sensor system 906 may transmit sensor data via one or more networks 938 to one or more computing devices 940 at a particular frequency, after a predetermined period of time, or in near real-time.

[0132] The vehicle 902 may also include one or more emitters 908 for emitting light and / or sound, as described above. The emitters 908 in this example include interior audio and video emitters for communicating with passengers of the vehicle 902. By way of example and not limitation, the interior emitters may include speakers, lighting, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The emitters 908 in this example also include exterior emitters. Without limitation, the exterior emitters in this example include lighting (e.g., turn signals, signs, light arrays, etc.) for announcing a direction of travel or other indicators of the vehicle's operation, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which may include acoustic beam steering technology.

[0133] Vehicle 902 may also include one or more communication connections 910 that enable communication between vehicle 902 and one or more other local or remote computing devices. For example, communication connections 910 may facilitate communication with other local computing devices on vehicle 902 and / or drive system 914. Communication connections 910 may also enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, signals, etc.). Communication connections 910 may also enable vehicle 902 to communicate with remotely operated computing devices or other remote services.

[0134] The communications connection 910 may include physical and / or logical interfaces for connecting the vehicle computing device 904 to another computing device or network, such as network 938. For example, the communications connection 910 may enable Wi-Fi based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing devices.

[0135] In at least one example, the vehicle 902 can include one or more drive systems 914. In some examples, the vehicle 902 can have a single drive system 914. In at least one example, when the vehicle 902 has multiple drive systems 914, the individual drive systems 914 can be located at opposite ends of the vehicle 902 (e.g., the front and the rear, etc.). In at least one example, the drive system 914 can include one or more sensor systems for detecting conditions surrounding the drive system 914 and / or the vehicle 902. By way of example and not limitation, the sensor systems can include one or more wheel encoders (e.g., rotary encoders) that sense the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) that measure the orientation and acceleration of the drive module, cameras or other imaging sensors, ultrasonic sensors that acoustically detect objects surrounding the drive system, LIDAR sensors, RADAR sensors, etc. Some sensors, such as wheel encoders, can be unique to the drive system 914. In some cases, sensor systems on drive system 914 may overlap or complement corresponding systems on vehicle 902 (e.g., sensor system 906).

[0136] The drive system 914 can include many vehicle systems, such as a high voltage battery, a motor for propelling the vehicle, a steering system (which can be electric) including an inverter that converts direct current from the battery to alternating current for use in other vehicle systems, a steering motor, and a steering rack, a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the exterior surroundings of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, the drive system 914 can include a drive system controller that can receive and pre-process data from the sensor systems and control the operation of various vehicle systems. In some examples, the drive system controller can include one or more processors and a memory communicatively coupled to the one or more processors. The memory can store one or more components for performing various functions of the drive system 914. Additionally, drive system 914 also includes one or more communication connections that enable each drive system to communicate with one or more other local or remote computing devices.

[0137] In at least one example, the direct connection 912 can provide a physical interface for coupling one or more drive systems 914 with the body of the vehicle 902. For example, the direct connection 912 can provide for the transfer of energy, fluid, air, data, etc. between the drive systems 914 and the vehicle. In some examples, the direct connection 912 can further releasably secure the drive systems 914 to the body of the vehicle 902.

[0138] In at least one example, the localization component 920, the perception component 922, the one or more maps 924, the one or more system controllers 926, the prediction component 928, the attribute component 930, the destination prediction component 932, the location prediction component 934, and the planning component 936 can process the sensor data as described above and can transmit their respective outputs over one or more networks 938 to one or more computing devices 940. In at least one example, the localization component 920, the one or more maps 924, the one or more system controllers 926, the prediction component 928, the attribute component 930, the destination prediction component 932, the location prediction component 934, and the planning component 936 can transmit their respective outputs to the one or more computing devices 940 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0139] In some examples, vehicle 902 may transmit sensor data over network 938 to one or more computing devices 940. In some examples, vehicle 902 may transmit raw sensor data to computing device 940. In other examples, vehicle 902 may transmit processed sensor data and / or representations of the sensor data to computing device 940. In some examples, vehicle 902 may transmit sensor data at a particular frequency, after a predetermined period of time, in near real-time, etc. In some examples, vehicle 902 may transmit sensor data (raw or processed) to computing device 940 as one or more log files.

[0140] The computing device 940 may include a processor 942 and a memory 944 that stores a training component 946 .

[0141] In some examples, the training component 946 can include functionality to train one or more models to determine predictive information, as discussed herein. In some examples, the training component 946 can communicate information generated by the one or more models to the vehicle computing device 904 to modify how to control the vehicle 902 in response to different situations.

[0142] For example, the training component 946 can train one or more machine learning models to generate the predictive components discussed herein. In some examples, the training component 946 can include functionality to search the data log and determine attribute and / or location (e.g., in any one or more reference frames) information associated with the objects. Log data corresponding to particular scenarios (e.g., pedestrians approaching and crossing a crosswalk area, pedestrians running red lights, objects offset from the center line and rounding a curve, etc.) can represent training data. The training data can be input to the machine learning model, and known results (e.g., ground truth, such as known "future" attributes) can be used to adjust the weights and / or parameters of the machine learning model to minimize error.

[0143] For example, some or all aspects of the components discussed herein may include any model, algorithm, and / or machine learning algorithm. For example, in some examples, the components in memory 944 (and memory 918 described above) may be implemented as a neural network. In some examples, the training component 946 may utilize a neural network to generate and / or execute one or more models for determining segmentation information from sensor data, as discussed herein.

[0144] As described herein, an exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to generate an output. Each layer of a neural network may comprise another neural network, or may comprise any number of layers (convolutional or not). As can be understood in the context of the present disclosure, a neural network may utilize machine learning, which may refer to a broad class of algorithms in which an output is generated based on trained parameters.

[0145] Although discussed in the context of neural networks, any type of machine learning consistent with this disclosure may be used. For example, machine learning algorithms may include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally weighted scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage selection operator (LASSO), elastic nets, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative binary tree 3 (ID3), chi-squared automated interaction detection (CH2), etc.). AID), decision stumps, conditional decision trees), Bayesian algorithms (e.g., Naïve Bayes, Gaussian Naïve Bayes, Multinomial Naïve Bayes, Average Ordinary Dependent Classifier (AODE), Bayesian Belief Networks (BNN), Bayesian Networks), clustering algorithms (e.g., k-means, k-medians, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning Algorithms (e.g., Perceptron, Backpropagation, Hopfield Networks, Radial Basis Function Network (RBFN)), deep learning algorithms (Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN), Convolutional Neural Networks (CNN), Stacked Auto-Encoders), dimensionality reduction algorithms (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), ensemble algorithms (e.g., Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, Stacked Generalization (Blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees (GBRT), RandomThese may include, but are not limited to, Support Vector Machine (SVM), supervised learning, unsupervised learning, semi-supervised learning, etc.

[0146] Additional example architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, and PointNet.

[0147] The processor 916 of the vehicle 902 and the processor 942 of the computing device 940 may be any suitable processor capable of executing instructions to perform operations and process data as described herein. By way of example and not limitation, the processors 916 and 942 may comprise one or more central processing units (CPUs), graphics processing units (GPUs), or other devices or portions of devices that process electronic data and convert it to other electronic data that can be stored in registers and / or memory. In some examples, integrated circuits (such as ASICs), gate arrays (such as FPGAs), and other hardware devices may also be considered processors so long as they are configured to implement coded instructions.

[0148] The memories 918 and 944 are exemplary non-transitory computer-readable media. The memories 918 and 944 may store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functions attributed to the various systems described herein. In various implementations, the memories may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or other types of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, but those shown in the accompanying drawings are merely examples for the discussion herein.

[0149] 9 is illustrated as a distributed system, it should be noted that in alternative examples, components of the vehicle 902 can be associated with the computing device 940 and / or components of the computing device 940 can be associated with the vehicle 902. That is, the vehicle 902 can perform one or more of the functions associated with the computing device 940, and vice versa. Additionally, aspects of the prediction component 928 (and subcomponents) can be performed on any of the devices discussed herein.

[0150] 10 illustrates an example process 1000 of capturing sensor data, determining attributes associated with an object, determining a predicted location based on the attributes, and controlling a vehicle based on the predicted location. For example, some or all of process 1000 may be performed by one or more components of FIG. 9 as described herein. For example, some or all of process 1000 may be performed by vehicle computing device 904. Additionally, any operations set forth in example process 1000 may be performed in parallel, in an order different from that shown in process 1000, with any of the operations of process 1000 shown omitted, and / or combined with any of the operations discussed herein.

[0151] In operation 1002, the process may include receiving sensor data of the environment. In some examples, operation 1002 may include receiving and / or capturing time of flight data, LIDAR data, image data, RADAR data, etc. of the environment. In some examples, operation 1002 may be performed by a vehicle (e.g., an autonomous vehicle) as the vehicle traverses the environment.

[0152] At operation 1004, the process may include determining that an object is in the environment based at least in part on the sensor data. For example, operation 1004 may include classifying the object as a pedestrian in the environment. In some examples, operation 1004 may include determining whether the object (e.g., a pedestrian) is on a sidewalk, on a road, running a red light, etc.

[0153] In operation 1006, the process may include determining whether the object is associated with a destination in the environment. For example, operation 1006 may include accessing map data of the environment and determining whether a crosswalk area is within a threshold distance of the object. If there is one crosswalk area and the object is on a sidewalk, operation 1006 may include identifying a location across the drivable area as the destination. If the object is on a road and is proximate to one crosswalk, operation 1006 may include disambiguating between the two destinations. In some examples, operation 1006 may include determining a likelihood that the object will approach and / or cross a particular crosswalk area based at least in part on attributes associated with the object. In some examples, operation 1006 provides such destinations to the pedestrian regardless of the presence of a crosswalk area proximate to the pedestrian.

[0154] In some examples, operation 1006 may include inputting attributes to a destination prediction component (e.g., destination prediction component 320) to determine a destination associated with the object in the environment. In some examples, the attributes input to the destination prediction component 320 may be the same as or similar to the attributes determined below in operations 1008 and 1010. In some examples, attributes may be determined for the object prior to determining the destination in the environment. Also, in some examples, attributes may be determined in parallel using a reference frame based on different destinations in the environment to determine likely destinations in the environment.

[0155] If the object is not associated with the destination (eg, "NO" at operation 1006), operation 1006 may continue with operation 1002 to capture additional data within the environment.

[0156] If the object is then associated with the destination (eg, “YES” at operation 1006 ), the operation may continue to operation 1008 .

[0157] At operation 1008, the process may include determining a first attribute associated with the object, the first attribute associated with a first time. In some examples, the attribute may include a position of the object (e.g., a pedestrian) at a time (e.g., the position may be expressed in a frame of reference as discussed herein), a size (e.g., length, width, and / or height) of the object or a bounding box associated with the object, a speed (e.g., a magnitude and / or angle relative to a first axis (or other reference line)) of the pedestrian at a time, an acceleration of the pedestrian at a time, an indication of whether the pedestrian is in a drivable area (e.g., whether the pedestrian is on a sidewalk or on a road), an indication of whether the pedestrian is in a crosswalk area, an indication of whether the pedestrian is running a red light, a state of an indicator controlling the area (e.g., whether the crosswalk is controlled by a light, The attributes may include, but are not limited to, one or more of: vehicle context (such as the presence of vehicles in the environment and attributes associated with the vehicles); flux through the crosswalk area in a period of time (such as the number of objects (such as vehicles and / or additional pedestrians) passing through the crosswalk area in a period of time); object association (e.g., whether a pedestrian is moving among a group of pedestrians); distance to the crosswalk in a first direction (e.g., global x-direction), distance to the crosswalk in a second direction (e.g., global y-direction), distance to a road within the crosswalk area (e.g., shortest distance to a road within the crosswalk area), distance to other objects, etc.

[0158] At operation 1010, the process may include determining a second attribute associated with the object, the second attribute associated with a second time after the first time. In some examples, operation 1010 may be omitted (such that only attributes associated with the first time may be determined and / or used), while in some examples attributes associated with additional or different time instances may be determined.

[0159] At operation 1012, the process may include determining a predicted location of the object at a third time after the second time based at least in part on the first attribute, the second attribute, and the destination. In some examples, operation 1012 may include inputting the attribute information into a location prediction component (e.g., location prediction component 404) and receiving as an output a predicted location associated with the object in the environment. As described herein, in some examples, the attribute and / or the predicted location may be expressed in one or more frames of reference based at least in part on the location of the object at the first time and / or the second time and the location of the destination in the environment.

[0160] In operation 1014, the process may include controlling the vehicle based at least in part on the predicted position. In some examples, operation 1014 may include generating a trajectory to stop or otherwise control the vehicle to safely traverse an environment.

[0161] 11 illustrates an example process of capturing sensor data, determining that a first object and a second object are in an environment, determining attributes associated with the second object, determining a predicted position based on the attributes and the reference line, and controlling a vehicle based on the predicted position. For example, some or all of the process 1100 may be performed by one or more components of FIG. 9 as described herein. For example, some or all of the process 1100 may be performed by the vehicle computing device 904. Additionally, any operations set forth in the illustrated process 1100 may be performed in parallel, in a different order than shown in process 1100, with any of the operations of the illustrated process 1100 omitted, and / or combined with any of the operations discussed herein.

[0162] At operation 1102, the process may include receiving sensor data of the environment. In some examples, operation 1102 may include receiving and / or capturing time of flight data, LIDAR data, image data, RADAR data, etc. of the environment. In some examples, operation 1102 may be performed by a vehicle (e.g., an autonomous vehicle) as the vehicle traverses the environment.

[0163] At operation 1104, the process may include determining that a first object is in the environment based at least in part on the sensor data. For example, operation 1104 may include determining a target object that is subject to the predictive action as discussed herein. By way of example, determining the target object may include selecting an object as the target object from a plurality of objects in the environment. In some examples, the target object may be selected based on a likelihood of an intersection between the target object's path and a vehicle (e.g., vehicle 902) capturing the sensor data, a distance between the target object and the vehicle (e.g., vehicle 902) capturing the sensor data, etc.

[0164] At operation 1106, the process may include determining whether the second object is proximate to the first object in the environment. In some examples, operation 1106 may include determining whether the second object is within a threshold distance of the first object. In some examples (e.g., in a crowded environment), operation 1106 may include determining N objects that are closest to the first object, where N is an integer. Without limitation, in at least some examples, such determination excludes objects with certain characteristics, such as objects of different classifications, objects with opposite directions of motion, etc.

[0165] If the second object is not proximate to the first object (e.g., “NO” at operation 1106), the process may return to operation 1102. However, in some examples, the process may continue to operation 1112 where a predicted location of the first object is determined without attributes associated with the second object (e.g., the predicted location of the first object may be determined based, at least in part, on attributes associated with the first object). That is, the predicted location of the first object may be determined in some examples regardless of whether the second object is proximate to the first object and / or regardless of whether attributes have been determined for any second object.

[0166] If the second object is proximate to the first object (eg, “YES” at operation 1106 ), processing continues at operation 1108 .

[0167] At operation 1108, the process may include determining a first attribute associated with the second object, the second attribute associated with the first time. In some examples, attributes may be determined for the first object, the second object, and / or other objects in the environment. For example, the attributes may include, but are not limited to, one or more of: the speed of the object at a time; the acceleration of the object at a time; the position of the object at a time (e.g., in global or local coordinates); a bounding box associated with the object at a time (e.g., representing the range, roll, pitch, and / or yaw of the object); the state of lighting associated with the object at a first time (e.g., headlights, brake lights, hazard lights, turn lights, backlights, etc.); a wheel direction indication for the object; the distance of the object to a map element at a time (e.g., distance to a stop line, turn line, speed bump, yield line, intersection, roadway, etc.); the distance relative to other objects in one or more frames of reference; a classification of the object (car, vehicle, animal, truck, bicycle, etc.); features associated with the object (whether the object is changing lanes, whether it is a double-parked vehicle, etc.); lane features, etc.

[0168] At operation 1110, the process may include determining a second attribute associated with the second object, the second attribute associated with a second time after the first time. In some examples, operation 1110 may be omitted (such that only the attributes associated with the first time may be used), while in some examples, attributes associated with additional or different time instances may be determined.

[0169] At operation 1112, the process may include determining a predicted location of the first object at a third time after the second time based at least in part on the first attribute and the second attribute, where the predicted location is relative to a reference line in the environment. In some examples, operation 1112 may include inputting attribute information associated with the first object and / or the second object into a location prediction component (e.g., location prediction component 802) to determine a predicted location associated with the first object.

[0170] In some examples, operation 1112 may include receiving or otherwise determining a reference line most closely associated with the predicted location and representing the predicted location relative to the reference line. For example, operation 1112 may include determining a similarity score between the predicted location and a candidate reference line, selecting a reference line based on the similarity score, or any other mechanism.

[0171] In operation 1114, the process may include controlling the vehicle based at least in part on the predicted position. In some examples, operation 1114 may include generating a trajectory to stop the vehicle or otherwise control the vehicle to safely traverse the environment.

[0172] (Example section) A: A system comprising: one or more processors; and one or more computer-readable media storing instructions executable by the one or more processors that, when executed, cause the system to perform operations comprising: capturing sensor data of an environment using sensors of an autonomous vehicle; determining that an object is within the environment based at least in part on the sensor data; determining that the object is associated with a destination within the environment based at least in part on the map data and the sensor data; determining a first attribute associated with the object, the first attribute associated with a first time; determining a second attribute associated with the object, the second attribute associated with a second time after the first time; inputting the first attribute, the second attribute, and the destination into a machine learning model, the first attribute and the second attribute represented by a frame of reference based at least in part on the destination; receiving from the machine learning model a predicted position of the object at a third time after the second time; and controlling the autonomous vehicle based at least in part on the predicted position of the object in the environment at the third time.

[0173] B: The system of paragraph A, wherein the object is a pedestrian and the destination is associated with a perimeter of a crosswalk area in the environment and faces a drivable surface associated with the pedestrian.

[0174] C: The system of paragraph A or B, further comprising: determining that the object is associated with a destination based at least in part on inputting the first attribute and the second attribute into a destination prediction component; and receiving the destination from the destination prediction component, the destination prediction component comprising another machine learning model.

[0175] D: The system of any of paragraphs A to C, wherein the operation further comprises: a predicted position associated with the object at a third time comprising a lateral offset based at least in part on the frame of reference and a distance along an axis of the frame of reference representing a difference between the position of the object at the second time and the predicted position.

[0176] E: Any of the systems of paragraphs A to D, further comprising: an operation of establishing a frame of reference, wherein the first position of the object at the second time is associated with coordinates of the frame of reference, the first axis being based at least in part on the destination and the surface of material, the second axis being perpendicular to the first axis, and the predicted position being based at least in part on the frame of reference.

[0177] F: A method comprising: receiving sensor data representing an environment; determining, based at least in part on the sensor data, that an object is within the environment; determining a location within the environment, the location associated with a pedestrian crossing area; determining a first attribute associated with the object, the first attribute associated with a first time; determining a second attribute associated with the object, the second attribute associated with a second time after the first time; inputting the first attribute, the second attribute, and the location into a machine learning model; and receiving from the machine learning model a predicted location associated with the object at a third time after the second time.

[0178] G: The method of paragraph F, further comprising capturing sensor data using a sensor on the vehicle and controlling the vehicle based at least in part on a predicted position of the object in the environment at a third time.

[0179] H: The method of paragraph F or G, wherein the location is a first location, further comprising: determining the first location based at least in part on at least one of map data representing the environment or sensor data; determining a threshold area associated with the first location; determining a second location of the object within the environment; determining that the second location of the object is within the threshold area; and selecting the location as a destination associated with the object based at least in part on the second location being within the threshold area and at least in part on the first attribute or at least one of the second attributes.

[0180] I: Any of the methods of paragraphs F to H, wherein the position is a first position, and further comprising establishing a frame of reference, wherein a second position of the object at a second time is associated with coordinates of the frame of reference, a first axis being based at least in part on the coordinates and the first position, a second axis being perpendicular to the first axis, and the first attribute being based at least in part on the frame of reference.

[0181] J: The method of paragraph I, further comprising determining a velocity of the object at a second time and determining an angle between a velocity vector representing the velocity and the first axis, wherein the second attribute comprises the angle.

[0182] K: The method of paragraph I or J, wherein the position is a first position and a predicted position associated with the object at a third time comprises a lateral offset relative to the second axis and a distance along the first axis representing a difference between the second position and the predicted position of the object at the second time.

[0183] L: The method of any of paragraphs F-K, further comprising determining a number of objects entering the crosswalk area over a period of time, wherein the second attribute comprises the number of objects.

[0184] M: Any of the method of paragraphs F to L, wherein the object is a first object, and further comprising determining that a second object is within the environment based at least in part on the sensor data; determining at least one of a position, velocity, or acceleration associated with the second object as object context; and determining a predicted position associated with the object further based at least in part on the object context.

[0185] N: The method of any of paragraphs F-M, further comprising binning at least a portion of the predicted locations to determine binned predicted locations.

[0186] O: The method of any of paragraphs F-N, wherein the first attribute comprises at least one of: a position of the object at the first time; a velocity of the object at the first time; a path of the object at the first time; a first distance between the object and a first portion of the crosswalk area at the first time; a second distance between the object and a second portion of the crosswalk area at the first time; an acceleration of the object at the first time; an indication of whether the object is in a drivable area; a state of a control area indicator; a vehicle context; or an association of the object.

[0187] P: A non-transitory computer-readable medium having instructions stored thereon that, when executed, cause one or more processors to perform operations including: receiving sensor data representative of an environment; determining, based at least in part on the sensor data, that an object is within the environment; determining a location within the environment, the location associated with at least one of a crosswalk area or a non-driveable area of ​​the environment; determining a first attribute associated with the object, the first attribute associated with a first time; determining a second attribute associated with the object, the second attribute associated with a second time after the first time; inputting the first attribute, the second attribute, and the location into a machine learning model; and receiving from the machine learning model a predicted location associated with the object at a third time after the second time.

[0188] Q: The non-transitory computer-readable medium of paragraph P, wherein the location is a first location, and the operations further include determining the first location based at least in part on at least one of map data representing the environment or sensor data representing the environment, determining a threshold area associated with the first location, determining a second location of an object within the environment, determining that the second location of the object is within the threshold area, and selecting the first location as a destination associated with the object based at least in part on the second location of the object being within the threshold area and at least one of the first attribute or the second attribute.

[0189] R: The non-transitory computer readable medium of paragraph P or Q, wherein the position is a first position, and the operation is establishing a frame of reference, wherein a second position of the object at a second time is associated with coordinates of the frame of reference, a first axis is based at least in part on the coordinates and the first position, a second axis is perpendicular to the first axis, and the first attribute is based at least in part on the frame of reference.

[0190] S: The non-transitory computer-readable medium of paragraph R, wherein the location is a first location and a predicted location associated with the object at a third time comprises a lateral offset along a second axis and a distance along the first axis representing a difference between the second location and the predicted location of the object at the second time.

[0191] T: The non-transitory computer-readable medium of any of paragraphs P to S, further comprising determining that the object is not associated with a crosswalk area and determining that the location is associated with a non-drivable area of ​​the environment.

[0192] U: A system comprising: one or more processors; and one or more computer-readable media having instructions stored thereon executable by the one or more processors, the instructions, when executed, causing the system to perform operations comprising: capturing sensor data of an environment using sensors on an autonomous vehicle; determining that an object is within the environment based at least in part on the sensor data; receiving a reference line associated with the object in the environment; determining a first attribute associated with the object, the first attribute being associated with a first time; determining a second attribute associated with the object, the second attribute being associated with a second time after the first time; inputting the first attribute, the second attribute, and the reference line into a machine learning model; receiving from the machine learning model a predicted position of the object at a third time after the second time, the predicted position relative to the reference line in the environment; and controlling the autonomous vehicle based at least in part on the predicted position of the object in the environment at the third time.

[0193] V: The system of paragraph U, wherein the object is a first object, and the action is determining a third attribute associated with a second object proximate to the first object, the third attribute being associated with a first time, and a fourth attribute associated with the second object, the fourth attribute being associated with a second time, and inputting the third attribute and the fourth attribute into a machine learning model to determine a predicted location of the first object at the third time.

[0194] W: The system of paragraph V, wherein at least one of the first attribute, the second attribute, the third attribute, or the fourth attribute is: the velocity of the second object at the first time, the acceleration of the second object at the first time, the position of a second object at a first time; a bounding box associated with a second object at a first time; a lighting condition associated with a second object at a first time; a first distance between the second object and the map element at a first time; a second distance between the first object and the second object; Second object classification, or a feature associated with a second object; A system comprising at least one of the following:

[0195] X: Any of the systems of paragraphs U to W, where the predicted position comprises a distance along a reference line and a lateral offset from the reference line.

[0196] Y: Any of the systems of paragraphs U to X, wherein the machine learning model is a first machine learning model and the reference line is received from a second machine learning model trained to output the reference line.

[0197] Z: A method comprising: receiving sensor data representing an environment; determining that an object is within the environment; receiving a reference line associated with the object; determining a first attribute associated with the object, the first attribute being associated with a first time; determining a second attribute associated with the object, the second attribute being associated with a second time after the first time; inputting the first attribute, the second attribute, and the reference line into a machine learning model; and receiving from the machine learning model a predicted position of the object at a third time after the second time, the predicted position relative to the reference line in the environment.

[0198] AA: The method of paragraph Z, further comprising capturing sensor data using a sensor of the vehicle and controlling the vehicle based at least in part on a predicted position of an object in the environment at a third time.

[0199] AB: The method of paragraph AA, wherein the object is one of a number of objects in the environment, and further comprising selecting the object as the target object based at least in part on a distance between the object and the vehicle in the environment.

[0200] AC: The method of any of paragraphs Z to AB, wherein the object is one of a number of objects in an environment, the object being a target object, and further comprising selecting a number of the number of objects based at least in part on a number of objects proximate to the target object; determining attributes associated with the object; and inputting the attributes into a machine learning model to determine a predicted location.

[0201] AD: The method of paragraph AC, further comprising selecting an object based at least in part on a classification associated with the object.

[0202] AE: The method of any of paragraphs Z to AD, wherein the reference line corresponds to a centerline of the drivable area and the predicted location comprises a distance along the reference line and a lateral offset from the reference line.

[0203] AF: Any of the methods of paragraphs Z to AE, wherein the first attribute and the second attribute are expressed with respect to a frame of reference, and the coordinates of the frame of reference are based at least in part on the position of the object at the second time.

[0204] AG: The method of any of paragraphs Z to AF, wherein the first attribute comprises at least one of: a velocity of the object at the first time; an acceleration of the object at the first time; a position of the object at the first time; a bounding box associated with the object at the first time; a lighting condition associated with the object at the first time; a distance between the object and the map element at the first time; a classification of the object; and a feature associated with the object.

[0205] AH: The method of paragraph AG, wherein the object is a first object, the distance is a first distance, and determining that a second object is in proximity to the first object in the environment, the first attribute further comprising a second distance between the first object and the second object at a first time.

[0206] AI: A non-transitory computer readable medium having instructions stored thereon that, when executed, cause one or more processors to perform operations comprising: receiving sensor data representing an environment; determining that an object is within the environment based at least in part on the sensor data; receiving a reference line associated with the object; determining a first attribute associated with the object, the first attribute being associated with a first time; a second attribute associated with the object, the second attribute being associated with a second time after the first time; inputting the first attribute, the second attribute, and the reference line into a machine learning model; and receiving from the machine learning model a predicted position of the object at a third time after the second time, the predicted position relative to the reference line in the environment.

[0207] AJ: A non-transitory computer-readable medium of paragraph AI, wherein the object is a first object, and the operation further comprises determining that a second object is in proximity to the first object in the environment; determining a third attribute associated with the second object, the third attribute being associated with a first time; and determining a fourth attribute associated with the second object, the fourth attribute being associated with a second time; and inputting the third attribute and the fourth attribute into a machine learning model to determine a predicted location associated with the first object.

[0208] AK: The non-transitory computer-readable medium of paragraph AI or AJ, wherein the first attribute and the second attribute are expressed with respect to a frame of reference, and coordinates of the frame of reference are based at least in part on a position of the object at a second time.

[0209] AL: The non-transitory computer-readable medium of paragraph AK, wherein the predicted location is represented as a distance along a reference line and a lateral offset from the reference line.

[0210] AM: The non-transitory computer readable medium of any of paragraphs AI to AL, wherein the first attribute comprises at least one of a velocity of the object at the first time, an acceleration of the object at the first time, a position of the object at the first time, a bounding box associated with the object at the first time, a lighting condition associated with the object at the first time, a distance between the object and a map element at the first time, a classification of the object, or a feature associated with the object.

[0211] AN: The non-transitory computer readable medium of paragraph AM, wherein the object is a first object, the distance is a first distance, and the first attribute further comprises a second distance between the first object and a second object at a first time.

[0212] Although the example sections above are described with respect to certain implementations, it should be understood that in the context of this document, the contents of the example sections may also be implemented via methods, apparatus, systems, computer-readable media, and / or other implementations.

[0213] (Conclusion) One or more examples of the technology described herein have been described; however, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0214] In the description of the examples, reference is made to the accompanying drawings, which form a part of this specification, illustrating specific examples of the claimed subject matter. It is understood that other examples can be used and that modifications or substitutions, such as structural changes, can be made. Such examples, modifications or substitutions do not necessarily depart from the intended scope of the claimed subject matter. Although the procedures herein may be presented in a certain order, in some cases the order can be changed such that certain inputs are provided at different times or in a different order without changing the functionality of the described systems and methods. It is also possible to perform the disclosed procedures in a different order. Furthermore, the various calculations described herein do not have to be performed in the order disclosed, and other examples using alternative orders of calculations can be easily implemented. In addition to changing the order, the calculations can also be decomposed into sub-calculations with the same results.

Claims

1. 1. A method comprising: receiving sensor data representative of an environment; determining, based at least in part on the sensor data, that an object is within the environment; and determining at least one of a pedestrian crossing area or a non-driveable area within the environment; determining a first attribute associated with the object, the first attribute being associated with a first time; determining a second attribute associated with the object, the second attribute being associated with a second time after the first time; and inputting the first attribute, the second attribute, and the crosswalk area or the non-driveable area into a machine learning model, the first attribute and the second attribute being represented by a frame of reference defined at least in part by a predicted destination associated with the object; receiving from the machine learning model a predicted position associated with the object at a third time after the second time; and A method for providing the above.

2. capturing said sensor data using sensors on a vehicle; controlling the vehicle based at least in part on the predicted position associated with the object at the third time; and The method of claim 1 further comprising:

3. determining the crosswalk area or the non-traveling area based at least in part on at least one of map data representative of the environment or the sensor data; determining a threshold area associated with the pedestrian crossing area or the non-travelable area; determining a location of the object within the threshold area; predicting a destination associated with the object based at least in part on the location of the object within the threshold region and at least one of the first attribute or the second attribute; The method of claim 1 or 2, further comprising:

4. Establishing said frame of reference, comprising: the position of the object at the second time is associated with coordinates in the frame of reference; a first axis is based at least in part on the coordinate and the pedestrian crossing area or the non-travelable area; the second axis being perpendicular to the first axis; The method of claim 1 , wherein the first attribute is based at least in part on the frame of reference.

5. determining a velocity of the object at the second time; and determining an angle between a velocity vector representing the velocity and the first axis; Further equipped with The method of claim 4 , wherein the second attribute comprises the angle.

6. The predicted location associated with the object at the third time is: a lateral offset relative to the second axis; and a distance along the first axis representing the difference between the position of the object at the second time and the predicted position; The method of claim 4 comprising:

7. A method according to any one of claims 1 to 6, wherein the first attribute and the second attribute represent the number of objects that have passed through the crosswalk area or the impassable area within a given period of time. determining, based at least in part on the sensor data, that other objects are within the environment; and determining an object context indicative of whether the object is associated with the other object, the object context including at least one of a position, a velocity, or an acceleration associated with the other object; determining the predicted position associated with the object at the third time based at least in part further on the object context; The method of claim 1 , further comprising:

9. The method of claim 1 , further comprising binning at least a portion of the predicted locations to determine binned predicted locations.

10. The first attribute is a position of the object at the first time; the velocity of the object at the first time; a path of the object at the first time; a first distance between the object and a first portion of the pedestrian crossing area at the first time; a second distance between the object and a second portion of the pedestrian crossing area at the first time; the acceleration of the object at the first time; an indication of whether the object is in a drivable area; Status of indicators controlling the area Or object association, The method according to claim 1 , further comprising at least one of:

11. A computer program product comprising coded instructions which, when executed on a computer, implements the method of any one of claims 1 to 10.

12. 1. A system comprising: one or more processors; One or more non-transitory computer-readable media having instructions stored thereon that, when executed, cause the system to perform operations, including: receiving sensor data representative of an environment; determining, based at least in part on the sensor data, that an object is within the environment; and determining at least one of a pedestrian crossing area or a non-driveable area in the environment; determining a first attribute associated with the object, the first attribute being associated with a first time; determining a second attribute associated with the object, the second attribute being associated with a second time after the first time; and inputting the first attribute, the second attribute, and the crosswalk area or the non-driveable area into a machine learning model, the first attribute and the second attribute being represented by a frame of reference defined at least in part by a predicted destination associated with the object; receiving from the machine learning model a predicted position associated with the object at a third time after the second time; and A system equipped with

13. The operation determining the pedestrian crossing area or the non-traveling area based at least in part on at least one of map data representative of the environment or the sensor data representative of the environment; determining a threshold area associated with the pedestrian crossing area or the non-travelable area; determining a position of the object within the environment; determining that the position of the object is within the threshold region; determining the pedestrian crossing area or the non-travelable area as a destination associated with the object based at least in part on the location within the threshold area and at least one of the first attribute or the second attribute; The system of claim 12 further comprising:

14. The operation Establishing said frame of reference, the position of the object at the second time is associated with coordinates in the frame of reference; a first axis is based at least in part on the coordinate and the pedestrian crossing area or the non-travelable area; the second axis being perpendicular to the first axis; 14. The system of claim 12 or 13, wherein the first attribute is based at least in part on the frame of reference.

Citation Information

Patent Citations

  • Travel supporting device

    JP2018012360A

  • Mobile body, location estimation device, and computer program

    WO2019044499A1