Planning room for reversing vehicles

Abstract

Techniques are described herein for determining that a first vehicle is associated with a reverse condition and controlling a second vehicle based on the reverse condition. In some examples, the first vehicle may provide an indication that the first vehicle will be performing a reverse maneuver, such as by having reverse lights on the vehicle or being positioned at an angle relative to the road or parking space to enable reversing to a desired location. A planning system of the second vehicle (e.g., an autonomous vehicle) may receive sensor data and determine various of these indications to determine a probability that the vehicle is performing a reverse maneuver. The second vehicle may further determine an approximate trajectory of the reversing maneuver and provide appropriate room (e.g., time and / or space) to enable the second vehicle to perform the maneuver safely and efficiently.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Patent Application No. 16 / 456,987, filed June 28, 2019, and entitled "PLANNING ACCOMMODATIONS FOR REVERSING VEHICLES," the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Various methods, devices, and systems are utilized to guide autonomous vehicles through environments that include various static and dynamic objects. For example, autonomous vehicles utilize route planning methods, devices, and systems to guide autonomous vehicles through areas crowded with other moving and stationary (autonomous or not) vehicles, people, buildings, and other objects. In some examples, autonomous vehicles may make decisions to ensure safety for occupants and surrounding people and objects while traversing an environment. However, predicting the behavior of other vehicles or objects in an environment can present challenges. Summary of the Invention

[0003] The detailed description will now be described with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference number in different drawings indicates similar or identical components or features according to examples of the present disclosure. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 is an illustrated flow diagram for controlling an autonomous vehicle based on a predicted trajectory of the vehicle determined to be associated with a reverse state, according to an example of the disclosure. [Diagram 2]FIG. 1 is an illustration of an exemplary system that may be utilized to determine one or more trajectories based in part on an indication that a vehicle is associated with a reverse state, in accordance with an example of the present disclosure. [Diagram 3] FIG. 13 is an illustration of utilizing environmental features to determine a reverse attempt condition and subsequent trajectory estimation for a vehicle in accordance with an example of the present disclosure. [Figure 4] 1 is a block diagram of an example system for implementing the techniques described herein. [Diagram 5] FIG. 13 depicts an example process for determining that a vehicle is associated with a reverse state, predicting a trajectory for the vehicle based on the vehicle being associated with the reverse state, and controlling an autonomous vehicle based on the predicted trajectory, in accordance with an example of the disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] The present disclosure relates to techniques for determining that a vehicle is likely to perform maneuvers in reverse and accommodating these maneuvers by controlling an autonomous vehicle in a safe and efficient manner. In some examples, the vehicle may provide an indication that the vehicle will be performing a reverse maneuver, such as by using reverse lights on the vehicle and / or by positioning at an angle relative to a road or parking space to consider reversing to a desired location. The autonomous vehicle's planning system may utilize sensor data to determine a variety of these indications and then determine the probability that the vehicle is about to perform a reverse maneuver, determine a likely trajectory for the reverse maneuver, and provide the vehicle with the appropriate room (e.g., time and / or space) to safely and efficiently perform the maneuver.

[0006] The sensor data captured by an autonomous vehicle may include data captured by image sensors, such as individual images, videos, etc., as well as data captured by other sensors, such as lidar sensors, radar sensors, time-of-flight sensors, sonar sensors, etc. In some cases, the sensor data may be provided to a perception system configured to determine or classify types of objects in the environment (e.g., vehicles, pedestrians, bicycles, motorcycles, animals, parked cars, trees, buildings, etc.).

[0007] Sensor data may be captured by an autonomous vehicle as it traverses an environment. In some examples, an autonomous vehicle may include, among other sensors, one or more cameras configured to capture images (e.g., individual images and / or video) of the surrounding environment. Images captured by the one or more cameras may be analyzed to detect objects, either independently of or in conjunction with detection of objects by other sensors of the vehicle. Once detected, in some examples, the images may be cropped to an area of ​​interest surrounding the object, for example, based on a size (in centimeters, pixels, etc.) surrounding the individual object, based on a bounding box surrounding the object, etc.

[0008] In an example, an autonomous vehicle trajectory determination component may identify an object, such as a vehicle, in an image along with detecting features of the vehicle. For example, the trajectory determination component may segment an image to detect vehicle lights that may indicate the vehicle is in a reverse state. In some cases, lights indicating the vehicle is associated with a reverse state may be obscured in the image. In such cases, the trajectory determination component may identify another light (e.g., a different reverse indicator light, brake light, etc.) to determine a state (e.g., forward state, reverse state, etc.) associated with the vehicle and / or may utilize additional context related to the vehicle to determine whether the vehicle is associated with a reverse state.

[0009] In some examples, the trajectory determination component may then utilize the image representing the feature to determine whether the light indicates that the vehicle is associated with a reverse condition, such as by inputting a portion of the image representing the feature into a machine learning model trained to predict vehicle behavior based at least in part on characteristics of the feature. In some examples, a vehicle associated with a reverse condition may also be referred to as a vehicle in reverse gear, although it may be understood that a vehicle does not necessarily need to have a physical reverse gear to be "in reverse gear." For example, the vehicle may include an electric motor and may provide thrust in the reverse direction by changing the polarity of a voltage to reverse the direction of the motor.

[0010] In particular, the machine learning model may determine that the vehicle's reverse lights are on and may predict the vehicle's behavior based on the reverse lights being on. Additionally, the machine learning model may receive various features of the environment that may provide additional information about whether the vehicle is in a reverse state, a non-reverse state, or a "reverse intent state" in which the vehicle is not yet in reverse but is likely about to perform a reverse maneuver (e.g., within 3 seconds, within 5 seconds, within 1 block, within 3 blocks, within 25 feet (7.62 meters), within 50 feet (15.24 meters), etc.). In some examples, features of the environment that may provide clues as to whether the vehicle is in (or intends to enter) a reverse condition may include the orientation of the vehicle (e.g., relative to a lane of traffic), the speed of the vehicle relative to a speed limit at the vehicle's location in the environment, the speed of the vehicle relative to other vehicles in the surrounding environment, the particular lane the vehicle is currently in, lanes adjacent to the lane the vehicle is currently in (e.g., adjacent parking lanes, driveways, etc.), whether the vehicle is located at or near a junction, the distance between the vehicle and other vehicles in the environment, the speeds of other vehicles near the vehicle, etc. Features of the environment may be collected from different sensor modalities to provide the trajectory determination component with information about the environment to utilize in predicting the behavior of other vehicles.

[0011] The machine learning model may output an indication that the vehicle is in reverse in response to the vehicle's reverse lights being on along with an analysis of characteristics of the environment surrounding the vehicle. In some examples, the machine learning model may be a classifier that outputs an indication corresponding to the vehicle being in reverse, the vehicle being in a non-reverse state, or the vehicle being in a reverse attempt state. Alternatively, the machine learning model may output a binary indication of whether the vehicle is in reverse or a non-reverse state. Other examples are also contemplated, such as outputting a predicted trajectory for the vehicle based on the determination of the state of the vehicle.

[0012] For example, the trajectory determination component may predict a trajectory of the vehicle based at least in part on an indication of a reverse state of the vehicle and / or characteristics of the environment, as described above. In some examples, the characteristics of the environment may be utilized to determine a path or trajectory that the vehicle may follow once the vehicle begins to perform a reverse gear maneuver. For example, the trajectory determination component may determine a number of possible trajectories for the vehicle based on the characteristics of the environment surrounding the vehicle and / or the reverse indication. In some cases, the trajectory determination component may select a most likely trajectory (or several most likely trajectories) from the multiple trajectories and plan a trajectory for the autonomous vehicle to follow based on the predicted trajectory that is most likely to accommodate the vehicle to the predicted reverse maneuver.

[0013] For example, the trajectory determination component may predict a trajectory by utilizing the machine learning model described above, or another machine learning model, to generate a top-down image that includes a channel containing information associated with features of the environment, such as traffic lanes, map data, objects in the environment, and the like. The machine learning model utilized to generate the predicted trajectory may output coordinates (e.g., x and y coordinates) associated with the vehicle at one or more times in the future (e.g., 1 second, 2 seconds, 3 seconds, and the like). In some examples, the machine learning model may output the coordinates associated with the vehicle, as well as probability information associated with each coordinate. Additionally, the machine learning model may output a heat map associated with the prediction probability. In some examples, at least one predicted trajectory may be determined based at least in part on the heat map. Additional details regarding predicting trajectories utilizing a top-down approach may be found in U.S. Patent Application Serial No. 16 / 151,607, which is incorporated by reference herein in its entirety. Other approaches to predicting a trajectory for a vehicle determined to be in reverse (or attempting to reverse) are also contemplated, such as predicting that the vehicle will proceed in reverse at the vehicle's observed speed for a predetermined amount of time (e.g., 1 second, 2 seconds, 3 seconds, etc.) until it is determined that the reverse condition has changed.

[0014] In some examples, the trajectory determination component may control the autonomous vehicle to maintain its current position (e.g., by stopping the autonomous vehicle), or may control the autonomous vehicle to follow a trajectory that continues along the route to the autonomous vehicle's intended destination, but that bypasses the predicted trajectory of the vehicle performing the reversing maneuver. In addition to these examples, the trajectory determination component may control the autonomous vehicle to reverse to provide additional space in the predicted trajectory of the vehicle performing the reversing maneuver.

[0015] As an illustration of a specific example, consider a vehicle preparing to parallel park from a lane of traffic next to a lane of a parking space. Behind that vehicle, an autonomous vehicle is moving in the lane of traffic and may detect that the vehicle has its reverse lights on (and / or other context information such as wheel direction, etc.) in preparation to perform a parallel parking maneuver. The autonomous vehicle may determine one or more predicted trajectories that the vehicle may follow to perform the parallel parking maneuver (e.g., a first attempt that is successful in performing the parallel parking maneuver, and a trajectory that may cause the vehicle to fail to perform the parallel parking maneuver and start over by re-entering the lane of traffic). The autonomous vehicle may evaluate the angle of the vehicle relative to the lane of traffic and / or the parking lane, along with the position of the parked vehicle, to determine, for example, the predicted trajectory. Then, in a first example, the autonomous vehicle may be controlled to wait for the vehicle to perform the maneuver by stopping in the lane of traffic at a position that allows the vehicle to perform the parallel parking maneuver (e.g., behind the parking space the vehicle is entering).

[0016] In another example, an autonomous vehicle may be controlled to enter a lane of different traffic on the opposite side of a parking lane to bypass the predicted trajectory of a parallel parking maneuver, for example, by entering the lane of different traffic at a location behind the parking space the vehicle is entering.

[0017] The techniques discussed herein may improve the functioning of a computing device in several ways. For example, the trajectory determination component may utilize as little as one image (or a cropped portion of one image) to make a reliable determination of a vehicle's reverse operation in an environment. As a result, significantly fewer processing resources are utilized compared to conventional techniques that require complex image analysis algorithms applied to multiple images or videos, and / or data provided by other sensors to determine the vehicle's reverse behavior. Because a reverse indication may be determined from one image, the trajectory determination component may be able to determine trajectories for more vehicles in reverse and / or other vehicles or objects in the environment in general than would be possible if multiple images or videos were required. In some cases, the described techniques are more accurate than other reverse determination mechanisms. For example, a determination of whether a vehicle is in reverse may be inaccurate if the reverse lights are obscured by other objects in the environment, the vehicle's reverse lights are out or not working in another situation and / or are completely obscured, or the vehicle has not yet engaged reverse gear to display illuminated reverse lights. These decisions may be improved by utilizing additional information on the surrounding environment and other information about the vehicle itself (e.g., angle relative to traffic lanes or parking lanes), as described herein.

[0018] By controlling the vehicle based in part on a determination of whether the other vehicle is in reverse gear, utilizing both indications provided by the other vehicle and information gathered about the surrounding environment, vehicle safety may be improved by determining that the other vehicle is reversing more quickly and earlier, thus allowing the vehicle to make earlier trajectory decisions that provide accommodations for a reversing maneuver by the other vehicle. Additionally, as just described, controlling the vehicle based in part on determining a reversing maneuver by another vehicle may reduce processing resources as only one image at a time is evaluated by the processing resources during the reversing decision.

[0019] Additionally, techniques for controlling a vehicle based in part on determining a reversing maneuver by another vehicle can increase confidence that the vehicle can avoid a collision with oncoming traffic and / or pedestrians by determining a reversing maneuver earlier and with greater accuracy, which can improve safety outcomes, performance, and / or accuracy. These and other improvements to computer functions are discussed herein.

[0020] The techniques described herein may be implemented in a number of ways. Exemplary implementations are provided below with reference to the following figures. Although discussed in the context of an autonomous vehicle, the methods, apparatus, and systems described herein may be applied to a variety of systems (e.g., sensor systems or robotic platforms) and are not limited to autonomous vehicles. In one example, the techniques described herein may be employed in a driver-controlled vehicle that may provide an indication to the vehicle's operator of whether such a system is safe to perform various operations. In another example, the techniques may be employed in an aviation or nautical context, or in any system that contains objects or entities that may be associated with behaviors that are unknown to the system. Such techniques may also be employed in manufacturing and assembly contexts, for example, to inspect components as they move through an assembly line. Additionally, the techniques described herein may be employed with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

[0021] FIG. 1 is an illustrated flow diagram 100 for controlling an autonomous vehicle based on a predicted trajectory of the vehicle determined to be in reverse gear, according to an example of the disclosure.

[0022] Operation 102 includes detecting a feature of the vehicle depicted in the image, the feature being a feature type indicating whether the vehicle is in reverse gear. For example, the feature type may be a light on the vehicle indicating that the vehicle is in reverse gear, an audio output by the vehicle that the vehicle is in reverse gear, etc. In some cases, even though the vehicle may be depicted in the image, the feature itself may not be depicted (e.g., when the feature is obscured or the feature is an audio (e.g., a backup beeper, a backup alarm, or a vehicle motion alarm)), although the process described may still continue to determine a predicted trajectory of the vehicle based on other information, as described above and below.

[0023] In an example, image 104 may be captured by image sensor 106 of autonomous vehicle 108 as the autonomous vehicle traverses an environment. Image 104 may also represent vehicle 110 traversing the environment. In some cases, image 104 may be cropped around vehicle 110 to generate cropped image 112 that may be utilized to detect one or more features of vehicle 110. In this case, autonomous vehicle 108 may detect features 114(A), 114(B), and 114(C) (collectively features 114) that correspond to lights of vehicle 110 that may indicate vehicle 110 is in reverse gear, vehicle 110 braking, blinkers to indicate vehicle 110 intends to turn left or right, hazard lights of vehicle 110, etc.

[0024] Other features are also contemplated, such as wheel angle, detection of double parked vehicles (e.g., a third vehicle in the environment), etc. Along with feature 114, additional information about the environment may also be detected utilizing image sensor 106 and / or other sensors of autonomous vehicle 108. As described above and in more detail below, features of the environment that may provide information about whether vehicle 110 is in reverse gear (or intends to engage reverse gear, also referred to as a "reverse intent state") may include the orientation of the vehicle (e.g., relative to a lane of traffic), the speed of vehicle 110 relative to a speed limit at the location of vehicle 110 in the environment, the speed of vehicle 110 relative to other vehicles in the surrounding environment, the particular lane in which vehicle 110 is currently in, lanes adjacent to the lane in which vehicle 110 is currently in (e.g., adjacent parking lanes, driveways, etc.), whether vehicle 110 is positioned at or near a junction, the distance between vehicle 110 and other vehicles in the environment, the speed of other vehicles near vehicle 110, etc. Vehicles 110 and / or features 114 may be detected, for example, utilizing a semantic segmentation algorithm trained to identify objects and object features, although other examples of detecting vehicles 110 and features 114 are also contemplated. Additional details regarding determining vehicle lighting status may be found in U.S. patent application Ser. No. 15 / 982,658, which is incorporated herein by reference in its entirety.

[0025] Operation 116 includes determining that the vehicle is in a reverse state. For example, a machine learning model (or multiple models) may be utilized to determine whether the vehicle is in reverse gear. In some examples, the features detected in operation 102 (and / or additional features of the environment surrounding the vehicle 110, sensor data captured or received in operation 102 with or without the detected features, etc.) may be input to a machine learning model trained to predict vehicle behavior based at least in part on the feature type of the particular feature. Continuing with the above example, the cropped image 112 may be input to a machine learning model trained to predict that the vehicle will perform a reverse maneuver based on the vehicle's reverse lights being on (e.g., illuminated). The machine learning model may be a classifier trained in some cases to classify the reversing state of the vehicle 110 into one or more of a reverse state, a non-reverse state, or a reverse attempt state to name some example classifications. Additional details regarding machine learning models that may be utilized to determine when a vehicle is in reverse gear can be found in the discussion of Figures 2 and 4.

[0026] Operation 118 includes predicting a trajectory of the vehicle based at least in part on the vehicle being in reverse and characteristics of the environment. For example, consider example 120 in which it is determined that vehicle 110 is in reverse gear. Autonomous vehicle 108 may identify various features of the environment, such as vehicle 122 and parking lane 124, that may be relevant to generating a predicted trajectory 126 for vehicle 110. For example, autonomous vehicle 108 may determine that vehicle 110 is likely to enter parking lane 124 while moving forward around vehicle 122 to park in parking lane 124, rather than continuing to reverse straight ahead or reversing into traffic lane 128. In examples, the characteristics of the environment may be determined from map data of the environment (e.g., traffic lanes, parking lanes, junctions, sidewalks, etc.), image data or other sensor data representing objects in the environment (e.g., cars, pedestrians, bicycles, construction zones, etc.), etc. The features of the environment may be utilized to determine a predicted trajectory based on objects that the vehicle 110 is likely to avoid or go around, such as other vehicles, sidewalks, pedestrians, etc., while also determining objects that the vehicle 110 is likely to pass through or enter, such as parking spaces, driveways, lanes of particular traffic (e.g., when performing a three-point turn), loading docks, or even objects for connecting to vehicle accessories such as trailers. For example, a top-down approach may be utilized to generate a heat map associated with the predicted probability of objects in the environment, which may then be utilized to determine a predicted trajectory for the vehicle 110, as discussed herein. Additional details of predicting a trajectory are discussed in connection with FIG. 2 and throughout this disclosure.

[0027] The operation 130 includes controlling the autonomous vehicle to accommodate the trajectory. Example 132 illustrates the autonomous vehicle 108 determining a trajectory 134 that accommodates a predicted trajectory 126 of the vehicle 110 (e.g., an output of the operation 118) based on a determination that the vehicle 110 is in reverse gear. The trajectory 134 of the autonomous vehicle 108 includes maneuvering into a traffic lane 128 while providing the vehicle 110 with space to continue executing a reverse maneuver predicted by the predicted trajectory 126. In some examples, to provide suitable space to accommodate the predicted trajectory 126 of the vehicle 110, the autonomous vehicle 108 may also determine a width of the vehicle 110 and generate the trajectory 134 based on the width of the vehicle 110 applied to the length of the predicted trajectory 126. Trajectory 134 may also include a buffer zone (e.g., an additional 20 percent of the width of vehicle 110) extending beyond the width of vehicle 110 along the predicted trajectory 126, which may provide additional space for slight variations in the actual trajectory that vehicle 110 may follow and / or instill confidence by the driver of vehicle 110 that the autonomous vehicle 108 will not end up colliding with vehicle 110. In this manner, autonomous vehicle 108 may safely and effectively adapt to the predicted trajectory 126 of vehicle 110, including reversing maneuvers, while providing the autonomous vehicle 108 with a mechanism for continuing to a desired destination with fewer interruptions than existed in previous techniques.

[0028] FIG. 2 is an illustration of a system 200 that may be utilized to determine one or more trajectories based in part on an indication that a vehicle is in reverse gear, according to an example of this disclosure.

[0029] The first example 202 illustrates a scenario in which an autonomous vehicle 204 is traveling along a trajectory 206 through an environment. The example 202 also includes a vehicle 208 illustrated at a position at time t0. A trajectory 210 of the vehicle 208 is shown at a time t0 (e.g., in the past) that represents the maneuvers that the vehicle 208 was performing up to time t0. -n From time t to time t0. In some examples, -nThe time from to t0 may correspond to an amount of time that the vehicle 208 has been tracked to provide information about the behavior of the vehicle 208, such as 1 second, 3 seconds, 5 seconds, 10 frames of video, 30 frames of video, etc. In the context of a top-down prediction implementation, the operation may include inputting data representative of the environment over time (e.g., data representative of the environment at T=-2.0 seconds, t=-1.5 seconds, t=-1.0 seconds, t=-0.5 seconds). At time t0, the autonomous vehicle 204 may capture an image 212 (or other data) of the vehicle 208. The image 212 may be utilized to determine how to control the autonomous vehicle 204 to continue progressing after time t0, such as based on a predicted trajectory of the vehicle 208.

[0030] Image 212 may, in some instances, be a cropped image based on a bounding box associated with vehicle 208. For example, the bounding box may be generated by a classifier, detector, or segmenter that may identify an object (such as a vehicle) in the image data. Although images are utilized in the examples described in association with system 200, other indicators may be associated with the vehicle, such as other types of visual signals (e.g., hand signals, display screens, etc.), auditory signals, etc., that provide an indication of the surrounding environment of the vehicle's behavior as may be determined by any combination of sensor data or other signals and may be relied upon to make decisions about the vehicle.

[0031] To predict a trajectory for the vehicle 208, the autonomous vehicle 204 may input the images 212 (or other indicators, sensor data, signals, data (e.g., map data), etc.) to the machine learning component 214, which may then output a reverse indicator 216 corresponding to an indication as to whether the vehicle 208 is in reverse gear. In some examples, the machine learning component 214 may output a probability associated with the vehicle 208 being in reverse, based on the illumination of lights on the vehicle 208, etc. Alternatively, or in addition, the machine learning component 214 may output a classification among a number of classifications corresponding to possible reverse indications, such as the vehicle 208 being in reverse, the vehicle 208 being in a non-reverse state, and / or the vehicle 208 being in a reverse intent state. Although generally described with respect to the classifier outputting a reversing indication 216 into a reverse state, a not-reversing state, and / or a reverse attempted state, the machine learning component may alternatively output a binary indication of whether the vehicle is reversing or not reversing.

[0032] The machine learning component 214 may also receive environmental features that may be utilized to determine the behavior of the vehicle 208, including determining the reverse indicator 216. The environmental features may be received from sensors of the autonomous vehicle 204 and / or from map data associated with the environment surrounding the autonomous vehicle 204 and the vehicle 208. For example, the machine learning component 214 may receive vehicle attributes 222 associated with the vehicle 208. The vehicle attributes 222 may include indicator visibility 218, which may indicate, for example, whether an indicator (e.g., lights) that the vehicle 208 is in reverse gear is visible. In some cases, if the indicator visibility 218 indicates that the lights of the vehicle 208 are obscured, not operating, or not depicted in the image 212, other features may be relied upon more heavily (e.g., weighted more highly) in making a prediction of the upcoming trajectory for the vehicle 208.

[0033] The vehicle attributes 222 may also include a vehicle orientation 224 of the vehicle 208. The vehicle orientation 224 may be determined from the imagery 212 and may be based on, for example, the angle of the vehicle 208 relative to the lane of traffic in which the vehicle 208 is located and / or relative to a lane adjacent to the lane of traffic in which the vehicle 208 is located, such as a parking lane or another lane of traffic. In some cases, the machine learning component 214 may utilize this information to predict a reversing condition for the vehicle 208 if the vehicle 208 is positioned at an angle that is not generally parallel to the direction of travel of the lane of traffic.

[0034] Additionally, the vehicle attributes 222 may include a vehicle speed 226 associated with the vehicle 208. The vehicle speed 226 may be determined, for example, based at least in part on lidar data, radar data, time-of-flight data, image data, or other techniques for determining the speed of objects in the environment. The machine learning component 214 may compare the vehicle speed 226 to one or more of a speed limit for the environment, or the speeds of other vehicles in the environment, to determine a likelihood that the vehicle 208 is performing a reversing maneuver or intends to perform a reversing maneuver. In many cases, a vehicle performing a reversing maneuver will proceed at a speed much lower than the permitted speed limit for the drivable area. Additionally, a vehicle intending to perform a reversing maneuver (e.g., looking for a parking space to back into) will also typically move at a speed lower than the permitted speed limit and / or other vehicles in the environment. Thus, the machine learning component 214 may utilize the vehicle speed 226 to determine whether the vehicle 208 is performing a reverse maneuver and / or whether the vehicle 208 intends to perform a reverse maneuver and, accordingly, output a reverse status for the vehicle 208. Additional vehicle attributes 222 are also contemplated.

[0035] Additionally, the machine learning component 214 may receive map data 228 associated with the autonomous vehicle 204 and the environment surrounding the vehicle 208. For example, the map data 228 may include a current lane type 230, an adjacent lane type 232, and / or an indication of a junction 234. The current lane type 230 may indicate a current lane occupied by the vehicle 208. For example, the vehicle 208 may occupy a driveway as the current lane type 230. The machine learning component 214 may use the current lane type 230 to determine a predicted trajectory associated with the vehicle 208 backing out of a driveway, for example. The adjacent lane type 232 may indicate a lane closest to the current lane occupied by the vehicle 208, such as a parking lane, other traffic lane, etc. In an example, the machine learning component 214 may use the adjacent lane type 232 to predict a reversing state of the vehicle 208 during a parallel parking maneuver, backing into a right-angle parking space, a three-point turn maneuver, etc. The junction indication 234 may indicate a junction closest to the vehicle 208 that the machine learning component 214 may use to predict reverse conditions for the vehicle 208 for maneuvers such as backing up around a corner. Additional map data 228 may also be used by the machine learning component 214 to predict reverse conditions for the vehicle 208.

[0036] Additionally, machine learning component 214 may receive surrounding vehicle attributes 236, which may correspond to information related to vehicles other than autonomous vehicle 204 and vehicle 208, as well as how other vehicles are reacting to a maneuver being performed by vehicle 208 at time t0. Surrounding vehicle attributes 236 may include, for example, distance 238 from vehicle 208 and / or speed 240 of nearest vehicle 208. Distance 238 and / or speed 240 may be determined based at least in part on lidar data, radar data, time-of-flight data, image data, or other data or techniques for determining relative distance or speed of objects in an environment.

[0037] In many cases, when other vehicles (or drivers of other vehicles) anticipate that vehicle 208 may be performing a reversing maneuver, the other vehicles may provide additional space to vehicle 208. Thus, machine learning component 214 may determine that other vehicles are providing additional distance 238 between themselves and vehicle 208 and may use this information to predict whether vehicle 208 is performing a reversing maneuver or intends to perform a reversing maneuver. Similarly, in many cases, when other vehicles anticipate that vehicle 208 may be performing a reversing maneuver, the other vehicles may proceed at a slower speed (e.g., to wait for the vehicle to complete the reversing maneuver) or at a faster speed (e.g., to proceed before the vehicle performs the anticipated reversing maneuver). Thus, machine learning component 214 may determine that other vehicles are proceeding at a different speed 240 closest to vehicle 208 and may use this information to predict whether vehicle 208 is performing a reversing maneuver or intends to perform a reversing maneuver. Additional surrounding vehicle attributes 236 may also be utilized by the machine learning component 214 to predict a reverse condition for the vehicle 208.

[0038] The vehicle attributes 222, the map data 228, and / or the surrounding vehicle attributes 236 may be utilized in a variety of ways by the machine learning component 214 to output the reverse indicator 216. For example, the machine learning component 214 may utilize data fusion techniques, such as by applying a Kalman filter or by utilizing a convolutional neural network, to combine data received from multiple sensor modalities related to the vehicle attributes 222, the map data 228, and / or the surrounding vehicle attributes 236. The machine learning component 214 may additionally utilize a classifier to select from a reverse state, a non-reverse state, and / or a reverse attempted state to generate the reverse indicator 216 based on the information received from the data fusion layer of the machine learning component 214.

[0039] Reverse indicator 216, determined using any of image 212, vehicle attributes 222, map data 228, and / or surrounding vehicle attributes 236, may be output to orbit determination component 220, which may use reverse indicator 216 to predict an upcoming trajectory for vehicle 208 and may adjust trajectory 206 of autonomous vehicle 204 based on the predicted trajectory for vehicle 208. In some examples, orbit determination component 220 may also receive features of the environment, including vehicle attributes 222, map data 228, and / or surrounding vehicle attributes 236, which may be used to refine the predicted trajectory of vehicle 208 and / or to adjust trajectory 206. The features of the environment may be received from sensors of autonomous vehicle 204 and / or from map data associated with the environment surrounding autonomous vehicle 204 and vehicle 208.

[0040] In some examples, trajectory determination component 220 may combine inputs received from reverse indicator 216, vehicle attributes 222, map data 228, and / or surrounding vehicle attributes 236 to predict a trajectory for vehicle 208, such as by utilizing a weighted average of different environmental attributes in conjunction with reverse indicator 216. In some examples, inputs received from reverse indicator 216, vehicle attributes 222, map data 228, and / or surrounding vehicle attributes 236 may be combined using one or more data fusion techniques, such as by applying a Kalman filter or by utilizing a convolutional neural network, to generate a predicted trajectory for vehicle 208. For example, map data 228 may be combined with vehicle attributes 222 and / or surrounding vehicle attributes 236 to determine candidate parking positions (or other reverse maneuver potential positions), and a physics-based model may be used to determine the likelihood that vehicle 208 will back into a parking space at the candidate parking position.

[0041] In another example, the images 212, vehicle attributes 222, map data 228, and / or surrounding vehicle attributes 236 may be input to a machine learning model that outputs a heat map indicative of possible trajectories of the vehicle 208. As described above, the trajectory determination component 220 may utilize a machine learning model to predict a trajectory by generating a top-down image that includes channels containing information associated with features of the environment, such as the vehicle attributes 222, the map data 228, the surrounding vehicle attributes 236, etc. The machine learning model utilized to generate the predicted trajectory may be based on a time t corresponding to one or more times in the future (e.g., 1 second, 2 seconds, 3 seconds, etc.). n In some examples, the machine learning model of the trajectory determination component 220 may output coordinates (e.g., x and y coordinates) associated with the vehicle 208. In some examples, the machine learning model of the trajectory determination component 220 may output coordinates associated with the vehicle 208, as well as probability information associated with each coordinate. Additionally, the machine learning model may output a heat map associated with the predicted probability. In some examples, at least one predicted trajectory may be determined based at least in part on the heat map. Additional details regarding predicting trajectories utilizing a top-down approach may be found in U.S. Patent Application No. 16 / 151,607, which is incorporated by reference herein in its entirety. Other approaches to predicting a trajectory for a vehicle 208 determined to be in a reverse state (or reverse attempted state) are also contemplated, such as predicting that the vehicle will proceed in reverse at the vehicle's observed speed for a predetermined amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, etc.) until it is determined that the reverse state has changed. Other examples are also contemplated.

[0042] Example 242 corresponds to the same time (t0) as example 202, but in this case represents a predicted trajectory 244 of vehicle 208 based on inputs to trajectory determination component 220 of vehicle 208. In this case, reverse indicator 216 may be based on a determination that vehicle 208 is in reverse gear (e.g., based on the reverse lights of vehicle 208 being on) and / or additional features of the environment, as described above.

[0043] The trajectory determination component 220 also determines the location of a perpendicular parking space that the vehicle 208 can back into that is closest to the current lane that the vehicle 208 occupies, the orientation of the vehicle 208 relative to the current lane that the vehicle 208 occupies and / or the parking lane, and the time t -n The vehicle 208's speed from time t to time t, as well as environmental features such as the distance and / or speed of other vehicles in the surrounding environment relative to the vehicle 208, may be utilized to generate a predicted trajectory 244. The predicted trajectory 244 may be calculated based on the predicted trajectory for a future time t, such as 1 second, 3 seconds, 5 seconds, and so on. n In some examples, the orbit determination component 220 also decelerates to a time t n Based on the predicted trajectory 244, the trajectory 206 of the autonomous vehicle 204 may be altered to trajectory 246, such as to wait for the vehicle 208 until to enable the vehicle 208 to complete the reversing maneuver indicated by the predicted trajectory 244.

[0044] FIG. 3 is an illustration 300 of utilizing environmental features to determine a reverse attempt condition and subsequent trajectory estimation for a vehicle, according to an example of this disclosure.

[0045] Example 302 represents autonomous vehicle 304 and vehicle 306 traversing an environment at time t1. Autonomous vehicle 304 may be traversing the environment along trajectory 308, which may correspond to following vehicle 306 as it traverses the environment along trajectory 310. In example 302, autonomous vehicle 304 and vehicle 306 may be passing through a parking lane 312 that includes vehicles in various parking spaces in parking lane 312.

[0046] Example 314 represents autonomous vehicle 304 and vehicle 306 at time t2 after time t1, still near parking lane 312. Autonomous vehicle 304 and vehicle 306 move forward along parking lane 312 from time t1 to time t2. However, at time t2, vehicle 306 may apply brakes to slow down, causing vehicle 306's brake lights 316 to illuminate. In response to detecting vehicle 306's brake lights 316 illuminating, autonomous vehicle 304 may alter trajectory 308 to trajectory 318 in which autonomous vehicle 304 slows down and / or stops to avoid a collision with vehicle 306.

[0047] Although vehicle 306 has not provided an indication that vehicle 306 is currently in reverse gear at time t2, autonomous vehicle 304 may utilize other information to determine that vehicle 306 is more likely to perform a reverse maneuver (e.g., is in a reverse attempt state) and accommodate the anticipated reverse maneuver by providing time and / or space to vehicle 306. For example, at time t2, autonomous vehicle 304 may analyze the speed of vehicle 306 (e.g., relative to the speed limits of the environment and / or other vehicles in the environment), map data indicating the proximity of vehicle 306 to parking lane 312, and / or image data indicating individual parking spaces in parking lane 312 that may be unoccupied, to increase the likelihood that vehicle 306 may imminently perform a reverse maneuver, despite not currently indicating that vehicle 306 is in reverse gear.

[0048] For example, examples 320 and 322 represent autonomous vehicle 304 and vehicle 306 at time t3, where example 320 represents the autonomous vehicle 304 at time t3. 3A , and example 322 corresponds to time t 3B(e.g., generally at the same time, although two examples are presented for illustrative purposes). In example 320, the vehicle 306 moves forward along a trajectory 324 in which the vehicle 306 begins to turn away from the parking lane 312. The autonomous vehicle 304 may combine observations of the vehicle 306's trajectory 324 with map data associated with the parking lane 312 and / or image data indicative of individual parking spaces in the parking lane 312 that may be unoccupied, to increase the likelihood that the vehicle 306 is about to perform a reverse maneuver. In some examples, the autonomous vehicle 304 may also detect (e.g., from the image data or map data) that at the vehicle 306's current location, there is no space for the vehicle 306 to make a left turn in the forward direction, which may increase the likelihood that the vehicle 306 is about to perform a reverse maneuver. The increased likelihood that the vehicle 306 is about to perform a reverse maneuver may cause the vehicle 306 to be labeled as being in a reverse attempt state.

[0049] In example 322, autonomous vehicle 304 may generate a predicted trajectory 326 based on observations of vehicle 306's behavior and / or characteristics of the environment surrounding vehicle 306, and following a determination that vehicle 306 is in a reverse attempt state. Predicted trajectory 326 may correspond to a reversing maneuver to be performed by vehicle 306 to enter a particular space in parking lane 312. Based on predicted trajectory 326, autonomous vehicle 304 may change trajectory 318 that autonomous vehicle 304 was following at time t2 to trajectory 328 that may accommodate the reversing maneuver of predicted trajectory 326.

[0050] In some examples, the trajectory 328 may cause the autonomous vehicle 304 to slow down, maintain position (e.g., stop), and / or detour from the predicted trajectory 326. In examples, the autonomous vehicle 304 continues to evaluate the behavior of the vehicle 306 after time t3 to determine the likelihood that the vehicle 306 will actually perform a reverse maneuver. The likelihood that the vehicle 306 will perform a reverse maneuver may be increased in response to a light on the vehicle 306 indicating a reverse condition being illuminated, for example, as discussed in connection with FIG. 2. In some cases, an identification that the vehicle 306 is in a reverse condition, such as by a reverse light being illuminated, may cause the autonomous vehicle 304 to change a label associated with the vehicle 306 from a reverse attempting state to a reverse state.

[0051] 4 illustrates a block diagram of an example system 400 for implementing the techniques described herein. In at least one example, the system 400 may include a vehicle 402, such as an autonomous vehicle, a semi-autonomous vehicle, or a manually controlled vehicle.

[0052] The vehicle 402 may include a vehicle computing device 404, one or more sensor systems 406, one or more emitters 408, one or more communication connections 410, at least one direct connection 412, and one or more drive systems 414.

[0053] Vehicle computing device 404 may include one or more processors 416 and memory 418 communicatively coupled to the one or more processors 416. In the illustrated example, vehicle 402 is an autonomous vehicle, however, vehicle 402 may be any other type of vehicle or robotic platform. In the illustrated example, memory 418 of vehicle computing device 404 stores a localization component 420, a perception component 422, one or more maps 424, one or more system controllers 426, a trajectory determination component 428, a feature identifier component 430, a trajectory prediction component 432, and a planning component 434. While depicted in FIG. 4 as residing in memory 418 for illustrative purposes, it is contemplated that localization component 420, perception component 422, one or more maps 424, one or more system controllers 426, orbit determination component 428, feature identifier component 430, trajectory prediction component 432, and planning component 434 may additionally or alternatively be accessible to vehicle 402 (e.g., stored on or otherwise accessible by memory remote from vehicle 402).

[0054] In at least one example, the localization component 420 may include functionality for receiving data from the sensor system 406 to determine the position and / or orientation of the vehicle 402 (e.g., one or more of x, y, z position, roll, pitch, or yaw). For example, the localization component 420 may include and / or request / receive a map of the environment and may continually determine the position and / or orientation of the autonomous vehicle within the map. In some instances, the localization component 420 may receive image data, lidar data, radar data, time of flight data, IMU data, GPS data, wheel encoder data, etc., utilizing SLAM (simultaneous localization and mapping), CLAMS (calibration, localization, and mapping simultaneously), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc., to accurately determine the position of the autonomous vehicle. In some instances, the localization component 420 may provide data to various components of the vehicle 402 to determine an initial position of the autonomous vehicle for generating a trajectory, determining to retrieve map data, etc., as discussed herein.

[0055] In some instances, the perception component 422 may include functionality for performing object detection, segmentation, and / or classification. In some examples, the perception component 422 may provide processed sensor data indicative of the presence of an entity or object proximate to the vehicle 402 and / or a classification of the entity as an entity type (e.g., car, wheel, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, stop light, stop sign, lane marker, vehicle light, unknown, etc.). In additional or alternative examples, the perception component 422 may provide processed sensor data indicative of one or more characteristics associated with a detected entity (e.g., a tracked object) and / or the environment in which the entity is located. In some examples, characteristics associated with an entity may include, but are not limited to, x-position (global and / or local position), y-position (global and / or local position), z-position (global and / or local position), orientation (e.g., roll, pitch, yaw), entity type (e.g., classification), entity velocity, entity acceleration, entity width (size), etc. Characteristics associated with an environment may include, but are not limited to, the presence of other entities in the environment, the state of other entities in the environment, time of day, day of the week, season, weather conditions, darkness / light indication, etc.

[0056] In examples where the perception component 422 performs detection, the perception component 422 may output detection of objects in the images and / or features of the detected objects. Such detection may include a two-dimensional bounding box (which may then be used to crop the image) and / or a mask of the detected object. In some examples, such detection may utilize machine learning techniques (e.g., scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG), etc.) followed by a support vector machine (SVM) to classify objects depicted in images received from the cameras of the sensor system 406. Alternatively, or in addition, the detection may utilize deep learning techniques based on convolutional neural networks (CNN) to classify objects depicted in images received from the cameras of the sensor system 406. In examples, the detection may utilize one or more of the object detection techniques (or others) to detect objects depicted in images and / or features of objects depicted in images according to the described techniques.

[0057] The memory 418 may further include one or more maps 424 that may be utilized by the vehicle 402 to navigate within the environment and / or to determine the location of features in the environment. For purposes of this discussion, a map may be any number of data structures modeled in two, three, or N dimensions that can provide information about the environment, such as, but not limited to, topology (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some instances, the map may include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., lidar information, radar information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and / or intensities), reflectivity information (e.g., specularity information, retroreflectivity information, BRDF information, BSSRDF information, etc.). In one example, the map may include a three-dimensional mesh of the environment. In some instances, the map may be stored in a tiled format, with each tile of the map representing a discontinuous portion of the environment, and may be loaded into the working memory as needed. In at least one example, the one or more maps 424 may include at least one map (e.g., an image and / or a mesh). In some examples, the vehicle 402 may be controlled based at least in part on the map 424. That is, the map 424 may be utilized in conjunction with the localization component 420, the perception component 422, the trajectory determination component 428, or the planning component 434 to determine a position of the vehicle 402, identify objects and / or environmental features in the environment, and / or generate a route and / or trajectory for navigating within the environment.

[0058] In some examples, one or more maps 424 may be stored on a remote computing device (such as computing device 438) accessible via network 436. In some examples, multiple maps 424 may be stored, for example, based on characteristics (e.g., type of entity, time of day, day of the week, season of the year, etc.). Storing multiple maps 424 may increase the speed at which data in the maps may be accessed while having similar memory requirements. In some examples, one or more maps 424 may store sizes or dimensions of objects associated with individual locations in the environment. For example, as vehicle 402 traverses the environment, and a map representing the area nearest vehicle 402 is loaded into memory, one or more sizes or dimensions of objects associated with the location may also be loaded into memory.

[0059] In general, the orbit determination component 428 may determine that the vehicle in the environment surrounding the vehicle 402 is in reverse gear, predict the trajectory of the vehicle based on the determination that the vehicle is in reverse gear, and generate a trajectory for the vehicle 402 that adapts to the predicted trajectory of the vehicle based on the data provided by the sensor system 406. In some instances, the orbit determination component 428 may provide information generated by the feature identifier component 430 and / or the trajectory prediction component 432 to the planning component 434 to determine when and / or how to control the vehicle 402 to traverse the environment. As discussed herein, the orbit determination component 428 may receive image data, map data, lidar data, etc. to determine information about objects in the environment.

[0060] The feature identifier component 430 may generate an indication of whether the vehicle is in reverse gear from an image (or cropped image) of the vehicle and from information about features of the environment from the image or other sensor modalities. For example, the feature identifier component 430 may utilize a machine learning model trained to classify the vehicle's light status into categories such as reverse, not reverse, or obscured, to name a few. In some examples, the technique may be performed continuously in the environment to determine the reverse status of objects in the environment and / or may be associated with prerequisites such as the location of the vehicle, the presence of objects in the environment, etc. In some examples, the feature identifier component 430 may output a tag (or flag, additional information, etc.) associated with the vehicle indicating that the vehicle is in reverse to the planning component 434 as long as the light status is detected and / or the vehicle is classified as being in reverse. Additionally, in some cases, feature identifier component 430 may output a tag associated with the vehicle that the vehicle is in a reverse attempt state to planning component 434, as described herein. In an example, if the machine learning model utilized by feature identifier component 430 is a classifier, the machine learning model may be trained using training image data that is annotated to indicate ground truth states of the features (e.g., vehicle light status, vehicle attributes 222, map data 228, and / or surrounding vehicle attributes 236, as described in connection with FIG. 2).Additionally, the machine learning model utilized by the feature identifier component 430 may be trained utilizing training image data annotated to indicate ground truth conditions of the direction of vehicle movement over the amount of time closest to the time the image was captured or the time the vehicle would otherwise be classified as in reverse or attempting to reverse, e.g., 1 second before and / or after the time the image was captured, 3 seconds before and / or after the time the image was captured, 5 seconds before and / or after the time the image was captured, or other amount of time. In some examples, the training data may correspond to a particular implementation of the techniques discussed herein. That is, the training data may correspond to a designed input for performing the techniques with a vehicle in an environment.

[0061] The feature identifier component 430 may identify features of the environment that may be relevant to determining that another vehicle is in reverse gear and / or may be relevant to determining a predicted trajectory of the other vehicle. For example, the feature identifier component 430 may identify features in the environment, such as traffic lanes, parking lanes, bicycle lanes, junctions, and / or sidewalks, from the map 424 and / or may identify features in the environment, such as other vehicles, pedestrians, bicycles, etc., from sensor data captured by the sensor system 406. These features may be input into a machine learning model to predict vehicle behavior, such as to provide a reverse indicator associated with the vehicle. In some examples, the feature identifier component 430 may provide information regarding vehicle features and / or environmental features relevant to determining that the vehicle is in reverse gear to the planning component 434 for use in controlling the vehicle 402.

[0062] From the indication that the vehicle is in reverse gear and / or from characteristics of the environment, the trajectory prediction component 432 may determine a predicted trajectory that the vehicle may follow to perform a reversing maneuver. For example, based on the indication that the vehicle is in reverse gear and characteristics of the environment, the trajectory prediction component 432 may determine that the vehicle is performing a parallel parking maneuver, that the vehicle is backing up into a parking space, that the vehicle is backing up around a corner, or that the vehicle is backing up into a driveway, to name a few.

[0063] In some cases, the trajectory prediction component 432 may receive an indication from the feature identifier component 430 that the vehicle is not currently in reverse gear, but may determine based on the vehicle's behavior that the vehicle is likely to transition into reverse gear to perform a maneuver (e.g., is in a reverse intent state), as described above. In some examples, the trajectory prediction component 432 may provide information regarding the vehicle's predicted trajectory, including the reverse maneuver, to the planning component 434, for use, for example, in controlling the vehicle 402 to safely and effectively accommodate the vehicle's reverse maneuver.

[0064] In general, the planning component 434 may determine a path for the vehicle 402 to follow to traverse the environment. For example, the planning component 434 may determine various routes and trajectories, as well as various levels of detail, including providing room for other vehicles determined to be likely to perform a reverse maneuver. For example, the planning component 434 may determine a route for traveling from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this discussion, the route may be a sequence of waypoints for traveling between the two locations. As non-limiting examples, the waypoints include streets, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the planning component 434 may generate instructions for directing the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 434 may determine how to direct the autonomous vehicle from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instructions may be a trajectory or a portion of a trajectory. In some examples, in accordance with the receding horizon technique, multiple trajectories can be generated substantially simultaneously (e.g., within technical tolerances) in which one of the multiple trajectories is selected for the vehicle 402 to navigate.

[0065] In some instances, the planning component 434 may generate one or more trajectories for the vehicle 402 based at least in part on a determination that the vehicle is performing or is likely to perform a reversing maneuver, as discussed herein. The planning component 434 may generate one or more trajectories that accommodate a reversing maneuver by another vehicle while continuing to follow a sequence of waypoints along a route. In some examples, the planning component 434 may evaluate one or more trajectories for the vehicle 402 using temporal logic, such as linear temporal logic and / or signal temporal logic.

[0066] In at least one example, vehicle computing device 404 may include one or more system controllers 426 that may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 402. System controller 426 may communicate with and / or control corresponding systems of drive system 414 and / or other components of vehicle 402.

[0067] As may be appreciated, the components discussed herein (e.g., localization component 420, perception component 422, one or more maps 424, one or more system controllers 426, orbit determination component 428, feature identifier component 430, trajectory prediction component 432, and planning component 434) are described as being separated for illustrative purposes. However, the operations performed by the various components may be combined or performed in any other component. As an example, functionality described in relation to orbit determination component 428, feature identifier component 430, and / or trajectory prediction component 432 may be performed by perception component 422 to reduce the amount of data communicated by the system.

[0068] In at least one example, the sensor system 406 may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, brightness, depth, time of flight, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. The sensor system 406 may include multiple instances of each of these or other types of sensors. For example, the lidar sensors may include individual lidar sensors positioned at the corners, front, back, sides, and / or top of the vehicle 402. As another example, the camera sensors may include multiple cameras positioned at various locations about the exterior and / or interior of the vehicle 402. The sensor system 406 may provide input to the vehicle computing device 404. Additionally or alternatively, the sensor system 406 may transmit sensor data via one or more networks 436 to one or more computing devices at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0069] The vehicle 402 may also include one or more emitters 408 for emitting light and / or sound, as described above. The emitters 408 in this example include interior audio and visual emitters for communicating with occupants of the vehicle 402. By way of example and not limitation, the interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), etc. The emitters 408 in this example also include exterior emitters. By way of example and not limitation, the exterior emitters in this example include lights or other indicators of vehicle action (e.g., indicator lights, signs, light arrays, etc.) to signal direction of movement, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which include acoustic beam steering technology.

[0070] Vehicle 402 may also include one or more communications connections 410 that facilitate communication between vehicle 402 and one or more other local or remote computing devices. For example, communications connections 410 may facilitate communication with other local computing devices on vehicle 402 and / or drive system 414. Communications connections 410 may also enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Communications connections 410 may also facilitate vehicle 402 communicating with remote teleoperated computing devices or other remote services.

[0071] The communication connections 410 may include physical and / or logical interfaces for connecting the vehicle computing device 404 to another computing device or network, such as the network 436. For example, the communication connections 410 may facilitate Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communications protocol that facilitates each computing device interfacing with other computing devices.

[0072] In at least one example, the vehicle 402 may include one or more drive systems 414. In some examples, the vehicle 402 may have a single drive system 414. In at least one example, if the vehicle 402 has multiple drive systems 414, the individual drive systems 414 may be positioned at opposite ends of the vehicle 402 (e.g., front and rear, etc.). In at least one example, the drive system 414 may include one or more sensor systems for detecting conditions surrounding the drive system 414 and / or the vehicle 402. By way of example and not limitation, the sensor systems may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive module, cameras or other image sensors, ultrasonic sensors for acoustically detecting objects in the vicinity of the drive system, lidar sensors, radar sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive system 414. In some cases, a sensor system on the drive system 414 may overlap or supplement a corresponding system on the vehicle 402 (eg, sensor system 406).

[0073] The drive system 414 may include many of the vehicle systems, including a high voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights for illuminating the exterior surroundings of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charge ports, etc.). Additionally, the drive system 414 may include a drive system controller that may receive and pre-process data from the sensor systems and for controlling the operation of the various vehicle systems. In some examples, the drive system controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more components for performing various functions of the drive system 414. In addition, the drive system 414 also includes one or more communication connections that facilitate communication by the respective drive system with one or more other local or remote computing devices.

[0074] In at least one example, the direct connection 412 may provide a physical interface for coupling one or more drive systems 414 to the body of the vehicle 402. For example, the direct connection 412 may enable the transfer of energy, fluid, air, data, etc. between the drive systems 414 and the vehicle. In some instances, the direct connection 412 may further releasably secure the drive systems 414 to the body of the vehicle 402.

[0075] In at least one example, the localization component 420, the perception component 422, the one or more maps 424, the one or more system controllers 426, the orbit determination component 428, the feature identifier component 430, the orbit prediction component 432, and the planning component 434 may process the sensor data as described above and transmit their respective outputs over one or more networks 436 to one or more computing devices 438. In at least one example, the localization component 420, the perception component 422, the one or more maps 424, the one or more system controllers 426, the orbit determination component 428, the feature identifier component 430, the orbit prediction component 432, and the planning component 434 may transmit their respective outputs to the computing device 438 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0076] In some examples, the vehicle 402 may transmit the sensor data over the network 436 to one or more computing devices 438. In some examples, the vehicle 402 may transmit raw sensor data to the computing device 438. In other examples, the vehicle 402 may transmit processed sensor data and / or representations of the sensor data to the computing device 438. In some examples, the vehicle 402 may transmit the sensor data at a particular frequency, after a predetermined period of time, in near real-time, etc. In some cases, the vehicle 402 may transmit the sensor data (raw or processed) to the computing device 438 as one or more log files.

[0077] The computing device 438 may include a processor 440 and a memory 442 that stores a training component 444 .

[0078] In some instances, the training component 444 may include functionality for training one or more models to detect objects and / or object features, determine (classify or regress) objects and / or object features, determine the location of objects in the environment, etc. As noted herein, the sensor data may be annotated or represented to represent the operations to be reversed, and such annotated or represented data may be identified as training data. The training data may be input to a machine learning model, where known results (e.g., ground truths such as known "future" attributes) may be utilized to tune the weights and / or parameters of the machine learning model to minimize error.

[0079] In some instances, the training component 444 may communicate information generated by one or more models to the vehicle computing device 404 to refine how to control the vehicle 402 in response to different situations.

[0080] For example, some or all aspects of the components discussed herein may include any models, algorithms, and / or machine learning algorithms. For example, in some instances, the components in memory 442 (and memory 418 discussed above) may be implemented as neural networks. In some examples, training component 444 may generate and / or execute one or more models using neural networks to improve various aspects of object trajectory estimation for use in trajectory planning of vehicle 402.

[0081] As described herein, a typical neural network is a biologically inspired algorithm that passes input data through a succession of connected layers to produce an output. Each layer in a neural network may also include another neural network, or may include any number of layers (whether convolutional or not). As may be understood in the context of the present disclosure, a neural network may utilize machine learning, which may refer to a broad class of such algorithms in which an output is generated based on learned parameters.

[0082] Although discussed in the context of neural networks, any type of machine learning may be utilized consistently with this disclosure. For example, machine learning algorithms include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute value shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative dichotomizer 3 (ID3), chi-squared automated interaction detection (CHAID), decision strain, conditional decision tree), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average unidependent estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, backprocessing, etc.), and so on. The learning algorithms may include, but are not limited to, deep learning algorithms (e.g., deep Boltzmann machines (DBM), deep belief networks (DBN), convolutional neural networks (CNN), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (blending), gradient boosting machines (GBM), gradient boosted regression trees (GBRT), random forests), support vector machines (SVM), supervised learning, unsupervised learning, semi-supervised learning, and others.

[0083] Additional examples of architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, PointNet, and others.

[0084] The processor 416 of the vehicle 402 and the processor 440 of the computing device 438 may be any suitable processor capable of processing data and executing instructions to perform operations as described herein. By way of example and not limitation, the processors 416 and 440 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts it to other electronic data that may be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement encoded instructions.

[0085] The memories 418 and 442 are examples of non-transitory computer-readable media. The memories 418 and 442 may store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods described herein and the functions attributed to the various systems. In various implementations, the memories may be implemented utilizing any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which the ones shown in the accompanying figures are merely examples relevant to the discussion herein.

[0086] 4 is illustrated as a distributed system, it should be noted that in alternative examples, components of vehicle 402 may be associated with computing device 438 and / or components of computing device 438 may be associated with vehicle 402. That is, vehicle 402 may perform one or more of the functions associated with computing device 438, and vice versa. Additionally, aspects of orbit determination component 428 and / or planning component 434 may be implemented on any of the devices discussed herein.

[0087] 5 illustrates an example process 500 for determining that a vehicle is in reverse gear, predicting a trajectory for the vehicle based on the vehicle being in reverse gear, and controlling an autonomous vehicle based on the predicted trajectory, according to an example of the disclosure. For example, some or all of the process 500 may be performed by one or more components in FIG. 4 as described herein. For example, some or all of the process 500 may be performed by the vehicle computing device 404, the computing device 438, or any other computing device or combination of computing devices. Additionally, any of the operations described in the example process 500 may be performed in parallel, in a different order other than depicted in the process 500, omitted, combined with other processes, or otherwise.

[0088] At operation 502, the process may include capturing an image representing the vehicle in an environment. For example, the image may be captured by a sensor, such as a camera, of the autonomous vehicle traversing the environment.

[0089] At operation 504, the process may include extracting features of the vehicle depicted in the image. In some examples, the features may be vehicle lights indicating that the vehicle is in reverse gear. The features may be detected in some cases by semantically segmenting the image and labeling the portion of the image that includes the feature. Operation 804 may include cropping the portion of the image that includes the feature for further analysis. Although the vehicle features are described with reference to process 500, it should be understood that other features, such as any of the vehicle attributes 222, map data 228, and / or surrounding vehicle attributes 236 described in connection with FIG. 2, may be utilized to predict vehicle behavior.

[0090] At operation 506, the process may include inputting the features into a machine learning model trained to predict vehicle behavior based at least in part on the feature type of the feature. For example, the machine learning model may be a classifier trained to classify a reverse state of the vehicle into one or more of a reverse state, a no-reverse state, or a reverse attempted state, to name a few example classifications. The machine learning model may alternatively or additionally be trained to predict vehicle behavior based on auditory signals or environmental features that may indicate, for example, that the vehicle is in reverse gear.

[0091] At operation 508, the process may include receiving an indication from the machine learning model that the vehicle is in reverse. In some cases, the machine learning model may output a tag for a duration corresponding to the length of time that the illuminated lights indicate that the vehicle is in reverse.

[0092] At operation 510, the process may include predicting a first trajectory of the vehicle based at least in part on the indication that the vehicle is in reverse and characteristics of the environment. For example, various characteristics such as the orientation of the vehicle (e.g., relative to a lane of traffic), the speed of the vehicle relative to a speed limit at the vehicle's location in the environment, the speed of the vehicle relative to other vehicles in the surrounding environment, the particular lane in which the vehicle is currently located, lanes adjacent to the lane in which the vehicle is currently located (e.g., adjacent parking lanes, driveways, etc.), whether the vehicle is located at or near a junction, the distance between the vehicle and other vehicles in the environment, the speeds of other vehicles near the vehicle, etc. may be analyzed to determine a predicted trajectory of the vehicle. Although operation 510 is generally described herein as being performed following operation 508, operation 510 may in some cases be combined with operation 508 such that a predicted trajectory for the vehicle is both generated by and received from a machine learning model.

[0093] At operation 512, the process may include a determination as to whether there is space to execute the first trajectory, such as if the autonomous vehicle continues along the current trajectory. For example, the autonomous vehicle may proceed along the current trajectory and stop behind the vehicle as if the vehicle were stopped at a stop light, such as by providing half a car length between the autonomous vehicle and the vehicle. However, such distance (e.g., half a car length) provided by the current trajectory may not be sufficient to allow the vehicle to perform a reversing maneuver, such as parallel parking or backing up into a parking space.

[0094] If there is sufficient space to execute the first trajectory, the process may include operation 514, in which the autonomous vehicle is controlled to continue along the second trajectory (e.g., the autonomous vehicle's current trajectory). For example, the autonomous vehicle may receive an indication that another vehicle is predicted to execute a reversing maneuver, but the current trajectory, e.g., along a lane of traffic that is not impeded by the predicted reversing maneuver, may provide sufficient space for the other vehicle to execute the reversing maneuver. Thus, the autonomous vehicle may be controlled to continue along the current trajectory.

[0095] However, if there is not enough space for the vehicle to execute the first trajectory, the process may include operation 516 in which the autonomous vehicle is controlled to follow a third trajectory to accommodate the first trajectory of the vehicle. As described above, the third trajectory determined to accommodate the first trajectory of the vehicle may include maintaining a current position, such as by stopping the autonomous vehicle. Alternatively, or in addition, the third trajectory determined to accommodate the first trajectory of the vehicle may include controlling the autonomous vehicle to follow a trajectory that continues along the route to the intended destination of the autonomous vehicle but that bypasses the predicted trajectory of the vehicle executing the reverse maneuver.

[0096] Illustrative clauses A: A system including one or more processors and one or more computer-readable media storing instructions executable by the one or more processors, the instructions, when executed, causing the system to perform operations including: utilizing an image sensor on an autonomous vehicle to capture image data of an environment; determining that the image data represents a vehicle within the environment; determining a first feature associated with the vehicle, the first feature being of a vehicle type indicating whether the vehicle is associated with a reverse condition; inputting the first feature to a machine learning model trained to predict vehicle behavior based at least in part on the feature type; receiving an indication from the machine learning model that the vehicle is associated with a reverse condition; predicting a trajectory of the vehicle based at least in part on the indication that the vehicle is associated with a reverse condition and a second feature of the environment determined from the map data; and controlling the autonomous vehicle to provide the vehicle with space to execute the trajectory based at least in part on the vehicle's trajectory.

[0097] B: The system of clause A, wherein the feature type indicates a light associated with the vehicle, and the machine learning model is trained using training image data annotated to indicate a ground truth state of the light's on state at the time the training image data is captured and a direction of training vehicle movement for that vehicle over a certain amount of time proximate to the time the training image data was captured.

[0098] C: The system of clause A, wherein the vehicle is a first vehicle, and the operations further include determining a first speed of the first vehicle and determining a first difference between the first speed of the first vehicle and a speed limit at the first vehicle's position in the environment or a second difference between the first speed of the first vehicle and a second speed of a second vehicle in the environment, and the machine learning model is further trained to predict vehicle behavior based at least in part on the difference in vehicle speeds, and one or more of the first difference or the second difference are input into the machine learning model.

[0099] D: The system of clause A, wherein the trajectory is a first trajectory, and controlling the autonomous vehicle includes determining a width associated with the vehicle, associating the width with a length of the first trajectory, and determining a second trajectory for the autonomous vehicle to traverse the environment such that the autonomous vehicle circumnavigates the width along the length of the first trajectory.

[0100] E: The system of clause A, wherein controlling the autonomous vehicle includes maintaining a position of the autonomous vehicle for a duration associated with the vehicle executing the trajectory.

[0101] F: A method comprising: receiving sensor data captured by sensors in an environment; determining that the sensor data represents a vehicle; determining a feature associated with the vehicle, the feature being of a feature type indicating whether the vehicle is associated with a reverse state; inputting the feature to a machine learning model trained to predict vehicle behavior based at least in part on the feature type; receiving an indication from the machine learning model that the vehicle is associated with a reverse state; and predicting a trajectory associated with the vehicle based at least in part on the indication that the vehicle is associated with a reverse state.

[0102] G: The method of clause F, wherein the indication that the vehicle is associated with a reverse state includes an angle of the vehicle relative to a second feature of the environment, and predicting the vehicle's trajectory is further based at least in part on the angle of the vehicle relative to the second feature.

[0103] H: The method of clause F, wherein predicting the vehicle's trajectory includes determining that the vehicle is preparing to parallel park, and the method further includes controlling the autonomous vehicle to maintain a stopped position for a duration for the vehicle to parallel park or to enter a traffic lane to traverse around the vehicle in the environment.

[0104] I: The method of clause F, further including determining an area nearest the vehicle that the vehicle is permitted to enter in reverse, and wherein predicting the trajectory is further based on a position of the area relative to the vehicle.

[0105] J: The method of clause I, wherein the area includes one or more of a lane of traffic, a parking space, or a driveway.

[0106] K: The method of clause I, further including determining that the vehicle is at a location within a threshold distance of the region, determining a first speed of the vehicle at the location, and determining a difference between the first speed of the vehicle and one or more of a second speed of another vehicle or a speed limit associated with the location, wherein the machine learning model is further trained to predict vehicle behavior based at least in part on environmental features, and one or more of the location, the first speed, and the difference are input into the machine learning model.

[0107] L: The method of clause F, wherein the features are indicative of lights associated with the vehicle, and the machine learning model includes a classifier trained to classify a reversing state of the vehicle into one or more of a reversing state, a non-reversing state, or a reversing attempted state.

[0108] M: One or more computer-readable media storing instructions that, when executed by one or more processors, perform operations including receiving sensor data captured by sensors in the environment; determining that the sensor data represents a vehicle; determining a feature associated with the vehicle, where the feature is of a feature type that indicates whether the vehicle is associated with a reverse state; inputting the feature into a machine learning model trained to predict vehicle behavior based at least in part on the feature type; receiving an indication from the machine learning model that the vehicle is associated with a reverse state; predicting a trajectory associated with the vehicle based at least in part on the indication that the vehicle is associated with a reverse state; and determining a maneuver for the autonomous vehicle to perform based at least in part on the reverse state.

[0109] N: The one or more computer-readable media of clause M, wherein the operations further include determining a width associated with the vehicle and associating the width with a length of the track, and the operations are associated with the autonomous vehicle circumnavigating at least the width along the length of the track.

[0110] O: The one or more computer-readable media of clause M, wherein the operation includes controlling the autonomous vehicle to maintain a current position of the autonomous vehicle for a duration associated with the vehicle executing the trajectory.

[0111] P: The one or more computer-readable media of clause M, wherein the feature is a first feature and the action further includes determining an angle of the vehicle relative to a second feature of the environment, and the machine learning model is further trained to predict vehicle behavior based at least in part on the environmental feature, and one or more of the angle or the second feature are input into the machine learning model.

[0112] Q: One or more computer-readable media of clause M, wherein the operations further include determining an area nearest the vehicle that the vehicle is permitted to enter in reverse, and predicting the trajectory is further based on a position of the area relative to the vehicle.

[0113] R: One or more computer-readable media of clause Q, wherein the region includes one or more of a traffic lane, a parking space, or a driveway.

[0114] S: The operations further include determining that the vehicle is located within a threshold distance of the area, determining a first speed of the vehicle at the location, and determining a difference between the first speed of the vehicle and one or more of a second speed of another vehicle or a speed limit associated with the location, the machine learning model being further trained to predict vehicle behavior based at least in part on environmental features, and one or more of the location, the first speed, and the difference are input into the machine learning model.

[0115] T: The one or more computer-readable media of clause M, wherein the features are indicative of lights associated with the vehicle, and the machine learning model includes a classifier trained to classify a reversing state of the vehicle into one or more of a reversing state, a non-reversing state, or a reversing attempted state.

[0116] Although the above example clauses are described with respect to one particular implementation, in the context of this document, it should be understood that the contents of the example clauses can also be implemented via methods, devices, systems, computer-readable media, and / or other implementations. In addition, any of the examples A-T can be implemented alone or in combination with any other of the examples A-T.

[0117] conclusion One or more examples of the technology described herein have been described, and various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0118] In describing the examples, reference is made to the accompanying drawings, which form a part hereof, which show, by way of illustration, specific examples of the claimed subject matter. It should be understood that other examples may be utilized and that modifications or variations, such as structural changes, may be made. Such examples, modifications or variations do not necessarily depart from the intended scope of the claimed subject matter. Although individual examples are described herein as having certain features or components, the features and components of the individual examples may be combined and utilized together. Although operations herein may be represented in a certain order, in some cases, the order may be changed such that certain inputs are provided at different times or in a different order without changing the functionality of the systems and methods described. The procedures disclosed may also be performed in different orders. In addition, the various calculations herein need not be performed in the order disclosed, and other examples utilizing alternative orders of calculations may be readily implemented. In addition to rearrangements, calculations may also be decomposed into sub-calculations with the same results.

Claims

1. 1. A system including one or more processors and one or more computer-readable media storing instructions executable by the one or more processors, comprising: The instructions, when executed, cause the system to: Receiving sensor data captured by sensors in the environment; determining that the sensor data is indicative of a vehicle; determining a feature associated with the vehicle, the feature being of a feature type indicating whether the vehicle is associated with a reverse state; and inputting the features into a machine learning model trained to predict vehicle behavior based at least in part on the feature type, the machine learning model being trained using training image data that has been annotated to indicate a light status at a time that training image data is captured and a direction of training vehicle movement for the vehicle over an amount of time proximate to the time that the training image data was captured; receiving an indication from the machine learning model that the vehicle is associated with the reverse state, the indication that the vehicle is associated with the reverse state including an angle of the vehicle relative to a second feature of the environment determined from map data; predicting a trajectory associated with the vehicle based at least in part on the indication that the vehicle is associated with the reverse condition; and determining a maneuver for the autonomous vehicle to perform based at least in part on the reverse condition. system.

2. the vehicle is a first vehicle, The operation includes: Determining a first velocity of the first vehicle; determining a first difference between the first speed of the first vehicle and a speed limit at a location of the first vehicle in the environment or a second difference between the first speed of the first vehicle and a second speed of a second vehicle in the environment; Further comprising: the machine learning model is further trained to predict the vehicle behavior based at least in part on a difference in vehicle speed, and one or more of the first difference or the second difference are input to the machine learning model. The system of claim 1 .

3. The operation includes: determining a width associated with the vehicle; relating said width to a length of said track; Further comprising: the operation is associated with the autonomous vehicle traversing at least the width along the length of the trajectory.

3. A system according to claim 1 or 2.

4. the maneuvering includes maintaining a position of the autonomous vehicle for a duration associated with the vehicle executing the trajectory. The system of claim 3.

5. The feature is a first feature, The operation includes: determining an angle of the vehicle relative to a second characteristic of the environment; the machine learning model is further trained to predict the vehicle behavior based at least in part on environmental features, and one or more of the angle or the second feature are input to the machine learning model. A system according to any one of claims 1 to 4.

6. The operation includes: determining an area nearest to the vehicle that the vehicle is permitted to enter in reverse, the predicting of the trajectory being further based on a position of the area relative to the vehicle; determining that the vehicle is located within a threshold distance of the region; determining a first velocity of the vehicle at the location; determining a difference between the first speed of the vehicle and one or more of a second speed of another vehicle or a speed limit associated with the location; Further comprising: the machine learning model is further trained to predict vehicle behavior based at least in part on environmental features, and one or more of the position, the first velocity, and the difference are input into the machine learning model. A system according to any one of claims 1 to 5.

7. Receiving sensor data captured by sensors in the environment; determining that the sensor data is indicative of a vehicle; determining a feature associated with the vehicle, the feature being of a feature type indicating whether the vehicle is associated with a reverse state; and inputting the features into a machine learning model trained to predict vehicle behavior based at least in part on the feature type, the machine learning model being trained using training image data that has been annotated to indicate a light status at a time that training image data is captured and a direction of training vehicle movement for the vehicle over an amount of time proximate to the time that the training image data was captured; receiving an indication from the machine learning model that the vehicle is associated with the reverse condition; and predicting a trajectory associated with the vehicle based at least in part on the indication that the vehicle is associated with the reverse condition; and A method comprising:

8. the indication that the vehicle is associated with the reverse condition includes an angle of the vehicle relative to a second characteristic of the environment determined from map data; and predicting the trajectory of the vehicle is further based at least in part on the angle of the vehicle relative to the second characteristic. The method according to claim 7.

9. predicting the trajectory of the vehicle includes determining that the vehicle is preparing to parallel park; The method comprises: controlling the autonomous vehicle to maintain a parked position for a duration for the vehicle to parallel park or to enter a traffic lane to traverse around the vehicle in the environment.

9. The method according to claim 7 or 8.

10. determining an area nearest the vehicle into which the vehicle is permitted to enter in reverse; predicting the trajectory is further based on a position of the region relative to the vehicle. The method according to any one of claims 7 to 9.

11. the area includes one or more of a lane of traffic, a parking space, or a driveway; The method of claim 10.

12. the position is a first position, The method comprises: determining that the vehicle is at a second location within a threshold distance of the region; determining a first velocity of the vehicle at the second location; determining a difference between the first speed of the vehicle and one or more of a second speed of another vehicle or a speed limit associated with the second location; Further comprising: the machine learning model is further trained to predict a vehicle behavior based at least in part on environmental features, and one or more of the first position, the second position, the first velocity, and the difference are input to the machine learning model.

12. The method according to claim 10 or 11.

13. the feature is indicative of a light associated with the vehicle; the machine learning model includes a classifier trained to classify a reversing state of the vehicle into one or more of a reverse state, a non-reverse state, or a reverse attempted state. The method according to any one of claims 7 to 12.

14. A computer program comprising instructions which, when executed on a computer, implements the method according to any one of claims 7 to 13.

15. 1. A system including one or more processors and one or more computer-readable media storing instructions executable by the one or more processors, comprising: The instructions, when executed, cause the system to: Receiving sensor data captured by sensors in the environment; determining that the sensor data represents a first vehicle; determining a feature associated with the first vehicle, the feature being of a feature type indicating whether the first vehicle is associated with a reverse condition; and Determining a first velocity of the first vehicle; determining a first difference between the first speed of the first vehicle and a speed limit at a location of the first vehicle in the environment or a second difference between the first speed of the first vehicle and a second speed of a second vehicle in the environment; inputting one or more of the features, the first difference, and the second difference into a machine learning model trained to predict a vehicle behavior based at least in part on the feature type, the machine learning model being trained to predict the vehicle behavior based at least in part on a difference in vehicle speeds; receiving an indication from the machine learning model that the vehicle is associated with the reverse condition; and predicting a trajectory associated with the vehicle based at least in part on the indication that the vehicle is associated with the reverse condition; and determining a maneuver for the autonomous vehicle to perform based at least in part on the reverse condition. system.

16. 1. A system including one or more processors and one or more computer-readable media storing instructions executable by the one or more processors, comprising: The instructions, when executed, cause the system to: Receiving sensor data captured by sensors in the environment; determining that the sensor data is indicative of a vehicle; determining a first characteristic associated with the vehicle, the first characteristic being of a characteristic type indicating whether the vehicle is associated with a reverse state; and determining an angle of the vehicle relative to a second characteristic of the environment; and inputting at least one of the first feature, the angle, and the second feature into a machine learning model trained to predict a vehicle behavior based at least in part on the feature type, the machine learning model being trained to predict the vehicle behavior based at least in part on environmental features; and receiving an indication from the machine learning model that the vehicle is associated with the reverse condition; and predicting a trajectory associated with the vehicle based at least in part on the indication that the vehicle is associated with the reverse condition; and determining a maneuver for the autonomous vehicle to perform based at least in part on the reverse condition. system.

17. 1. A system including one or more processors and one or more computer-readable media storing instructions executable by the one or more processors, comprising: The instructions, when executed, cause the system to: Receiving sensor data captured by sensors in the environment; determining that the sensor data is indicative of a vehicle; determining a feature associated with the vehicle, the feature being of a feature type indicating whether the vehicle is associated with a reverse state; and determining an area nearest the vehicle into which the vehicle is permitted to enter in reverse; determining that the vehicle is located within a threshold distance of the region; determining a first velocity of the vehicle at the location; determining a difference between the first speed of the vehicle and one or more of a second speed of another vehicle or a speed limit associated with the location; inputting at least one of the features, the position, the first velocity, and the difference into a machine learning model trained to predict a vehicle behavior based at least in part on the feature type, the machine learning model being trained to predict the vehicle behavior based at least in part on environmental features; receiving an indication from the machine learning model that the vehicle is associated with the reverse condition; and predicting a trajectory associated with the vehicle based at least in part on at least one of the indication that the vehicle is associated with the reverse state or a location of the region relative to the vehicle; determining a maneuver for the autonomous vehicle to perform based at least in part on the reverse condition. system.

18. Receiving sensor data captured by sensors in the environment; determining that the sensor data is indicative of a vehicle; determining a feature associated with the vehicle, the feature being of a feature type indicating whether the vehicle is associated with a reverse state; and determining an area nearest the vehicle into which the vehicle is permitted to enter in reverse; inputting the features into a machine learning model trained to predict vehicle behavior based at least in part on the feature type; receiving an indication from the machine learning model that the vehicle is associated with the reverse state, the indication that the vehicle is associated with the reverse state including an angle of the vehicle relative to a second feature of the environment determined from map data; predicting a trajectory associated with the vehicle based at least in part on at least one of the indication that the vehicle is associated with the reverse state or a location of the region relative to the vehicle; A method comprising:

19. Receiving sensor data captured by sensors in the environment; determining that the sensor data is indicative of a vehicle; determining a feature associated with the vehicle, the feature being of a feature type indicating whether the vehicle is associated with a reverse state; and determining an area nearest the vehicle into which the vehicle is permitted to reverse, the area including one or more of a lane of traffic, a parking space, or a driveway; inputting the features into a machine learning model trained to predict vehicle behavior based at least in part on the feature type; receiving an indication from the machine learning model that the vehicle is associated with the reverse state, the indication that the vehicle is associated with the reverse state including an angle of the vehicle relative to a second feature of the environment determined from map data; predicting a trajectory associated with the vehicle based at least in part on at least one of the indication that the vehicle is associated with the reverse state or a location of the region relative to the vehicle; A method comprising:

20. Receiving sensor data captured by sensors in the environment; determining that the sensor data is indicative of a vehicle; determining a feature associated with the vehicle, the feature being of a feature type indicating whether the vehicle is associated with a reverse state; and determining an area nearest the vehicle into which the vehicle is permitted to enter in reverse; inputting the features into a machine learning model trained to predict vehicle behavior based at least in part on the feature type; receiving an indication from the machine learning model that the vehicle is associated with the reverse state, the indication that the vehicle is associated with the reverse state including an angle of the vehicle relative to a second feature of the environment determined from map data; predicting a trajectory associated with the vehicle based at least in part on at least one of the indication that the vehicle is associated with the reverse state or a first location of the region associated with the vehicle; determining that the vehicle is at a second location within a threshold distance of the region; determining a first velocity of the vehicle at the second location; determining a difference between the first speed of the vehicle and one or more of a second speed of another vehicle or a speed limit associated with the second location; Including, The method, wherein the machine learning model is further trained to predict vehicle behavior based at least in part on environmental features, and one or more of the first position, the second position, the first velocity, and the difference are input into the machine learning model.

21. Receiving sensor data captured by sensors in the environment; determining that the sensor data is indicative of a vehicle; determining a feature associated with the vehicle, the feature being of a feature type indicating whether the vehicle is associated with a reverse condition, the feature being indicative of a light associated with the vehicle; inputting the features into a machine learning model trained to predict a vehicle behavior based at least in part on the feature type, the machine learning model including a classifier trained to classify a reversing state of the vehicle into one or more of a reverse state, a non-reverse state, or a reverse attempted state; receiving an indication from the machine learning model that the vehicle is associated with the reverse condition; and predicting a trajectory associated with the vehicle based at least in part on the indication that the vehicle is associated with the reverse condition; and A method comprising:

22. Receiving sensor data captured by sensors in the environment; determining that the sensor data is indicative of a vehicle; determining a feature associated with the vehicle, the feature being of a feature type indicating whether the vehicle is associated with a reverse state; and inputting the features into a machine learning model trained to predict vehicle behavior based at least in part on the feature type; receiving an indication from the machine learning model that the vehicle is associated with the reverse state, the indication that the vehicle is associated with the reverse state including an angle of the vehicle relative to a second feature of the environment determined from map data; predicting a trajectory associated with the vehicle based at least in part on at least one of the indication that the vehicle is associated with the reverse condition or the angle of the vehicle associated with the second characteristic; A method comprising:

Images