Right angle interrupt training

Abstract

Techniques related to detecting when a vehicle is about to enter a lane region ahead of another vehicle are described. In an example, a computing device onboard a first vehicle can receive sensor data related to the environment of the first vehicle. Based at least in part on attributes determined from the sensor data, the computing device can determine that a second vehicle near the first vehicle (e.g., by making a U-turn, performing an N-point turn, or exiting a parking lot or driveway) is predicted to enter the lane region ahead of the first vehicle from a different direction of travel. In an example, the computing device can determine instructions for controlling the first vehicle based at least in part on determining that the second vehicle is predicted to enter the lane region ahead of the first vehicle.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Patent Application No. 16 / 803,644, entitled "PERPENDICULAR CUT-IN TRAINING," filed February 27, 2020, and U.S. Patent Application No. 16 / 803,705, entitled "PERPENDICULAR CUT-IN DETECTION," filed February 27, 2020, each of which is incorporated by reference in its entirety. [Background technology]

[0002] In general, predictive systems utilize information associated with objects in an environment to infer future actions, such as trajectories, of those objects. Such information can then be used to determine, for example, how to control a vehicle in the environment. Summary of the Invention

[0003] The detailed description will now be described with reference to the accompanying figures, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different figures indicates similar or identical components or features. [Brief description of the drawings]

[0004] [Figure 1] FIG. 2 illustrates an example environment for predicting whether a vehicle will enter a lane region in front of another vehicle, as described herein. [Diagram 2] FIG. 2 illustrates another example environment for predicting whether a vehicle will enter a lane region in front of another vehicle as described herein. [Diagram 3] FIG. 2 illustrates another example environment for predicting whether a vehicle will enter a lane region in front of another vehicle as described herein. [Figure 4] FIG. 2 illustrates another example environment for predicting whether a vehicle will enter a lane region in front of another vehicle as described herein. [Diagram 5] FIG. 2 illustrates another example environment for predicting whether a vehicle will enter a lane region in front of another vehicle as described herein. [Figure 6] FIG. 1 is a block diagram illustrating an example system for performing the techniques described herein. [Figure 7] FIG. 2 illustrates an example process for training a model to predict whether a vehicle will enter a lane region in front of another vehicle, as described herein. [Figure 8] FIG. 2 illustrates an example process described herein for predicting whether a vehicle will enter a lane region in front of another vehicle and controlling the other vehicle based on such prediction. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] The techniques described herein are directed to training and using models to predict certain behaviors of objects (e.g., pedestrians, animals, bicyclists, other vehicles, etc.) in a vehicle's environment. For example, a model can be trained using machine learning to predict when a vehicle that is proximate and substantially perpendicular to another vehicle, such as an autonomous vehicle, will enter (e.g., "cut in") into a lane region in front of the autonomous vehicle, such as by exiting a driveway, exiting a parking lot at a right angle, making a U-turn, etc. In at least one example, the model can be trained based on training data previously collected by one or more vehicles in the environment. In at least one example, the model can be provided to the autonomous vehicle and utilized by a computing device onboard the autonomous vehicle to predict how objects (e.g., other vehicles) that are proximate and substantially perpendicular to the autonomous vehicle are likely to behave.

[0006] For example, a computing device onboard an autonomous vehicle can utilize the model to predict when a vehicle that is close to and substantially perpendicular to the autonomous vehicle will enter (e.g., "cut in") a lane region ahead of the autonomous vehicle, such as by exiting a driveway, exiting a parking lot at a right angle, making a U-turn, etc. These scenarios present unique challenges, particularly in that such actors (other vehicles performing these maneuvers) may start from unstructured parts of the map (e.g., parts without lane markers, such as driveways or parking stalls), plus the fact that the actor's movements may include transitions from multi-point turns that may not follow the rules of the road to structured and / or complex movements. A computing device onboard an autonomous vehicle can utilize such predictions to determine how to navigate the autonomous vehicle in an environment. That is, a computing device onboard an autonomous vehicle can adapt driving maneuvers based on predictions output by the model. Thus, the autonomous vehicle can utilize the model to (i) predict when a vehicle that is close to and substantially perpendicular to the autonomous vehicle will cut in, and (ii) determine a trajectory to navigate the autonomous vehicle to accommodate the other vehicle (e.g., slowing down the autonomous vehicle thereby increasing the distance between the autonomous vehicle and the other vehicle, causing the autonomous vehicle to yield to the other vehicle and / or stopping the autonomous vehicle, causing the autonomous vehicle to perform a lane change maneuver, causing the autonomous vehicle to safely go around the other vehicle, and / or causing the autonomous vehicle to perform any other combination of maneuvers that the autonomous vehicle is capable of performing).

[0007] The techniques described herein are directed to training models, for example using machine learning, to enable a vehicle, such as an autonomous vehicle, to predict the behavior of objects in its environment. The techniques described herein can utilize noisy sensor data, and more specifically, noisy prediction data (e.g., processed sensor data), to predict the behavior of objects in an environment associated with the vehicle more accurately than with existing prediction techniques. Furthermore, as described above, with respect to vehicles that are perpendicular or substantially perpendicular to the autonomous vehicle, such vehicles may not be aligned with the road network and / or may not follow the rules of the road that the autonomous vehicle is trained to perceive and understand. Thus, existing prediction techniques may be insufficient to detect and safely bypass such vehicles. That is, the techniques described herein provide a technical improvement over existing prediction technologies. In addition to improving the accuracy with which sensor data can be used to predict the behavior of objects in the vehicle's environment, the techniques described herein can reduce false positives and improve safety outcomes.

[0008] The techniques described herein can be implemented in several ways. Exemplary implementations are provided below with reference to the following figures. Although exemplary implementations are discussed in the context of an autonomous vehicle, the methods, apparatus, and components described herein can be applied to a variety of components (e.g., sensor components or robotic platforms) and are not limited to autonomous vehicles. In one example, the techniques described herein can be utilized in driver-controlled vehicles, where such components can provide indications to the vehicle driver about whether it is safe to perform various operations. In another example, these techniques can be utilized in an aviation or nautical context, or in any component that includes an object or entity that may be associated with behaviors that are unknown to the component. In some examples, these techniques can be utilized in contexts other than autonomous vehicles and / or perception components. Additionally, the techniques described herein can be used with real (e.g., captured using sensors) data, simulated (e.g., generated by a simulator) data, or any combination of the two.

[0009] FIG. 1 illustrates an example environment 100 for predicting whether a vehicle will enter a lane region in front of another vehicle, as described herein.

[0010] In the exemplary environment 100, the vehicle 102 may be located in a travel lane 104 of a driving surface 106. In at least one example, the vehicle 102 may be an autonomous vehicle configured to operate in accordance with the Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions throughout the entire journey without a driver (or passenger) being expected to control the vehicle at any time. In such an example, the vehicle 102 may be unmanned, as the vehicle 102 may be configured to control all functions from starting to stopping, including all parking functions. This is by way of example only, and the components and methods described herein may be incorporated into any ground, air, or water vehicle, including vehicles ranging from those that must be manually controlled by a driver at all times to those that are partially or fully autonomously controlled. That is, in the example shown, the vehicle 102 is an autonomous vehicle, but the vehicle 102 may be any other type of vehicle.

[0011] In at least one example, the travel lane 104 may be associated with a lane region and the travel surface 106 may be associated with a road. In at least one example, another vehicle 108 may be detected in the environment 100. In some examples, the other vehicle 108 may be close enough to the vehicle 102 that the other vehicle 108 is within a threshold distance of the vehicle 102. In some examples, the other vehicle 108 may be associated with a different travel direction than the vehicle 102. The travel direction of the vehicle 102 is indicated by an arrow at the end of the track 110 on which the vehicle 102 is traveling. In some examples, the other vehicle 108 may be associated with a travel direction that is opposite the travel direction of the vehicle 102. In other examples, the travel direction may differ from the other travel direction by being offset from the other travel direction by more than a threshold.

[0012] In some examples, another vehicle 108 can be positioned at an angle relative to the vehicle 102. In at least one example, the angle can be within a specified offset of 90 degrees, whereby the another vehicle 108 is substantially perpendicular to the vehicle 102. That is, for purposes of this discussion, a vehicle is "substantially perpendicular" to another vehicle if the angle between the two vehicles is within a specified offset of 90 degrees. As a non-limiting example, the specified offset can be 50 degrees, whereby the vehicle 102 and the another vehicle 108 (or any two vehicles) can be "substantially perpendicular" as long as the angle between the two is between 40 degrees and 140 degrees. In at least one example, the specified offset can be learned using machine learning techniques described herein. In some examples, the specified offset can be set manually, can be based on map data (e.g., can be location dependent), etc.

[0013] 1, another vehicle 108 is positioned substantially perpendicular to vehicle 102. As a non-limiting example, another vehicle 108 may be exiting a parking lot (e.g., a parking lot at a substantially perpendicular position) or parking stall, an alley, a driveway, or any other location that is substantially perpendicular to vehicle 102. In some examples, such parking lots or parking stalls, alleys, driveways, etc. may be annotated in a map associated with environment 100. In other examples, such parking lots or parking stalls, alleys, driveways, etc. may not be annotated in the map.

[0014] In at least one example, the vehicle 102 may include one or more computing devices, which may be onboard the vehicle 102. In at least one example, the computing devices may include components for controlling the vehicle 102. For example, in at least one example, a perception component may perform object detection, segmentation, and / or classification based at least in part on sensor data received from a sensor component of the vehicle 102. In at least one example, a prediction component may receive sensor data from the sensor component, map data associated with a map (e.g., of the environment 100), and / or perception data (e.g., processed sensor data) output from the perception component, and may output predictions associated with one or more objects in the environment 100 of the vehicle 102. In at least one example, a planning component may determine an output for use in controlling the vehicle 102 based at least in part on sensor data received from the sensor component, map data, and / or any decisions made by other components of the vehicle 102. Further details associated with such computing devices and associated components are described below with reference to FIG.

[0015] As described above, the vehicle 102 may be associated with one or more sensor components. In at least one example, the sensor components may include LIDAR sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., global positioning components (GPS), compasses, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), wheel encoders, microphones, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), time-of-flight (ToF) sensors, etc. Such sensor data may include, but is not limited to, LIDAR data, RADAR data, ultrasonic transducer data, sonar data, position data (e.g., global positioning component (GPS), compass, etc.), inertial data (e.g., inertial measurement unit data, accelerometer data, magnetometer data, gyroscope data, etc.), camera data (e.g., RGB data, IR data, intensity data, depth data, etc.), wheel encoder data, microphone data, environmental sensor data (e.g., temperature sensor data, humidity sensor data, light sensor data, pressure sensor data, etc.), ToF sensor data, etc.

[0016] In at least one example, the perception component can detect another vehicle 108, and the prediction component can predict whether the other vehicle 108 will enter a lane region of the travel lane 104 ahead of the vehicle 102. In some examples, the prediction component can utilize a model trained by a machine learning mechanism to output a prediction indicating whether the other vehicle 108 will enter a lane region of the travel lane 104 ahead of the vehicle 102. In at least one example, the prediction component can analyze features (also referred to as attributes, characteristics, parameters, data, etc.) associated with the environment 100 to determine whether the other vehicle 108 is likely to enter the lane region of the travel lane 104. Such features that can be associated with the characteristics of the environment 100 can include annotations in a map associated with the environment 100 indicating that the other vehicle 108 is located in a parking lot (e.g., a parking lot at a substantially right angle) or parking stall, an alley, a driveway, etc.

[0017] In at least one example, the prediction component may analyze features associated with the other vehicle 108 to determine whether the other vehicle 108 is likely to enter a lane region of the travel lane 104. For example, such features may be associated with the characteristics of the other vehicle 108, including a position of the other vehicle 108 relative to the vehicle 102 (e.g., direction of travel, angular offset, etc.), an instantaneous speed of the other vehicle 108, an indication of whether the other vehicle 108 has a driver, an indication of the direction the driver is looking (e.g., the driver's head position), an indication of whether a door associated with the other vehicle 108 is open or closed, an indication of whether the engine of the other vehicle 108 is running or off, a wheel angle associated with the wheels of the other vehicle 108 (e.g., the position of the driver's head), and the like. Further details for determining such are described in U.S. Patent Application No. 16 / 709,263, filed December 10, 2019, the entirety of which is incorporated herein by reference), which may include, but is not limited to, an indication of whether the brake lights of another vehicle 108 are turned on, an indication of whether the headlights of another vehicle 108 are turned on, an indication of whether the reverse lights of another vehicle 108 are turned on, an indication of whether the turn signals of another vehicle 108 are turned on, or a light state associated with another vehicle 108.

[0018] Further details associated with such models, and techniques for training such models, are described below with reference to FIG.

[0019] 1, if another vehicle 108 follows its associated trajectory 112, the other vehicle 108 will enter the lane region of the travel lane 104 ahead of the vehicle 102 (e.g., as the other vehicle 108 executes a right turn into the traffic flow). That is, if the other vehicle 108 follows its associated trajectory 112, the other vehicle 108 will execute a cut-in, which allows the planning component associated with the vehicle 102 to modify the trajectory 110 of the vehicle 102 to move the vehicle 102 taking into account the other vehicle 202. As described above, the planning component may cause the vehicle 102 to slow down and increase the distance between the vehicle 102 and another vehicle 108, cause the vehicle 102 to slow down and yield to the other vehicle 108, and / or to stop, cause the vehicle 102 to perform a lane change, cause the vehicle 102 to safely go around the other vehicle 108, and / or cause any other combination of maneuvers that the vehicle 102 is capable of performing.

[0020] FIG. 2 illustrates another example environment 200 for predicting whether a vehicle will enter a lane region in front of another vehicle as described herein.

[0021] The exemplary environment 200 includes a vehicle 102, which may be disposed in a travel lane 104 of a travel surface 106. In at least one example, the travel lane 104 may be associated with a lane region and the travel surface 106 may be associated with a road, as described above with reference to FIG. 1. In at least one example, another vehicle 202 may be detected within the environment 200. In some examples, the other vehicle 202 may be close enough to the vehicle 102 that the other vehicle 202 is within a threshold distance of the vehicle 102. Furthermore, in some examples, the other vehicle 202 may be associated with a different travel direction than the vehicle 102. The travel direction of the vehicle 102 is indicated by an arrow at the end of the track 110 on which the vehicle 102 is traveling. In some examples, the other vehicle 202 may be associated with a travel direction that is opposite to the travel direction of the vehicle 102. In some examples, the other vehicle 202 may be disposed at an angle relative to the vehicle 102. In at least one example, the angle can be within a specified offset of 90 degrees, such that the other vehicle 202 is substantially perpendicular to the vehicle 102 .

[0022] 2, another vehicle 202 is positioned substantially perpendicular to vehicle 102. As a non-limiting example, another vehicle 108 may be exiting a parking lot (e.g., a parking lot at a substantially perpendicular position) or parking stall, an alley, a driveway, or any other location that is substantially perpendicular to vehicle 102. In some examples, such parking lots or parking stalls, alleys, driveways, etc. may be annotated in a map associated with environment 200. In other examples, such parking lots or parking stalls, alleys, driveways, etc. may not be annotated in the map.

[0023] In at least one example, the perception component can detect another vehicle 202, and the prediction component can output a prediction indicating whether the other vehicle 202 will enter a lane region of the travel lane 104 ahead of the vehicle 102. In some examples, the prediction component can utilize a model trained by a machine learning mechanism to predict whether the other vehicle 202 will enter a lane region of the travel lane 104 ahead of the vehicle 102. In at least one example, the prediction component can analyze features associated with the environment 200 and / or the other vehicle 202, as described above, to determine whether the other vehicle 202 is likely to enter the lane region of the travel lane 104.

[0024] 2, if another vehicle 202 follows the trajectory 204 associated with it, the other vehicle 202 will enter the lane region of the travel lane 104 ahead of the vehicle 102 (e.g., as the other vehicle 202 executes a left turn into the traffic flow). That is, if the other vehicle 202 follows the trajectory 204 associated with it, the other vehicle 202 will execute a cut-in, whereby the planning component associated with the vehicle 102 can modify the trajectory 110 of the vehicle 102 to move the vehicle 102 to take into account the other vehicle 202. As described above, the planning component can cause the vehicle 102 to slow down to increase the distance between the vehicle 102 and the other vehicle 202, to slow down to give way to the other vehicle 202, and / or to stop, and / or to execute any other combination of maneuvers that the vehicle 102 is capable of executing.

[0025] 1 and 2 show examples of the technology described herein whereby another vehicle 108 and 202, respectively, are exiting an alley, a parking lot (e.g., a substantially perpendicular parking lot) or parking stall, a driveway, or other location that is substantially perpendicular to the vehicle 102 and / or the driving surface 106. In FIG. 1, the other vehicle 108 is located in the alley, a parking lot (e.g., a substantially perpendicular parking lot) or parking stall, a driveway, or other location on the same side of the driving surface 106 as the vehicle 102. That is, the other vehicle 108 is located substantially perpendicular to the driving lane 104 in which the vehicle 102 is located and on the same side of the driving surface 106 as the driving lane 104. In FIG. 2, the other vehicle 202 is located in the alley, a parking lot (e.g., a substantially perpendicular parking lot) or parking stall, a driveway, or other location on the opposite side of the driving surface 106 from the vehicle 102. That is, another vehicle 108 is positioned substantially perpendicular to the travel lane 104 in which the vehicle 102 is positioned and on the opposite side of the travel surface 106 from the travel lane 104 .

[0026] In some examples, another vehicle 108 or 202 may be positioned in front of the vehicle 102 and positioned substantially parallel (e.g., in the same direction) as the vehicle 102. In such examples, the other vehicle 108 or 202 may execute a left or right turn across the travel lane 104. The vehicle 102 may utilize the features described above to anticipate such a maneuver, and the planning component may determine how to control the vehicle 102 accordingly.

[0027] FIG. 3 illustrates another example environment 300 for predicting whether a vehicle will enter a lane region in front of another vehicle as described herein.

[0028] The exemplary environment 300 includes a vehicle 102, which may be disposed in a travel lane 104 of a travel surface 106. In at least one example, the travel lane 104 may be associated with a lane region and the travel surface 106 may be associated with a road, as described above with reference to FIG. 1. In at least one example, another vehicle 302 may be detected within the environment 300. In some examples, the other vehicle 302 may be close enough to the vehicle 102 that the other vehicle 302 is within a threshold distance of the vehicle 102. Furthermore, in some examples, the other vehicle 302 may be associated with a different travel direction than the vehicle 102. The travel direction of the vehicle 102 is indicated by an arrow at the end of the track 110 on which the vehicle 102 is traveling. In some examples, the other vehicle 302 may be associated with a travel direction that is opposite the travel direction of the vehicle 102 (e.g., as shown in FIG. 3). In some examples, the other vehicle 302 may be positioned at an angle relative to the vehicle 102 .

[0029] In at least one example, the perception component can detect another vehicle 302, and the prediction component can predict whether the other vehicle 302 will enter a lane region of the travel lane 104 ahead of the vehicle 102. In some examples, the prediction component can utilize a model trained by a machine learning mechanism to output a prediction indicating whether the other vehicle 302 will enter a lane region of the travel lane 104 ahead of the vehicle 102. In at least one example, the prediction component can analyze features associated with the environment 300 and / or the other vehicle 302, as described above, to determine whether the other vehicle 302 is likely to enter the lane region of the travel lane 104.

[0030] 3, if another vehicle 302 follows the trajectory 304 associated with it, the other vehicle 302 will enter the lane region of the travel lane 104 in front of the vehicle 102 (e.g., as the other vehicle 302 performs a U-turn). That is, if another vehicle 302 follows the trajectory 304 associated with it, the other vehicle 302 will perform a cut-in, whereby the planning component associated with the vehicle 102 can modify the trajectory 110 of the vehicle 102 to move the vehicle 102 to take into account the other vehicle 302. As described above, the planning component can cause the vehicle 102 to slow down to increase the distance between the vehicle 102 and the other vehicle 302, to slow down to give way to the other vehicle 302, and / or to stop, and / or to perform any other combination of maneuvers that the vehicle 102 is capable of performing.

[0031] 3 shows another vehicle 302 performing a U-turn, in alternative examples, the other vehicle 302 may be performing a turn across the travel lane 104 (e.g., onto a driveway, street, alley, etc.). In such examples, the vehicle 102 may utilize the same features to determine whether the other vehicle 302 is likely to enter a lane region of the travel lane 104 ahead of the vehicle 102, and the planning component may determine how to control the vehicle 102 accordingly.

[0032] FIG. 4 illustrates another example environment 400 for predicting whether a vehicle will enter a lane region in front of another vehicle as described herein.

[0033] The exemplary environment 400 includes a vehicle 102, which may be located in a travel lane 104 of a travel surface 106. In at least one example, the travel lane 104 may be associated with a lane region and the travel surface 106 may be associated with a road, as described above with reference to FIG. 1. In at least one example, another vehicle 402 may be detected within the environment 400. In some examples, the other vehicle 402 may be close enough to the vehicle 102 that the other vehicle 402 is within a threshold distance of the vehicle 102. Furthermore, in some examples, the other vehicle 402 may be associated with a different travel direction than the vehicle 102. The travel direction of the vehicle 102 is indicated by an arrow at the end of the track 110 on which the vehicle 102 is traveling. In some examples, the other vehicle 402 may be associated with a travel direction that is opposite to the travel direction of the vehicle 102 (e.g., as shown in FIG. 4). In some examples, the other vehicle 402 may be positioned at an angle relative to the vehicle 102 .

[0034] In at least one example, the perception component can detect another vehicle 402, and the prediction component can predict whether the other vehicle 402 will enter a lane region of the travel lane 104 ahead of the vehicle 102. In some examples, the prediction component can leverage a model trained by a machine learning mechanism to output a prediction indicating whether the other vehicle 402 will enter a lane region of the travel lane 104 ahead of the vehicle 102. In at least one example, the prediction component can analyze features associated with the environment 400 and / or the other vehicle 402, as described above, to determine whether the other vehicle 402 is likely to enter the lane region of the travel lane 104.

[0035] 4, if another vehicle 402 follows the trajectory 404 associated with it, the other vehicle 402 will enter the lane region of the travel lane 104 in front of the vehicle 102 (e.g., when the other vehicle 402 performs a three-point turn or any N-point turn). That is, if the other vehicle 402 follows the trajectory 404 associated with it, the other vehicle 402 will perform a cut-in, whereby the planning component associated with the vehicle 102 can modify the trajectory 110 of the vehicle 102 to move the vehicle 102 to take the other vehicle 402 into account. As described above, the planning component can cause the vehicle 102 to slow down to increase the distance between the vehicle 102 and the other vehicle 402, to slow down to give way to the other vehicle 402, and / or to stop, and / or to perform any other combination of maneuvers that the vehicle 102 is capable of performing.

[0036] FIG. 5 illustrates another example environment 500 for predicting whether a vehicle will enter a lane region in front of another vehicle as described herein.

[0037] The exemplary environment 500 includes a vehicle 102, which may be disposed in a travel lane 104 of a travel surface 106. In at least one example, the travel lane 104 may be associated with a lane region and the travel surface 106 may be associated with a road, as described above with reference to FIG. 1. In at least one example, another vehicle 502 may be detected within the environment 500. In some examples, the other vehicle 502 may be close enough to the vehicle 102 that the other vehicle 502 is within a threshold distance of the vehicle 102. Furthermore, in some examples, the other vehicle 502 may be associated with a different travel direction than the vehicle 102. The travel direction of the vehicle 102 is indicated by an arrow at the end of the track 110 on which the vehicle 102 is traveling. In some examples, the other vehicle 502 may be associated with a travel direction that is opposite to the travel direction of the vehicle 102. In some examples, the other vehicle 502 may be disposed at an angle relative to the vehicle 102. As shown in FIG. 5, another vehicle 502 may be substantially perpendicular to the vehicle 102 .

[0038] In at least one example, the perception component can detect another vehicle 502, and the prediction component can predict whether the other vehicle 502 will enter a lane region of the travel lane 104 ahead of the vehicle 102. In some examples, the prediction component can leverage a model trained by a machine learning mechanism to output a prediction indicating whether the other vehicle 502 will enter a lane region of the travel lane 104 ahead of the vehicle 102. In at least one example, the prediction component can analyze features associated with the environment 500 and / or the other vehicle 502, as described above, to determine whether the other vehicle 502 is likely to enter the lane region of the travel lane 104.

[0039] 5, if another vehicle 502 follows the trajectory 504 associated with it, the other vehicle 502 will enter the lane region of the travel lane 104 ahead of the vehicle 102 (e.g., when the other vehicle 502 reverses from its current position and travels in the same travel direction as the vehicle 102). That is, if the other vehicle 502 follows the trajectory 504 associated with it, the other vehicle 502 will perform a cut-in, which allows the planning component associated with the vehicle 102 to modify the trajectory 110 of the vehicle 102 to move the vehicle 102 taking into account the other vehicle 502. As described above, the planning component may cause the vehicle 102 to slow down and increase the distance between the vehicle 102 and another vehicle 502, cause the vehicle 102 to change lanes, cause the vehicle 102 to slow down and yield to another vehicle 502 and / or stop, cause the vehicle 102 to safely go around the other vehicle 502, and / or perform any other combination of maneuvers that the vehicle 102 is capable of performing.

[0040] Although FIG. 5 shows another vehicle 502 backing out of a driveway, parking lot, alley, etc. on the same side of the riding surface 106 as the vehicle 102, the techniques described herein are also applicable in relation to another vehicle 502 backing out of a driveway, parking lot, alley, etc. on the opposite side of the riding surface 106 as the vehicle 102.

[0041] 1-5 illustrate various examples of other vehicles in close proximity to the vehicle 102 that are entering or are otherwise performing a "cut-in maneuver" as described herein. Such examples include left turns, right turns, U-turns, three-point turns, and the like. Further or alternative examples are within the scope of this disclosure. For example, the techniques described herein can be used to determine how to control the vehicle 102 when another vehicle is performing a K-turn, any N-point turn, and the like, or when another vehicle is turning to weave in and out of traffic. That is, the techniques described herein are not limited to the examples described in FIGS. 1-5 and may be applicable to any example in which another vehicle in close proximity to the vehicle 102 and associated with the same or different travel direction as the vehicle 102 is entering or is expected to enter a lane area of ​​the travel lane 104 in front of the vehicle 102. Additionally, the techniques described herein are applicable to scenarios in which another vehicle is entering a lane region from a location on the road (e.g., on the driving surface 106) or off the road (e.g., not on the driving surface 106).

[0042] FIG. 6 is a block diagram illustrating an example system 600 for performing the techniques described herein. In at least one example, a vehicle 602 can include one or more vehicle computing devices 604, one or more sensor components 606, one or more emitters 608, one or more communication connections 610, at least one direct connection 612, and one or more driving systems 614. In at least one example, the vehicle 602 can correspond to the vehicle 102 described above with reference to FIG. 1. As described above with reference to FIG. 1, in the example shown, the vehicle 602 is an autonomous vehicle, but the vehicle 602 can be any other type of vehicle. Although only a single vehicle 602 is shown in FIG. 6, in a practical application, the example system 600 can include multiple vehicles, which in some examples can include a fleet of vehicles.

[0043] The vehicle computing device 604 may include a processor 616 and a memory 618 communicatively coupled to the processor 616. In the illustrated example, the memory 618 of the vehicle computing device 604 stores a localization component 620, a perception component 622, a prediction component 624, a planning component 626, and one or more system controllers 628. In addition, the memory 618 may include a storage 630, which may store maps, models, previous outputs, and the like. The map may be any number of data structures that may provide information about the environment, such as, but not limited to, topology (such as junctions, lanes, merge zones, and the like), streets, mountains, roads, terrain, and the environment in general. The map may be associated with a real environment or a simulated environment. The model may include a machine-learned model, as described below. In some examples, the storage 630 may store previous outputs.

[0044] In at least one example, the localization component 620 can determine an attitude (position and orientation) of the vehicle 602 relative to a local and / or global map based at least in part on sensor data received from the sensor component 606 and / or map data associated with the map (e.g., of the map). In at least one example, the localization component 620 can include or be associated with a calibration component that can substantially simultaneously perform operations for calibrating (determining various intrinsic and extrinsic parameters associated with any one or more of the sensor components 606), locating, and mapping.

[0045] In at least one example, the perception component 622 can perform object detection, segmentation, and / or classification based at least in part on the sensor data received from the sensor component 606. In at least one example, the perception component 622 can receive raw sensor data (e.g., from the sensor component 606). In at least one example, the perception component 622 can receive the sensor data and can utilize one or more processing algorithms to perform object detection, segmentation, and / or classification for objects identified in the sensor data. In some examples, the perception component 622 can associate a boundary region (or otherwise instance segmentation) with the identified object and can associate a confidence score associated with the classification of the identified object with the identified object. In some examples, the object can be colored based on its perceived class when rendered via a display.

[0046] The prediction component 624 may receive sensor data from the sensor component 606, map data associated with a map (e.g., of the map may be in storage 630), and / or perception data (e.g., processed sensor data) output from the perception component 622, and may output a prediction associated with one or more objects in the environment of the vehicle 602. In at least one example, the planning component 626 may determine an output for use in controlling the vehicle 602 based at least in part on the sensor data received from the sensor component 606, the map data, and / or any decisions made by other components of the vehicle 602.

[0047] Further details of localization, perception, prediction, and / or planning components that may be used may be found in U.S. Pat. No. 9,612,123, issued Apr. 4, 2017, and U.S. Pat. No. 10,353,390, issued Jul. 16, 2019, the entire contents of both of which are incorporated herein by reference. In some examples (e.g., where the vehicle 602 is not an autonomous vehicle), one or more of the aforementioned components may be omitted from the vehicle 602. Although the components described above are shown as "on board" the vehicle 602, in other embodiments, the components may be located and / or accessible remotely to the vehicle 602. Additionally, although components are described above as "components," such components may include one or more components that may be part of a system to perform the operations attributed to each of those components.

[0048] In at least one example, the localization component 620, the perception component 622, the prediction component 624, and / or the planning component 626 can process the sensor data as described above and can transmit their outputs to the computing device 634 via the network 632. In at least one example, the localization component 620, the perception component 622, the prediction component 624, and / or the planning component 626 can transmit their outputs to the computing device 634 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0049] In at least one example, the vehicle computing device 604 may include one or more system controllers 628 that may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 602. These system controllers 628 may communicate with and / or control corresponding systems of the driving system 614 and / or other systems of the vehicle 602.

[0050] In at least one example, the sensor component 606 can include LIDAR sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), wheel encoders, audio sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ToF sensors, etc. The sensor component 606 can provide input to the vehicle computing device 604. In some examples, the sensor component 606 can pre-process at least some of the sensor data before transmitting them to the vehicle computing device 604. In at least one example, the sensor component 606 can transmit the sensor data over the network 632 to the computing device 634 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0051] The vehicle 602 may also include one or more emitters 608 for emitting light and / or sound, as described above. The emitters 608 in this example include interior audio and visual emitters for communicating with occupants of the vehicle 602. By way of example and not limitation, the interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.). The emitters 608 in this example also include exterior emitters. By way of example and not limitation, the exterior emitters in this example include light emitters (e.g., indicator lights, signs, light arrays, etc.) for visually communicating with pedestrians, other drivers, other nearby vehicles, etc., one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians, other drivers, other nearby vehicles, etc. In at least one example, emitters 608 may be positioned at various locations around the exterior and / or interior of vehicle 602.

[0052] Vehicle 602 may also include communications connections 610 that enable communication between vehicle 602 and other local or remote computing devices. For example, communications connections 610 may facilitate communication with other local computing devices on vehicle 602 and / or with driving system 614. Communications connections 610 may also enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Communications connections 610 also enable vehicle 602 to communicate with remote teleoperated computing devices or other remote services.

[0053] The communication connections 610 may include physical and / or logical interfaces for connecting the vehicle computing device 604 to another computing device or network, such as network 632. For example, the communication connections 610 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as BLUETOOTH, or any suitable wired or wireless communication protocol that enables each computing device to interface with other computing devices.

[0054] The direct connection 612 may directly connect the driving system 614 to other systems of the vehicle 602 .

[0055] In at least one example, the vehicle 602 may include a driving system 614. In some examples, the vehicle 602 may have a single driving system 614. In at least one example, if the vehicle 602 has multiple driving systems 614, the individual driving systems 614 may be located at opposite ends of the vehicle 602 (e.g., the front and the rear, etc.). In at least one example, the driving system 614 may include sensor components for sensing conditions around the driving system 614 and / or the vehicle 602. By way of example and not limitation, the sensor components may include wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the driving system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the position and acceleration of the driving system, cameras or other imaging sensors, ultrasonic sensors for acoustically detecting objects in the surroundings of the driving system, LIDAR sensors, RADAR sensors, etc. Some sensors, such as the wheel encoders, may be unique to the driving system 614. In some cases, sensor components on the driving system 614 may overlap or complement corresponding systems in the vehicle 602 (e.g., the sensor component 606).

[0056] Driving system 614 can include many of the vehicle systems including a high voltage battery, a motor for propelling vehicle 602, an inverter for converting direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking forces to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights for illuminating the exterior surroundings of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, an on-board charging system, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, driving system 614 can include a driving system controller, which can receive and pre-process data from the sensor components and control the operation of various vehicle systems. In some examples, the driving system controller can include a processor and a memory communicatively coupled to the processor. The memory may store one or more components for performing various functionality of the driving system 614. In addition, the driving system 614 also includes communication connections that enable each driving system to communicate with other local or remote computing devices.

[0057] 6, the vehicle computing device 604, the sensor components 606, the emitters 608, and the communication connections 610 are shown on-board the vehicle 602. However, in some examples, the vehicle computing device 604, the sensor components 606, the emitters 608, and the communication connections 610 may be implemented outside the actual vehicle (i.e., not on-board the vehicle 602).

[0058] As described above, the vehicle 602 can transmit sensor data over the network 632 to the computing device 634. In some examples, the vehicle 602 can transmit raw sensor data to the computing device 634. In other examples, the vehicle 602 can transmit processed sensor data and / or a representation of the sensor data (e.g., data output from the localization component 620, the perception component 622, the prediction component 624, and / or the planning component 626) to the computing device 634. In some examples, the vehicle 602 can transmit the sensor data to the computing device 634 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0059] Computing device 634 can receive sensor data (raw or processed) from vehicle 602 and / or other data collection devices, as well as data from one or more third party sources and / or systems. In at least one example, computing device 634 can include a processor 636 and a memory 638 communicatively coupled to processor 636. In the illustrated example, memory 638 of computing device 634 stores training component 640, which can include annotation component 642, map storage 644 (e.g., storing one or more maps), training data storage 646 (e.g., storing training data accessible to training component 640), and model storage 648 (e.g., models output by training component 640). In some examples, one or more of these systems and / or storage repositories can be associated with vehicle 602 or other computing devices associated with system 600 instead of or in addition to being associated with memory 638 of computing device 634.

[0060] In at least one example, the training component 640 can train a model that can be used for various operations described herein. That is, the training component 640 can train a model that can be used to predict whether a first vehicle will enter a travel lane in front of another vehicle (e.g., vehicle 602). In at least one example, the annotation component 642 can receive or otherwise determine annotated data based at least in part on sensor data received from one or more vehicles and / or log data associated with the sensor data, as described below. Details associated with training a model, as used herein, are described below with reference to FIG. 7. In at least one example, the resulting model can be stored in the model storage 648 and / or the storage 630 on the vehicle 602 and can be accessed in near real-time by one or more components of the vehicle computing device 604.

[0061] The processor 616 of the vehicle 602 and the processor 636 of the computing device 634 may be any suitable processor capable of processing data and executing instructions to perform the operations described herein. By way of example and not limitation, the processors 616 and 636 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts the electronic data into other electronic data that may be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to execute encoded instructions.

[0062] The memories 618 and 638 are examples of non-transitory computer readable media. The memories 618 and 638 can store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods and functions attributed to the various systems described herein. In various embodiments, the memories can be implemented using any suitable memory technology, such as Static Random Receive Memory (SRAM), Synchronous Dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, of which the ones shown in the accompanying figures are merely examples relevant to the discussion herein.

[0063] 6 is depicted as a distributed system, it should be noted that in some examples components of the vehicle 602 may be associated with the computing device 634 and / or components of the computing device 634 may be associated with the vehicle 602. That is, the vehicle 602 may perform one or more of the functions associated with the computing device 634, and vice versa.

[0064] Figures 7 and 8 are flow charts illustrating example processes that include the techniques described herein. The processes illustrated in Figures 7 and 8 are described with reference to the system 600 illustrated in Figure 6 for convenience and ease of understanding. However, the processes illustrated in Figures 7 and 8 are not limited to being performed using the system 600. Moreover, the system 600 described herein is not limited to performing the processes illustrated in Figures 7 and 8.

[0065] Processes 700 and 800 are illustrated as a collection of blocks in a logic flow graph that represent sequences of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by a processor, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as limiting, and any number of the described blocks may be combined in any order and / or in parallel to implement a process. In some embodiments, one or more blocks of the processes may be omitted entirely. Moreover, processes 700 and 800 may be combined in whole or in part with each other or with other processes.

[0066] FIG. 7 shows an example process 700 for training a model to predict whether a vehicle will enter a lane region in front of another vehicle (e.g., perform a cutting-in maneuver), as described herein.

[0067] Block 702 illustrates receiving sensor data and / or log data associated with the sensor data. As described above, an individual vehicle, such as vehicle 602, may include a sensor component 606. In at least one example, the sensor component 606 may include a LIDAR sensor, a RADAR sensor, an ultrasonic transducer, a sonar sensor, a position sensor (e.g., GPS, compass, etc.), an inertial sensor (e.g., inertial measurement unit, accelerometer, magnetometer, gyroscope, etc.), a camera (e.g., RGB, IR, intensity, depth, etc.), a wheel encoder, an audio sensor, an environmental sensor (e.g., temperature sensor, humidity sensor, light sensor, pressure sensor, etc.), a ToF sensor, etc. The sensor component 606 may provide input to the vehicle computing device 604. In some examples, the sensor component 606 may pre-process at least some of the sensor data before transmitting the sensor data to the vehicle computing device 604. In at least one example, the sensor component 606 can transmit sensor data over the network 632 to the computing device 634 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0068] In at least one example, a plurality of vehicles (e.g., a fleet of vehicles) may transmit sensor data over the network 632 to the computing device 634. In some examples, such sensor data may be associated with a driving log (i.e., “log data”) that may indicate how individual vehicles of the plurality of vehicles moved over time in their respective environments. That is, such a driving log may be a driving log history that indicates the movement of individual vehicles of the plurality of vehicles over time. In some examples, such a driving log may include sensor data received over time (which may include the output of individual sensors at particular instances of time). In some examples, such a driving log may include outputs based on sensor data (e.g., processed sensor data) associated with a time period. That is, in some examples, the log data may include raw sensor data received from the sensor component 606 as well as one or more downstream outputs (e.g., perception output, prediction output, planner output, control output, etc.) related to individual messages during a driving mission of a vehicle, such as the vehicle 602.

[0069] In at least one example, the training component 640 of the computing device 634 can receive sensor data and / or associated driving logs from multiple vehicles.

[0070] Block 704 illustrates mining the sensor data and / or log data to detect events associated with a cut-in maneuver (e.g., any one or more of the maneuvers described in detail herein (whether or not the cut-in maneuver involves a second vehicle)), where the detected events are annotated in the sensor data and / or log data to generate annotated data. In at least one example, the training component 640 can mine (e.g., analyze) the sensor data and / or log data to detect when a vehicle performs a cut-in maneuver (e.g., the cut-in maneuver can be associated with a left turn (e.g., from a parking lot (e.g., a parking lot at a substantially right-angle position) or parking stall, alley, driveway, etc.), a right turn (e.g., from a parking lot (e.g., a parking lot at a substantially right-angle position) or parking stall, alley, driveway, etc.), a U-turn, a K-turn, an N-point turn, etc.). For example, the training component 640 can analyze the sensor data and / or logged data to detect when the vehicle is not aligned with the driving surface, e.g., when the vehicle is substantially perpendicular to the driving surface and subsequently aligned with or substantially parallel to the driving surface. In at least one example, the annotation component 642 can automatically annotate the sensor data and / or logged data to indicate the occurrence of a cut-in maneuver.

[0071] In at least one example, the annotation component 642 can determine a time period associated with a cut-in maneuver. That is, the annotation component 642 can access sensor data and / or log data leading to a detected event (e.g., associated with a cut-in maneuver) and can determine when the detected event begins and ends. For a moving vehicle (e.g., a vehicle making a U-turn, K-point turn, N-point turn, etc.), the annotation component 642 can backtrack (e.g., in the log data) to the first time the vehicle begins to deviate from its lane of travel to determine a start time associated with the event. For example, the annotation component 642 can determine that the position of the vehicle is changing from being substantially parallel to the centerline of the driving surface (e.g., road, lane, etc.) to being within a specified offset of 90 degrees of the centerline (e.g., being substantially perpendicular to the centerline) and can determine a start time associated with the event accordingly. For a stopped vehicle (e.g., parked, idling in a driveway or alley, etc.), the annotation component 642 can use the change in speed to determine when the vehicle begins to perform a cut-in maneuver. For example, the annotation component 642 can determine the first time that the vehicle transitions from a zero speed to a non-zero speed to determine a start time associated with the event. In at least one example, the end time can correspond to when the event occurs.

[0072] Block 706 depicts generating training data. In at least one example, the training component 640 can access sensor data and / or log data leading to a detected event (e.g., associated with a cut-in maneuver) and can generate a multi-channel image associated with such sensor data and / or log data. In at least one example, the annotation component 642 can determine when an event associated with a cut-in maneuver begins and ends, as described above, and can access sensor data and / or log data from the start time to just before the occurrence of the event. For example, the training component 640 can access sensor data and / or log data from 1-10 seconds prior to the detected event, which can be used to generate a multi-channel image representative of the environment and / or vehicle conditions prior to the occurrence of the detected event.

[0073] In at least one example, the sensor data and / or logged data can be encoded into one or more multi-channel images, each channel can correspond to a feature, which for a vehicle detected as performing a cut-in maneuver can include, but is not limited to, a note in a map associated with the environment indicating a parking spot (e.g., a parking spot at a substantially right angle) or parking stall, an alley, a driveway, etc.), a position of the vehicle (e.g., a direction of travel, an angular offset from another vehicle, etc.), an instantaneous speed of the vehicle, an indication of whether the vehicle has a driver, an indication of the direction the driver is looking (e.g., the direction of the driver's head), an indication of whether a door associated with the vehicle is open or closed, an indication of whether the engine of the vehicle is running or off, a wheel angle associated with the wheels of the vehicle, an indication of whether the brake lights of the vehicle are on, an indication of whether the headlights of the vehicle are on, an indication of whether the reverse lights of the vehicle are on, an indication of whether the turn signal of the vehicle is on, or a light state associated with the vehicle, etc. In at least one example, each channel can be associated with a top-down view of the environment. That is, the channels can represent sensor outputs or information based on sensor outputs that are modeled in a top-down view. The channels can then be input into a model, such as a multi-layer model, a convolutional neural network, or the like. Further details associated with top-down convolutional neural networks are described in U.S. Patent Application No. 16 / 420,050, filed May 22, 2019, and U.S. Patent Application No. 16 / 504,147, filed July 5, 2019, the entire contents of both of which are incorporated herein by reference. Of course, while described above with respect to one multi-channel image, any number of multi-channel images (which may represent a sequence of times prior to the most recent time step) and / or any other data structure are contemplated.

[0074] In some examples, the training data and associated sensor data and / or log data may be referred to as ground truth.

[0075] Block 708 illustrates inputting training data into a model for determining an output associated with an indication of whether the vehicle is likely to perform a cut-in maneuver. In at least one example, the training data can be analyzed by the training component 640 using a machine learning mechanism. For example, in at least one example, a model can be trained using a neural network (e.g., a convolutional neural network, etc.) and the training data. That is, the training data can be input into a model (e.g., a top-down convolutional neural network, etc.) for determining an output associated with a prediction of whether the vehicle will enter a lane in front of another vehicle or will otherwise perform a maneuver described herein as a "cut-in maneuver." Such a "prediction" can be a detection of a cut-in maneuver, a prediction of a subsequent lane or position in the environment, or something in between. In some examples, the output can be a binary indicator (e.g., "yes" or "no") indicating whether the vehicle will enter the lane of travel of another vehicle in front of the other vehicle. In some examples, the output can be associated with a confidence score. In some examples, the output may be a percentage or other metric indicating the certainty that the vehicle will enter the other vehicle's lane in front of the other vehicle. As mentioned above, in some examples, the output may be related to a predicted lane or position in the environment.

[0076] Block 710 illustrates determining a difference between the output and an expected output associated with the training data. In at least one example, the resulting model can determine an output associated with a prediction of whether the vehicle will enter a lane in front of another vehicle or will otherwise perform a cut-in maneuver as described herein. The training component 640 can compare the output to an expected output associated with the training data (which can be based at least in part on the annotated data) to determine a difference. In at least one example, the training component 640 can modify one or more parameters of the model based at least in part on the difference (e.g., via gradient descent, backpropagation, etc.).

[0077] Block 712 depicts determining whether the difference is greater than or equal to a threshold. In at least one example, the training component 640 can determine whether the difference is greater than or equal to a threshold. If the difference is greater than or equal to the threshold, the training component 640 can modify the parameters of the model, as shown in block 714, and the process 700 can return to block 702. If the difference is less than the threshold, the training component 640 can transmit the model to model storage 648 and / or to storage on the vehicle, such as storage 630 on the vehicle 602. In some examples, as shown in block 716, the model can be transmitted to a vehicle, such as the vehicle 602, which can be configured to be controlled by the output of the model. In such examples, the resulting model can be stored in storage 630 on the vehicle 602 and can be accessed in near real-time by one or more components of the vehicle computing device 604, as described below with reference to FIG. 8.

[0078] In at least one example, the training component 640 can train the model using machine learning techniques. Block 708 makes a reference to one or more neural networks that can be used to train the model. Additional or alternative machine learning algorithms can be used to train the model. Such machine learning algorithms include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatter plot smoothing (LOESS)), example-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative dichotomy 3 (ID3), chi-squared automated interaction detection (CHAID), decision stumps, conditional decision trees), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, polynomial naive Bayes, average-one dependence estimator (AODE), Bayesian belief networks (BNN), Bayesian networks), clustering algorithms (e.g., k-means, k-means-based regression ... The learning algorithms may include, but are not limited to, algorithms such as: median, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, backpropagation, Hopfield networks, radial basis function networks (RBFN)), deep learning algorithms (e.g., deep Boltzmann machines (DBM), other deep belief networks (DBN), artificial neural networks (ANN), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), support vector machines (SVM), supervised learning, unsupervised learning, semi-supervised learning, etc.

[0079] FIG. 8 illustrates an example process 800 described herein for predicting whether a vehicle will enter a lane region in front of another vehicle and controlling the other vehicle based on such prediction.

[0080] Block 802 illustrates receiving sensor data associated with the vehicle's environment. As discussed above, an individual vehicle, such as vehicle 602, may include a sensor component 606. In at least one example, the sensor component 606 may include a LIDAR sensor, a RADAR sensor, an ultrasonic transducer, a sonar sensor, a position sensor (e.g., GPS, compass, etc.), an inertial sensor (e.g., inertial measurement unit, accelerometer, magnetometer, gyroscope, etc.), a camera (e.g., RGB, IR, intensity, depth, etc.), a wheel encoder, an audio sensor, an environmental sensor (e.g., temperature sensor, humidity sensor, light sensor, pressure sensor, etc.), a ToF sensor, etc. The sensor component 606 may provide input to the vehicle computing device 604.

[0081] Block 804 depicts determining whether another vehicle is detected in proximity to the vehicle. As discussed above, the vehicle 602 may include one or more components associated with a vehicle computing device 604 onboard the vehicle 602. In at least one example, the localization component 620 may determine an attitude (position and orientation) of the vehicle 602 relative to a local and / or global map based at least in part on sensor data received from the sensor component 606 and / or map data associated with (e.g., of) the map. In at least one example, the localization component 620 may include or be associated with a calibration component that may substantially simultaneously perform operations for calibrating (determining various intrinsic and extrinsic parameters associated with any one or more of the sensor components 606), locating, and mapping.

[0082] In at least one example, the perception component 622 can perform object detection, segmentation, and / or classification based at least in part on the sensor data received from the sensor component 606. In at least one example, the perception component 622 can receive raw sensor data (e.g., from the sensor component 606). In at least one example, the perception component 622 can receive the sensor data and can utilize one or more processing algorithms to perform object detection, segmentation, and / or classification for objects identified in the sensor data. In some examples, the perception component 622 can associate a boundary region (or otherwise instance segmentation) with the identified object and can associate a confidence score associated with the classification of the identified object with the identified object. In some examples, the object can be colored based on its perceived class when rendered via a display.

[0083] In at least one example, the prediction component 624 may analyze the positioning data (e.g., output from the localization component 620) and the perception data (e.g., output from the perception component 622) to determine whether the perception data indicates that another vehicle is within a threshold distance of the vehicle 602. That is, the prediction component 624 may determine whether another vehicle is proximate to the vehicle 602. If no vehicle is determined to be proximate to the vehicle 602, the process 800 may return to block 802. If another vehicle is determined to be proximate to the vehicle 602, the prediction component 624 may determine an output indicating whether the other vehicle is predicted to enter a lane in front of the vehicle 602, as described below.

[0084] Block 806 depicts determining whether another vehicle is predicted to perform a cut-in maneuver. In at least one example, the prediction component 624 may use a model trained via machine learning techniques, as described above with reference to FIG. 7, to output a prediction indicating whether another vehicle will enter the lane of travel of the vehicle 602 in front of the vehicle 602 or will otherwise perform a cut-in maneuver. That is, the prediction component 624 may use the model to generate a prediction regarding whether another vehicle is likely to perform a cut-in maneuver (e.g., enter the lane of travel in front of the vehicle 602).

[0085] In at least one example, the prediction component 624 can convert sensor data (e.g., received in block 802) into a top-down representation of the environment. For example, the prediction component 624 can generate a multi-channel image associated with such sensor data. As discussed above, each channel may correspond to a feature that may include, but is not limited to, an annotation in a map associated with the environment indicating a parking spot (e.g., a parking spot at a substantially right angle) or parking stall, an alley, a driveway, etc.), a position of the vehicle and / or another vehicle (e.g., direction of travel, angular offset from another vehicle, etc.), an instantaneous speed of the other vehicle, an indication of whether a driver is in the other vehicle, an indication of the direction the driver is looking (e.g., the direction of the driver's head), an indication of whether a door associated with the other vehicle is open or closed, an indication of whether the engine of the other vehicle is running or off, a wheel angle associated with the wheels of the other vehicle, an indication of whether the brake lights of the other vehicle are on, an indication of whether the headlights of the other vehicle are on, an indication of whether the reverse lights of the other vehicle are on, an indication of whether the turn signals of the other vehicle are on, or a lighting state associated with the other vehicle, etc. In at least one example, each channel may be associated with a top-down view of the environment. That is, the channels can represent sensor output or information based on sensor output that is modeled in a top-down view.As mentioned above, multiple multi-channel images (or any other data structure) representing the period preceding the most recent time step are possible.

[0086] In at least one example, the model can be trained to analyze one or more features (also referred to as attributes, characteristics, parameters, data, etc.) associated with the environment with which the sensor data is associated and / or another vehicle to output a prediction regarding whether another vehicle will enter the lane of the vehicle 602 in front of the vehicle 602. In at least one example, the prediction component 624 can receive (e.g., from a respective model) an output indicating the likelihood that another vehicle will enter the lane of the vehicle 602 in front of the vehicle 602. As discussed above, in some examples, the output can be a binary indicator (e.g., “yes” or “no”) indicating whether the other vehicle will enter the lane of the vehicle 602 in front of the vehicle 602. In some examples, the output can be a percentage or other metric indicating the certainty of whether the other vehicle will enter the lane of the vehicle 602 in front of the vehicle 602. In such an example, the prediction component 624 can compare the output percentage or other metric to a threshold, and if the output percentage or other metric is equal to or greater than the threshold, the prediction component 624 can determine that another vehicle is predicted to enter the vehicle's lane in front of the vehicle 602. If the output percentage or other metric is less than the threshold, the prediction component 624 can determine that another vehicle is predicted to enter the vehicle's lane in front of the vehicle 602.

[0087] In at least one example, if another vehicle is predicted to enter the lane in front of the vehicle 602 (e.g., perform a cut-in maneuver), a planning component 626 associated with the vehicle 602 may receive an indication to that effect. The planning component 626 may determine a first command for controlling the vehicle 602 taking into account the other vehicle, as shown in block 808, and may cause the vehicle 602 to be controlled based at least in part on the first command, as shown in block 810. That is, as described above, the planning component 626 may utilize the output of the model to determine how to navigate the vehicle 602 in its environment. For example, the planning component 626 may generate one or more trajectories that may be used by other components associated with the vehicle 602 to adapt driving maneuvers based on predictions output by the model. In some examples, such trajectories may be modifications of a previous trajectory. In some examples, such trajectories may be new trajectories based on the first command. In consideration of the foregoing, planning component 626 may determine a trajectory for navigating vehicle 602 to accommodate another vehicle, for example, by slowing down vehicle 602 thereby increasing the distance between vehicle 602 and another vehicle, slowing down vehicle 602 to yield to the other vehicle and / or stopping, causing vehicle 602 to perform a lane change, causing vehicle 602 to safely go around the other vehicle, and / or any other combination of maneuvers that vehicle 602 is capable of performing.

[0088] In some examples, the output can be used by the prediction component 624 to predict how another vehicle is likely to move, and the prediction component 624 can output a trajectory representing such prediction. In such examples, the predicted trajectory associated with the other vehicle can be provided to the planning component 626 for the planning component to determine a trajectory for navigating the vehicle 602.

[0089] In at least one example, if another vehicle is not predicted to enter the lane in front of the vehicle 602, the planning component 626 associated with the vehicle 602 may receive an indication to that effect. In such an example, the planning component 626 may determine a second command for controlling the vehicle 602, as shown in block 812, and cause the vehicle 602 to be controlled at least in part based on the second command, as shown in block 814. In such an example, if another vehicle is not predicted to enter the lane in front of the vehicle 602, the planning component 626 may plan a trajectory (which may or may not take into account the other vehicle) to otherwise control the vehicle 602 to travel through the environment. In some examples, such a trajectory may be the same trajectory previously determined (e.g., before determining that another vehicle is not predicted to perform a cut-in maneuver). In some examples, the trajectory may be a new trajectory based on the second command.

[0090] 1-5, discussed above, illustrate various examples of other vehicles proximate to a vehicle, such as vehicle 602, that are entering or are expected to enter a lane region of a travel lane ahead of vehicle 602 or are otherwise performing a "cut-in maneuver" as described herein. Such examples include left turns, right turns, U-turns, three-point turns, and the like. As discussed above, further or alternative examples are within the scope of this disclosure (e.g., K-turns, any N-point turns, and the like). That is, process 800, discussed above, may be applicable to any example in which another vehicle proximate to vehicle 602 and associated with the same or different travel direction as vehicle 602 is entering or is expected to enter a lane region of a travel lane ahead of the vehicle.

[0091] Illustrative clauses

[0092] A. A method comprising: receiving log data relating to vehicles in an environment; detecting an event in the log data, the event corresponding to a cut-in maneuver performed by the vehicle; determining a time period for the event; accessing a portion of the log data corresponding to a time period; generating training data based at least in part on transforming a portion of the log data corresponding to a time period into a top-down representation of the environment; inputting training data into the model; determining, by the model, an output including an indication that another vehicle is about to perform a cut-in maneuver; determining a difference between the output and an expected output associated with the training data; Varying one or more parameters of the model based on the difference; transmitting the model to an autonomous vehicle configured to be controlled by other outputs of the model; A method comprising:

[0093] B. The log data includes raw sensor data and downstream outputs determined based at least in part on the raw sensor data, the downstream outputs including sensory data, predictive data, or planner data; The method described in Section A.

[0094] C. determining that the position of the vehicle changes from being substantially parallel to the centerline of the driving surface to within a specified offset of 90 degrees of the centerline; determining a beginning of a time period based at least in part on determining when a position of the vehicle changes from substantially parallel to the centerline to within a specified offset of 90 degrees of the centerline; The method of any one of clauses A to B, further comprising:

[0095] D. Determining that the speed of the vehicle changes from zero to a non-zero speed prior to the event; determining a start of the time period based at least in part on determining when a speed of the vehicle changes from zero to a non-zero speed prior to the event; The method according to any one of clauses A to C, further comprising:

[0096] E. A system comprising: one or more processors; When executed by one or more processors, the system: receiving log data relating to vehicles in an environment; detecting an event in the log data, the event corresponding to a cut-in maneuver performed by the vehicle; generating training data based at least in part on transforming a portion of the log data corresponding to the events into a top-down representation of the environment; inputting training data into the model; transmitting the model to an autonomous vehicle configured to be controlled by other outputs of the model; and Including, the system.

[0097] F. The operation includes determining a time period for an event; accessing a portion of the log data corresponding to a time period; A top-down view of the environment is associated with a portion of the log data that corresponds to a time period, A system as described in clause E.

[0098] G. Operation determining that the position of the vehicle changes from being substantially parallel to a centerline of the driving surface to within a specified offset of 90 degrees of the centerline; determining a beginning of a time period based at least in part on determining when a position of the vehicle changes from substantially parallel to the centerline to within a specified offset of 90 degrees of the centerline; The system of claim F, further comprising:

[0099] H. Operation: determining that a speed of the vehicle changes from zero to a non-zero speed prior to the event; determining a start of the time period based at least in part on determining when a speed of the vehicle changes from zero to a non-zero speed prior to the event; The system of any one of clauses F and G, further comprising:

[0100] I. Operation determining, by the model, an output including an indication that another vehicle is about to perform a cut-in maneuver; determining a difference between the output and an expected output associated with the training data; Varying one or more parameters of the model based on the difference; The system of any of clauses EH, further comprising:

[0101] J. The model includes a convolutional neural network. A system as described in any one of clauses E-I.

[0102] K. The top-down representation includes a multi-channel image, where each channel is the position of the vehicle relative to the centerline of the driving surface; The instantaneous speed of the vehicle, Indication of whether the driver is in the vehicle; Indication of whether the vehicle doors are open or closed; Indication of whether the vehicle's engine is running or off; Indication of whether the vehicle's brake lights are on; Indication of whether the vehicle's headlights are on or not, An indication of whether the vehicle's reversing lights are on, or Indication of whether the vehicle's turn signals are on The system according to any one of clauses E to J, which corresponds to at least one of the above.

[0103] L. Cut-in operation is Turning left from an alley, parking lot, or driveway; Turning right from an alley, parking lot, or driveway; U-turn, N-point turn, K-turn, or Reversing into the driving lane, The system of any one of clauses E-K, including any one of the following:

[0104] M. when executed by one or more processors, causes the one or more processors to receiving log data relating to vehicles in an environment; detecting an event in the log data, the event corresponding to a cut-in maneuver performed by the vehicle; generating training data based at least in part on transforming a portion of the log data corresponding to a time period into a top-down representation of the environment; inputting training data into a model, the model being trained to output an indication of whether another vehicle is about to perform a cut-in maneuver; One or more non-transitory computer-readable media storing instructions for performing operations including:

[0105] N. The operations further include transmitting the model to an autonomous vehicle configured to be controlled by other outputs of the model; One or more non-transitory computer-readable media as described in clause M.

[0106] O. Operation determining a time period for the event; accessing a portion of the log data corresponding to a time period; Further comprising: A top-down view of the environment is associated with a portion of the log data that corresponds to a time period, One or more non-transitory computer-readable media according to clause M or N.

[0107] P. determining the period is based on at least one of a change in position of the vehicle or a speed of the vehicle; One or more non-transitory computer-readable media as described in clause O.

[0108] Q.How does it work? determining a difference between the output of the model and an expected output associated with the training data; Varying one or more parameters of the model based on the difference; Further comprising: One or more non-transitory computer-readable media according to any of clauses MP.

[0109] R. Models include convolutional neural networks, One or more non-transitory computer-readable media according to any of clauses MQ.

[0110] S. The top-down representation includes a multi-channel image, where each channel is the position of the vehicle relative to the centerline of the driving surface; The instantaneous speed of the vehicle, Indication of whether the driver is in the vehicle; Indication of whether the vehicle doors are open or closed; Indication of whether the vehicle's engine is running or off; Indication of whether the vehicle's brake lights are on; Indication of whether the vehicle's headlights are on or not, An indication of whether the vehicle's reversing lights are on, or Indication of whether the vehicle's turn signals are on One or more non-transitory computer-readable media according to any of clauses M-R, corresponding to at least one of.

[0111] T. Cut-in driving is Turning left from an alley, parking lot, or driveway; Turning right from an alley, parking lot, or driveway; U-turn, N-point turn, K-turn, or Reversing into the driving lane, One or more non-transitory computer-readable media as described in any of clauses M through S, including any one of:

[0112] U. A method comprising: receiving sensor data relating to an environment of a first vehicle; detecting a second vehicle proximate to the first vehicle based at least in part on the sensor data, the second vehicle being substantially perpendicular to the first vehicle; determining, based at least in part on the sensor data, an attribute related to the environment or the second vehicle that is indicative of whether the second vehicle is entering a lane region in front of the first vehicle; determining an indication of whether the second vehicle is entering a lane region in front of the first vehicle based at least in part on the attribute; determining instructions for controlling the first vehicle based at least in part on the indication; A method comprising:

[0113] V. the second vehicle is substantially perpendicular to the first vehicle based at least in part on an angle between the second vehicle and the first vehicle that is within a specified offset of 90 degrees; The method described in clause U.

[0114] W. The specified offset is 50 degrees, The method described in clause V.

[0115] X. The second vehicle is located in a parking lot, driveway, or alley substantially at a right angle to the first vehicle; A method according to any one of clauses U to W.

[0116] Y. the second vehicle is associated with a different heading than the first vehicle; A method according to any one of clauses U to X.

[0117] Z. Attributes a position of the second vehicle relative to the first vehicle; the instantaneous speed of the second vehicle; an indication of whether a driver is in a second vehicle; an indication of whether a door associated with the second vehicle is open or closed; an indication of whether the engine of the second vehicle is running or off; an indication of whether the brake lights of the second vehicle are on; an indication of whether the headlights of the second vehicle are on; an indication of whether the second vehicle's reverse lights are on; or Indication of whether the second vehicle's turn signals are on At least one of A method according to any one of clauses U to Y.

[0118] AA. A system comprising: one or more processors; When executed by one or more processors, the system: receiving sensor data relating to an environment of a first vehicle; determining, based at least in part on the sensor data, that a second vehicle proximate to the first vehicle is predicted to enter a lane region ahead of the first vehicle; determining instructions for controlling the first vehicle based at least in part on determining that a second vehicle proximate to the first vehicle is predicted to enter a lane area ahead of the first vehicle; and one or more non-transitory computer-readable media storing instructions for performing operations including:

[0119] AB. determining that the second vehicle is predicted to enter a lane region ahead of the first vehicle based at least in part on determining one or more attributes related to at least one of the second vehicle or the environment; The system described in clause AA.

[0120] AC. One or more attributes a position of the second vehicle relative to the first vehicle; the instantaneous speed of the second vehicle; an indication of whether a driver is in a second vehicle; an indication of whether a door associated with the second vehicle is open or closed; an indication of whether the engine of the second vehicle is running or off; an indication of whether the brake lights of the second vehicle are on; an indication of whether the headlights of the second vehicle are on; an indication of whether the second vehicle's reverse lights are on; or Indication of whether the second vehicle's turn signals are on At least one of The system described in clause AB.

[0121] AD. The first vehicle and the second vehicle are positioned at an angle between 90 degrees and a specified offset; A system as described in any one of clauses AA to AC.

[0122] AE. The second vehicle is located in a parking lot or driveway on the same side of the road as the first vehicle or on the opposite side of the road as the first vehicle; The system described in clauses AD.

[0123] AF. determining instructions for controlling the first vehicle is based at least in part on output from the machine learning model, the output including a binary indication of whether the second vehicle is entering the lane region ahead of the first vehicle or a percentage indicating a certainty as to whether the second vehicle is entering the lane region ahead of the first vehicle. A system as described in any one of clauses AA to AE.

[0124] AG. The instruction causes the first vehicle to perform at least one of slowing down, stopping, or changing lanes; A system as described in any one of clauses AA to AF.

[0125] AH. when executed by one or more processors, causes the one or more processors to receiving sensor data relating to an environment of a first vehicle; Transforming the sensor data into a top-down representation of the environment; inputting the top-down representation into a machine learning model; determining, based at least in part on inputting the top-down view into a machine learning model, that a second vehicle proximate to the first vehicle is predicted to enter a lane region ahead of the first vehicle; determining instructions for controlling the first vehicle based at least in part on determining that a second vehicle proximate to the first vehicle is predicted to enter a lane area ahead of the first vehicle; One or more non-transitory computer-readable media storing instructions for performing operations including:

[0126] AI. Determining that a second vehicle proximate to a first vehicle is predicted to enter a lane region ahead of the first vehicle is based at least in part on at least one attribute, the attribute being: the instantaneous speed of the second vehicle; an indication of whether the driver is in the second vehicle and, if so, an indication of the driver's head direction; an indication of whether the engine of the second vehicle is running or off; a wheel angle relative to a wheel of a second vehicle, or an indication of whether at least one of a brake light, a headlight, a backup light, or a turn signal of the second vehicle is illuminated; one or more non-transitory computer-readable media as described in clause AH,

[0127] AJ. The second vehicle is associated with a different heading than the first vehicle; One or more non-transitory computer-readable media as described in clause AH or AI.

[0128] AK. The second vehicle and the first vehicle are positioned at an angle within a specified offset range of 90 degrees; One or more non-transitory computer-readable media as described in any of clauses AH-AJ.

[0129] AL. The second vehicle is located either (i) in a parking lot or driveway on the same side of the road as the first vehicle, or (ii) in a parking lot or driveway on the opposite side of the road as the first vehicle; One or more non-transitory computer-readable media as described in any of clauses AH-AK.

[0130] AM. Operation is inputting an indication to a prediction component that a second vehicle proximate to the first vehicle is predicted to enter a lane area ahead of the first vehicle; receiving a predicted trajectory for the second vehicle from the prediction component; determining instructions based at least in part on the predicted trajectory; Further comprising: One or more non-transitory computer-readable media as described in any of clauses AH-AL.

[0131] AN. The command causes the first vehicle to perform at least one of a maneuver to slow down or change lanes; One or more non-transitory computer-readable media as described in any of clauses AH-AM.

[0132] Although the above example clauses are described with respect to one particular implementation, it should be understood in the context of this document that the example clauses may also be implemented via methods, devices, systems, computer-readable media, and / or other implementations. In addition, any of the examples A-AN may be implemented alone or in combination with any one or more of the examples A-AN.

[0133] conclusion

[0134] Having described one or more examples of the technology described herein, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0135] In describing the examples, reference is made to the accompanying drawings, which form a part of this specification, which show by way of illustration specific examples of the claimed components. It is understood that other examples can be utilized and modifications or variations, such as structural changes, can be made. Such examples, modifications or variations do not necessarily depart from the scope of the claimed components. Although the steps in this specification may be expressed in a specific order, in some cases the order can be changed such that certain inputs are provided at different times or in a different order without changing the functionality and method of the described system. The disclosed procedures may also be performed in different orders. In addition, the various calculations in this specification need not be performed in the order disclosed, and other examples utilizing alternative calculation orders can be easily implemented. In addition to being reordered, the calculations can also be decomposed into sub-calculations that achieve the same results.

Claims

1. 1. A system comprising: one or more processors; When executed by the one or more processors, the system further comprises: receiving sensor data associated with an environment of a first vehicle operating within a structured domain; detecting, by the first vehicle, a second vehicle within a threshold distance of the first vehicle based at least in part on the sensor data, the second vehicle being in an unstructured area; determining, by the first vehicle, an attribute associated with the second vehicle indicative of whether the second vehicle in the unstructured region is entering a lane region occupied by the first vehicle in the structured region based at least in part on the sensor data; determining an indication of whether the second vehicle will enter the lane region occupied by the first vehicle based at least in part on the attributes and the second vehicle's presence in the unstructured region, wherein determining the indication is performed at least in part by a machine learning model trained to predict object behavior in an environment transitioning from unstructured to structured regions; controlling the first vehicle based at least in part on the indication; One or more non-transitory computer-readable media storing instructions for performing operations including: A system comprising:

2. The attribute is: a position of the second vehicle relative to the first vehicle; the instantaneous speed of the second vehicle; an indication of whether a driver is within the second vehicle; an indication of whether a door associated with the second vehicle is open or closed; an indication of whether the engine of the second vehicle is running or off; an indication of whether the brake lights of the second vehicle are illuminated; an indication of whether the headlights of the second vehicle are on; an indication of whether the second vehicle's backup lights are on; or an indication of whether a turn signal of the second vehicle is on; At least one of: The system of claim 1 .

3. the first vehicle and the second vehicle are positioned at an angle between 90 degrees and a specified offset, the specified offset being 50 degrees; 3. A system according to claim 1 or 2.

4. the second vehicle is located in a parking lot, driveway, or alley on the same side of the road as the first vehicle or on the opposite side of the road as the first vehicle; The system of claim 3.

5. determining the indication for controlling the first vehicle is based at least in part on an output from the machine learning model, the output including a binary indication of whether the second vehicle is entering the lane region in front of the first vehicle or a metric indicating a certainty as to whether the second vehicle is entering the lane region in front of the first vehicle. A system according to any one of claims 1 to 4.

6. the indication causes the first vehicle to perform at least one of slowing down, stopping, or changing lane. A system according to any one of claims 1 to 5.

7. A method for detecting a vehicle environment comprising: receiving sensor data associated with an environment of a first vehicle operating within a structured domain; detecting, by the first vehicle, a second vehicle within a threshold distance of the first vehicle based at least in part on the sensor data, the second vehicle being in an unstructured area; determining, by the first vehicle, an attribute associated with the second vehicle indicative of whether the second vehicle in the unstructured region is entering a lane region occupied by the first vehicle in the structured region based at least in part on the sensor data; determining an indication of whether the second vehicle will enter the lane region occupied by the first vehicle based at least in part on the attributes and the second vehicle being in the unstructured region, wherein determining the indication is performed at least in part by a machine learning model trained to predict object behavior in an environment transitioning from unstructured to structured regions; controlling the first vehicle based at least in part on the indication; A method comprising:

8. determining a top-down representation of the environment based at least in part on the sensor data; inputting the top-down representation into the machine learning model; determining, based at least in part on inputting the top-down representation into the machine learning model, that the second vehicle is entering the lane region in front of the first vehicle; Further comprising: The method of claim 7.

9. The attribute is: the instantaneous speed of the second vehicle; an indication of whether a driver is in the second vehicle and, if the driver is in the second vehicle, an indication of the driver's head direction; an indication of whether the engine of the second vehicle is running or off; a wheel angle relative to a wheel of the second vehicle; or an indication of whether at least one of the brake lights, headlights, back-up lights, or turn signals of the second vehicle are illuminated; Including, 9. The method according to claim 7 or 8.

10. inputting an indication that the second vehicle is about to enter the lane region ahead of the first vehicle into a prediction component; receiving a predicted trajectory for the second vehicle from the prediction component; further determining the indication for controlling the first vehicle based at least in part on the predicted trajectory; Further comprising: The method according to any one of claims 7 to 9.

11. the second vehicle is associated with a different heading than the first vehicle; The method according to any one of claims 7 to 10.

12. the second vehicle is substantially perpendicular to the first vehicle based at least in part on an angle between the second vehicle and the first vehicle being within a specified offset of 90 degrees; The method according to any one of claims 7 to 11.

13. the second vehicle is located in a parking lot, driveway, or alley substantially perpendicular to the first vehicle; The method according to any one of claims 7 to 12.

14. One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 7 to 13.

Images