Using predictive models to solve scenario challenges in vehicle routing

Abstract

A method is disclosed, comprising: receiving information from one or more vehicles configured to operate in an autonomous driving mode, the information identifying different types of difficulties encountered when the one or more vehicles are operating in the autonomous driving mode along one or more routes, wherein the different types of difficulties include sensor difficulties that affect autonomous driving; storing the information in a memory together with metadata, the metadata identifying one or more details about each of the different types of difficulties including the sensor difficulties, wherein the metadata also includes a weighted difficulty value for each of the different types of difficulties that indicates a difficulty level associated with a driving environment; evaluating the stored information and the metadata considering multiple route options; and selecting a given route option from the multiple route options for use when the vehicle is operating in the autonomous driving mode based on the information associated with the sensor difficulties and the metadata.

Description

[0001] This application is a divisional application of the following application: application number is 201880081076.8, application date is November 26, 2018, and the name of the invention is "Using predictive models to solve scenario difficulties in vehicle routes." Technical Field

[0002] The present application relates to using predictive models to solve scenario difficulties in vehicle routing. Background Art

[0003] Current navigation systems take into account factors such as distance and driving time. However, such systems only provide limited options to drivers and passengers of vehicles, particularly autonomous vehicles. Such autonomous vehicles can operate in a fully autonomous mode, where the passenger can provide some initial input, such as a pickup or destination location, and the vehicle will maneuver itself to that location. In other cases, the vehicle can operate in a semi-autonomous or manual mode, where the driver controls some or all aspects of the vehicle's maneuvering. Summary of the invention

[0004] One aspect of the present disclosure provides a method for determining a driving route of an autonomous vehicle. The method includes receiving data from one or more vehicles at one or more computing devices, the data providing information about a road segment traveled by one or more vehicles, and the data including one or more difficulty signals from one or more vehicles. The data is used to determine the difficulty value of the road segment, which is based on the difficulty signal from one or more vehicles indicating the departure from the autonomous driving mode to the manual driving mode. In other examples, the difficulty signal may indicate a request for assistance from a remote operator, multiple unprotected turns along the road segment, a portion of the driving area where the field of view of the autonomous driving vehicle is blocked, a threshold or an abnormal number of obstacles, etc. A route cost is calculated for each of one or more possible routes between a starting point and a destination, wherein the route cost is based at least on the difficulty value of the road segment along the route, and based on the route cost of each of the one or more possible routes, it is determined which route of the one or more possible routes is to be traveled. In some examples, the method also includes driving the autonomous driving vehicle along the determined route. The difficulty signal may be associated with a temporal feature. For example, calculating the difficulty value for a particular road segment may include comparing an expected date or time for traveling along the particular road segment with a date or time associated with a difficulty signal for the particular road segment. Determining the difficulty value may include determining a probability of encountering a difficulty corresponding to at least one difficulty signal.

[0005] In some examples, the method may further include: calculating a segment cost for the segment, wherein the segment cost is calculated based on a difficulty value, a distance, and a time required to travel the segment, wherein the route cost is calculated based on the segment cost of the segment along the route. In such an example, calculating the segment cost may include: calculating a difficulty score based on the difficulty value of the segment; calculating a distance score based on the distance of the segment; calculating a time score based on the time required to travel the segment; weighting the difficulty score, the distance score, and the time score; and calculating the segment cost as a function of the weighted score. Weighting the difficulty score, the distance score, and the time score may include setting coefficients for the difficulty score, the distance score, and the time score, wherein the coefficients may be modified to adjust the relative weights. Input may be received from a passenger of the autonomous vehicle, and the coefficients may be adjusted based on the input received from the passenger.

[0006] Another aspect of the present disclosure provides a system for determining a driving route of an autonomous vehicle. The system includes one or more memories, and one or more processors in communication with the one or more memories. The one or more processors may be configured to receive data from one or more vehicles, the data providing information about a road segment traveled by one or more vehicles and including difficulty signals from one or more vehicles, and determine the difficulty value of the road segment based on the data, the difficulty value based on the difficulty signal from one or more vehicles indicating the disengagement from the autonomous driving mode to the manual driving mode. In some examples, the difficulty signal may alternatively or additionally indicate a request for assistance from a remote operator, a plurality of unprotected turns along the road segment, a portion of the driving area where the field of view of the autonomous driving vehicle is blocked, the number of obstacles on the road segment, etc. The one or more processors may further calculate the route cost of each of one or more possible routes between a starting point and a destination, wherein the route cost is based at least on the difficulty value of the road segment along the route, and determine which of the one or more possible routes to travel based on the route cost of each of the one or more possible routes.

[0007] Another aspect of the present disclosure provides a method, comprising: receiving, by one or more computing devices, information from one or more vehicles configured to operate in an autonomous driving mode, the information identifying different types of difficulties encountered when the one or more vehicles operate along one or more routes in the autonomous driving mode, wherein the different types of difficulties include sensor difficulties that affect autonomous driving; storing the information in a memory together with metadata, the metadata identifying one or more details about each of the different types of difficulties including the sensor difficulties, wherein the metadata also includes a weighted difficulty value for each of the different types of difficulties that indicates a difficulty level associated with a driving environment; evaluating, by the one or more computing devices, the stored information and the metadata considering multiple route options; and selecting, by the one or more computing devices, a given route option from the multiple route options for use when the vehicle operates in the autonomous driving mode based on the information associated with the sensor difficulties and the metadata.

[0008] Another aspect of the present disclosure provides a method for operating a vehicle in an autonomous driving mode, the method comprising: controlling, by one or more processors, driving operations of the vehicle in the autonomous driving mode; receiving, by one or more processors, sensor information about objects or conditions in an external environment of the vehicle from a perception system of the vehicle; generating, by one or more processors, information about difficulties encountered when operating in the autonomous driving mode and associated metadata, the information identifying at least one type of difficulty encountered by the vehicle when operating along one or more routes in the autonomous driving mode, wherein the at least one type of difficulty includes sensor difficulties of sensors of the perception system, wherein the metadata also includes a weighted difficulty value for each of the at least one type of difficulty indicating a difficulty level associated with the driving environment; and sending the generated information to a remote system for selecting a route from the multiple route options based on the information associated with the sensor difficulties and the metadata. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a functional diagram of an example vehicle according to an exemplary embodiment.

[0010] Figure 2 is an example of map information according to aspects of the present disclosure.

[0011] Figure 3 are example exterior views of a vehicle according to aspects of the present disclosure.

[0012] Figure 4 is a schematic diagram of an example system according to an exemplary embodiment.

[0013] Figure 5 According to various aspects of the present disclosure Figure 4 Functional diagram of the system.

[0014] Figure 6 is an example data structure according to aspects of the present disclosure.

[0015] Figure 7 is an example flow chart according to aspects of the present disclosure.

[0016] Figure 8 is with Figure 2 Examples of possible route options associated with the map information.

[0017] Fig. 9 yes Figure 8 Examples of road segments for possible route options.

[0018] Fig.10 is with Fig. 9 An example representation of difficulty levels associated with road segments.

[0019] Fig.11 yes Figure 8 Example calculation of route costs for possible route options. DETAILED DESCRIPTION

[0020] The technology generally involves using collected data to improve the system behavior of an autonomous vehicle. In particular, the data collected in this way can be included in a predictive model and used to determine a route to be taken based on the time, distance and difficulty of traveling along the route.

[0021] Metrics can be learned from many vehicles, which can include autonomous vehicles, semi-autonomous vehicles, and manually driven vehicles. These metrics are collected and considered together. Such metrics can include, for example, distance, time, and difficulty. The difficulty metric takes into account factors such as the difficulty of a particular turn, whether there are any obstacles (e.g., a construction site), the number of crosswalks, whether the road is rough, narrow, etc. These metrics are automatically learned from records of previous trips, thus allowing the system to improve itself over time.

[0022] Such a metric can be associated with specific features, such as temporal features (e.g., hour of day, day of week), geographic features (e.g., precise location), and general features (e.g., speed limits, nearby features (e.g., crosswalks, intersections, schools)). For example, while a particular turn may be difficult during the morning rush hour, the same turn may be less difficult in the early afternoon and even less difficult in the evening. By associating such features with a metric, such features can be taken into account when determining an optimal route.

[0023] The difficulty metric may be determined based on a signal indicating the difficulty of navigating a particular road segment. Such a signal may indicate the likelihood of disengaging from an autonomous driving mode, the likelihood of being stuck for too long, traffic density, etc. According to some examples, a difficulty score is calculated for each road segment and then the scores for all road segments of the route are summed. The score may be based on the number of previous disengagements, the number of previous requests for remote operator assistance, the number of unprotected left or right turns, whether portions of the driving area are occluded, user complaints, measured passenger discomfort, failure responses (the vehicle cannot operate normally and needs to stop), etc.

[0024] It is also possible to calculate the score of each section of the route based on the expected amount of time to complete the trip along the section and the distance of the section. These scores can be integrated with the difficulty score. For example, the difficulty score, time score and distance score can all be used to calculate the cost of the section. Such costs can be input into the prediction model and used to determine the best route.

[0025] When determining the best route, dynamic weights can be used to adjust various trade-offs in routing, such as weighing additional driving distance / time against easier driving conditions. As just one example of a trade-off, a driver may be willing to drive a longer distance to avoid making a difficult left turn. The dynamic weights can be, for example, coefficients that can be adjusted based on the scenario. For example, in the first scenario where the customer is in the vehicle, the coefficients can be adjusted to be more conservative, such as providing an easier but potentially longer route. On the other hand, in the second scenario where the test driver is in the vehicle, the coefficients can be adjusted to be more adventurous, such as providing a faster and more difficult route.

[0026] Using the above model, a driving route can be selected not only based on time and distance, but also based on difficulty. Because the difficulty information is collected from many vehicle logs, it is accurate and does not impose an input burden on the user.

[0027] Example System

[0028] like Figure 1 As shown, a vehicle 100 according to one aspect of the present disclosure includes various components. Although certain aspects of the present disclosure are particularly useful for specific types of vehicles, the vehicle can be any type of vehicle, including, but not limited to, cars, trucks, motorcycles, buses, recreational vehicles, etc. The vehicle can have one or more computing devices, such as a computing device 110 that includes one or more processors 120, memory 130, and other components typically found in a general-purpose computing device.

[0029] The memory 130 stores information accessible to one or more processors 120, including instructions 134 and data 132 that can be executed or used by the processor 120. The memory 130 can be of any type capable of storing information accessible to the processor, including computing device readable media or other media that stores data that can be read by electronic devices, such as hard drives, memory cards, ROM, RAM, DVD or other optical disks, and other writable and read-only memories. The systems and methods may include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media.

[0030] Instructions 134 may be any set of instructions to be executed directly (e.g., machine code) or indirectly (e.g., scripts) by a processor. For example, instructions may be stored as computing device code on a computing device readable medium. In this regard, the terms "instructions" and "programs" may be used interchangeably herein. Instructions may be stored in an object code format for direct processing by a processor, or in any other computing device language, including a collection of scripts or independent source code modules that are interpreted on demand or pre-compiled. The functions, methods, and routines of instructions will be described in more detail below.

[0031] Data 132 may be retrieved, stored, or modified by processor 120 according to instructions 134. For example, the data may be stored in a computing device register in the form of a relational database, as a table with a plurality of different fields and records, an XML document, or a flat file, although the claimed subject matter is not limited to any particular data structure. The data may also be formatted in any format readable by a computing device.

[0032] The one or more processors 120 may be any conventional processor, such as a commercially available CPU. Alternatively, the one or more processors may be dedicated devices, such as ASICs or other hardware-based processors. Figure 1 The processor, memory, and other elements of computing device 110 are shown functionally as being within the same block, but one of ordinary skill in the art will appreciate that a processor, computing device, or memory may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. For example, the memory may be a hard drive or other storage medium located in a different housing than computing device 110. Thus, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0033] The computing device 110 may include all components typically used in conjunction with a computing device such as a processor and the aforementioned memory, as well as user input 150 (e.g., a mouse, keyboard, touch screen, and / or microphone) and various electronic displays (e.g., a monitor with a screen or any other electronic device operable to display information). In this example, the vehicle includes an interior electronic display 152 and one or more speakers 154 to provide information or an audio-visual experience. In this regard, the interior electronic display 152 may be located within the cabin of the vehicle 100 and may be used by the computing device 110 to provide information to passengers within the vehicle 100.

[0034] The computing device 110 may also include one or more wireless network connections 156 to facilitate communication with other computing devices, such as client computing devices and server computing devices described in detail below. The wireless network connections may include short-range communication protocols such as Bluetooth, Bluetooth Low Energy (LE), cellular connections, and various configurations and protocols including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, as well as various combinations of the foregoing.

[0035] In one example, computing device 110 may be a control computing device of an autonomous driving computing system or incorporated into vehicle 100. The autonomous driving computing system may be able to communicate with various components of the vehicle to control the movement of vehicle 100 according to the primary vehicle control code of memory 130. For example, returning Figure 1 , the computing device 110 can communicate with various systems of the vehicle 100, such as a deceleration system 160, an acceleration system 162, a steering system 164, a signal system 166, a navigation system 168, a positioning system 170, a perception system 172, and a power system 174 (i.e., the engine or motor of the vehicle) to control the movement, speed, etc. of the vehicle 100 according to the instructions 134 of the memory 130. Similarly, although these systems are shown as being external to the computing device 110, in fact, these systems can also be incorporated into the computing device 110 as an autonomous driving computing system for controlling the vehicle 100.

[0036] As an example, computing device 110 may interact with one or more actuators of deceleration system 160 and / or acceleration system 162 (e.g., brakes, accelerator pedal, and / or engine or motor of the vehicle) to control the speed of the vehicle. Similarly, computing device 110 may use one or more actuators of steering system 164, such as a steering wheel, steering shaft, and / or a gear and rack in a rack and pinion system, to control the direction of vehicle 100. For example, if vehicle 100 is configured for use on a road, such as a car or truck, the steering system may include one or more actuators to control the angle of the wheels to steer the vehicle. Signaling system 166 may be used by computing device 110 to signal the vehicle's intentions to other drivers or vehicles, such as by illuminating a turn signal or brake light when necessary.

[0037] Navigation system 168 may be used by computing device 110 to determine and follow a route to a location. In this regard, navigation system 168 and / or data 132 may store detailed map information, such as a highly detailed map that identifies the shape and height of roads, lane lines, intersections, crosswalks, speed limits, traffic signals, buildings, signs, real-time traffic information, vegetation, or other such objects and information.

[0038] Figure 2 is an example of map information 200 for a geographic area including multiple roads. The map information 200 may include information on the shape, location, and other characteristics of identification features, such as lane lines 210, 212, 214, traffic lights 220, 222, stop signs 250, 251, 252, and other features not shown, such as crosswalks, sidewalks, and avoidance signs 260. The map information 200 may also include distance information for each of the multiple roads and regulations for various roads, such as speed limits, prohibited operations, etc. As just one example of prohibited operations, between 8 a.m. and 4 p.m. on weekdays, avoid turning right at a red light. According to some examples, the map information may also include traffic information, such as historical traffic information. Such traffic information may be stored in association with time limits, for example, indicating times of day / week when traffic is busier.

[0039] Although the map information is depicted here as an image-based map, the map information need not be entirely image-based (e.g., raster-based). For example, the map information may include one or more road maps or a graphical network of information, such as roads, lanes, intersections, and connections between these features. Each feature may be stored as graphical data and may be associated with information such as a geographic location and whether it is linked to other related features, e.g., a stop sign may be linked to roads and intersections, etc. In some examples, the associated data may include a grid-based index of the road map to allow efficient lookup of certain road map features.

[0040] Positioning system 170 may be used by computing device 110 to determine a relative or absolute position of a vehicle on a map or on the earth. For example, positioning system 170 may include a GPS receiver to determine the latitude, longitude, and / or altitude position of the device. Other positioning systems, such as laser-based positioning systems, inertial assisted GPS, or camera-based positioning, may also be used to identify the location of the vehicle. The location of the vehicle may include an absolute geographic location, such as latitude, longitude, and altitude, as well as relative position information, such as relative to other surrounding vehicles, which can typically be determined with less noise than the absolute geographic location.

[0041] Positioning system 170 may also include other devices in communication with computing device 110, such as accelerometers, gyroscopes, or another direction / speed detection device to determine the direction and speed of the vehicle or changes in the direction and speed of the vehicle. By way of example only, an acceleration device may determine its pitch, yaw, or roll (or changes thereof) relative to the direction of gravity or a plane perpendicular to the direction of gravity. The device may also track increases or decreases in speed and the direction of such changes. Provision of position and orientation data of the device as described herein may be automatically provided to computing device 110, other computing devices, and combinations of the foregoing.

[0042] Perception system 172 also includes one or more components for detecting objects external to the vehicle, such as other vehicles, obstacles in the road, traffic signals, signs, trees, etc. For example, perception system 172 may include lasers, sonar, radar, cameras, and / or any other detection devices that record data that can be processed by computing device 110. In the case where the vehicle is a passenger car such as a minivan, the minivan may include lasers or other sensors mounted on the roof or at other convenient locations. For example, Figure 3310 is an example external view of the vehicle 100. In this example, the roof housing 310 and the dome housing 312 may include LIDAR sensors and various cameras and radar units. In addition, the housing 320 located at the front end of the vehicle 100 and the housings 330, 332 on the driver and passenger sides of the vehicle may store the LIDAR sensors, respectively. For example, the housing 330 is located in front of the driver's door 360. The vehicle 100 also includes housings 340, 342 for radar units and / or cameras that are also located on the roof of the vehicle 100. Additional radar units and cameras (not shown) may be located at the front and rear ends of the vehicle 100, and / or other locations along the roof or roof housing 310.

[0043] The computing device 110 can control the direction and speed of the vehicle by controlling various components. For example, the computing device 110 can use data from the detailed map information and navigation system 168 to completely autonomously navigate the vehicle to the destination location. The computing device 110 can use the positioning system 170 to determine the location of the vehicle, and the perception system 172 to detect objects and react to objects when necessary to safely reach the location. To do so, the computing device 110 can accelerate the vehicle (e.g., by increasing the fuel or other energy provided to the engine by the acceleration system 162), decelerate (e.g., by reducing the fuel provided to the engine by the deceleration system 160, changing gears and / or applying brakes), change direction (e.g., by turning the front wheels or rear wheels of the vehicle 100 by the steering system 164) and signal such changes (e.g., by illuminating the turn signal of the signal system 166). Therefore, the acceleration system 162 and the deceleration system 160 can be part of a drivetrain, which includes various components between the engine of the vehicle and the wheels of the vehicle. Likewise, by controlling these systems, computing device 110 may also control the vehicle's powertrain in order to autonomously maneuver the vehicle.

[0044] The computing device 110 of the vehicle 100 may also send information to or receive information from other computing devices, such as other vehicles, control centers, databases, user devices, and the like. Figure 4 and Figure 5 are a visual diagram and a functional diagram, respectively, of an example system 400 that includes a plurality of computing devices 410, 420, 430, 440 and a storage system 450 connected via a network 460. The system 400 also includes a vehicle 100 and vehicles 100A, 100B that may be configured the same as or similar to the vehicle 100. Although only a small number of vehicles and computing devices are described for simplicity, a typical system may include many more vehicles and computing devices.

[0045] like Figure 4As shown, each of computing devices 410, 420, 430, 440 may include one or more processors, memory, data, and instructions. Such processors, memory, data, and instructions may be configured similarly to one or more processors 120, memory 130, data 134, and instructions 132 of computing device 110.

[0046] The network 460 and intermediate nodes may include a variety of configurations and protocols, including short-range communication protocols such as Bluetooth, Bluetooth LE, the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, as well as various combinations of the foregoing. Any device capable of sending and receiving data to and from other computing devices (e.g., modems and wireless interfaces) may facilitate such communications.

[0047] In one example, the one or more computing devices 110 may include one or more servers having multiple computing devices, for example, a load balancing server group, which exchanges information with different nodes of the network for receiving data from other computing devices, processing data, and sending data to other computing devices. For example, the one or more computing devices 410 may include one or more server computing devices that can communicate with the computing device 110 of the vehicle 100 or a similar computing device of the vehicle 100A and computing devices 420, 430, 440 via the network 460. For example, the vehicles 100, 100A, and 100B may be part of a fleet of vehicles that may be dispatched to various locations by the server computing devices. In this regard, the server computing device 410 may be used as a dispatching system. In addition, the vehicles of the fleet may periodically send location information provided by the various positioning systems of the vehicles to the server computing devices, as well as other information related to the status of the vehicles discussed further below, and the one or more server computing devices may track the location and status of each vehicle in the fleet.

[0048] Additionally, server computing device 410 may send and present information to users (e.g., users 422, 432, 442) on displays (e.g., displays 424, 434, 444 of computing devices 420, 430, 440) using network 460. In this regard, computing devices 420, 430, 440 may be considered client computing devices.

[0049] like Figure 4As shown, each client computing device 420, 430, 440 can be a personal computing device intended for use by a user 422, 432, 442, and has all the components commonly used in conjunction with a personal computing device, including one or more processors (e.g., a central processing unit (CPU)), memory (e.g., RAM and an internal hard drive) to store data and instructions, a display such as a display 424, 434, 444 (e.g., a monitor with a screen, a touch screen, a projector, a television, or other device operable to display information), and a user input device 426, 436, 446 (e.g., a mouse, keyboard, touch screen, or microphone). The client computing device may also include a camera for recording a video stream, a speaker, a network interface device, and all the components for connecting these elements to each other.

[0050] Although each of client computing devices 420, 430, and 440 may include a full-size personal computing device, they may alternatively include mobile computing devices capable of wirelessly exchanging data with a server over a network such as the Internet. By way of example only, client computing device 420 may be a mobile phone or a device such as a wireless-enabled PDA, a tablet PC, a wearable computing device or system, or a netbook capable of obtaining information via the Internet or other network. In another example, client computing device 430 may be a wearable computing device such as a Figure 4 As examples, the user can input information using a keypad, a keypad, a microphone, visual signals using a camera, or a touch screen, etc.

[0051] As with memory 130, storage system 450 may be any type of computerized storage device capable of storing information accessible to server computing device 410, such as a hard drive, memory card, ROM, RAM, DVD, CD-ROM, writable and read-only memory. In addition, storage system 450 may include a distributed storage system in which data is stored on multiple different storage devices that may be physically located in the same or different geographic locations. Storage system 450 may be connected to computing devices via network 460, such as Figure 4 and Figure 5 As shown, and / or may be directly connected to or incorporated into any computing device 110, 410, 420, 430, etc.

[0052] Storage system 450 can store various types of information, as described in more detail below. This information can be retrieved or accessed by a server computing device (e.g., one or more server computing devices 410) to perform some or all of the features described herein. As an example, the information of storage system 450 can include reports from other vehicles about difficulties encountered during previous driving. Such difficulties can include, for example, an obstructed sensor view, an obstructed road, an unprotected turn, rough terrain, a need for the driver to take over control of the vehicle, a request for assistance, or any other difficulty. Such reports of difficulties can be stored in association with metadata that identifies what the specific difficulty is and when the difficulty occurs. In some examples, the report can also be stored in association with a value indicating the severity of the difficulty.

[0053] Figure 6 Only one example of the types and arrangements of information that may be stored in storage system 450 is provided. As shown, chart 600 includes a plurality of columns listing vehicle signal identifier 610, signal type 620, time stamp 630, and location 640. It should be understood that additional information may also be listed, and that certain columns may be omitted. As just one example, chart 600 may also store, for each signal, an associated difficulty value indicating the level of difficulty imposed by the driving environment. It should also be understood that the columns and rows of chart 600 may be ordered in any order, or different organizational structures may be used together.

[0054] In the example diagram 600, a vehicle signal identifier 610 may identify a specific signal transmitted by a vehicle. Each unique signal may have a unique identifier. The identifier may be used to distinguish a specific signal and aid in the identification of specific information. The signal identifier may be included with the signal transmitted by the vehicle, assigned by the server computing device 410, or composed of some combination of information from the vehicle, the server computing device 410, or other sources. The signals in the diagram 600 may be from one vehicle or from multiple different vehicles.

[0055] Signal type 620 indicates the type of difficulty encountered by the vehicle when sending the signal. Some example types of difficulties are listed in chart 600, including the need for a human driver to manually take over control of the vehicle, requesting assistance from a remote operator, extended waits (e.g., to make a turn or other maneuver), unprotected turns, obstructed sensor views, emergency brake applications, restricted sensor views, and unprotected lane crossings. Other examples include user complaints, measured passenger discomfort, fault responses, etc. Other types of signals indicating driving difficulties caused by the driving environment may also be stored, and multiple signals of the same type may be stored.

[0056] Timestamp 630 may indicate the time at which the signal was sent by the vehicle. This information may be particularly useful for time-related difficulties. For example, it may be determined from the signal that an extended wait to make a turn at a particular location is only experienced between 2 and 4 p.m. on school days. As another example, the sensor field of view is only obstructed in the spring and summer months before the leaves fall from the trees that obstruct the view. Machine learning techniques may be used to identify such time patterns. Timestamp 630 may also be used to update information. For example, when determining an associated difficulty level, signals that exceed a predetermined threshold (e.g., a number of months or days) may be deleted or may be attributed to a lower weight. Although the timestamp information shown is expressed in "month: day: year: hour: minute: second," it should be understood that any other convention for expressing time may be used.

[0057] The location information 640 may identify the location where the signal was sent. In the example shown, the location information is indicated by a region identifier, a street identifier, and a segment identifier, although other formats may be used. The region identifier may include a value indicating a particular city, state, country, or other region. The street identifier may include a value or other indicia corresponding to a particular road, which is divided into segments for analysis purposes. Thus, the segment identifier indicates a particular segment of a street. It should be understood that the location format used in the diagram 600 is merely an example, and any of a variety of other formats may be used, such as geographic coordinates (e.g., latitude, longitude, altitude) or other descriptors.

[0058] In some examples, storage system 450 may also store other information related to the difficulty of navigating the driving environment, such as a model based on information about difficulty signals. For example, the model may be a predictive model that identifies which road sections may present difficulty at a given time, such as requiring a transition from an autonomous driving mode to a manual driving mode.

[0059] Storage system 450 may also store information that may be provided to a client computing device for display to a user. For example, storage system 450 may store predetermined distance information for determining an area where a vehicle may be parked given a pickup or destination location. Storage system 450 may also store graphics, icons, and other items that may be displayed to a user.

[0060] The vehicle 100 may use information from the storage system 450 to determine the route that the vehicle 100 is traveling. For example, given a starting point and a destination, the vehicle 100 may use several different routes. However, some routes may be more difficult to travel than other routes. The information about the difficulty of driving through the driving environment stored in the storage system 450 may be used to determine which route will present the least difficulty at a given time. In some examples, such difficulty information may be considered in conjunction with other driving information, such as time, distance, tolls, etc. Time information may take into account traffic, speed limits, and other information that affects driving time. According to some examples, the difficulty information and other driving information may be weighted. For example, by adjusting the weights given to the difficulty information and the time information, respectively, a preference for easier driving despite a longer driving time may be accommodated. By adjusting the weights given to the distance and difficulty, respectively, a preference for the shortest distance despite potential difficult conditions may be accommodated.

[0061] Example Method

[0062] In addition to the operations described above and shown in the accompanying drawings, various operations will now be described. It should be understood that the following operations need not be performed in the exact order described below. Instead, the various steps may be processed in a different order or simultaneously, and steps may also be added or omitted.

[0063] Figure 7 A flow chart 700 is presented, which illustrates an example method for selecting a route based on the difficulty of a driving environment. In block 710, data is received that provides information about a plurality of road segments. The information may include signals from one or more vehicles traveling through the road segment, such as the above combined Figure 6 The signal in question. The signal may indicate a difficulty associated with navigating the road segment, wherein such difficulty is a result of the driving environment. Examples of such difficulties include requiring manual takeover, requesting assistance from a remote operator, unprotected maneuvers, limited / obstructed / obstructed vision, etc. In some examples, the difficulty may be represented by user complaints, measured passenger discomfort, a failure response where the vehicle can no longer operate normally and needs to be pulled over, etc. Passenger discomfort may be measured, for example, by survey responses, changes in passenger heart rate, temperature or other physiology, changes in passenger seating position, control adjustments, etc.

[0064] In block 720, a difficulty value is determined for each of the plurality of road segments based on the received information. For example, the difficulty value may be a score or other value that can be used to compare the difficulty level associated with driving through the first road segment with the difficulty level of driving through the second road segment. For example, each different type of difficulty indicated by the vehicle signal may have a value. In some instances, the value of one type of difficulty may be higher than the value of another type. For example, the difficulty value associated with requesting assistance from a remote operator may be different from the difficulty value associated with an unprotected turn. In other examples, the difficulty value may vary based on other information, so that the values ​​of two signals of the same type are different. These difficulty values ​​may be used to calculate the difficulty value of the road segment. For example, the difficulty values ​​of various signals may be factored into a prediction model that predicts the difficulties that will be encountered on the road segment if traveling. Such predictions may be time-dependent, so that the predicted difficulty level may vary for a particular road segment according to the time of day, the time of week, the season, or other time information. For a particular road segment, multiple difficulties may be predicted to occur at a particular travel time. Therefore, the difficulty value may be calculated as a function of multiple predicted difficulties.

[0065] In block 730, a route cost for each of the one or more possible routes between the origin and the destination is calculated based on the difficulty values ​​associated with the plurality of road segments. For example, the route cost may be the sum of the difficulty values ​​for each road segment along the route. In some examples, the route cost may also incorporate other travel variables, such as time, distance, traffic, etc.

[0066] In block 740, a driving route for the autonomous vehicle is selected based on the calculated route cost. Thus, the control system of the autonomous vehicle can cause the vehicle to drive along the selected route.

[0067] Figure 8 An example of a plurality of possible routes 810, 820, 830 between an origin O and a destination D is shown. Routes 810-830 are mapped relative to map information (e.g. Figure 2 The map information 200) is shown.

[0068] Fig. 9Route 810, 820, 830 are shown, wherein each route includes multiple sections. For example, the first route 810 includes sections 1a-1h, the second route 820 includes sections 2a-2f, and the third route 830 includes sections 3a-3g. The sections can be defined for each road indicated on the map, or only for each possible route between the starting point and the destination. The sections can divide the road or route into any number of discrete sub-parts for analysis. For example, additional smaller sections can be used for more fine-grained analysis with potential difficulties, and fewer larger sections can be used for faster analysis consuming less memory and computing resources. In some examples, the size of the sections can be different from each other. The sections can be assigned based on any factors, such as absolute distance, distance between turns, estimated travel time, etc.

[0069] Fig.10 Possible routes 810-830 are shown, wherein the relative difficulty of each section is shown. In this example, the increased difficulty is represented by the increased line thickness. Along route 810, sections 1a and 1b have relatively low difficulty, while section 1c has a very high associated difficulty level. This may be attributed to the difficulty of maneuvering the vehicle to make such a sharp left turn. Other factors may include visibility during the turn and whether the turn is unprotected. For example, when the vehicle makes a sharp left turn, if other vehicles traveling along the road where the vehicle turns do not have to stop or avoid, the difficulty of the turn increases. In the example shown, other difficult sections of route 810 include section 1f. This may be attributed to factors other than the turn, such as a high pedestrian volume, rugged terrain, limited visibility, etc.

[0070] Along route 820, sections 2a and 2b are also shown as having relatively low relative difficulty. Because these sections correspond to the same road sections as sections 1a and 1b of route 810, it can be expected that they have the same difficulty level. However, sections 2c-2e have a higher associated difficulty level, although still lower than section 1c. Such difficulty may be the result of having to maneuver around a vehicle attempting to make a turn in section 1c, waiting to make an unprotected left turn in section 2d, or any of a variety of other possibilities. The difficulty of section 2e may be the result of other driving conditions, such as obstructed vision (e.g., buildings, signs, leaves, parked vehicles or any other objects), road construction, etc.

[0071] Along route 830, only segment 3c is shown as having an associated difficulty level. This difficulty level corresponds to the difficulty level of segment 2c and may be a result of the same driving conditions. Although segment 3d includes a left turn, such a turn may have a lower associated difficulty level because it is a protected turn where vehicles traveling in the opposite direction will stop according to stop signs 251, 252.

[0072] Fig.11 An example calculation of the route cost for each of the possible routes 810, 820, 830 is shown. For each route segment, a difficulty value attributed to the route segment is listed. Fig.10 The difficulty is indicated graphically in the example of Fig.11 However, it should be understood that any convention for indicating difficulty levels may be used. Fig.10 810, segment 1c of route 10 is indicated as having the highest associated difficulty level. Other segments with relatively increasing difficulty include segment 1f of route 810, segments 2c-2e of route 820, and segment 3c of route 830. Since route 820 does not have a segment corresponding to h, their values ​​are set to 0. The values ​​of each segment along the route can be summed to derive the total difficulty value of the route. For example, route 810 has a total difficulty value of 20, route 820 has a total difficulty value of 18, and route 830 has a total difficulty value of 11. Therefore, if difficulty is considered alone, route 830 may be the most ideal route with the lowest route cost of 11. However, as described above, in some examples, other factors, such as distance, time, etc., may be considered when calculating route costs.

[0073] The foregoing example describes an advantageous method for determining a driving route because it expands upon traditional considerations of factors such as time and distance to further consider the driving / riding experience. Thus, a passenger of an autonomous vehicle may be offered the option of selecting a smoother or less dangerous ride, even if it may take longer than an alternative route. This is particularly desirable for passengers who want to read, work, or relax during their journey.

[0074] Although multiple examples of different types of difficulties that a vehicle may encounter are described above, it should be understood that any number of more types of difficulties may also be incorporated into the route cost calculation. For example, on a particular street, garbage collection may be performed at a predetermined time, such as between 7 and 8 a.m. every Tuesday and Friday. During garbage collection, slow-moving and frequently stopped garbage trucks may become obstacles to vehicles traveling on the street. In addition, the presence of sanitation workers collecting garbage requires extra caution in the operation of vehicles traveling on the street. Therefore, difficulty signals from previous or historical driving along this street may indicate the number of obstacles (e.g., garbage trucks, garbage cans, sanitation workers, etc.) that exceed a predetermined threshold. Difficulty signals may also indicate slower driving times, increased waiting times, increased number of times the braking system is applied, unexpected maneuvers (e.g., side slips), etc. Any or all of these difficulty signals can be used to calculate the cost of a particular street during the scheduled garbage collection time.

[0075] As just one more example, a given road may be near a school. Thus, during the school day, particularly at arrival and departure times, there may be more pedestrians walking on or crossing the given road. Thus, during those school hours, there may be a greater number of obstacles present, requiring vehicles traveling on the given road to be extra careful. Difficulty signals from vehicles that have previously traveled the given road may indicate the presence of such additional obstacles during times corresponding to school hours. In some cases, such additional obstacles may result in a disengagement or request for assistance from a remote operator. These difficulty signals indicating the presence of such additional obstacles on a given road at a particular time may also be used to calculate the cost of traveling the given road.

[0076] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be understood as illustrative rather than limiting the subject matter defined by the claims. In addition, the examples described herein and sentences expressed as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; rather, the examples are intended to illustrate one of many possible embodiments. Moreover, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A method comprising: receiving, by one or more computing devices, information from one or more vehicles configured to operate in an autonomous driving mode, the information identifying different types of difficulties encountered when the one or more vehicles are operating in the autonomous driving mode along one or more routes, wherein the different types of difficulties include sensor difficulties that affect autonomous driving; storing the information in a memory along with metadata identifying one or more details about each of the different types of difficulties including the sensor difficulty, wherein the metadata further includes a weighted difficulty value for each of the different types of difficulties indicating a difficulty level associated with a driving environment; evaluating, by one or more computing devices, the stored information and the metadata considering a plurality of route options; as well as A given route option from the plurality of route options is selected, by one or more computing devices, based on the information and metadata associated with the sensor difficulty, for use when the vehicle is operating in the autonomous driving mode.

2. The method of claim 1, wherein: The information includes a vehicle signal identifier for each of the different types of difficulties that identifies a specific signal transmitted by a corresponding vehicle of the one or more vehicles.

3. The method of claim 2, wherein: The vehicle signal identifier is (i) included with a specific signal sent by the corresponding vehicle, (ii) assigned by one or more computing devices, or (iii) formed by a combination of information from the corresponding vehicle and one or more computing devices.

4. The method of claim 1, wherein: The metadata identifies at least one of a time at which the sensor difficulty was encountered or location information identifying a location at which the sensor difficulty was encountered.

5. The method of claim 4, wherein: The location information includes at least one of an area identifier, a street identifier, or a road segment identifier.

6. The method of claim 4, wherein: The location information includes geographic coordinates.

7. The method of claim 1 further comprising identifying a temporal pattern based on temporal information associated with the sensor difficulty.

8. The method of claim 1 further comprising generating a predictive model based on information stored in the memory to identify which road segments may present difficulties.

9. The method of claim 1, wherein: The information stored in the memory includes a signal type indicative of a type of difficulty encountered by each of the one or more vehicles that sent the corresponding information.

10. The method of claim 9, wherein: The type of difficulty is an obstructed sensor field of view or a restricted sensor field of view.

11. The method of claim 1 , further comprising transmitting the selected route option to a specific vehicle for operation in an autonomous driving mode.

12. A method of operating a vehicle in an autonomous driving mode, the method comprising: Controlling driving operations of the vehicle in an autonomous driving mode by one or more processors; receiving, by one or more processors, sensor information from a perception system of the vehicle regarding objects or conditions in an environment external to the vehicle; generating, by one or more processors, information regarding difficulties encountered while operating in an autonomous driving mode and associated metadata, the information identifying at least one type of difficulty encountered by the vehicle while operating in the autonomous driving mode along one or more routes, wherein the at least one type of difficulty includes sensor difficulties of sensors of a perception system, wherein the metadata further includes a weighted difficulty value for each of the at least one type of difficulty indicating a difficulty level associated with the driving environment; as well as The generated information is sent to a remote system for selecting a route from a plurality of route options based on the information and metadata associated with the sensor difficulties.

13. The method of claim 12, further identifying a temporal pattern based on temporal information associated with the sensor difficulty.

14. The method of claim 13, wherein: The generated information includes a vehicle signal identifier for each of the at least one type of difficulty.

15. The method of claim 13, wherein: The generated information includes one or both of a time or a location at which each of the at least one type of difficulty was encountered.

16. The method of claim 15, wherein: The location includes at least one of an area identifier, a street identifier, or a road segment identifier.

17. The method of claim 13, wherein: The sensor difficulty is one of an obstructed sensor field of view or a limited sensor field of view.

18. The method of claim 13, further comprising: receiving, from a remote system, a routing signal based on difficulty information obtained from one or more other vehicles; as well as Another driving operation of the vehicle is controlled by the one or more processors in the autonomous driving mode based on the received routing signal.

Images