Methods, apparatus and storage media for autonomous vehicles

Abstract

Methods, apparatus, and storage media for autonomous vehicles. For example, a computer system can deploy autonomous vehicles to one or more locations or areas, assign transportation tasks to each autonomous vehicle, provide navigation instructions to each autonomous vehicle, assign maintenance tasks to each autonomous vehicle, and / or assign other tasks to each autonomous vehicle. Furthermore, the computer system can be used to monitor the operation of autonomous vehicles. For example, the computer system can collect information from each autonomous vehicle, process the collected information, and present the information to one or more users, thereby enabling users to always be aware of information related to the operation of the autonomous vehicles.

Description

[0001] This application is a divisional application of the application filed on August 1, 2019, with application number 201910706491.4 and invention title "Remote Operation of Autonomous Vehicle". Technical Field

[0002] This specification relates to computer systems for remotely monitoring and controlling the operation of autonomous vehicles. Background Technology

[0003] Autonomous vehicles can be used to transport people and / or goods (e.g., parcels, articles, or other items) from one place to another. As an example, an autonomous vehicle may navigate to a person's location, wait for the person to board the vehicle, and then travel to a designated destination (e.g., a location chosen by the person). As another example, an autonomous vehicle may navigate to a cargo's location, wait for the cargo to be loaded onto the vehicle, and then travel to a designated destination (e.g., the cargo's delivery location). Summary of the Invention

[0004] A computer system is capable of controlling the operation of one or more autonomous vehicles. For example, the computer system can deploy autonomous vehicles to one or more locations or areas, assign transportation tasks to each of the autonomous vehicles, provide navigation instructions to each of the autonomous vehicles, assign maintenance tasks to each of the autonomous vehicles, and / or assign other tasks to each of the autonomous vehicles. Furthermore, the computer system can be used to monitor the operation of the autonomous vehicles. For example, the computer system can collect information from each of the autonomous vehicles, process the collected information, and present the information to one or more users, thereby ensuring that users are always aware of information related to the operation of the autonomous vehicles. The computer system may include one or more devices located on a communication network (e.g., a centralized network, a peer-to-peer network, and a decentralized network). In some embodiments, the computer system is a centralized computer system.

[0005] In one aspect, the computer device receives first vehicle telemetry data from each of a plurality of vehicles. For each of the plurality of vehicles, the vehicle telemetry data includes an indication of the vehicle's geographical location. At least one of the plurality of vehicles is an autonomous vehicle. Further, a user interface is presented by a display device associated with the computer device. The user interface is generated based on the first vehicle telemetry data. The user interface includes a graphical map and one or more first display elements. Each first display element at least indicates the corresponding geographical location of the corresponding vehicle among the plurality of vehicles. The computer system receives first user input that selects a specific vehicle among the plurality of vehicles. In response to receiving the first user input, the computer device acquires second vehicle telemetry data from the selected vehicle, and the display device presents on the user interface a visual representation of at least the portion of the first vehicle telemetry data relevant to the selected vehicle or the portion of the second vehicle telemetry data relevant to the selected vehicle.

[0006] Implementation of this aspect may include one or more of the following features.

[0007] In some embodiments, for each of a plurality of vehicles, the first vehicle telemetry data includes geographic coordinates corresponding to the geographic location of the vehicle.

[0008] In some embodiments, for each of a plurality of vehicles, the first vehicle telemetry data includes an indication of the vehicle’s altitude.

[0009] In some embodiments, the second vehicle telemetry data includes at least one of video or images captured by sensors of the selected vehicle.

[0010] In some embodiments, the second vehicle telemetry data includes an indication of the speed of the selected vehicle.

[0011] In some embodiments, the second vehicle telemetry data includes an indication of the orientation of the selected vehicle.

[0012] In some embodiments, the second vehicle telemetry data includes indications of the operational status of the selected vehicle's computer system.

[0013] In some embodiments, the second vehicle telemetry data includes indications of the condition of one or more batteries of the selected vehicle.

[0014] In some embodiments, the second vehicle telemetry data includes an indication of the energy consumption of the selected vehicle.

[0015] In some embodiments, the second vehicle telemetry data includes indications of the operational status of the selected vehicle.

[0016] In some embodiments, the second vehicle telemetry data includes information relating to the environment of the selected vehicle. In some embodiments, the information relating to the environment of the selected vehicle includes indications of one or more objects located near the selected vehicle. In some embodiments, the indications relating to the environment of the selected vehicle include indications of weather in the environment. In some embodiments, the information relating to the environment of the selected vehicle includes at least one of indications of one or more parking spaces in the environment or indications of landmarks in the environment.

[0017] In some embodiments, the second vehicle telemetry data includes information related to the route of the selected vehicle.

[0018] In some embodiments, at least one of the first vehicle telemetry data and / or the second vehicle telemetry data is text data. In some embodiments, at least one of the first vehicle telemetry data or the second vehicle telemetry data includes one or more data items in JavaScript Object Notation (JSON) format.

[0019] In some embodiments, the computer device further receives an indication of abnormal operation of the first vehicle from a plurality of vehicles. In response to receiving the indication of abnormal operation of the first vehicle, a display device presents an alarm related to the abnormal operation of the first vehicle on a user interface. In some embodiments, the indication of abnormal operation of the first vehicle is at least one of an indication of interruption of network connection between the first vehicle and the computer device or an indication that the path of the first vehicle is blocked.

[0020] In some embodiments, the user interface includes a second display element that indicates one or more vehicles among a plurality of vehicles in a list.

[0021] In some embodiments, a visual representation of at least the portion of the telemetry data of a first vehicle relating to the selected vehicle or the portion of the telemetry data of a second vehicle relating to the selected vehicle includes presenting video captured by sensors of the selected vehicle and one or more graphic elements overlaid on the video. The one or more graphic elements indicate one or more objects detected in the video.

[0022] In some embodiments, for at least one of a plurality of autonomous vehicles, the user interface includes instructions for the assigned tasks of the autonomous vehicle (e.g., drive to customer / package, deliver customer / package, charge, idle, reposition, etc.).

[0023] In some embodiments, the computer device further receives second user input, which includes one or more search criteria regarding a plurality of vehicles. The computer device determines one or more vehicles from the plurality of vehicles that meet the one or more search criteria. A display device displays a visual representation of the one or more vehicles that meet the one or more search criteria on the user interface. In some embodiments, the one or more search criteria include indications of service facilities associated with one or more autonomous vehicles. Determining one or more vehicles that meet the one or more search criteria includes identifying one or more vehicles located near service facilities.

[0024] On the other hand, a user interface is presented on the display device of the autonomous vehicle. The user interface includes a visual representation of the environment surrounding the autonomous vehicle, a first display element indicating the physical position of the autonomous vehicle relative to the environment, and one or more second display elements. Each second display element indicates a corresponding operational attribute of the autonomous vehicle. User input specifying the operation to be performed by the autonomous vehicle is received through the user interface. In response to receiving the user input, the specified operation is performed using the autonomous vehicle.

[0025] Implementation of this aspect may include one or more of the following features.

[0026] In some embodiments, at least one second display element indicates the route of the autonomous vehicle.

[0027] In some embodiments, at least one second display element indicates the occurrence of an event related to the autonomous vehicle. In some embodiments, the event is traffic congestion in the environment of the autonomous vehicle. In some embodiments, the event is road construction in the environment of the autonomous vehicle. In some embodiments, the event is a decrease in the speed of a traffic pattern along the path of the autonomous vehicle.

[0028] In some embodiments, at least one second display element comprises video captured by sensors of the autonomous vehicle. In some implementations, at least one second display element may include one or more graphic elements overlaid on the video, the one or more graphic elements indicating one or more detected objects in the video.

[0029] In some embodiments, at least one second display element indicates an object detected by the autonomous vehicle.

[0030] In some embodiments, at least one second display element indicates the classification of an object detected by the autonomous vehicle. In some embodiments, the classification may be at least one of a vehicle, a pedestrian, or a stationary object. In some embodiments, at least one second display element indicates a confidence measure associated with the classification of the object detected by the autonomous vehicle.

[0031] In some embodiments, the autonomous vehicle further determines the occurrence of a triggering event related to the autonomous vehicle, acquires video captured by the autonomous vehicle's sensors, identifies a portion of the video corresponding to the triggering event, and presents that portion of the video in a user interface on a display device. In some embodiments, the triggering event is at least one of the following events: detecting an object approaching the autonomous vehicle, detecting an object along the path of the autonomous vehicle, activating emergency braking of the autonomous vehicle, or activating evasive maneuvers of the autonomous vehicle.

[0032] In some embodiments, the autonomous vehicle detects multiple objects in its vicinity. One or more second display elements indicate a subset of the multiple objects.

[0033] In some embodiments, performing a specified operation using an autonomous vehicle includes displaying additional information related to the autonomous vehicle.

[0034] In some embodiments, performing the specified operation using an autonomous vehicle includes modifying the route of the autonomous vehicle.

[0035] In some embodiments, performing the specified operation using an autonomous vehicle includes modifying the destination of the autonomous vehicle.

[0036] On the other hand, the computer system receives a request from an autonomous vehicle located remotely to the computer system for remotely controlling the operation of the autonomous vehicle. The request includes information relating to one or more objects located near the autonomous vehicle. A user interface generated based on the received request is presented on the computer system's display device. The user interface includes a visual representation of the environment surrounding the autonomous vehicle, a first display element indicating the physical position of the autonomous vehicle relative to the environment, and a second display element indicating one or more objects located near the autonomous vehicle. The computer system receives user input specifying a route for the autonomous vehicle to traverse one or more objects in the environment. In response to receiving the user input, command signals are transmitted from the computer system to the autonomous vehicle. The command signals include instructions for navigating the specified route for the autonomous vehicle.

[0037] Implementation of this aspect may include one or more of the following features.

[0038] In some embodiments, information relating to one or more objects includes at least one of videos or images captured by sensors of an autonomous vehicle.

[0039] In some embodiments, for each object, the information associated with one or more objects includes an indication of the object's classification and an indication of the object's position relative to the autonomous vehicle.

[0040] In some embodiments, the indication of the object's position relative to the autonomous vehicle includes text information indicating the object's position relative to the autonomous vehicle. In some embodiments, the text information describing the object's position relative to the autonomous vehicle includes one or more data items in JavaScript Object Notation (JSON) format.

[0041] In some embodiments, the second display element includes: a graphical representation of a plurality of areas surrounding the autonomous vehicle; and a graphical indication that at least one of one or more objects is positioned within at least one of the areas. In some embodiments, presenting the user interface includes generating the second display element based on text data to indicate the position of each object relative to a plurality of two-dimensional boxes of a graphical grid. In some embodiments, the text data indicating the position of an object relative to the plurality of boxes of the graphical grid includes one or more data items in JavaScript Object Notation (JSON) format.

[0042] In some embodiments, receiving user input specifying a path for an autonomous vehicle to traverse one or more objects in the environment includes detecting physical gestures performed by the user relative to a second display element using an input device. In some embodiments, the display device is a touch-sensitive display device, and the physical gestures are detected using the touch-sensitive display device.

[0043] In some embodiments, the specified path is modified based on one or more reference paths. In some embodiments, the one or more reference paths correspond to traffic patterns on roads in the environment surrounding the autonomous vehicle. In some embodiments, modifying the specified path includes aligning the specified path with one or more reference paths.

[0044] On the other hand, the autonomous vehicle receives sensor data from one or more of its sensors. The autonomous vehicle generates a request for remote control of the autonomous vehicle by a computer system located remotely to the vehicle. The request includes a quality metric determined by the autonomous vehicle as associated with the network connection between the autonomous vehicle and the computer system. When the quality metric is determined to be greater than a threshold quality level, a first data item representing sensor data is included in the request, which meets one or more conditions associated with the threshold quality level. The remote control request is transmitted from the autonomous vehicle to the computer system.

[0045] Implementation of this aspect may include one or more of the following features.

[0046] In some embodiments, sensor data includes at least one of video, images, or proximity data captured by one or more sensors.

[0047] In some embodiments, the quality metric corresponds to the available bandwidth of the network connection.

[0048] In some embodiments, the quality metric corresponds to the latency associated with the network connection.

[0049] In some embodiments, the quality metric corresponds to the reliability of the network connection.

[0050] In some embodiments, determining a quality metric associated with a network connection includes: transmitting a beacon signal using the network connection; and determining the quality metric based on the transmission of the beacon signal.

[0051] In some embodiments, determining quality metrics based on beacon signal transmission includes determining the available bandwidth of a network connection based on the beacon signal transmission.

[0052] In some embodiments, one or more conditions associated with the threshold quality level include at least one of the data size or complexity of the sensor data.

[0053] In some embodiments, the autonomous vehicle determines a second quality metric associated with the network connection between the autonomous vehicle and the computer system. When the second quality metric is determined to be below a threshold quality level, a second data item representing sensor data is included in the request, the second data item meeting one or more conditions associated with the threshold quality level. The second data item has a smaller data size or lower complexity compared to the first data item. In some embodiments, the first data item includes video with a higher resolution compared to the video included in the second data item. In some embodiments, the first data item includes video with a higher frame rate compared to the video included in the second data item. In some embodiments, the first data item includes images with a higher resolution compared to the images included in the second data item. In some embodiments, the first data item includes a greater number of images compared to the second data item. In some embodiments, the first data item includes a portion of sensor data with a higher spatial resolution compared to the portion of that sensor data included in the second data item. In some embodiments, the first data item includes a portion of sensor data with a higher temporal resolution compared to the portion of that sensor data included in the second data item. In some embodiments, the first data item includes at least one of video or images of the autonomous vehicle's environment. In some embodiments, the second data item includes text data describing the autonomous vehicle's environment, rather than at least one of video or images. In some embodiments, text data includes one or more data items in JavaScript Object Notation (JSON), Hypertext Markup Language (HTML), or Extensible Markup Language (XML) data formats.

[0054] In some embodiments, the autonomous vehicle determines a condition that impedes its movement. A remote control request is generated in response to the determination of this condition. In some embodiments, the condition impeding the autonomous vehicle corresponds to an object blocking its path. In some embodiments, the condition impeding the autonomous vehicle corresponds to the closure of a road along its path.

[0055] In some embodiments, the autonomous vehicle receives a command signal that includes instructions for navigating a specified path, and the autonomous vehicle executes the command signal to navigate the specified route.

[0056] These and other aspects, features, and implementations can be expressed as methods, apparatus, systems, components, program products, devices or steps for performing functions, and in other ways.

[0057] These and other aspects, features, and implementations will become apparent from the following description, including the claims. Attached Figure Description

[0058] Figure 1 An example of an autonomous vehicle with autonomous capabilities is shown.

[0059] Figure 2 An exemplary “cloud” computing environment is shown.

[0060] Figure 3 The computer system is shown.

[0061] Figure 4 An example architecture for an autonomous vehicle is shown.

[0062] Figure 5 Examples of inputs and outputs that can be used by the perception module are shown.

[0063] Figure 6 An example of a LiDAR system is shown.

[0064] Figure 7 The LiDAR system in operation is shown.

[0065] Figure 8 The operation of the LiDAR system is shown in more detail.

[0066] Figure 9 A block diagram showing the relationship between the inputs and outputs of the planning module is provided.

[0067] Figure 10 The directed graph used in route planning is shown.

[0068] Figure 11 A block diagram showing the inputs and outputs of the control module is provided.

[0069] Figure 12 A block diagram showing the controller's inputs, outputs, and components is provided.

[0070] Figures 13-15 An example of a computer system for controlling and monitoring the operation of a queue of autonomous vehicles is shown.

[0071] Figures 16-19 This illustrates an example of dynamic information exchange between an autonomous vehicle and a computer system, based on the network connectivity between the two.

[0072] Figures 20-22 An example visual representation depicting the relative positions of obstacles and autonomous vehicles is shown.

[0073] Figures 23-29 An example graphical user interface for monitoring and controlling one or more autonomous vehicles is shown.

[0074] Figures 30-36 An example graphical user interface for monitoring and controlling the operation of autonomous vehicles is shown.

[0075] Figure 37 A flowchart illustrates an example process for monitoring and controlling the operation of autonomous vehicles.

[0076] Figure 38 Another example process for monitoring and controlling the operation of autonomous vehicles is illustrated in a flowchart.

[0077] Figure 39 Another example process for monitoring and controlling the operation of autonomous vehicles is illustrated in a flowchart.

[0078] Figure 40 Another example process for monitoring and controlling the operation of autonomous vehicles is illustrated in a flowchart. Detailed Implementation

[0079] In the following description, numerous specific details are set forth for illustrative purposes in order to provide a full understanding of the invention. However, it will be apparent that the invention can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the invention.

[0080] In the accompanying drawings, for ease of description, a specific arrangement or order of schematic elements is shown, such as schematic elements representing devices, modules, instruction blocks, and data elements. However, those skilled in the art should understand that the specific order or arrangement of the schematic elements in the drawings does not imply a requirement for a specific processing order or sequence, or separation of processes. Furthermore, the inclusion of schematic elements in the drawings does not imply that such elements are required in all embodiments, or that in some embodiments, features represented by such elements may not be included in or combined with other elements.

[0081] Furthermore, in the accompanying drawings, connecting elements such as solid or dashed lines or arrows are used to illustrate connections, relationships, or associations between two or more other schematic elements. The absence of any such connecting elements does not imply the absence of any connection, relationship, or association. In other words, some connections, relationships, or associations between elements are not shown in the drawings so as not to obscure this disclosure. Additionally, for ease of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, where a connecting element represents communication of signals, data, or instructions, those skilled in the art will understand that such an element represents one or more signal paths (e.g., a bus) that may be required to enable communication.

[0082] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description to provide a thorough understanding of the various described embodiments. However, it will be apparent to those skilled in the art that the various described embodiments can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail to avoid unnecessarily obscuring aspects of the embodiments.

[0083] Several features are described below, which can be used independently of each other or in any combination with other features. However, any single feature may not solve any of the problems discussed above, or may only solve one of the problems discussed above. Some of the problems discussed above may not be fully solved by any of the features described herein. Although headings are provided, information relating to a particular heading is not found in the section containing that heading, but may be found elsewhere in the specification. Embodiments are described herein based on the following overview:

[0084] 1. General Overview

[0085] 2. Hardware Overview

[0086] 3. Autonomous Vehicle Architecture

[0087] 4. Autonomous Vehicle Input

[0088] 5. Autonomous Vehicle Planning

[0089] 6. Autonomous Vehicle Control

[0090] 7. Remotely monitor and control the operation of autonomous vehicles.

[0091] 8. Example procedures for monitoring and controlling the operation of a fleet of autonomous vehicles.

[0092] General Overview

[0093] A computer system can control the operation of one or more autonomous vehicles (e.g., a fleet of autonomous vehicles). For example, the computer system can deploy autonomous vehicles to one or more locations or areas, assign transportation tasks to each autonomous vehicle (e.g., pick up and transport passengers, pick up and transport goods, etc.), provide navigation instructions to each autonomous vehicle (e.g., provide a path or route between two locations, provide instructions to pass through objects near the autonomous vehicle, etc.), assign maintenance tasks to each autonomous vehicle (e.g., charge its battery at a charging station, receive repairs at a service station, etc.), and / or assign other tasks to each autonomous vehicle.

[0094] Furthermore, computer systems can be used to monitor the operation of autonomous vehicles. For example, a computer system can collect information from each part of an autonomous vehicle (e.g., vehicle telemetry data, such as data on the vehicle's speed, orientation, status, or other aspects of its operation), process the collected information, and present that information to one or more users (e.g., in the form of an interactive graphical user interface) so that users are always aware of information about the operation of the autonomous vehicle.

[0095] In some embodiments, the computer system includes one or more devices located on a communication network (e.g., a centralized network, a peer-to-peer network, and a decentralized network). Further, the computer system and the autonomous vehicle exchange information dynamically based on the network conditions between them. For example, if the quality of the network connection between the computer system and the autonomous vehicle is high (e.g., high available bandwidth, low latency, and / or high reliability), the computer system can obtain higher quality data (e.g., more detailed data, larger data size, and / or more complex data) from the autonomous vehicle. As another example, if the quality of the network connection between the computer system and the autonomous vehicle is low (e.g., low available bandwidth, high latency, and / or lower reliability), the computer system can obtain lower quality data (e.g., less detailed data, smaller data size, and / or less complex data) from the autonomous vehicle.

[0096] The topics described herein offer several technical benefits. For example, some implementations can improve the efficiency and effectiveness of the entire autonomous vehicle fleet and individual autonomous vehicles. As an example, by dynamically adjusting data transmission between the computer system and the autonomous vehicle based on the quality of the network connection between them, it becomes more likely that the computer system and the autonomous vehicle will exchange relevant information in a timely and reliable manner, even if the performance of the communication network changes. For instance, when the autonomous vehicle is located in an area with good network connectivity, it can transmit detailed information about its operation to the computer system. This allows the computer system to control and / or monitor the operation of the autonomous vehicle in a more accurate and / or comprehensive manner. However, when the autonomous vehicle is located in an area with poor network connectivity, it can transmit less detailed information about its operation to the computer system, allowing the computer system to continue receiving specific information (e.g., information that is relatively more important for the security control and / or monitoring of the operation of the autonomous vehicle) without exceeding the capacity of the network connection. Thus, the computer system and the autonomous vehicle can communicate with each other more efficiently and reliably. This enables autonomous vehicles to operate more efficiently (e.g., by enabling computer systems to generate instructions in a timely manner based on information obtained from the autonomous vehicle, even in congested networks). It also enables autonomous vehicles to operate more safely (e.g., by enabling computer systems to respond more quickly to hazards detected by the autonomous vehicle, such as obstacles or impending collisions, even in congested networks).

[0097] Hardware Overview

[0098] Figure 1 An example of an autonomous vehicle 100 with autonomous capabilities is shown.

[0099] As used herein, the term “autonomy” refers to the function, feature, or facility that enables a vehicle to be operated without real-time human intervention (unless specifically requested by the vehicle).

[0100] As used in this article, an autonomous vehicle (AV) is a vehicle with autonomous capabilities.

[0101] As used in this article, "vehicle" includes means of transferring goods or people. Examples include cars, buses, trains, airplanes, drones, trucks, boats, ships, submersibles, spacecraft, etc. Driverless cars are an example of AV.

[0102] As used herein, a “track” refers to a route or path generated by an AV for navigating from a first spatiotemporal location to a second spatiotemporal location. In embodiments, the first spatiotemporal location refers to an initial or starting location, and the second spatiotemporal location refers to a destination, final location, target, target location, or target position. In some examples, a track consists of one or more segments (e.g., road segments), and each segment consists of one or more blocks (e.g., portions of streets or intersections). In embodiments, spatiotemporal locations correspond to real-world locations. For example, a spatiotemporal location is a pick-up or drop-off point for picking up or dropping off people or goods.

[0103] As used herein, a “sensor” includes one or more physical components that detect information about the environment surrounding the physical component. Some of these physical components may include electronic components such as analog-to-digital converters, buffers (such as RAM and / or non-volatile memory), and data processing components such as ASICs (Application-Specific Integrated Circuits), microprocessors, and / or microcontrollers.

[0104] "One or more" includes: functions performed by a single element; functions performed by more than one element, for example, in a distributed manner; several functions performed by a single element; several functions performed by several elements; or any combination of the foregoing.

[0105] It will also be understood that, although in some instances the terms first, second, etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact, without departing from the scope of the various described embodiments. Both the first contact and the second contact are contacts, but they are not the same contact.

[0106] The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the described embodiments and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and includes any one of the associated listed items and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “comprises,” and / or “comprising,” when used in this application, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0107] As used herein, depending on the context, the term "if" may optionally be interpreted as "when," "after," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrases "if determined" or "if detected (the stated condition or event)" may optionally be interpreted as "after determining," "in response to determination," "after detecting (the stated condition or event)," or "in response to detecting (the stated condition or event)."

[0108] As used herein, an AV system refers to the AV and the array of hardware, software, stored data, and real-time generated data that support the operation of the AV. In embodiments, the AV system is incorporated into an AV. In embodiments, the AV system is distributed across several locations. For example, some of the software in the AV system is similar to that described below. Figure 3 The cloud computing environment described is implemented in the cloud computing environment 300.

[0109] In general, this document describes techniques applicable to any vehicle with one or more autonomous capabilities, including fully autonomous vehicles, highly autonomous vehicles, and conditionally autonomous vehicles, such as so-called Level 5, Level 4, and Level 3 vehicles, respectively (see SAE International Standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, which is incorporated herein by reference in its entirety for further details regarding the classification of vehicle autonomy levels). The techniques described herein can also be applied to partially autonomous vehicles and driver-assisted vehicles, such as so-called Level 2 and Level 1 vehicles (see SAE International Standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems). In embodiments, one or more of the Level 1, Level 2, Level 3, Level 4, and Level 5 vehicle systems can automate certain vehicle operations (e.g., steering, braking, and map usage) under certain operating conditions based on the processing of sensor inputs. The technology described in this article can benefit vehicles of any level, ranging from fully autonomous vehicles to manually operated vehicles.

[0110] refer to Figure 1 The AV system 120 autonomously or semi-autonomously operates the AV 100 along the trajectory 198 through the environment 190 to the destination 199 (sometimes referred to as the final location), while avoiding objects (e.g., natural obstacles 191, vehicles 193, pedestrians 192, cyclists and other obstacles) and complying with road rules (e.g., operating or driving convention rules).

[0111] In one embodiment, the AV system 120 includes a device 101 configured to receive and act on operation commands from a computer processor 146. In this embodiment, the computer processor 146 is referred to below. Figure 3 The processor 304 described is similar. Examples of device 101 include steering control 102, brake 103, gears, accelerator pedal or other acceleration control mechanism, windshield wipers, side door locks, window controls, and turn indicators.

[0112] In an embodiment, the AV system 120 includes sensors 121 for measuring or inferring attributes of the state or condition of the AV 100, such as the AV's position, linear velocity and angular velocity and acceleration, and orientation (e.g., the orientation of the front end of the AV 100). Examples of sensors 121 include GPS, an inertial measurement unit (IMU) for measuring the vehicle's linear acceleration and angular velocity, a wheel speed sensor for measuring or estimating wheel slip ratio, a wheel braking pressure or braking torque sensor, an engine torque or wheel torque sensor, and steering angle and angular velocity sensors.

[0113] In this embodiment, sensor 121 also includes sensors for sensing or measuring properties of the environment of the AV. For example, a monocular or stereo video camera 122 employing visible light, infrared, or thermal (or both) spectra, LiDAR 123, radar, ultrasonic sensors, time-of-flight (TOF) depth sensors, velocity sensors, temperature sensors, humidity sensors, and precipitation sensors.

[0114] In one embodiment, the AV system 120 includes a data storage unit 142 and a memory unit 144 for storing machine instructions associated with data collected by the computer processor 146 or by the sensor 121. In another embodiment, the data storage unit 142 is related to the data collected by the computer processor 146 or the data collected by the sensor 121. Figure 3 The ROM 308 or storage device 310 is similar. In this embodiment, the memory 144 is similar to the main memory 306 described below. In this embodiment, the data storage unit 142 and the memory 144 store historical, real-time, and / or predictive information related to the environment 190. In this embodiment, the stored information includes maps, driving performance, traffic congestion updates, or weather conditions. In this embodiment, data related to the environment 190 is transmitted from a remote database 134 to the AV 100 via a communication channel.

[0115] In an embodiment, AV system 120 includes communication devices 140 for communicating measured or inferred attributes of the status and condition of other vehicles (such as position, linear velocity and angular velocity, linear acceleration and angular acceleration, and linear and angular directions of travel) to AV 100. These devices include vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication devices and devices for wireless communication via point-to-point or ad hoc networks or both. In an embodiment, communication devices 140 communicate across the electromagnetic spectrum (including radio and optical communications) or other media (e.g., air and acoustic media). The combination of vehicle-to-vehicle (V2V) communication and vehicle-to-infrastructure (V2I) communication (and in some embodiments, one or more other types of communication) is sometimes referred to as vehicle-to-all (V2X) communication. V2X communication typically adheres to one or more communication standards for communicating with autonomous vehicles or between autonomous vehicles.

[0116] In an embodiment, the communication device 140 includes a communication interface. For example, this may be a wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near-field, infrared, or radio interface. The communication interface transmits data from a remote database 134 to the AV system 120. In an embodiment, such as... Figure 2 As described, a remote database 134 is embedded in a cloud computing environment 200. A communication interface 140 transmits data collected from sensor 121 or other data related to the operation of AV 100 to the remote database 134. In some embodiments, the communication interface 140 transmits information related to remote operation to AV 100. In some embodiments, AV 100 communicates with other remote (e.g., "cloud") servers 136.

[0117] In this embodiment, a remote database 134 also stores and transmits digital data (e.g., data such as road and street locations). Such data is stored in a memory 144 located on the AV100 or transmitted from the remote database 134 to the AV100 via a communication channel.

[0118] In one embodiment, a remote database 134 stores and transmits historical information (e.g., speed and acceleration distribution) about the vehicle's driving attributes, such as those it previously traveled along trajectory 198 at similar times of day. In one implementation, such data may be stored in a memory 144 located on the AV100, or transmitted from the remote database 134 to the AV100 via a communication channel.

[0119] The computing device 146 located on the AV100 algorithmically generates control actions based on real-time sensor data and existing information, thereby allowing the AV system 120 to perform its autonomous driving capabilities.

[0120] In one embodiment, the AV system 120 includes a computer peripheral device 132 coupled to a computing device 146 for providing information and alerts to a user of the AV 100 (e.g., a passenger or a remote user) and receiving input from that user. In this embodiment, the peripheral device 132 is referred to below. Figure 3 The display 312, input device 314, and cursor controller 316 discussed are similar. Coupling can be wireless or wired. Any two or more interface devices can be integrated into a single device.

[0121] Figure 2 An exemplary "cloud" computing environment is illustrated. Cloud computing is a service delivery model used to enable convenient, on-demand network access to a shared pool of configurable computing resources, such as networks, network bandwidth, servers, processing power, storage, applications, virtual machines, and services. In a typical cloud computing system, one or more large cloud data centers house the machines used to deliver services provided by the cloud. Now refer to Figure 2 The cloud computing environment 200 includes cloud data centers 204a, 204b, and 204c, which are interconnected through cloud 202. Data centers 204a, 204b, and 204c provide cloud computing services to computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to cloud 202.

[0122] A cloud computing environment 200 includes one or more cloud data centers. Typically, a cloud data center (e.g., Figure 2 The cloud data center 204a) shown refers to the physical arrangement of servers to form a cloud, for example... Figure 2 The cloud 202 shown refers to a specific portion of the cloud. For example, servers are physically arranged in rooms, groups, rows, and racks within a cloud data center. A cloud data center has one or more zones that include one or more server rooms. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementations, servers located in zones, rooms, racks, and / or rows are arranged in groups based on the physical infrastructure requirements of the data center facility (including power, energy, heat, thermal, and / or other requirements). In this embodiment, server nodes and Figure 3 The computer system described herein is similar. Data center 204a has multiple computing systems distributed across multiple racks.

[0123] Cloud 202 includes cloud data centers 204a, 204b, and 204c, and the networks and network resources (e.g., network devices, nodes, routers, switches, and network cables) that interconnect cloud data centers 204a, 204b, and 204c, and helps cloud computing systems 206a-f to more easily access cloud computing services. In embodiments, a network represents any combination of one or more local area networks, wide area networks, or the Internet, coupled using wired or wireless links deployed via terrestrial or satellite connections. Data exchanged on the network is transmitted using any number of network layer protocols, such as Internet Protocol (IP), Multiprotocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), and Frame Relay. Further, in embodiments where the network represents a combination of multiple subnets, each of the subnets uses a different network layer protocol. In some embodiments, a network represents one or more interconnected Internets, such as the public Internet.

[0124] The computing systems 206a-f or cloud computing device consumers are connected to the cloud 202 via network links and network adapters. In embodiments, the computing systems 206a-f are implemented as various computing devices, such as servers, desktop computers, laptop computers, tablets, smartphones, IoT devices, autonomous vehicles (including cars, drones, shuttle buses, trains, buses, etc.), and consumer electronics. In embodiments, the computing systems 206a-f are implemented in other systems or as part of other systems.

[0125] Figure 3 Computer system 300 is illustrated. In implementations, computer system 300 is a dedicated computing device. A dedicated computing device is hardwired and is used to perform technologies or includes digital electronic devices (such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs)) that are persistently programmed to perform technologies, or may include one or more general-purpose hardware processors programmed to execute technologies according to program instructions within firmware, memory, other storage, or combinations thereof. Such dedicated computing devices may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to implement these technologies. In various embodiments, the dedicated computing device is a desktop computer system, a portable computer system, a handheld device, a network device, or any other device containing hardwired and / or program logic for implementing these technologies.

[0126] In an embodiment, computer system 300 includes a bus 302 or other communication mechanism for transmitting information and a hardware processor 304 coupled to the bus 302 and used for processing information. The hardware processor 304 is, for example, a general-purpose microprocessor. Computer system 300 also includes main memory 306 (such as random access memory (RAM) or other dynamic storage device) coupled to the bus 302 for storing information and instructions to be executed by processor 304. In one implementation, main memory 306 is used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 304. Such instructions, when stored in a non-transitory storage medium accessible to processor 304, present computer system 300 as a dedicated machine customized to perform the operations specified in those instructions.

[0127] In an embodiment, the computer system 300 further includes a read-only memory (ROM) 308 or other static storage device coupled to the bus 302 and used to store static information and instructions for the processor 303. A storage device 310 is provided, coupled to the bus 302 for storing information and instructions; this storage device may be, for example, a magnetic disk, optical disk, solid-state drive, or three-dimensional cross-point memory.

[0128] In this embodiment, computer system 300 is coupled to display 312 via bus 302 to display information to a computer user. Display 312 may be, for example, a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, light-emitting diode (LED) display, or organic light-emitting diode (OLED) display. Input device 314, including alphanumeric and other keys, is coupled to bus 302 to transmit information and command selections to processor 304. Another type of user input device is a cursor controller 316, such as a mouse, trackball, touch-enabled display, or cursor arrow keys, which transmits directional information and command selections to processor 304 and controls cursor movement on display 312. This input device typically has two degrees of freedom on two axes (a first axis (e.g., x-axis) and a second axis (e.g., y-axis)), allowing the device to specify a position in a plane.

[0129] According to one embodiment, the techniques described herein are executed by computer system 300 in response to processor 304 executing one or more sequences of one or more instructions contained in main memory 306. Such instructions are read into main memory 306 from another storage medium (such as storage device 310). Execution of the sequence of instructions contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hardwired circuitry is used in place of software instructions, or in combination with software instructions.

[0130] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media include non-volatile and / or volatile media. Non-volatile media include, for example, optical discs, magnetic disks, solid-state drives, or three-dimensional cross-point memory, such as storage device 310. Volatile media include dynamic memory, such as main memory 306. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical medium with a perforated pattern, RAM, PROMs and EPROMs, flash memory-EPROMs, NV-RAMs, or any other memory chip or cassette memory.

[0131] Storage media differ from transmission media, but can be used in conjunction with them. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including wires containing bus 302. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0132] In embodiments, various forms of media involve carrying one or more sequences of one or more instructions to processor 304 for execution. For example, the instructions are initially carried on a disk or solid-state drive of a remote computer. The remote computer loads these instructions into its dynamic memory and transmits them over a telephone line using a modem. A modem local to computer system 300 receives data over the telephone line and converts the data into an infrared signal using an infrared transmitter. An infrared detector receives the data carried in the infrared signal, and appropriate circuitry places the data on bus 302. Bus 302 carries the data to main memory 306, from which processor 304 fetches and executes the instructions. The instructions received by main memory 306 may optionally be stored on storage device 310 before or after execution by processor 304.

[0133] Computer system 300 also includes a communication interface 318 coupled to bus 302. Communication interface 318 provides bidirectional data communication coupling to network link 320, which connects to local network 322. For example, communication interface 318 is an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem for providing data communication connectivity to a corresponding type of telephone line. As another example, communication interface 318 is a local area network (LAN) card for providing data communication connectivity to a compatible LAN. In some implementations, a wireless link is also implemented. In any such implementation, communication interface 318 transmits and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0134] Network link 320 typically provides data communication to other data devices via one or more networks. For example, network link 320 provides a connection via local network 322 to host computer 324 or to a cloud data center or facility operated by Internet Service Provider (ISP) 326. ISP 326 then provides data communication services via a worldwide packet data communication network (now commonly referred to as the "Internet" 328). Both local network 322 and Internet 328 use electrical, electromagnetic, or optical signals carrying digital data streams. Signals through various networks, as well as signals on network link 320 and through communication interface 318, are exemplary forms of transmission media carrying digital data to and from computer system 300. In embodiments, network 320 includes cloud 202 or a portion of cloud 202 as described above.

[0135] Computer system 300 sends messages and receives data including program code through networks(s), network links 320, and communication interfaces 318. In an embodiment, computer system 300 receives code for processing. The received code is executed by processor 304 upon receipt and / or stored in storage device 310 and / or other non-volatile memory for later execution.

[0136] Autonomous Vehicle Architecture

[0137] Figure 4 An autonomous vehicle (e.g., is shown) Figure 1 The example architecture 400 of the AV100 shown is illustrated. Architecture 400 includes a prediction module 402 (sometimes referred to as a prediction circuit), a planning module 404 (sometimes referred to as a planning circuit), a control module 406 (sometimes referred to as a control circuit), a positioning module 408 (sometimes referred to as a positioning circuit), and a database module 410 (sometimes referred to as a database circuit). Each module plays a role in the operation of the AV100. Modules 402, 404, 406, 408, and 410 together serve as… Figure 1 This is a portion of the AV system 120 shown. In some embodiments, any of modules 402, 404, 406, 408, and 410 is a combination of computer software and computer hardware.

[0138] In use, the planning module 404 receives data representing the destination 412 and determines data representing the trajectory 414 (sometimes referred to as the route), along which the AV100 can travel to (e.g., arrive at) the destination 412. To allow the planning module 404 to determine the data representing the trajectory 414, the planning module 404 receives data from the prediction module 402, the positioning module 408, and the database module 410.

[0139] Prediction module 402 uses one or more sensors 121 (e.g., also as Figure 1 (As shown in the diagram) identifies nearby physical objects. Objects are categorized (e.g., grouped into types such as pedestrians, bicycles, motor vehicles, traffic signs, etc.) and data representing the categorized objects 416 are provided to the planning module 404.

[0140] The planning module 404 also receives data representing the location 418 of the AV from the positioning module 408. The positioning module 408 determines the location of the AV by calculating the location using data from the sensor 121 and data from the database module 410 (e.g., geographic data). For example, the positioning module 408 uses data from GNSS (Global Navigation Satellite System) sensors and geographic data to calculate the longitude and latitude of the AV. In embodiments, the data used by the positioning module 408 includes high-precision maps of road geometry properties, maps describing road network connectivity properties, maps describing road physical properties (such as traffic speed, traffic volume, number of vehicle and bicycle lanes, lane width, lane traffic direction, or lane marking type and location, or combinations thereof), and maps describing the spatial location of road features (such as pedestrian crossings, traffic signs, or other various types of traffic signs).

[0141] Control module 406 receives data representing trajectory 414 and data representing AV position 418, and operates AV control functions 420a-c (e.g., steering, throttle, braking, ignition) in a manner that would cause AV 100 to travel along trajectory 414 to destination 412. For example, if trajectory 414 includes a left turn, control module 406 will operate control functions 420a-c in such a manner that the steering angle of the steering function will cause AV 100 to turn left, and the throttle and braking will cause AV 100 to stop before the turn and wait for pedestrians or vehicles.

[0142] Autonomous vehicle input

[0143] Figure 5The prediction module 402 is shown. Figure 4 The input used is 502a-d (e.g., Figure 1 The sensor 121 shown is an example, along with outputs 504a-d (e.g., sensor data). One input 502a is a LiDAR (light detection and ranging) system (e.g., Figure 1 (LiDAR 123 shown). LiDAR is a technique that uses light (e.g., pulses of light such as infrared light) to obtain data about physical objects within its line of sight. A LiDAR system produces LiDAR data as output 504a. For example, LiDAR data is a collection of 3D or 2D points (also known as a point cloud) used to construct a representation of environment 190.

[0144] Another input 502b is a radar system. Radar is a technology that uses radio waves to acquire data about nearby physical objects. Radar can acquire data about objects that are not within the line of sight of a LiDAR system. The radar system 502b generates radar data as output 504b. For example, radar data is one or more radio frequency electromagnetic signals used to construct a representation of the environment 190.

[0145] Another input 502c is the camera system. The camera system uses one or more cameras (e.g., a digital camera using a light sensor such as an electrical-coupled device (CCD)) to acquire information about nearby physical objects. The camera system produces camera data as output 504c. Camera data typically takes the form of image data (e.g., data in image data formats such as RAW, JPEG, PNG, etc.). In some examples, the camera system has multiple independent cameras, such as a camera for stereo vision, which enables the camera system to perceive depth. Although the objects perceived by the camera system are described herein as “nearby,” this is relative to the AV. In use, the camera system can be configured to “see” objects at a distance, such as objects located up to one kilometer or more in front of the AV. Accordingly, the camera system may have features such as sensors optimized for perceiving distant objects and lenses.

[0146] Another input 502d is the Traffic Light Detection (TLD) system. The TLD system uses one or more cameras to acquire information about traffic lights, street signs, and other physical objects that provide visual navigation information. The TLD system produces TLD data as output 504d. TLD data is typically in the form of image data (e.g., data in image data formats such as RAW, JPEG, PNG, etc.). The TLD system differs from systems that incorporate cameras in that it uses cameras with a wide field of view (e.g., using a wide-angle lens or fisheye lens) to acquire information about as many physical objects as possible that provide visual navigation information, allowing the AV100 to access all relevant navigation information provided by these objects. For example, the field of view of a TLD system can be approximately 120 degrees or greater.

[0147] In some embodiments, outputs 504a-d are combined using sensor fusion technology. Thus, each output 504a-d is provided to other systems of the AV100 (e.g., provided to systems such as...). Figure 4 The planning module 404 shown, or the combined output, can be provided to other systems in the form of a single or multiple combined output of the same type (e.g., using the same combination technique or combining the same output or both) or different types (e.g., using different corresponding combination techniques or combining different corresponding outputs or both). In some embodiments, an early fusion technique is used. The early fusion technique is characterized by combining the outputs before one or more data processing steps are applied to the combined outputs. In some embodiments, a late fusion technique is used. The late fusion technique is characterized by combining the outputs after one or more data processing steps are applied to the respective outputs.

[0148] Figure 6 An example of the LiDAR system 602 is shown (e.g., as shown in the image). Figure 5 The input 502a is shown. The LiDAR system 602 emits light 604a-c from a light emitter 606 (e.g., a laser emitter). The light emitted by the LiDAR system is typically outside the visible spectrum; for example, infrared light is typically used. Some of the emitted light 604b encounters a physical object 608 (e.g., a vehicle) and is reflected back to the LiDAR system 602. (The light emitted by the LiDAR system typically does not penetrate physical objects, e.g., physical objects existing in a solid state.) The LiDAR system 602 also has one or more photodetectors 610 that detect the reflected light. In an embodiment, one or more data processing systems associated with the LiDAR system generate an image 612 representing the field of view 614 of the LiDAR system. Image 612 includes information representing the boundary 616 of the physical object 608. Thus, image 612 is used to determine the boundary 616 of one or more physical objects near the AV.

[0149] Figure 7 A LiDAR system 602 in operation is shown. In the scenario shown in this figure, AV100 receives camera system output 504c in the form of image 702 and LiDAR system output 504a in the form of LiDAR data points 704. In use, AV100's data processing system compares image 702 with data points 704. Specifically, physical objects 706 identified in image 702 are also identified in data points 704. Thus, AV100 perceives the boundaries of physical objects based on the contours and density of data points 704.

[0150] Figure 8 The operation of the LiDAR system 602 is shown in more detail. As described above, the AV100 detects the boundaries of physical objects based on features of data points detected by the LiDAR system 602. Figure 8 As shown, a flat object (such as ground 802) will reflect light 804a-d emitted by LiDAR system 602 in a consistent manner. In other words, because LiDAR system 602 emits light at consistent intervals, ground 802 will reflect light back to LiDAR system 602 at the same consistent intervals. When AV100 travels on ground 802, if there are no obstructions, LiDAR system 602 will continue to detect light reflected from the next effective surface point 806. However, if object 808 obstructs the path, light 804e-f emitted by LiDAR system 602 will be reflected from points 810a-b in a manner inconsistent with the expected consistency. From this information, AV100 can determine the presence of object 808.

[0151] Autonomous Vehicle Planning

[0152] Figure 9 The planning module 404 is shown (e.g.) Figure 4 A block diagram 900 illustrates the relationship between the inputs and outputs of the planning module 404. Typically, the output of the planning module 404 is a route 902 from a starting point 904 (e.g., a source location or initial location) to an ending point 906 (e.g., a destination or final location). Route 902 is typically defined by one or more segments. For example, a segment refers to the distance traveled over at least a portion of a street, road, highway, driveway, or other physical area suitable for motor vehicle travel. In some examples, such as if the AV100 is an off-road capable vehicle (such as a four-wheel drive (4WD) or all-wheel drive (AWD) car, SUV, or pickup truck), route 902 includes “off-road” segments such as unpaved roads or open ground.

[0153] In addition to route 902, the planning module also outputs lane-level route planning data 908. Lane-level route planning data 908 is used to traverse segments of route 902 based on segment conditions at a specific time. For example, if route 902 includes a multi-lane highway, lane-level route planning data 908 includes trajectory design data 910, which the AV100 can use to select lanes from multiple lanes, for example, based on proximity to an exit, the presence of other vehicles in one or more lanes, or other factors that change within minutes or less. Similarly, in some implementations, lane-level route planning data 908 includes speed constraints 912 specific to segments of route 902. For example, if a segment includes pedestrians or unexpected traffic, speed constraints 912 can restrict the AV to travel at a speed lower than expected (e.g., based on the speed limit data used for that segment).

[0154] In this embodiment, the input to the planning module 404 includes database data 914 (e.g., from...). Figure 4 The database module 410 shown), current location data 916 (for example, such as...) Figure 4 The AV location 418 shown), destination data 918 (for example, for such...) Figure 4 The destination 412 shown) and object data 920 (e.g., from such destination 412 ... Figure 4 The perception module 402 shown perceives the classified objects 416. In some embodiments, the database data 914 includes rules used in planning. Rules are specified using formal language, such as Boolean logic. In any given situation encountered by the AV100, at least some of the rules apply to that situation. A rule applies to a given situation if it has conditions satisfied based on information available to the AV100 (e.g., information about the surrounding environment). Rules may have priorities. For example, the rule "Move to the leftmost lane if the road is a highway" may have a lower priority than "Move to the rightmost lane if there is an exit within one mile".

[0155] Figure 10 Route planning is shown (e.g., by planning module 404). Figure 4 The directed graph 1000 used in the execution. Overall, as... Figure 10 The directed graph 1000 shown is used to determine the route between any starting point 1002 and ending point 1004. In the real world, the distance separating the starting point 1002 and the ending point 1004 may be relatively large (e.g., in two different urban areas) or relatively small (e.g., two intersections adjacent to a city block or two lanes of a multi-lane road).

[0156] In an embodiment, the directed graph 1000 has nodes 1006a-d, which represent different locations between the starting point 1002 and the ending point 1004 that can be occupied by the AV 100. In some examples, for instance, when the starting point 1002 and the ending point 1004 represent different urban areas, nodes 1006a-d represent road segments. In some examples, for instance, when the starting point 1002 and the ending point 1004 represent different locations on the same road, nodes 1006a-d represent different locations on that road. Thus, the directed graph 1000 includes information at different levels of granularity. In an embodiment, a directed graph with high granularity is also a subgraph of another directed graph with a larger proportion. For example, the majority of the information in a directed graph where the starting point 1002 is far from the ending point 1004 (e.g., many miles apart) is low-granularity and based on stored data, but some high-granularity information is also included for the portion of the graph that represents physical locations within the field of view of the AV 100.

[0157] Nodes 1006a-d differ from objects 1008a-b in that objects cannot overlap with nodes. In an embodiment, at a lower granularity, objects 1008a-b represent areas that cannot be traversed by motor vehicles, such as areas without streets or roads. At a higher granularity, objects 1008a-b represent physical objects within the field of view of the AV100, such as other motor vehicles, pedestrians, or other entities with which the AV100 cannot share physical space. In an embodiment, some or all of objects 1008a-b are static objects (e.g., objects that do not change position, such as streetlights or utility poles) or dynamic objects (e.g., objects that can change position, such as pedestrians or other vehicles).

[0158] Nodes 1006a-d are connected by edges 1010a-c. If two nodes 1006a-b are connected by edge 1010a, then AV100 may travel between one node 1006a and the other node 1006b, for example, without needing to travel to an intermediate node before reaching the other node 1006b. (When we describe AV100 traveling between nodes, we mean AV100 traveling between two physical locations identified by the respective nodes.) Edges 1010a-c are generally bidirectional in the sense that AV100 travels from a first node to a second node or from a second node to a first node. In an embodiment, edges 1010a-c are unidirectional in the sense that AV can travel from a first node to a second node but cannot travel from a second node to a first node. Edges 1010a-c are unidirectional when they represent, for example, lanes of a one-way street, street, road, or highway, or other features that allow travel in only one direction due to legal or physical constraints.

[0159] In an embodiment, the planning module 404 uses a directed graph 1000 to identify the path 1012, which consists of nodes and edges between the start point 1002 and the end point 1004.

[0160] Edge 1010a-c has an associated cost 1014a-b. Cost 1014a-b is a value representing the resource that would be spent if AV100 selected this edge. A typical resource is time. For example, if one edge 1010a represents twice the physical distance of another edge 1010b, then the associated cost 1014a of the first edge 1010a could be twice the associated cost 1014b of the second edge 1010b. Other factors affecting time include anticipated traffic, the number of intersections, speed limits, etc. Another typical resource is fuel economy. Two edges 1010a-b can represent the same physical distance, but one edge 1010a may require more fuel than the other edge 1010b (e.g., due to road conditions, anticipated weather, etc.).

[0161] When the planning module 404 identifies the path 1012 between the starting point 1002 and the ending point 1004, the planning module 404 typically selects a path that is optimized for cost, such as the path that has the lowest total cost when the costs of each edge are added together.

[0162] Autonomous Vehicle Control

[0163] Figure 11 The control module 406 is shown (e.g., as shown in the image). Figure 4 A block diagram 1100 showing the inputs and outputs (as shown in the diagram). The control module operates according to a controller 1102, which includes, for example, one or more processors similar to processor 304 (e.g., one or more computer processors, such as microprocessors or microcontrollers or both), short-term and / or long-term data storage similar to main memory 306, ROM 1308, and storage device 210 (e.g., memory random access memory or flash memory or both), and instructions stored in memory that, when executed (e.g., by one or more processors), perform the operation of controller 1102.

[0164] In one embodiment, controller 1102 receives data representing a desired output 1104. The desired output 1104 typically includes velocity, for example, speed and orientation. The desired output 1104 may be based, for example, from planning module 404 (e.g., as...). Figure 4(As shown) The data received. Based on the desired output 1104, the controller 1102 generates data that can be used as throttle input 1106 and steering input 1108. Throttle input 1106 represents the magnitude of engaging the throttle of AV 100 (e.g., acceleration control), for example, by engaging the steering pedal or engaging another throttle control to achieve the desired output 1104. In some examples, throttle input 1106 also includes data that can be used to engage braking (e.g., deceleration control) of AV 100. Steering input 1108 represents the steering angle, for example, the angle at which the steering control of the AV (e.g., steering wheel, steering angle actuator, or other function for controlling the steering angle) should be positioned to achieve the desired output 1104.

[0165] In one embodiment, controller 1102 receives feedback that is used to adjust the inputs provided to the throttle and steering. For example, if AV100 encounters an disturbance 1110, such as a mountain, the measured speed 1112 of AV100 will decrease to the desired output speed. In another embodiment, any measured output 1114 is provided to controller 1102 to perform necessary adjustments, such as based on the difference 1113 between the measured speed and the desired output. The measured output 1114 includes measured position 1116, measured speed 1118 (including speed and orientation), measured acceleration 1120, and other outputs that can be measured by the sensors of AV100.

[0166] In this embodiment, information about the disturbance 1110 is detected beforehand (e.g., by a sensor such as a camera or LiDAR sensor) and provided to the predictive feedback module 1122. The predictive feedback module 1122 then provides information to the controller 1102, which can use this information to make appropriate adjustments. For example, if the sensors of AV100 detect (“see”) a mountain, this information can be used by the controller 1102 to prepare to engage the throttle at an appropriate time to avoid significant deceleration.

[0167] Figure 12 A block diagram 1200 showing the inputs, outputs, and components of controller 1102 is provided. Controller 1102 has a speed analyzer 1202 that influences the operation of throttle / brake controller 1204. For example, depending on feedback received by controller 1102 and processed by speed analyzer 1202, speed analyzer 1202 uses throttle / brake 1206 to instruct throttle / brake controller 1204 to engage acceleration or deceleration.

[0168] Controller 1102 also has a lateral tracking controller 1208 that influences the operation of steering controller 1210. For example, depending on feedback received by controller 1102 and processed by lateral tracking controller 1208, lateral tracking controller 1208 instructs steering controller 1204 to adjust the position of steering angle actuator 1212.

[0169] Controller 1102 receives several inputs for determining how to control the throttle / brake 1206 and the steering angle actuator 1212. Planning module 404 provides information used by controller 1212, for example, to select orientation when AV100 begins operation and to determine which road segment to traverse when AV100 reaches an intersection. Positioning module 408 provides controller 1102 with information, for example, describing the current position of AV100, enabling controller 1102 to determine whether AV100 is in a position expected based on how the throttle / brake 1206 and steering angle actuator 1212 are controlled. In an embodiment, controller 1102 receives information from other inputs 1214 (e.g., information received from a database or computer network).

[0170] Remote monitoring and control of autonomous vehicle operation

[0171] In some embodiments, a computer system controls the operation of one or more autonomous vehicles (e.g., a fleet of autonomous vehicles). For example, the computer system may deploy autonomous vehicles to one or more locations or areas, assign transportation tasks to each of the autonomous vehicles (e.g., pick up and transport passengers, pick up and transport goods, etc.), provide navigation instructions to each of the autonomous vehicles (e.g., provide a path or route between two locations, provide instructions to traverse objects near the autonomous vehicle, etc.), assign maintenance tasks to each of the autonomous vehicles (e.g., replace their batteries at a charging station, receive maintenance at a service station, etc.), and / or assign other tasks to each of the autonomous vehicles.

[0172] Furthermore, computer systems can be used to monitor the operation of autonomous vehicles. For example, a computer system can collect information from each of the autonomous vehicles (e.g., vehicle telemetry data, such as data on the vehicle's speed, orientation, status, or other aspects of vehicle operation), process the collected information, and present that information to one or more users (e.g., in the form of an interactive graphical user interface), so that users can always be informed about the operation of the autonomous vehicles.

[0173] Figure 13An example computer system 1300 is shown for monitoring and controlling the operation of autonomous vehicle convoys 1302a-d. In this example, the computer system 1300 is located remotely from each of the autonomous vehicles 1302a-d and communicates with the autonomous vehicles 1302a-d (e.g., via a wireless communication network). In some embodiments, the computer system 1300 is positioned relative to... Figure 1 The remote server 136 and / or relative to Figure 1 and Figure 3 The cloud computing environment 300 is implemented in a similar manner. In some embodiments, one or more of the autonomous vehicles 1302a-d are implemented in a manner similar to that described above. Figure 1 The autonomous vehicle 100 described is implemented in this way.

[0174] Each of the autonomous vehicles 1302a-d is located within a geographic region 1304. The geographic region 1304 may correspond to a specific political region (e.g., a specific country, state, county, province, city, town, district, or other political region), a specific predefined region (e.g., a region with specific predefined boundaries, such as a geofenced region determined by software), a momentarily defined region (e.g., a region with dynamic boundaries, such as a group of streets affected by heavy traffic), or any other region.

[0175] Each of the autonomous vehicles 1302a-d is capable of transmitting information about its operation to the computer system 1300. This information may include vehicle telemetry data (e.g., data from one or more measurements, readings, and / or samples acquired by one or more sensors of the vehicle).

[0176] Vehicle telemetry data can include a variety of information. As an example, vehicle telemetry data can include data obtained using one or more imaging sensors (e.g., light detectors, camera modules, etc.). For instance, this data can include one or more videos or images captured by the imaging sensors of an autonomous vehicle.

[0177] As another example, vehicle telemetry data may include information about the current status of the autonomous vehicle. For example, this data may include information about the autonomous vehicle's position (e.g., determined by a positioning module with GNSS sensors), speed (e.g., determined by a velocity sensor), altitude (e.g., determined by an altimeter), and / or orientation (e.g., determined by a compass or gyroscope). The data may also include information about the status of the autonomous vehicle and / or one or more of its sub-components. For example, this data may include information indicating that the autonomous vehicle is operating normally, or information indicating one or more anomalies related to the operation of the autonomous vehicle (e.g., error indications, warnings, malfunction indications, etc.). As another example, this data may include information indicating that one or more specific sub-components of the autonomous vehicle are operating normally, or information indicating one or more anomalies related to these sub-components.

[0178] As another example, vehicle telemetry data may include information about the historical status of the autonomous vehicle. For instance, this data may include information about the historical location, speed, altitude, and / or orientation of the autonomous vehicle. The data may also include information about the historical status of the autonomous vehicle and / or one or more of its sub-components.

[0179] As another example, vehicle telemetry data may include information about current and / or historical environmental conditions observed by the autonomous vehicle. For instance, the data may include information about traffic conditions on roads observed by the autonomous vehicle, road closures or obstacles observed by the autonomous vehicle, objects or hazards observed by the autonomous vehicle, or other information.

[0180] Autonomous vehicles 1302a-d can transmit information to computer system 1300 according to different frequencies, rates, or patterns. For example, autonomous vehicles 1302a-d can transmit information periodically (e.g., in a cyclical manner, such as at a specific frequency). As another example, autonomous vehicles 1302a-d can transmit information intermittently or sporadically. As another example, autonomous vehicles 1302a-d can transmit information if one or more triggering conditions are met (e.g., when a specific type of information is collected by the autonomous vehicle, at a specific type of time, when a specific event occurs, etc.). As another example, autonomous vehicles 1302a-d can transmit information continuously or substantially continuously.

[0181] In some embodiments, autonomous vehicles 1302a-d transmit a subset of the information they collect. As an example, each autonomous vehicle 1302a-d can collect information (e.g., using one or more sensors) and selectively transmit a subset of the collected information to computer system 1300. In some embodiments, autonomous vehicles 1302a-d can transmit all or substantially all of the information they collect. As an example, each autonomous vehicle 1302a-d can collect information (e.g., using one or more sensors) and selectively transmit all or substantially all of the collected information to computer system 1300.

[0182] In some embodiments, the autonomous vehicle 1302a-d dynamically transmits different types of information to the computer system 1300 (depending on the situation). As an example, in some embodiments, the autonomous vehicle 1302a-d may periodically, intermittently, or continuously transmit a first dataset to the computer system 1300. When a request for additional information is received from the computer system 1300, the autonomous vehicle 1302a-d may transmit a second dataset to the computer system 1300. The first and second datasets may have different sizes and / or complexities. For example, the first dataset may be relatively smaller and / or less complex (e.g., containing less information about the operation of the autonomous vehicle), while the second dataset may be relatively larger and / or more complex (e.g., containing more information about the operation of the autonomous vehicle). This can be beneficial, for example, because it enables computer system 1300 to receive specific information about each of the autonomous vehicles 1302a-d (e.g., to continuously monitor the autonomous vehicles 1302a-d based on a relatively small dataset) and to request additional information on demand (e.g., if additional information is needed to monitor and / or control the operation of the autonomous vehicles 1302a-d in a more accurate or comprehensive manner), without exceeding the network connectivity between them.

[0183] As an example, such as Figure 13As shown, each of the autonomous vehicles is capable of transmitting corresponding first vehicle telemetry data 1306a-d to computer system 1300. The first vehicle telemetry data 1306a-d can be relatively “basic,” “simple,” “lightweight,” or “general.” As an example, the first vehicle telemetry data 1306a-d may be relatively smaller in size and / or less complex in nature (e.g., compared to the second vehicle telemetry data described in more detail below). As another example, the first vehicle telemetry data 1306a-d may include a relatively smaller subset of data collected by the sensors of the autonomous vehicles 1302a-d. As another example, the first vehicle telemetry data 1306a-d may include a relatively abstract representation of the data collected by the sensors of the autonomous vehicles 1302a-d (e.g., a generalized, abbreviated, edited, and / or simplified representation of the collected data).

[0184] Computer system 1300 may use the collected information to facilitate the control and / or monitoring of autonomous vehicles 1302a-d. As an example, computer system 1300 may generate an interactive graphical user interface (GUI) incorporating some or all of the collected information and present this GUI to a user (e.g., using a display device associated with or otherwise communicating with computer system 1300). The user may interact with the GUI to view information about each of the autonomous vehicles 1302a-d and / or issue instructions to the autonomous vehicles 1302a-d (e.g., commands to assign new tasks to the autonomous vehicles, commands to modify or cancel previous commands, etc.). Example GUI relative to... Figure 23-29 and Figures 30-36 The instructions are shown and described in the figure. In some embodiments, the computer system 1300 is capable of automatically issuing one or more instructions to the autonomous vehicles 1302a-d based on collected information (e.g., automatically generating commands to assign new tasks to the autonomous vehicles, modifying or canceling previous commands, etc.). The issued commands are transmitted from the computer system 1300 to one or more of the autonomous vehicles 1302a-d for execution.

[0185] In some embodiments, additional information is selectively obtained from one or more of the autonomous vehicles 1302a-d. For example, such as Figure 14 As shown, computer system 1300 can receive user input 1308 indicating that a user of computer system 1300 has selected autonomous vehicle 1302a. User input 1308 may correspond to, for example, a user who interacts with a GUI presented by computer system 1300 and selects autonomous vehicle 1302a (e.g., indicating that the user wishes to view additional information about autonomous vehicle 1302a).

[0186] like Figure 15 As shown, in response, computer system 1300 transmits request information 1310 to the selected autonomous vehicle 1302a to request additional information. In response to receiving request information 1310, autonomous vehicle 1302b transmits second vehicle telemetry data 1312 to computer system 1300.

[0187] The second vehicle telemetry data 1312 may be more "detailed" than the first vehicle telemetry data 1306a previously transmitted by the autonomous vehicle 1302a. For example, the second vehicle telemetry data 1312 may be relatively larger and / or more complex compared to the first vehicle telemetry data 1306a. As another example, the second vehicle telemetry data 1312 may include a larger subset of data collected by the sensors of the autonomous vehicle 1302a compared to the first vehicle telemetry data 1306a. As another example, the second vehicle telemetry data 1312 may include a relatively less abstract representation of the data collected by the sensors of the autonomous vehicle 1302a (e.g., a more detailed overview or less simplified representation of the collected data, or the collected data itself). As another example, the second vehicle telemetry data 1312 may include the entirety of the data collected by the sensors of the autonomous vehicle 1302a, rather than a subset of the collected data.

[0188] For example, the first vehicle telemetry data 1306A may include one or more images with a specific resolution and size, and the second vehicle telemetry data 1312 may include one or more images with a higher resolution and / or size than the images in the first vehicle telemetry data 1306A.

[0189] As another example, the first vehicle telemetry data 1306A may include one or more videos with a specific resolution, size, and frame rate, and the second vehicle telemetry data 1312 may include one or more videos with a higher resolution, size, and / or frame rate than the videos in the first vehicle telemetry data 1306A.

[0190] As another example, the first vehicle telemetry data 1306A may include sensor measurements collected at a specific frequency or rate, and the second vehicle telemetry data 1312 may include one or more sensor measurements collected at a frequency or rate higher than that of the first vehicle telemetry data 1306A.

[0191] As another example, the first vehicle telemetry data 1306A may include one or more data items of a specific data type (e.g., 32-bit floating-point), and the second vehicle telemetry data 1312 may include one or more data items of a data type greater than that of the first vehicle telemetry data 1306A (e.g., 64-bit double-precision floating-point).

[0192] As another example, the first vehicle telemetry data 1306A may include one or more data items that express values ​​according to a certain level of precision, and the second vehicle telemetry data 1312 may include one or more data items that express values ​​according to a higher level of precision than the data items of the first vehicle telemetry data 1306A.

[0193] As another example, the first vehicle telemetry data 1306A may include one or more data items of relatively small size (e.g., text data, such as JavaScript Object Notation (JSON), Hypertext Markup Language (HTML), or Extensible Markup Language (XML) data formats), and the second vehicle telemetry data 1312 may include one or more data items of relatively large size (e.g., images and / or videos, detailed sensor data, large binary files, etc.).

[0194] In some embodiments, the computer system 1300 and the autonomous vehicles 1302a-d exchange information dynamically based on the state of their network connection. For example, if the network connection between the computer system 1300 and the autonomous vehicles 1302a-d is of high quality (e.g., high available bandwidth, low latency, and / or high reliability), the computer system 1302a-d can obtain a larger amount of data from the autonomous vehicles (e.g., more detailed data, larger data size, and / or more complex data). As another example, if the network connection between the computer system 1300 and the autonomous vehicles 1302a-d is of low quality (e.g., low available bandwidth, high latency, and / or lower reliability), the computer system 1300 can obtain a smaller amount of data from the autonomous vehicles (e.g., less detailed data, more basic data, smaller data size, and / or less complex data).

[0195] For illustration purposes, Figure 16Area 1304 with autonomous vehicle 1302a is shown. In this example, autonomous vehicle 1302a encounters obstacle 1600 on its path and has determined that it cannot safely and / or effectively navigate around obstacle 1600 in an automatic manner. Obstacle 1600 can be, for example, another vehicle (e.g., car, van, truck, motorcycle, bicycle), pedestrian, non-vehicle object (e.g., obstacle, traffic cone, road debris, etc.), hazard (e.g., ditch or pothole, road discontinuity, etc.), road closure, or any other obstruction that may impede the movement of autonomous vehicle 1302a. Obstacle 1600 can be detected by, for example, autonomous vehicle 1302a, other autonomous vehicles (e.g., other autonomous vehicles currently near or previously near obstacle 1600), and / or computer system 1300 (e.g., based on data collected by computer system 1300 from one or more autonomous vehicles or other information sources such as maps or traffic databases).

[0196] When it is determined that navigation around obstacle 1600 cannot be safely and / or effectively performed automatically, the autonomous vehicle 1302a prepares to transmit a remote control request (e.g., a request for instructions to traverse obstacle 1600) to the computer system 1300. Figure 17 As shown, the autonomous vehicle first transmits a beacon signal 1602 to the computer system 1300 and evaluates the quality of the network connection between the autonomous vehicle 1302a and the computer system 1300 based on this transmission. As an example, the beacon signal 1602 may include one or more data items having a known size and / or known content as a baseline, standard, or reproducible sample (used for testing (e.g., "test" data or "sampled" data")).

[0197] like Figure 18As shown, the transmission of beacon signal 1602 can be analyzed to determine the quality of the network connection between autonomous vehicle 1302a and computer system 1300. For example, based on the transmission of beacon signal 1602, autonomous vehicle 1302a and / or computer system 1300 can determine the available bandwidth of the network connection (e.g., the amount of information that the network connection can transmit in a given period), the latency associated with transmitting information using the network connection (e.g., the amount of time it takes for information to traverse the network connection), and / or the reliability of transmission using the network connection (e.g., the ability of the network connection to perform transmission in a continuous manner). For example, autonomous vehicle 1302a and / or computer system 1300 can measure the amount of time taken to transmit beacon signal 1602, the transmission rate, the consistency or regularity of the transmission rate, the integrity of beacon signal 1602 when it arrives at computer system 1300, and other factors. Furthermore, autonomous vehicle 1302a and / or computer system 1300 can determine whether any errors or irregularities occur during transmission. In some embodiments, the autonomous vehicle 1302a and / or the computer system 1300 determine one or more quality metrics (e.g., one or more numerical scores, or other objective or subjective descriptors) representing the quality of the network connection between the autonomous vehicle 1302a and the computer system 1300. As an example, if the quality metric is a numerical score, higher available bandwidth, lower latency, and higher reliability may correspond to a higher quality metric. Conversely, lower available bandwidth, higher latency, and lower reliability may correspond to a lower quality metric. In some embodiments, multiple quality metrics may be determined, each corresponding to a different aspect of network connection quality (e.g., available bandwidth, latency, reliability, etc.).

[0198] like Figure 19 As shown, when the quality metric of the network connection is determined, the autonomous vehicle 1302 generates a remote control request 1604 and transmits the request 1604 to the computer system 1300.

[0199] Request 1604 may include data describing obstacle 1600. As an example, request 1604 may include data indicating the location of obstacle 1600 (e.g., the location of obstacle 1600 relative to autonomous vehicle 1304A, or the absolute location of obstacle 1600, etc.). As another example, request 1604 may include data indicating the size, shape, and / or orientation of obstacle 1600.

[0200] Request 1604 may also include data describing one or more aspects of the autonomous vehicle 1304A. As an example, Request 1604 may include vehicle telemetry data, such as information about... Figures 13-15As described. For example, vehicle telemetry data may include data obtained using one or more imaging sensors (e.g., video or images), information about the current status of the autonomous vehicle (e.g., the autonomous vehicle's position, speed, altitude and / or orientation, the status of the autonomous vehicle and / or one or more of its sub-components, etc.), information about the prior status of the autonomous vehicle, and / or information about current and / or historical environmental conditions observed by the autonomous vehicle.

[0201] Autonomous vehicle 1304A may dynamically modify the information included in request 1604 based on quality metrics associated with network connectivity. For example, if the network connectivity quality between computer system 1300 and autonomous vehicle 1302a is high (e.g., associated with one or more higher quality metrics corresponding to higher available bandwidth, lower latency, and / or higher reliability), autonomous vehicle 1304A may include a larger amount of information (e.g., more detailed data, larger data size, and / or more complex data). As another example, if the network connectivity quality between computer system 1300 and autonomous vehicle 1302a is low (e.g., associated with one or more lower quality metrics corresponding to lower available bandwidth, higher latency, and / or lower reliability), autonomous vehicle 1304A may include a smaller amount of information (e.g., less detailed data, smaller data size, and / or less complex data).

[0202] As an example, if the network connection quality is low, the autonomous vehicle 1304A may include one or more images with a specific resolution and size. If the network connection quality is high, the autonomous vehicle 1304A may include one or more images with a higher resolution and / or size.

[0203] As another example, if the network connection quality is low, the autonomous vehicle 1304A may include one or more videos with a specific resolution, size, and frame rate. If the network connection quality is high, the autonomous vehicle 1304A may include one or more videos with a higher resolution, size, and / or frame rate.

[0204] As another example, if the network connection quality is low, the autonomous vehicle 1304A may include one or more sensor measurements collected at a specific frequency or rate. If the network connection quality is high, the autonomous vehicle 1304A may include one or more sensor measurements collected at a higher frequency or rate.

[0205] As another example, if the network connection quality is low, the autonomous vehicle 1304A may include one or more sensor measurements collected according to a specific temporal and / or spatial resolution. If the network connection quality is high, the autonomous vehicle 1304A may include one or more sensor measurements collected according to a higher temporal and / or spatial resolution.

[0206] As another example, if the network connection quality is low, the autonomous vehicle 1304A may include one or more data items of a specific data type (e.g., a 32-bit floating-point type). If the network connection quality is high, the autonomous vehicle 1304A may include one or more data items of a larger data type (e.g., a 64-bit double-precision type).

[0207] As another example, if the network connection quality is low, the autonomous vehicle 1304A may include one or more data items that express values ​​with a certain level of precision. If the network connection quality is high, the autonomous vehicle 1304A may include one or more data items that express values ​​with a higher level of precision.

[0208] As another example, if the network connection quality is low, the autonomous vehicle 1304A may include one or more data items of relatively small size (e.g., text data, data in formats such as JSON, HTML, or XML). If the network connection quality is high, the autonomous vehicle 1304A may include one or more data items of relatively large size (e.g., images and / or video, detailed sensor data, large binary files, etc.).

[0209] In some embodiments, the autonomous vehicle 1304A modifies the information included in request 1604 according to multiple different levels, tiers, or grades. For example, if the quality of the network connection corresponds to the highest level (e.g., associated with a quality metric above or equal to a first threshold level), the autonomous vehicle 1304A may include a first portion of the information containing the most information (e.g., the most detailed data, the largest data size, and / or the most complex data). If the quality of the network connection corresponds to a lower level (e.g., associated with a quality metric below the first threshold level but above or equal to a lower second threshold level), the autonomous vehicle 1304A may include a second portion of the information containing less data (e.g., less detailed data, smaller data size, and / or less complex data). If the quality of the network connection corresponds to an even lower level (e.g., associated with a quality metric below the second threshold level but above or equal to a lower third threshold level), the autonomous vehicle 1304A may include a third portion of the information containing even less data (e.g., less detailed data, smaller data size, and / or less complex data). Any level, tier, or grade may be defined in this manner (e.g., two, three, four, or more). In this way, data transmission between the autonomous vehicle 1302a and the computer system 1300 can be dynamically altered based on the quality of the network connection between them, thereby improving the effectiveness of communication.

[0210] In some embodiments, the network connection between the autonomous vehicle and the computer system can be established using a cellular network, such as a 4G network (e.g., a network using “fourth generation” broadband cellular network technology as defined by the International Telecommunication Union (ITU)) or a 5G network (e.g., a network using “fifth generation” broadband cellular network technology as defined by the ITU). The quality of the network connection can vary based on the type of cellular network used. As an example, a 5G network may have a peak data rate of approximately 20 Gbit / s (e.g., the maximum achievable data rate), a user experience data rate of approximately 1 Gbit / s (e.g., the data rate achievable across the coverage area), a latency of approximately 1 ms (e.g., the contribution of the radio network to packet travel time), mobility of approximately 500 km / h (e.g., the maximum speed required for “handover” and quality of service), a connection density of approximately 10⁶ / km² (e.g., the total number of devices per unit area), and a regional traffic capacity of approximately 10 (Mbit / s) / m² (e.g., the total traffic across the coverage area). In some embodiments, the network has a peak data rate higher than, equal to, or lower than approximately 20 Gbit / s. In some embodiments, the network has a user experience data rate of 1 Gbit / s or higher. In some embodiments, the network has a latency of 1 ms or higher. In some embodiments, the network has a mobility of 500 km / h or higher. In some embodiments, the network has a connection density of 10⁶ / km² or higher. In some embodiments, the network has a regional traffic capacity of 10 Mbit / s or higher. In some embodiments, beacon signals are used to determine the type of cellular network being used, and the performance of the network connection is determined based on appropriate technical specifications (e.g., 4G or 5G specifications defined by the ITU).

[0211] Upon receiving request 1604, computer system 1300 determines the location, shape, and orientation of obstacle 1600 (e.g., relative to autonomous vehicle 1302a) and presents a visual representation of obstacle 1600 and autonomous vehicle 1302a to the user (e.g., using a GUI generated by computer system 1300). The user can interact with the GUI to examine the nature of the obstacle and input commands to autonomous vehicle 1302a to traverse obstacle 1600. For example, the user can input a route for autonomous vehicle 1302a to navigate around and / or avoid obstacle 1600 while maintaining a sufficiently safe driving path. The input commands are transmitted from computer system 1300 to autonomous vehicle 1302a for execution.

[0212] In some embodiments, the computer system 1300 uses a simplified visual representation to present information about the obstacle 1600 and the autonomous vehicle 1302a (e.g., an abstract representation of the relative position and orientation of the obstacle 1600 and the autonomous vehicle 1302a). This can be useful, for example, because it allows the user to more easily understand the nature of the obstacle 1600 and provide navigation paths to bypass the obstacle 1600 without being overwhelmed by irrelevant information.

[0213] As an example, Figure 20 A visual representation 2000 depicting the relative positions of obstacles and autonomous vehicles is shown. The visual representation 2000 can be rendered, for example, using a GUI generated by the computer system 1300.

[0214] The visual representation 2000 includes an occupancy grid 2002 and a visual representation of the autonomous vehicle 2006, the occupancy grid defining a plurality of different two-dimensional boxes 2004. The occupancy grid 2002 represents the physical environment surrounding the autonomous vehicle (e.g., from a top-down view). For example, each box 2004 of the occupancy grid 2004 may indicate a specific area or region near the autonomous vehicle along its flight plane. In some embodiments, the occupancy grid 2002 may represent the physical environment surrounding the autonomous vehicle at a fixed or constant scale (e.g., each box represents a corresponding 1-inch by 1-inch area near the autonomous vehicle). In some embodiments, the occupancy grid 2002 may represent the physical environment surrounding the autonomous vehicle at a variable scale (e.g., boxes closer to the autonomous vehicle represent smaller areas near the autonomous vehicle, while boxes farther away from the autonomous vehicle represent larger areas).

[0215] Furthermore, one or more boxes 2004 occupying the grid 2004 can be depicted with contrasting colors, hues, or patterns to indicate the position of obstacles relative to the autonomous vehicle. For example, in Figure 20 In the example shown, several boxes 2004a are shaded to indicate that obstacles are located in the areas depicted by these boxes (e.g., generally in front of or to the right of the autonomous vehicle). Other boxes 2004b are not shaded to indicate that the areas depicted by these boxes are unobstructed. Accordingly, the user can view the visual representation 2000 and quickly and intuitively determine the position, shape, and orientation of obstacles relative to the autonomous vehicle.

[0216] Occupation grid 2002 can be generated based on information collected by autonomous vehicle 1302a. For example, autonomous vehicle 1302a can collect sensor data related to its environment (e.g., images, video, proximity data, etc.) and transmit some or all of the collected data to computer system 1300. Computer system 1300 can interpret the sensor data to determine the presence and location of one or more objects near autonomous vehicle 1302a and generate occupancy grid 2004 that visually represents these objects.

[0217] As another example, the autonomous vehicle 1302a is capable of collecting sensor data related to its environment, interpreting the sensor data to determine the presence and location of one or more objects in its vicinity, and generating one or more data items indicating the position of the objects relative to the autonomous vehicle 1302a. The autonomous vehicle 1302a is capable of transmitting the data items to the computer system 1300, which in turn is capable of generating an occupancy grid 2004 based on the data items.

[0218] In some embodiments, the data item includes a simplified representation of sensor data collected by the autonomous vehicle 1302a. For example, the autonomous vehicle 1302a can collect various sensor measurements (e.g., images, video, radar measurements, LiDAR measurements, ultrasonic measurements, etc.) and determine the presence and location of one or more objects based on these measurements. The data item can simply identify the location of each object relative to the autonomous vehicle 1302a, without including the sensor measurements themselves. In some embodiments, this information is expressed in the form of text data (text data such as JSON, HTML, or XML data formats). This is advantageous, for example, because it allows the autonomous vehicle 1302a to transmit information describing the location of objects near the autonomous vehicle to the computer system 1300 while reducing the amount of data transmitted over the network connection (e.g., transmitting simplified text data instead of comprehensive sensor data). Thus, the autonomous vehicle 1302a and the computer system 1300 can communicate in a more efficient manner.

[0219] Furthermore, users can interact with the visual representation 2000 to input paths for autonomous vehicles to navigate around and / or avoid obstacles. For example, such as... Figure 21As shown, a user can input a path 2008 for an autonomous vehicle to navigate around and / or avoid obstacles and continue moving forward. The path 2008 can be input in various different ways. As an example, the user can input the path 2008 using an input device such as a mouse, touchpad, or stylus (e.g., drawing or swapping the path 2008 relative to the visual representation 2000). As another example, the visual representation 2000 can be presented on a touch-sensitive display device (e.g., a touchscreen), and the user can input the path 2008 by drawing or tracing it onto the visual representation 2000 (e.g., by tracing their finger or stylus relative to the visual representation 2000). The input path 2008 can be transmitted from the computer system 1300 to the autonomous vehicle for execution, thereby enabling the autonomous vehicle to traverse obstacles and continue moving forward.

[0220] In some embodiments, the path entered by the user is modified before being transmitted to the autonomous vehicle. For example, the entered path can be modified for safety and / or practical considerations. As an example, the entered path can be modified to better take into account road configuration (e.g., lane markings, road boundaries, etc.), traffic flow (e.g., direction of traffic), traffic rules or regulations, and other considerations. In some embodiments, the entered path is “bited” into an alternative route adjacent to the entered path (e.g., a reference path similar to the entered path but with improved safety and / or practicality). This can be useful, for example, because it allows the user to quickly provide rough or general instructions related to the navigation of the autonomous vehicle (e.g., without worrying about the precise route the autonomous vehicle will take), while ensuring that the autonomous vehicle is given sufficiently precise instructions to safely and efficiently navigate obstacles. In some embodiments, the computer system 1300 determines alternative routes based on information collected by autonomous vehicles, other autonomous vehicles (e.g., other autonomous vehicles currently near an obstacle or that have previously navigated through the environment), or other information sources (e.g., services providing map data, imaging data such as satellite or aerial imagery, survey data, etc.).

[0221] As described above, computer system 1300 can generate an interactive GUI to present information to a user about one or more autonomous vehicles and / or enable the user to input instructions for one or more autonomous vehicles. Figures 23-29 as well as Figures 30-36 A sample GUI is shown.

[0222] Figure 23An example GUI 2300 for monitoring and controlling the operation of one or more autonomous vehicles is shown. The GUI 2300 can be generated by, for example, a computer system 1300 and presented to a user using a display device (e.g., a video monitor or screen). In some embodiments, the GUI 2300 can be used in a "command center" for remotely controlling multiple autonomous vehicles (e.g., an autonomous vehicle queue).

[0223] GUI 2300 includes a map portion 2302 that shows the locations of several autonomous vehicles. For example, map portion 2302 includes multiple display elements 2304 that indicate the position of the autonomous vehicles relative to a graphical map 2306. Users can interact with map portion 2302 to view different geographic areas (e.g., by zooming and / or panning graphical map 2306). Users can also interact with map portion 2302 to view information at different layers. For example, graphical map 2306 may include multiple different overlapping layers, each indicating a different type of information (e.g., roads, traffic, satellite / aerial imaging, terrain, etc.). Users can selectively show or hide one or more layers to view different types of information (viewing them individually or in combination with other layers).

[0224] GUI 2300 also includes a list section 2308 that displays information about several autonomous vehicles. As an example, for each autonomous vehicle, the list section 2308 can display the vehicle's identifier (e.g., name, serial number, or other identifier), its location (e.g., city, region, or location point), its current status (e.g., an indication that the vehicle is docked, in motion, picking up passengers, transporting passengers, or performing some other task), its "health" (e.g., an indication that the vehicle is operating normally or has encountered one or more anomalies or errors), and the quality of the available network connectivity (e.g., an indication of the strength of the signal transmitted by a wireless transmitter associated with a wireless communication network). In some embodiments, the autonomous vehicles shown in the list section 2308 directly correspond to those shown in the map section 2302. In some embodiments, a user can scroll through the list section 2308 (e.g., by scrolling up or down) to view information related to additional autonomous vehicles.

[0225] Furthermore, list portion 2308 includes a search input box 2310. Users can enter search terms into the search input box 2310 to filter the autonomous vehicles displayed in list portion 2308. As an example, users can enter one or more search terms (e.g., terms related to the identifier, location, status, health, or any other information of an autonomous vehicle), and in response, GUI 2300 can display autonomous vehicles matching these search terms. In some embodiments, users can enter a specific location or landmark (e.g., the name of a city or region, address, place, building, etc.), and in response, GUI 2300 can display autonomous vehicles located at or near the entered location or landmark.

[0226] like Figure 24 As shown, GUI 2300 can suggest one or more search terms to the user. For example, when the user selects the search input box 2310, GUI 2300 can display one or more suggested search terms 2312 near the search input box 2310. The user can select the suggested search terms 2312 to perform a search, or manually enter search terms into the search input box 2310.

[0227] Furthermore, users can select an autonomous vehicle to view additional information related to that vehicle. For example, users can use an input device (e.g., a mouse, touchpad, stylus, or touch-sensitive display device) to select the autonomous vehicle displayed in map section 2302 or list section 2308. Figure 25 As shown, in response, GUI 2300 adjusts map portion 2302 to focus on the selected autonomous vehicle (e.g., by zooming and / or panning graphical map 2306) and displays vehicle information portion 2314, which presents additional information related to the selected autonomous vehicle. As an example, vehicle information portion 2314 can display the selected autonomous vehicle's identifier, operating status, progress relative to the assigned task (e.g., travel progress between two locations), speed, battery status, orientation or direction, mileage, or other information related to the operation of the autonomous vehicle. As another example, vehicle information portion 2314 can display one or more images or videos acquired by the autonomous vehicle (e.g., an image or video feed showing the area around the autonomous vehicle). In some embodiments, the user can select from several different views (e.g., corresponding to different imaging sensors mounted on the autonomous vehicle). Further, map portion 2302 can display the autonomous vehicle's current route or travel path and its destination.

[0228] Users can interact with the vehicle information section 2314 to obtain further information about the autonomous vehicle. For example, the vehicle information section 2314 may include several selectable elements 2316, each element 2316 relating to different aspects of the autonomous vehicle (e.g., the autonomous vehicle's computer system, network connectivity, sensors, etc.). Users can select one of the selectable elements 2316 to view additional information about the selected aspect. For example, a user can select the selectable element 2316 corresponding to the autonomous vehicle's computer system. Figure 26 As shown, in response, GUI 2300 updates vehicle information section 2314 to display additional information about the computer system (e.g., CPU usage, memory usage, hard disk drive usage, hardware version, software version, etc.). Similarly, the user can select other selectable elements 2316 to view additional information about the autonomous vehicle.

[0229] In some embodiments, the GUI2300 displays information about safety checks performed on the autonomous vehicle (e.g., safety checks performed automatically by the autonomous vehicle and / or by a human operator, such as a safety operator). For example, as... Figure 27 As shown, GUI 2300 can display a security check section 2318, which includes several tasks to be executed as part of the security check and a list of the statuses of these tasks (e.g., "completed", "incomplete", or "N / A"). The security check section 2318 can be updated to indicate the performance of each task.

[0230] In some embodiments, users can manually specify a route or path for the autonomous vehicle. For example... Figure 28 As shown, the user can select selectable elements 2320 (e.g., indicating that the user wants to enter a new route for the autonomous vehicle). In response, the GUI 2300 updates the map portion 2302 to display visual representations 2322 of any object or other obstacles near the autonomous vehicle. Visual representations 2312 can be used with... Figures 20-22 Similar to those shown and described (e.g., having a two-dimensional occupancy grid representing the physical environment around an autonomous vehicle). To be consistent with... Figures 20-22 In a similar manner as described, the user can input a route 2324 (e.g., a path through obstacles) for the autonomous vehicle and instruct the autonomous vehicle to travel along that route (e.g., by selecting the "Send" element 2326).

[0231] like Figure 23As shown, GUI 2300 can display the locations of multiple different autonomous vehicles using map portion 2302 (e.g., using display element 2304 overlaid on graphical map 2306). Furthermore, the user can interact with map portion 2302 to view different geographic areas (e.g., by zooming and / or panning graphical map 2306). In some embodiments, GUI 2300 uses a single display element to display the approximate locations of multiple autonomous vehicles. This can be useful, for example, in improving the readability of GUI 2300. For example, if multiple autonomous vehicles are positioned adjacent to each other on map portion 2302, then the display elements of these autonomous vehicles can be combined into a single display element. Further, the combined display element may include an indication of the number of vehicles it represents.

[0232] As an example, Figure 29 The graphic map 2306 is shown relative to Figure 23 The graphical map shown is a scaled-down version of GUI 2300. GUI 2300 uses display element 2326 to indicate the general location of different autonomous vehicle groups (and the number of autonomous vehicles in each group), without showing the location of individual autonomous vehicles.

[0233] In some embodiments, GUI 2300 dynamically switches between displaying the locations of individual autonomous vehicles and displaying the general location of a group of autonomous vehicles. For example, if graphical map 2306 is zoomed in relatively close, GUI 2300 may display the locations of individual autonomous vehicles on graphical map 2306. If graphical map 2306 is subsequently zoomed out to a specific zoom level, GUI 2300 may instead display the general location of the group of autonomous vehicles (e.g., to avoid cluttering graphical map 2306).

[0234] Figure 30 Example GUIs 3000a (with a "portrait" orientation) and 3000b (with a "landscape" orientation) are shown for monitoring and controlling the operation of a single autonomous vehicle. GUIs 3000a and 3000b can be generated by, for example, the autonomous vehicle and presented to the passengers of the autonomous vehicle using a display device (e.g., a video monitor or screen installed inside the autonomous vehicle, such as on the internal console or seats of the autonomous vehicle).

[0235] GUIs 3000a and 3000b can present information about the operation of the autonomous vehicle and / or the environment surrounding it. This can be useful, for example, because it allows passengers of the autonomous vehicle to be aware of information related to its operation during operation, even if the passenger is not directly controlling the autonomous vehicle (e.g., to improve passenger comfort and safety).

[0236] GUIs 3000a and 3000b are capable of displaying simplified visual representations of the autonomous vehicle and / or its surrounding environment. This can be beneficial in, for example, improving the readability of GUIs 3000a and 3000b (e.g., by reducing complexity or clutter that might confuse or distract the user). For instance, GUIs 3000a and 3000b can display simplified graphical representations (e.g., specific roads, objects, etc.) showing the autonomous vehicle and selected environmental features, rather than displaying photorealistic representations of the autonomous vehicle and every object around it (e.g., using images or videos directly captured by imaging sensors). Furthermore, the graphical representations can be generated using high-contrast and high-visibility display elements to improve readability.

[0237] For example, such as Figure 30 As shown, GUIs 3000a and 3000b include an environment map portion 3002 representing the surroundings of the autonomous vehicle, and a display element 3004 indicating the location of the autonomous vehicle in the environment. As an example, the map portion 3002 can indicate one or more roads in the environment, the current or intended path of the autonomous vehicle, and one or more objects located near the autonomous vehicle.

[0238] Furthermore, the map portion 3002 can display information about the operation of the autonomous vehicle. For example, the map portion 3002 can indicate the current speed of the autonomous vehicle (e.g., in the form of text next to display element 3004). In some embodiments, the current speed is hidden in the view, and passengers can selectively display the current speed by selecting to display element 3004.

[0239] As another example, map portion 3002 may indicate whether there is deceleration or delay along the path or route of the autonomous vehicle. For example, if there is traffic congestion causing the autonomous vehicle to slow down, map portion 3002 may indicate deceleration by using contrasting colors or patterns (e.g., a red dashed line) to indicate path 3006.

[0240] As another example, map portion 3002 may indicate one or more events or conditions detected by the autonomous vehicle (e.g., using a "bubble" display element 3008 extending from display element 3004, or using a notification message 3010 overlaid on map portion 3002). Example events or conditions include detecting pedestrians, detours, obstacles, crosswalks, traffic signals, or other features, or determining that a lane change is necessary (e.g., for navigation or avoidance purposes). Additional example bubble display element 3008 and notification message 3010 are shown below GUI 3000b.

[0241] The map section 3002 may use symbolic icons 3012 to indicate the location and classification of objects (e.g., simplified representations of objects). Different icons can be used for objects of different classifications and types. For example, Figure 31 Several example symbols 3012a-3012b are shown, which can be used to represent objects of different categories or types. For example, icon 3012a can be used to represent motor vehicles (e.g., cars, trucks, trains, etc.), icon 3012b can be used to represent pedestrians, icon 3012c can be used to represent bicycles, icon 3012d can be used to represent unclassified objects, and icon 3012e can be used to identify building obstacles or traffic cone warning signs.

[0242] Furthermore, the opacity or shadow of icon 3012 can be altered to indicate the confidence level regarding the object's classification. For example, if the autonomous vehicle identifies the object as a vehicle with a high confidence level, the object's icon 3012 can be relatively darker or more opaque. As another example, if the autonomous vehicle identifies the object as a vehicle with a low confidence level, the object's icon 3012 can be relatively lighter or more transparent.

[0243] The amount of information presented on GUIs 3000a and 3000b can vary depending on the implementation. As an example, in some implementations, GUIs 3000a and 3000b may indicate all or nearly all of the objects detected in the surrounding environment. As another example, in some implementations, GUIs 3000a and 3000b may indicate a subset of the detected objects (e.g., objects closer to the autonomous vehicle, or objects that are more important or have higher priority). For illustration purposes, Figure 32 Four variations of GUI3000a are shown, each displaying different levels of detail about the autonomous vehicle and its surrounding environment. For example, the leftmost variation shows only the road in the environment, while the leftmost variation further shows pedestrian crossings, the autonomous vehicle's position in the environment, and its intended path. Further, the rightmost variation further shows objects detected in the environment (e.g., other vehicles, pedestrians, bicycles, etc.). Further still, the rightmost variation further shows information about detected events or conditions (e.g., in the form of bubble displays and notification messages). This is advantageous, for example, because it allows the autonomous vehicle to vary the amount of information displayed to passengers, preventing them from becoming overwhelmed or distracted.

[0244] In some embodiments, GUIs 3000a and 3000b include one or more videos with "augmented reality" overlay to enhance video clarity. As an example, Figure 33A GUI 3000a with video 3014 is shown. Video 3014 can be acquired using, for example, one or more video cameras or other imaging sensors of the autonomous vehicle. Further, video 3014 may include one or more graphical overlays emphasizing certain features depicted in video 3014. For example, video 3014 may include an overlay 3016 indicating the current or expected path of the autonomous vehicle (e.g., display elements simulating a 3D path). As another example, video 3014 may include an overlay 3018 identifying vehicles near the autonomous vehicle (e.g., a high-contrast color overlay). Other overlays may also be used (e.g., to identify other types of objects, obstacles, or other features shown in video 3014).

[0245] In some embodiments, GUIs 3000a and 3000b display video captured in real-time or near real-time. For example, video captured by the imaging sensor can be displayed immediately on GUIs 3000a and 3000b to provide a substantially simultaneous view of the environment surrounding the autonomous vehicle. In some embodiments, GUIs 3000a and 3000b display portions of video captured in the past (e.g., occurring seconds or minutes ago, etc.). This can be useful, for example, in providing a user with "instant playback" of a specific event (e.g., a near-collision, a sudden lane change, an evasive maneuver, etc.), thereby allowing the user to better understand why the autonomous vehicle responded in that manner. In some embodiments, the imaging sensor continuously captures video (e.g., in a video buffer), and portions of the video are selectively extracted for display based on the detection of a specific triggering event (e.g., a nearby object, a lane change, an evasive maneuver, an emergency braking, etc.).

[0246] In some embodiments, the GUI is used to display entertainment programs (e.g., movies, TV shows, videos, music, etc.) to passengers. For example, as Figure 34 As shown, GUI 3020 includes several selectable display elements 3022, each corresponding to a different program. The user can select one of the display elements 3012 to play back a specific program (e.g., such as...). Figure 35 (As shown). Furthermore, GUI3020 can interrupt playback to present information about the autonomous vehicle to passengers. For example, GUI3020 can present notification messages 3024 about the occurrence of a specific event or situation (e.g., a construction site is detected, an object is detected near the autonomous vehicle, a lane change is determined, etc.).

[0247] In some embodiments, the GUI is used to display safety and orientation information to passengers. This can be useful, for example, in educating passengers about the safe operation of autonomous vehicles. Figure 36Example GUI3026a shows a system that welcomes passengers aboard the autonomous vehicle and displays information about the upcoming journey, such as the start location, destination, distance between them, estimated travel time, and traffic levels along the route. As another example, Figure 36 A sample GUI 3026b is shown, instructing users to fasten their seatbelts during the trip preparation phase. As another example, Figure 36 Example GUI3026c shows a request from the user for instructions to start a trip.

[0248] Example procedures for monitoring and controlling the operation of autonomous vehicles.

[0249] Figure 37 An example process 3700 for monitoring and controlling the operation of an autonomous vehicle is shown. Process 3700 can be performed, at least in part, using one or more of the systems described herein (e.g., using information about...). Figures 1-36 The description includes one or more computer systems, AV systems, autonomous vehicles, graphical user interfaces, etc.

[0250] In process 3700, the computer device receives first vehicle telemetry data from each of the plurality of vehicles (step 3710). The vehicle telemetry data includes each of the plurality of vehicles and an indication of the geographic location of that vehicle. At least one of the plurality of vehicles is an autonomous vehicle.

[0251] In some embodiments, the first vehicle telemetry data includes relatively “basic,” “simple,” “lightweight,” or “general” data (e.g., compared to the second vehicle telemetry data described in detail below). As an example, the first vehicle telemetry data can be relatively small in size and / or relatively uncomplicated in nature. As another example, the first vehicle telemetry data may include a relatively small subset of data collected by the autonomous vehicle’s sensors. As yet another example, the first vehicle telemetry data may include a relatively abstract representation of the data collected by the autonomous vehicle’s sensors (e.g., a generalized, abbreviated, edited, and / or simplified representation of the collected data).

[0252] This document describes various examples of telemetry data for the first vehicle. As an example, for each of a plurality of vehicles, the telemetry data for the first vehicle may include geographic coordinates corresponding to the vehicle's geographic location and / or the vehicle's altitude.

[0253] The computer generates a user interface based on telemetry data from the first vehicle and presents the user interface on a display device associated with the computer device (step 3720). The user interface includes a graphical map and one or more first display elements. Each first display element indicates at least the corresponding geographical location of a given vehicle among a plurality of vehicles. In some embodiments, the user interface includes second display elements indicating one or more vehicles in a list. In some embodiments, for at least one autonomous vehicle, the user interface includes instructions for the assigned tasks of the autonomous vehicle (e.g., driving to a customer / package, delivering a customer / package, charging, idling, repositioning, etc.). Example user interfaces are shown in, for example... Figures 23-29 As shown in the image.

[0254] The computer device receives first user input from a plurality of vehicles to select a specific vehicle (step 3730). As an example, the user may use an input device (such as a mouse, touchpad, stylus, or touch-sensitive display device) to select the vehicle.

[0255] In response to receiving the first user input, the computer device acquires second vehicle telemetry data from the selected vehicle (step 3740). The second vehicle telemetry data may be more "detailed" compared to the first vehicle telemetry data. For example, the second vehicle telemetry data may be relatively larger and / or more complex than the first vehicle telemetry data. As another example, the second vehicle telemetry data may include a larger subset of data collected by the autonomous vehicle's sensors compared to the first vehicle telemetry data. As another example, the second vehicle telemetry data may include a relatively less abstract representation of the data collected by the autonomous vehicle's sensors (e.g., a more detailed overview or less simplified representation of the collected data, or the collected data itself). As yet another example, the second vehicle telemetry data may include the entirety of the data collected by the autonomous vehicle's sensors, rather than a subset of the collected data.

[0256] This document describes various examples of telemetry data for the first vehicle. This may include video, images, sensor data, CPU, battery, energy consumption, orientation, driving speed, general vehicle status (e.g., normal / faulty conditions), fault indications (e.g., brake failure, sensor failure, etc.), historical data, decision making, environmental information, traffic speed, object identification, geographic location, object color, vehicle / object orientation, route information, route selection decision making, etc.

[0257] As an example, the second vehicle telemetry data may include video and / or images captured by sensors of the selected vehicle (e.g., imaging sensors, such as light detectors or camera modules). As another example, the second telemetry data may include an indication of the speed of the selected vehicle and / or an indication of the orientation of the selected vehicle. As another example, the second vehicle telemetry data may include an indication of the operating status of the selected vehicle's computer system (e.g., CPU usage, memory usage, etc.). As another example, the second vehicle telemetry data may include an indication of the status of one or more batteries of the selected vehicle. As another example, the second vehicle telemetry data may include an indication of the energy consumption of the selected vehicle. As another example, the second vehicle telemetry data may include an indication of the operating status of the selected vehicle (e.g., overall vehicle status, such as normal / fault conditions, fault indicators (such as brakes or fault sensors), etc.).

[0258] In some embodiments, the second vehicle telemetry data includes information about the environment of the selected vehicle. As an example, this may include indications of one or more objects near the selected vehicle, indications of weather in the environment, indications of one or more parking spaces in the environment, and / or indications of landmarks in the environment.

[0259] In some embodiments, the telemetry data of the second vehicle includes information about the route of the selected vehicle (e.g., the current route, decisions made regarding route selection, etc.).

[0260] The computing device uses a display device to present a visual representation on a user interface of at least the portion of the first vehicle telemetry data relevant to the selected vehicle, or the portion of the second vehicle telemetry data relevant to the selected vehicle (step 3750). In some embodiments, the visual representation includes “raw” data (e.g., data acquired directly from sensors without substantial post-processing). In some embodiments, the visual representation includes “processed” data (e.g., data acquired from sensors and subsequently manipulated to, for example, summarize data, identify new trends or insights based on the data, or perform other data analyses).

[0261] In some embodiments, the visual representation includes “augmented reality” video. For example, this could include video captured by sensors of the selected vehicle, and one or more graphical elements overlaid on the video (e.g., indicating one or more detected objects in the video, such as other vehicles, pedestrians, objects, etc.).

[0262] In some embodiments, the telemetry data of the first vehicle and / or the telemetry data of the second vehicle are text data. For example, the telemetry data of the first vehicle and / or the telemetry data of the second vehicle may include one or more data items in JavaScript Object Notation (JSON), Hypertext Markup Language (HTML), or Extensible Markup Language (XML) format.

[0263] In some cases, the computer device receives indications of abnormal operation of the first vehicle from the first vehicle. In response, the computer device may display an alert regarding the abnormal operation of the first vehicle on a user interface via a display device. The indication of abnormal operation of the first vehicle may be an indication of an interruption of the network connection between the first vehicle and the computer device and / or an indication that the path of the first vehicle is blocked.

[0264] In some embodiments, the computer device receives second user input, which includes one or more search criteria regarding a plurality of vehicles. The computer device determines one or more vehicles that match the one or more search criteria and presents a visual representation of the one or more vehicles that match the one or more search criteria on a user interface. The one or more search criteria may include service facilities associated with one or more autonomous vehicles, and determining one or more vehicles that match the one or more search criteria may include identifying one or more vehicles located near service facilities.

[0265] Figure 38 Another example process 3800 for monitoring and controlling the operation of an autonomous vehicle is shown. Process 3800 may be performed at least in part using one or more of the systems described herein (e.g., using information about...). Figures 1-36 The described one or more computer systems, AV systems, autonomous vehicles, graphical user interfaces, etc.

[0266] In process 3800, the user interface is presented on the display device of the autonomous vehicle (step 3810). The user interface includes a visual representation of the environment surrounding the autonomous vehicle, a first display element indicating the physical position of the autonomous vehicle relative to the environment, and one or more second display elements. An example user interface is shown in, for example... Figures 30-36 As shown in the image.

[0267] Each second display element indicates a corresponding operational attribute of the autonomous vehicle. For example, this could include the vehicle's route, objects detected by the vehicle, detected events, video feeds, video playback, augmented reality video overlays, confidence indicators, deceleration indicators, etc.

[0268] As an example, at least one second display element may indicate the route of the autonomous vehicle. As another example, at least one second display element may indicate the occurrence of an event related to the autonomous vehicle. The event may be traffic congestion in the autonomous vehicle's environment, road construction in the autonomous vehicle's environment, a decrease in speed of traffic patterns along the autonomous vehicle's path, and / or any other event. As yet another example, at least one second display element may include video captured by the autonomous vehicle's sensors. The video may be "augmented reality" video. For example, at least one second display element may include one or more graphic elements overlaid on the video (e.g., indicating one or more detected objects in the video).

[0269] As another example, at least one second display element may indicate an object detected by the autonomous vehicle. As another example, at least one second display element may indicate the classification of the object detected by the autonomous vehicle. The classification may be at least one of vehicle, pedestrian, or stationary object. Further, at least one second display element may indicate a confidence measure associated with the classification of the object detected by the autonomous vehicle.

[0270] In some embodiments, "instant playback" video is displayed to the user. For example, the autonomous vehicle may determine the occurrence of a triggering event related to the autonomous vehicle (e.g., detection of an object approaching the autonomous vehicle, detection of an object along the path of the autonomous vehicle, activation of emergency braking of the autonomous vehicle, activation of evasive maneuvers of the autonomous vehicle, or any other event). Further, the autonomous vehicle may acquire video captured by its sensors (e.g., a camera module). The autonomous vehicle may determine the portion of the video corresponding to the triggering event (e.g., a video clip) and use a display device to present that portion of the video on the user interface.

[0271] In some embodiments, the autonomous vehicle detects multiple objects in its vicinity. One or more second display elements may indicate a subset of the detected objects.

[0272] User input specifying the operation to be performed by the autonomous vehicle is received via a user interface (step 3820). As an example, the user may use an input device such as a mouse, touchpad, stylus, or touch-sensitive display device to input the information.

[0273] In response to receiving user input, the specified operation is performed using the autonomous vehicle (step 3830). As an example, this may include displaying additional information related to the autonomous vehicle, modifying the autonomous vehicle's path, modifying the autonomous vehicle's destination, or performing any other action related to the autonomous vehicle.

[0274] Figure 39 Another example process 3900 for monitoring and controlling the operation of an autonomous vehicle is shown. Process 3900 can be performed at least in part using one or more of the systems described herein (e.g., using information about...). Figures 1-36 The description includes one or more computer systems, AV systems, autonomous vehicles, graphical user interfaces, etc.

[0275] In process 3900, the computer system receives a request from the autonomous vehicle to remotely control the operation of the autonomous vehicle (step 3910). The request includes information relating to one or more objects located near the autonomous vehicle. For example, the request may include at least one video or image captured by the autonomous vehicle's sensors (e.g., imaging sensors, such as light detectors or camera modules), an indication of the classification of each object (e.g., other vehicles, pedestrians, bicycles, etc.), an indication of the position of each object relative to the autonomous vehicle, and / or text data indicating the position of the objects relative to the autonomous vehicle. In some embodiments, the text data includes one or more data in JavaScript Object Notation (JSON), Hypertext Markup Language (HTML), or Extensible Markup Language (XML) data formats.

[0276] The computer system generates a user interface based on the received request and presents the user interface on a display device (step 3920). The user interface includes a visual representation of the environment surrounding the autonomous vehicle, a first display element indicating the physical position of the autonomous vehicle relative to the environment, and a second display element indicating one or more objects located near the autonomous vehicle.

[0277] In some embodiments, the second display element includes an occupancy grid that identifies the location of one or more objects near the autonomous vehicle. As an example, the second display element may include a graphical representation of multiple areas surrounding the autonomous vehicle, and for at least one area, may include graphical indications (e.g., contrasting shadows, colors, markings, patterns, etc.) that at least one of the objects is located within that area. In some embodiments, the graphical representation of multiple areas surrounding the autonomous vehicle includes a graphical grid defining multiple two-dimensional boxes (each corresponding to a different location around the autonomous vehicle), wherein one or more boxes have contrasting shadows, colors, markings, patterns, or other visual features to indicate the presence of an object at a particular location. In some embodiments, the second display element is generated based on text data, which, for each object, indicates the position of the object relative to the multiple two-dimensional boxes of the graphical grid. The text data may include one or more data items in JSON, HTML, or XML data format. Further, the text data may be received from the autonomous vehicle. An example occupancy grid is shown below. Figures 20-22To describe.

[0278] The computer system receives user input that specifies a path for the autonomous vehicle to traverse one or more objects in the environment (step 3930). In some embodiments, the computer system receives the user input by detecting a physical gesture performed by the user relative to a second display element (e.g., drawing a route relative to an occupancy grid). In some embodiments, the second display element is presented on a touch-sensitive display device, and the touch-sensitive display device is used to detect the physical gesture (e.g., by detecting the user drawing a route on the displayed occupancy grid).

[0279] In some embodiments, the user-specified path is modified based on one or more reference paths (e.g., "engaged" with reference paths, aligned with one or more reference paths, etc.). In some embodiments, the reference paths correspond to traffic patterns on roads in the environment surrounding the autonomous vehicle.

[0280] In response to receiving user input, the computer system transmits command signals to the autonomous vehicle, including instructions for the autonomous vehicle to traverse a specified route.

[0281] Figure 40 Another example process 4000 for monitoring and controlling the operation of an autonomous vehicle is shown. Process 4000 may be performed at least in part using one or more of the systems described herein (e.g., using information about...). Figures 1-36 The aforementioned one or more computer systems, AV systems, autonomous vehicles, graphical user interfaces, etc.

[0282] In process 4000, the autonomous vehicle receives sensor data from one or more sensors of the autonomous vehicle (step 4010). The sensor data may include, for example, one or more videos, images, proximity data, and / or other information captured by one or more sensors.

[0283] The autonomous vehicle generates a request for remote control of the autonomous vehicle by a computer system located remotely from the autonomous vehicle (step 4020). Generating the request includes: determining a quality metric associated with the network connection between the autonomous vehicle and the computer system; and when the quality metric is determined to be above a threshold quality level, including in the request a first data item representing sensor data (e.g., a data item that meets one or more conditions associated with the threshold quality level). Alternatively, when the quality metric is determined to be below the threshold quality level, including in the request another data item representing sensor data (e.g., a data item associated with the lower quality level).

[0284] In some embodiments, one or more conditions associated with the threshold quality level include at least one of the data size or complexity of the sensor data. For example, if the quality metric meets or exceeds the threshold quality level, the included sensor data may have a specific data size or complexity (e.g., a higher data size or complexity). Conversely, if the quality metric is below the threshold quality level, the included sensor data may have a different data size or complexity (e.g., a lower data size or complexity).

[0285] In some embodiments, quality metrics correspond to one or more aspects of a network connection, such as the available bandwidth of the network connection, the latency associated with the network connection, and / or the reliability of the network connection. A higher quality metric may correspond to a higher quality connection (e.g., greater available bandwidth, lower latency, and / or higher reliability), while a lower quality metric may correspond to a lower quality connection (e.g., less available bandwidth, higher latency, and / or lower reliability). In some embodiments, quality metrics are determined by transmitting beacon signals through the network connection (e.g., including the transmission of test data), and the quality metric is determined based on the transmission of beacon signals. In some embodiments, the available bandwidth of the network connection is determined based on the transmission of beacon signals.

[0286] The autonomous vehicle transmits a remote control request to the computer system (step 4030).

[0287] In some embodiments, the autonomous vehicle further determines a second quality metric associated with the network connection between the autonomous vehicle and the computer system (e.g., at a different time and / or location than the determination of the first quality metric). When the second quality metric is determined to be below a threshold quality level, the autonomous vehicle may include a second data item representing sensor data in a request. The second data item may meet one or more conditions associated with the threshold quality level.

[0288] Furthermore, the second data item may have a smaller data size or lower complexity than the first data item. For example, the first data item may include video with a higher resolution compared to the video included in the second data item. As another example, the first data item may include video with a higher frame rate compared to the video included in the second data item. As another example, the first data item may include images with a higher resolution compared to the images included in the second data item. As another example, the first data item may include a greater number of images than the second data item. As another example, the first data item may include a portion of sensor data with a higher spatial resolution compared to the portion of sensor data included in the second data item. As another example, the first data item may include a portion of sensor data with a higher temporal resolution compared to the portion of sensor data included in the second data item. As another example, the first data item may include at least one video or image of the autonomous vehicle's environment, and the second data item may include text data describing the autonomous vehicle instead of the at least one video or image. In some embodiments, the text data may include one or more data items in JSON, HTML, or XML data formats.

[0289] In some embodiments, the autonomous vehicle determines a condition that impedes its movement. In response to this determination, a remote control request is generated. This condition may correspond to, for example, an object blocking the autonomous vehicle's path or a closure of a road along the autonomous vehicle's path.

[0290] In some embodiments, the autonomous vehicle receives command signals from a computer system (e.g., in response to a request). Command signals may include instructions for the autonomous vehicle to traverse a defined path. The autonomous vehicle may execute command signals to traverse a specified route (e.g., to avoid one or more objects obstructing its movement).

[0291] In the preceding description, embodiments of the invention have been described with reference to numerous specific details, which may vary depending on implementation. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The scope of the invention is a unique and exclusive indicator, and the applicant expects the scope of the invention to be the literal and equivalent scope of the claims published in this application in a specific form, including any subsequent corrections. The definitions of any terms expressly set forth herein and included in such claims should be taken as they are used in the claims. Furthermore, when we use the term “further includes” in the preceding specification or the following claims, what follows that phrase may be an additional step or entity, or a sub-step / sub-entity of a previously stated step or entity.

Claims

1. A method for an autonomous vehicle, the method comprising: The autonomous vehicle receives sensor data from one or more sensors of the autonomous vehicle. The autonomous vehicle generates a request for remote control of the autonomous vehicle by a computer system located remotely from the autonomous vehicle, wherein generating the request includes: The autonomous vehicle determines a quality metric associated with the network connection between the autonomous vehicle and the computer system, and When it is determined that the quality metric is greater than a threshold quality level, a first data item representing the sensor data is included in the request, the first data item meeting one or more conditions associated with the threshold quality level; The request for remote control is transmitted from the autonomous vehicle to the computer system; Transmitting first vehicle telemetry data from the autonomous vehicle to the computer system; and In response to receiving a request from the computer system for additional information about an autonomous vehicle selected by the user from a plurality of autonomous vehicles, the selected autonomous vehicle transmits second vehicle telemetry data to the computer system, wherein the second vehicle telemetry data is relatively larger and / or more complex than the first vehicle telemetry data.

2. The method as described in claim 1, characterized in that, The sensor data includes at least one of video, image, or proximity data captured by the one or more sensors.

3. The method as described in claim 1, characterized in that, The quality metric corresponds to the available bandwidth of the network connection.

4. The method as described in claim 1, characterized in that, The quality metric corresponds to the latency associated with the network connection.

5. The method as described in claim 1, characterized in that, The quality metric corresponds to the reliability of the network connection.

6. The method as described in claim 1, characterized in that, Determining the quality metric associated with the network connectivity quality includes: Using the network connection to transmit beacon signals, and The quality metric is determined based on the transmission of the beacon signal.

7. The method as described in claim 6, characterized in that, Determining the quality metric based on the transmission of the beacon signal includes determining the available bandwidth of the network connection based on the transmission of the beacon signal.

8. The method as described in claim 1, characterized in that, The one or more conditions associated with the threshold quality level include at least one of the data size or complexity of the sensor data.

9. The method as described in claim 1, characterized in that, When it is determined that the quality metric is below the threshold quality level, a second data item representing the sensor data is included in the request. The second data item meets one or more conditions associated with the threshold quality level and has a smaller data size or lower complexity compared to the first data item.

10. The method as described in claim 9, characterized in that, The first data item includes videos with higher resolution compared to the videos included in the second data item.

11. The method as described in claim 9, characterized in that, The first data item includes videos with a higher frame rate compared to the videos included in the second data item.

12. The method as described in claim 9, characterized in that, The first data item includes an image with a higher resolution compared to the image included in the second data item.

13. The method as described in claim 9, characterized in that, The first data item includes a greater number of images than the number of images included in the second data item.

14. The method as described in claim 9, characterized in that, The first data item includes the portion of the sensor data that has a higher spatial resolution compared to the portion of the sensor data included in the second data item.

15. The method as described in claim 9, characterized in that, The first data item includes the portion of the sensor data that has a higher temporal resolution compared to the portion of the sensor data included in the second data item.

16. The method as described in claim 9, characterized in that, The first data item includes at least one video or image of the environment of the autonomous vehicle, and the second data item includes text data describing the environment of the autonomous vehicle instead of the at least one video or image.

17. The method as described in claim 16, characterized in that, The text data includes one or more data items in JavaScript Object Notation (JSON), Hypertext Markup Language (HTML), or Extensible Markup Language (XML) format.

18. The method of claim 1, further comprising: The autonomous vehicle determines the conditions that impede its movement, and The remote control request is generated in response to determining a condition that hinders the movement of the autonomous vehicle.

19. The method as described in claim 18, characterized in that, The condition that hinders the autonomous vehicle's movement corresponds to the object that blocks the autonomous vehicle's path.

20. The method as described in claim 18, characterized in that, The condition that hinders the movement of the autonomous vehicle corresponds to the closure of the road along the path of the autonomous vehicle.

21. The method of claim 1, further comprising: The autonomous vehicle receives command signals from the computer system, the command signals including instructions for navigating a specified path to the autonomous vehicle, and... The command signal is executed at the autonomous vehicle to navigate the specified path.

22. An apparatus for an autonomous vehicle, the apparatus comprising: One or more processors; Memory; as well as One or more programs, the programs being stored in memory and containing instructions for performing the method as described in any one of claims 1-21.

23. A non-transient computer-readable storage medium comprising one or more programs executed by one or more processors of a device, the one or more programs comprising instructions that, when executed by the one or more processors, cause the device to perform the method as claimed in any one of claims 1-21.

24. A computer program product comprising a computer program including instructions that, when executed by a processor, implement the method as described in any one of claims 1-21.

Images