Remote Operation Queuing for a Self-Driving Vehicle

Abstract

The remote operation system receives a request for remote operator assistance and adds the request to a queue of additional requests. The queue can be ordered based on, for example, the time of reception, priority, importance, etc. The remote operation system determines a remote operator from a set of remote operators for providing a response to a request in the queue, at least in part based on one or more of the status of the remote operator (e.g., indicating availability, whether they are in training, etc.), criteria associated with the remote operator (e.g., skills in responding to various requests, preferences regarding geographical areas, mission types, etc.), and information associated with the request received from the vehicle (e.g., mission type, sensor data, messages, vehicle status, etc.). If a request is not accepted within a threshold period, the request can be rerouted to additional remote operators.

Description

Technical Field

[0001] Cross - reference to Related Applications This PCT international application was filed on July 29, 2022, and claims the priority of U.S. Patent Application No. 17 / 876,975, entitled "Teleoperations Queueing for Autonomous Vehicles," which is hereby incorporated by reference in its entirety.

Background Art

[0002] Semi - autonomous and fully autonomous vehicles present a new set of technical challenges compared to vehicles operated by a driver. For example, autonomous vehicles may encounter scenarios that they have never faced before, or scenarios that are so complex that the autonomous vehicle cannot determine how to navigate through them with a sufficient level of certainty. In such situations, input from a remote operator can assist the autonomous vehicle in navigating through the scenario.

Summary of the Invention

[0003] A detailed description is set forth with reference to the accompanying drawings. In the figures, the left - most digit of a reference number identifies the figure in which that reference number first appears. The same reference number in different figures indicates a similar or identical item.

Brief Description of the Drawings

[0004]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

[0005] The present application relates to a technique for matching an autonomous vehicle that requests assistance to pass through an environment with a remote operator. In some examples, the autonomous vehicle can periodically transmit information associated with the status of the autonomous vehicle, such as the current mission of the vehicle, the position of the vehicle, the posture of the vehicle, speed, direction, etc., to the remote operator system. Such information is useful, for example, in determining a remote operator to assist the vehicle when the autonomous vehicle requests assistance to pass through the environment. In some examples, the remote operator can be associated with (or otherwise provide) criteria associated with the type of assistance the remote operator provides or desires to provide. In addition, the availability of the remote operator, such as whether the remote operator is online to provide assistance and / or whether the remote operator is unavailable, is useful when assigning requests to the remote operator. The remote operator is optimally matched with the autonomous vehicle that requests assistance as a means for quickly resolving incidents without significant delay.

[0006] An autonomous vehicle can be in communication with a remote operation system that receives information associated with the status of the autonomous vehicle. In some examples, the autonomous vehicle is configured to automatically transmit information according to a predetermined schedule (e.g., every second, every minute, etc.) and / or in response to the occurrence of a specific event (e.g., upon request of a fleet monitoring service, when the vehicle has traveled a threshold distance, upon detection of an object, when it is impossible to pass through a region, when the level of uncertainty drops or falls below a threshold level, a passenger issue, etc.). Such information can indicate any number of indicators including, but not limited to, informing the remote operation system of the status of the autonomous vehicle, whether the autonomous vehicle needs assistance, the type of assistance required, providing information about the surroundings of the vehicle, etc. As described above, such information transmitted by the autonomous vehicle can include the current state of the vehicle such as speed, direction (or heading), position, whether the remote operator is communicating with and / or controlling the autonomous vehicle, the health status of the components of the autonomous vehicle (e.g., brakes, microcontrollers, HVAC controllers, etc.), mission type (e.g., performing recharging, performing training, transporting passengers, picking up passengers, etc.), and so on. The remote operation system can communicate with any number of autonomous vehicles within a fleet and monitor their status.

[0007] The remote operation system is associated with one or more remote operators who respond to or otherwise provide support to the autonomous vehicle. In some examples, a remote operator can set criteria associated with the support they provide to the remote vehicle. For example, a remote operator can be associated with criteria including, for example, vehicle type, location, type of support, mission type, etc. As a non-limiting example, one operator can be associated with criteria related to responding to problems with internal components (e.g., HVAC, brakes, communication systems, etc.), another operator can have criteria indicating skills related to situation awareness and planning, while another operator can be associated with criteria related to passenger problems (e.g., medical emergencies, etc.).

[0008] Additionally, or alternatively, a remote operator may be able to set their status. The status can generally indicate the availability of the remote operator, such as whether the remote operator is on break, in a training mode (e.g., unable to accept all or specific types of requests), whether the remote operator is already providing support for another request, whether the remote operator is in the middle of accepting a request, etc. More generally, the status can indicate whether the remote operator is occupied or unoccupied for the purpose of receiving requests and providing support to the vehicle. As discussed herein, the criteria and / or status are used to match the vehicle requesting support with the remote operator.

[0009] The remote operation system has further insight into available remote operators. For example, a particular remote operator may be on duty or off duty. A remote operator on duty may be available, while a remote operator off duty may not be available. Using available remote operators, the remote operation system can filter remote operators when assigning requests. In some examples, the remote operation system determines available remote operators based on the remote operator being logged into the remote operation system and / or the remote operator providing an indication to that effect (e.g., the remote operator can switch their status to "available").

[0010] In some instances, from time to time, an autonomous vehicle may encounter an event that is inherently unpredictable, an event that poses a safety concern, an event of a type not previously encountered, or an event that, for example, requires a response to a spontaneous visual cue or instruction from a police officer or construction worker, and that cannot be confidently navigated through. Of course, such examples are provided for illustrative purposes only and are not intended to be limiting. In some instances, an autonomous vehicle may be unable to plan a path to navigate around an obstacle and / or may determine that the level of confidence associated with one or more maneuvers (e.g., the planned trajectory or path of the autonomous vehicle) and / or events (e.g., detection or classification of a particular object, prediction of an object's behavior, etc.) is insufficient to proceed autonomously (e.g., below a threshold confidence level). In such cases, the autonomous vehicle may be able to send a request for guidance to a remote operation system. In some instances, a passenger may be able to initiate such a request, for example, by pressing a button, speaking a phrase, or being recognized by the vehicle system as in need of assistance (e.g., using an interior camera and / or microphone).

[0011] A remote operator may be able to provide assistance to the autonomous vehicle and / or a passenger within the autonomous vehicle in any suitable manner. In some instances, the assistance can include transmitting processor-executable instructions from the remote operator's device to the autonomous vehicle via a network interface. These instructions can cause the autonomous vehicle to perform an action, cooperate with the autonomous vehicle to determine an action to perform, and / or confirm a potential action determined by the autonomous vehicle. In various instances, such instructions can also include an audiovisual message for display to one or more passengers.

[0012] The remote operation system is capable of receiving requests from a plurality of autonomous vehicles. When a request is received, the request can be ordered within a queue and transmitted to a remote operator for processing. In some examples, the request can be ordered within the queue based on the time the request was received. Additionally, or alternatively, the request can be prioritized based on specific safety considerations such as the vehicle's operating speed (e.g., operation on arterial roads versus operation in urban streets), the vehicle's occupancy status (not occupied or occupied, number of passengers, etc.), the length of the ride, traffic volume, or other factors. In any case, upon receiving a request, the remote operation system can select a remote operator to respond to the request.

[0013] In some instances, a request and a remote operator can be matched based on vehicle information, request details, the status of the remote operator, and / or the criteria of the remote operator. First, available remote operators can be filtered from a plurality of remote operators associated with the remote operation system. For the available remote operators, their status and / or criteria can be used as a way to filter requests and select an available remote operator to respond to the request. Selecting a remote operator in this manner can increase the response time and provide effective assistance. For example, if a remote operator has requested to provide assistance within a particular geographic area (e.g., region), that remote operator may be familiar and / or proficient in how to navigate within that particular geographic area. When a request associated with an autonomous vehicle within that particular geographic area in need of assistance is received, that remote operator can be selected (potentially, prior to one or more other remote operators) to provide assistance. This can enable the remote operator to quickly understand the situation and respond accordingly, as compared to a situation where the remote operator is unfamiliar with the geographic area and it could cause delays in guiding the autonomous vehicle. In at least some such instances, knowledge of a particular geographic area can be obtained based on the relative number of requests that the remote operator has responded to regarding that area.

[0014] Furthermore, in some examples, mission types can be used to filter requests. For example, an autonomous vehicle may be operating in a training mode by driving around a test facility. If such an autonomous vehicle issues a request for assistance, this request can be ignored, placed in a separate queue, or otherwise processed differently than vehicles not operating in the training mode. As a further example, if a remote operator is in the process of being trained on how to provide assistance to an autonomous vehicle, that remote operator can be excluded as being unable to respond to requests (or a particular type of request). This can ensure the safety of the autonomous vehicle and the quality of the assistance provided. In some examples, any number of filters can be applied to match remote operators to requests, and the order in which those filters are applied can be dynamic depending on the situation. In some examples, the order in which filters are applied can be hierarchical based on the relative priorities of the various filters being applied.

[0015] Once a remote operator is determined for a particular request, the remote operation system can send that request to that remote operator. In some examples, this includes displaying an indication of the request on a device being operated by the remote operator. The remote operator can have a predetermined amount of time to respond to the request (e.g., accept the request). If a response is received within the predetermined amount of time, the request is assigned to the remote operator. Alternatively, if no response is received within the predetermined amount of time, the request can be sent to one or more additional remote operators and / or returned to the queue for reallocation. In some examples, the first remote operator among those accepting the request in response can be assigned the request to provide assistance to the autonomous vehicle.

[0016] In response to a request being assigned to a remote operator, the remote operator's device can display information associated with the autonomous vehicle. Such information can include, for example, sensor data generated by the autonomous vehicle (e.g., image data, lidar data, etc.), and / or graphical representations of the sensor data (e.g., synthetic, computer-generated scenes based on the sensor data). This enables the remote operator to recognize the situation of the autonomous vehicle, whereby the remote operator can provide guidance to the autonomous vehicle with minimal delay. Of course, any other data transmitted from the vehicle, including, for example, internal camera data, microphone data, vehicle status, etc., is considered to be shown to the remote operator or relayed in some other form.

[0017] In some instances, when more than two remote operators match the request, the remote operation system can select a remote operator to load balance the request among the two remote operators. For example, if a first remote operator and a second remote operator are able to provide assistance for the request, the remote operation system can select the remote operator who has responded to the least amount of requests over a given period (e.g., shift, time, day, etc.), the remote operator with the fastest past response time (based on all past response times, most recent response time, etc.), the remote operator with the most constraints (i.e., on average, the remote operator who is more likely to match fewer requests), or the remote operation system can randomly select from among the remote operators who match the request. In at least some instances, it is possible to select a remote operator associated with a previously successful resolution of a similar request, and / or it is possible to select the remote operator with the highest percentage of successful resolutions of requests. Additionally, or alternatively, when more than two remote operators match the request, the remote operation system can select a remote operator based on experience, the type of request associated with the vehicle (e.g., construction), the amount of time since the remote operator last responded to the request, the proficiency of the remote operator with respect to the geographical area of the vehicle, the latency associated with the terminal associated with the remote operator, etc.

[0018] The techniques and systems described herein can be implemented in a plurality of ways. Exemplary embodiments are provided hereinafter with reference to the figures. The embodiments, examples, and illustrations described herein can be combined.

[0019] FIG. 1 is a schematic diagram of an environment 100 in which a vehicle 102 is traveling. The vehicle 102 can be an autonomous vehicle, such as an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration of the United States, which describes vehicles that can perform all safety-critical functions throughout a trip without the driver (or passengers) being expected to control the vehicle 102 at any point. In some examples, the vehicle 102 can be configured to control all functions from departure to route completion, including all parking functions, so the vehicle 102 can be without means for controlling the vehicle 102, such as a driver and / or a steering wheel, etc. In some examples, the techniques described herein can be incorporated into any land, air, or water vehicle (or autonomous vehicle) that includes vehicles ranging from those that always need to be manually controlled by a driver to those that are partially or fully autonomously controlled. In some examples, the vehicle 102 can correspond to an autonomous vehicle that is part of a group of autonomous vehicles.

[0020] In FIG. 1, an exemplary scenario 104 is shown, in which vehicle 102 can operate autonomously until the vehicle 102 faces an event (a series of conditions inside or outside the vehicle 102, including the environment and operating conditions of the vehicle 102) along road 106. Regarding that event, vehicle 102 can request assistance from, for example, a remote operator 108 located remotely from vehicle 102. For example, at 110, vehicle 102 may be traveling along road 106 in a normal manner. As part of this, vehicle 102 can continuously transmit the operating state data associated with vehicle 102, although such transmission does not need to be continuous and can be transmitted at the occurrence of an event or at some interval. The operating state data can include, but is not limited to, the current state of vehicle 102 such as speed, direction of travel, position, whether the remote operator is connected to / controlling vehicle 102, mission type (e.g., transporting passengers, picking up passengers, recharging the vehicle's battery, test mode), etc.

[0021] At 114, vehicle 102 may face a construction area 124 associated with a portion of road 106, and the traffic in the vicinity of construction area 124 may be under the instructions of construction workers who provide instructions to the traffic to bypass construction area 124. Due in part to the unpredictable nature of this type of event, vehicle 102 can request remote assistance from remote operation system 112. Thus, remote operation system 112 receives, at 114, a request for remote assistance to guide vehicle 102.

[0022] As described herein, the remote operation system 112 includes a request queue (which can be managed by one or more remote servers) and a remote operator (such as remote operator 108). The request queue helps organize requests in response to the vehicle 102 and other vehicles seeking assistance. In some examples, requests are organized and ordered in the request queue based on, for example, the time they are received, priority, geographical location, etc., or any combination thereof. The request queue is used by the remote operation system 112 to organize requests and assign them to remote operators. Of course, multiple request queues can be implemented simultaneously, and accordingly, each individual queue has separate criteria for ordering incoming requests and separate sets of operators for servicing those requests.

[0023] As part of assigning requirements to remote operator 108, remote operation system 112 can determine the availability, preferences, and / or status of remote operator 108 at 116. For example, the availability of remote operator 108, such as whether remote operator 108 is on-duty or off-duty, can be used to filter remote operators connected to and capable of providing assistance to remote operation system 112. The status can further indicate, for available remote operators, whether an individual remote operator is on break, in a training mode (and thus unable to provide assistance), whether the remote operator is already providing assistance to a vehicle, etc. Thus, here the status can indicate whether a remote operator is available or unavailable for the purpose of providing assistance to vehicle 102. Preferences can be set by remote operator 108 and can correspond to the type of assistance that remote operator 108 desires to provide or the case of assistance. For example, remote operator 108 may only wish to assist vehicles engaged in a particular mission type, vehicles within a particular geographic area, particular types of vehicles, vehicles requiring a given assistance (e.g., navigating a construction area, an accident, etc.). In such cases, remote operator 108 has the ability to customize the types of incidents to which they respond or provide assistance.

[0024] After determining the preference and / or status, at 118, scenario 104 can select a remote operator to respond to the request. For example, the remote operation system 112 can match the request to an operator based on availability, preference, and / or status. Further, such matching can be based on the operating state data received from the vehicle 102. For example, the operating state data can indicate the position of the vehicle, and that position is compared to the geographical location where the remote operator 108 desires to provide assistance. As discussed herein, the selected remote operator 108 can be selected from among a plurality of remote operators of the remote operation system 112, in which case each remote operator can have its own preference and status.

[0025] In 120, the remote operator 108 can be connected to the vehicle 102. For example, the remote operator 108 can interact with the vehicle 102 via a user interface that can include a remote operator interface. The remote operator interface can include one or more displays 122 configured to provide the remote operator 108 with data related to the vehicle 102, a subset of the group of vehicles, and / or operations of the group of vehicles. For example, the display 122 can be configured to show data related to sensor signals received from the vehicle 102, data related to the road 106, and / or additional data or information to facilitate providing assistance to the vehicle 102. In some examples, the remote operator 108 can first be provided with the opportunity to accept or reject a request, and based at least in part on the response of the remote operator 108, the remote operator 108 can be connected to the vehicle 102 or not be connected. For example, an indication enabling the remote operator 108 to accept or reject a request can be displayed on one or more of the displays 122. Further examples of connecting a remote operator or remotely controlling a vehicle are described, for example, in U.S. Patent No. 11,209,822 entitled "Techniques for Contacting a Teleoperator", which is hereby incorporated by reference in its entirety for all purposes.

[0026] The remote operator 108 may utilize one or more devices associated with the display 122, for example, that enable the remote operator 108 to provide information to the vehicle 102. Such information may be in the form of a remote operation signal that provides guidance to the vehicle 102. The remote operator device may include one or more of a touch-sensitive screen, a stylus, a mouse, a dial, a keypad, a microphone, a touch screen, and / or a gesture input system. In some examples, the remote operators 108 may be human remote operators and may be located at a remote operation center. However, in some examples, one or more of the remote operators 108 may be non-human, such as, for example, a computer system that utilizes artificial intelligence, machine learning, and / or other decision-making strategies, and in at least some examples, a computer system that has different or more powerful computing resources or algorithms than those available on the vehicle.

[0027] The remote operator 108 may provide assistance to the vehicle 102. For example, at 126, the remote operator 108 may determine an action 128 to provide assistance to the vehicle 102, such as bypassing the construction area 124. In some examples, the remote operator 108 may provide guidance to the vehicle 102 to avoid, detour around, or pass through an event. Additionally or alternatively, the remote operator 108 may communicate with the occupants of the vehicle 102 using, for example, a microphone, a speaker, and / or a haptic feedback device.

[0028] Although FIG. 1 and Scenario 104 show a single instance of providing assistance to vehicle 102 or to a single vehicle, remote operation system 112 is capable of receiving any number of requests from vehicles. For example, vehicle 102 can be part of a group of vehicles in communication with remote operation system 112, and remote operation system 112 is configured to process the requests and assign the requests to respective remote operators 108.

[0029] In such an example, vehicle 102 may be faced with separate events that result in requests for remote operator input simultaneously or nearly simultaneously. In such an example, remote operation system 112 may receive a request for assistance from vehicle 102. Requests for assistance can be prioritized within a request queue for processing by remote operator 108. Remote operator 108 can be filtered based on their preferences, status, operational state data associated with the vehicle, and / or the assistance requested. Additionally, or alternatively, remote operator 108 can be filtered based on experience level, training, experience with certain environments (e.g., geolocation, weather events, time of day, number of nearby objects including vehicles and pedestrians), situation, vehicle system, network connection speed or latency of a particular remote operator, etc. In some examples, preferences can be weighted differently from each other. For example, if a remote operator 108 is given proficiency with a particular geographical environment, the location of vehicle 102 can be weighted more heavily than other preferences.

[0030] FIG. 2 is a block diagram of an architecture 200 that provides data associated with the operation of vehicle 102 and includes a vehicle system 202 for controlling the operation of a system that connects with a remote operation system 112 to control the operation of vehicle 102. Vehicle system 202 can include one or more vehicle computing devices 204, one or more sensor systems 206, one or more emitters 208, one or more communication connections 210, at least one direct connection 212, and one or more drive systems 214.

[0031] Vehicle system 202 can be an autonomous vehicle configured to operate according to a Level 5 classification issued by the United States National Highway Traffic Safety Administration, which describes vehicles that can perform all safety-critical functions over the entire trip without the driver (or passengers) being expected to control the vehicle at any point. In such an example, vehicle system 202 can be configured to control all functions from start to stop, including all parking functions, so vehicle system 202 can be driverless. This is only an example, and the systems and methods described herein can be incorporated into any land, air, or water transportation vehicle, including vehicles that range from those that must always be manually controlled by a driver to those that are partially or fully autonomously controlled. That is, in the example shown, vehicle system 202 is an autonomous vehicle, but vehicle system 202 can be any other type of vehicle. Only a single vehicle system 202 is shown in FIG. 2, but in actual applications, an exemplary system can include multiple vehicles, which in some examples can include a fleet of vehicles.

[0032] The vehicle computing device 204 can include a processor 216 and a memory 618 communicatively coupled to the processor 216. In the illustrated example, the memory 218 of the vehicle computing device 204 stores a localization component 220, a perception component 222, a prediction component 224, a planning component 226, and one or more system controllers 228. Additionally, the memory 218 can include a storage 230, which can store maps, models, and the like. As described above, a map can be any number of data structures that can provide information about the environment, such as topology (junctions, lanes, merge zones, etc.), streets, mountains, roads, terrain, and the overall environment, but is not limited thereto. The map can be associated with an actual environment or a simulated environment.

[0033] In at least one example, the localization component 220 can determine the posture (position and orientation) of the vehicle system 202 in relation to a local and / or global map, based at least in part on sensor data received from the sensor system 206 and / or map data associated with (e.g., of) the map. In at least one example, the localization component 220 can include, or be associated with, a calibration system that can substantially simultaneously perform operations for calibrating (determining various intrinsic and extrinsic parameters associated with any one or more of the sensor systems 206), localizing, and mapping. Further details associated with such a system are described in U.S. Patent Application No. 15 / 675,487, filed Aug. 11, 2017, which is related to U.S. Patent Application No. 15 / 674,853, filed Aug. 11, 2017, the entire contents of both of which are incorporated herein by reference.

[0034] In at least one example, the perception component 222 can perform object detection, segmentation, and / or classification based at least in part on sensor data received from the sensor system 206. In at least one example, the perception component 222 can receive raw sensor data (e.g., from the sensor system 206). In at least one example, the perception component 222 can receive image data and can utilize one or more image processing algorithms to perform object detection, segmentation, and / or classification with respect to objects identified in the image data. In some examples, the perception component 222 can associate a bounding box (or otherwise instance segmentation) with the identified object and can associate a confidence score associated with the classification of the identified object with the identified object. In some examples, objects can be colored based on their perceived class when rendered via a display. The perception component 222 can perform similar processes with respect to one or more other modalities.

[0035] Prediction component 224 may receive sensor data from sensor system 206, map data associated with a map (e.g., of a map that may be in storage 230), and / or perception data (e.g., processed sensor data) output from perception component 222, and may output predictions associated with one or more objects in the environment of vehicle system 202. In at least one example, planning component 226 may determine a route and / or trajectory to use to control vehicle system 202 based at least in part on the sensor data received from sensor system 206 and / or any decisions made by perception component 222 and / or prediction component 224.

[0036] In some examples, the planning component 226 can fuse (e.g., combine) the operating state data from the data it obtains and / or the correlated data it determines. For example, the operating state data can include the display of sensor data and the position of the detected object, the trajectory of the detected object (e.g., the position, velocity, acceleration, and / or direction of travel of the object), the classification of the detected object (e.g., label) (e.g., including subclasses and subsets of classifications as discussed above), the identifier of the detected event, the confidence level (e.g., percentage, an indicator that the classification and / or identifier of the detected event is associated with a high prediction improbability or low reliability indicator), the rate of change of the confidence level over time, and / or the priority associated with the object and / or event, the data of the detected object / event, the route plan data including the route, the progress of the vehicle along the route, the mission type (e.g., stopping for additional passengers, transporting one passenger to a destination), the passenger input, the trajectory, the vehicle's posture, the geographical location of the autonomous vehicle, and / or the trajectory determined by the vehicle, the number of passengers in the vehicle, the passenger input (e.g., voice, passenger status), the indication of the vehicle and / or sensor health, the indication of the vehicle history (e.g., past routes, past assistance requests, past maintenance), the charge level of the vehicle's battery, the distance from the vehicle operation base or charging station of the vehicle group to the vehicle, the indication of whether a communication session is open between the vehicle and the remote operator device and / or another vehicle, the vehicle control data, the vehicle type (e.g., manufacturer, model, size, etc.), the road network data (e.g., data related to the overall or local map of the area associated with the vehicle operation, e.g., the position of the vehicle in the local map, and / or the indication of whether the vehicle data is canonical for that position (e.g., whether the vehicle speed exceeds or is below the speed limit indicated by the road network data, whether the vehicle is stopped at a position identified as a stop position, whether the vehicle is,Vehicle state information including, for events across the entire vehicle group, whether within a pre-defined distance (etc.), communication channel information (e.g., bandwidth and / or quality of connection, identification of the device to which the vehicle is connected, predicted degradation of the communication channel), and / or previous remote operator guidance to the vehicle (e.g., direct instructions, collaboration, and / or confirmation), and environmental data including traffic information, weather information, urban / regional events (e.g., obtained from social media, publications), time, and / or road network data (e.g., identifying the geographical location as associated with normal driving areas, drivable areas, speed limits associated with geographical regions, locations of events (e.g., location of an accident, location from which a large number of requests have been sent), and / or areas that are non-drivable, and / or including an action policy for the vehicle to operate within, a processor-executable map accessible to the vehicle) (e.g., in some examples, this can be included in the display of sensor data or it can be obtained via a network interface).

[0037] In some examples, the planning component 226 can use at least a portion of the sensor data and / or the operating state data to determine, for example, whether to send a request for the next action of the vehicle system 202, such as a trajectory and / or assistance.

[0038] Further details of the location system, perception system, prediction system, and / or planning system that may be used can be found in U.S. Patent No. 9,612,123, issued on April 4, 2017, and U.S. Patent No. 10,353,390, issued on July 16, 2019, the entire contents of both of which are incorporated herein by reference. In some examples (e.g., the vehicle system 202 is not an autonomous vehicle), one or more of the aforementioned systems may be omitted from the vehicle system 202. Although the systems described above are shown as being "mounted" in the vehicle system 202, in other embodiments, those systems may be remotely located and / or accessible to the vehicle system 202. Further, although those systems are described as "systems," such systems may include one or more components for performing the operations attributed to each of those systems.

[0039] In at least one example, the location component 220, perception component 222, prediction component 224, and / or planning component 226 can process sensor data as described above and can transmit their respective outputs to a computing device of the remote operation system 112 via the network 232. In at least one example, the location component 220, perception component 222, prediction component 224, and / or planning component 226 can transmit their respective outputs to a computing device of the remote operation system 112 at a specific frequency, after a predetermined period of time, in near real-time, and so on.

[0040] In at least one example, the vehicle computing device 204 can include one or more system controllers 228, and those system controllers 228 can be configured to control steering, propulsion, braking, safety, emitters, communications, and other systems of the vehicle system 202. These system controllers 228 can communicate with and / or control corresponding systems of the drive system 214 and / or other systems of the vehicle system 202.

[0041] In at least one example, the sensor system 206 can include rider sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), wheel encoders, audio sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ToF sensors, and the like. The sensor system 206 can include multiple instances of each of these or other types of sensors. The sensor system 206 can provide an input to the vehicle computing device 204. In some examples, the sensor system 206 can preprocess at least some of those sensor data before transmitting the sensor data to the vehicle computing device 204. In at least one example, the sensor system 206 can transmit the sensor data to the remote operation system 112 via the network 232 at a specific frequency, after a predetermined period of time, in near real-time, and so on.

[0042] As described above, the vehicle system 202 can also include one or more emitters 208 for emitting light and / or sound. The emitter 208 in this example includes internal audio and visual emitters for communicating with the passengers of the vehicle system 202. By way of example and not limitation, the internal emitters can include speakers, lights, signs, display screens, touch screens, tactile emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.), and the like. The emitter 208 in this example also includes external emitters. By way of example and not limitation, the external emitters in this example include light emitters (e.g., indicator lights, signs, light arrays, etc.) for visually communicating with pedestrians, other drivers, other nearby vehicles, etc., and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians, other drivers, other nearby vehicles, etc. In at least one example, the emitter 208 can be arranged at various positions around the outside and / or inside of the vehicle system 202.

[0043] The vehicle system 202 can also include a communication connection 210 that enables communication between the vehicle system 202 and other local or remote computing devices. For example, the communication connection 210 can facilitate communication with other local computing devices and / or the drive system 214 on the vehicle system 202. Also, the communication connection 210 can enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). The communication connection 210 also enables the vehicle system 202 to communicate with the remote operation system 112 or other remote services.

[0044] The communication connection 210 can include a physical interface and / or a logical interface for connecting the vehicle computing device 204 to another computing device or a network such as the network 232. For example, the communication connection 210 can enable Wi-Fi-based communication via a frequency defined by the IEEE 802.11 standard, short-range wireless frequencies such as BLUETOOTH (registered trademark), or any suitable wired or wireless communication protocol that enables each computing device to interface with other computing devices.

[0045] The direct connection 212 can directly connect the drive system 214 to other systems of the vehicle system 202.

[0046] In at least one example, the vehicle system 202 can include a drive system 214. In some examples, the vehicle system 202 can have a single drive system 214. In at least one example, when the vehicle system 202 has multiple drive systems 214, the individual drive systems 214 can be arranged at opposite ends of the vehicle system 202 (e.g., the front and rear, etc.). In at least one example, the drive system 214 can include a sensor system for detecting the situation around the drive system 214 and / or the vehicle system 202. By way of non-limiting example, the sensor system can include a wheel encoder (e.g., a rotary encoder) for sensing the rotation of the wheels of the drive module, an inertial sensor (e.g., an inertial measurement unit, an accelerometer, a gyroscope, a magnetometer, etc.) for measuring the position and acceleration of the drive module, a camera or other image sensor, an ultrasonic sensor for acoustically detecting objects around the drive module, a lidar sensor, a radar sensor, etc. Some sensors, such as wheel encoders, can be specific to the drive system 214. In some cases, the sensor system on the drive system 214 can overlap with or complement the corresponding systems of the vehicle system 202 (e.g., the sensor system 206).

[0047] The drive system 214 can include many of the vehicle systems, including a high-voltage battery, a motor for propelling the vehicle system 202, an inverter for converting direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights for illuminating the exterior surroundings of the vehicle), and one or more other systems (e.g., a cooling system, a safety system, an on-board charging system, other electrical components such as a DC / DC converter, a high-voltage junction, a high-voltage cable, a charging system, a charge port, etc.). Additionally, the drive system 214 can include a drive module controller, which can receive and preprocess data from sensor systems and control the operation of various vehicle systems. In some examples, the drive module controller can include a processor and a memory communicatively coupled to the processor. The memory may store one or more modules for performing various functionality of drive system 214. Additionally, drive system 214 also includes communication connections that enable each drive module to communicate with other local or remote computing devices.

[0048] 2, the vehicle computing device 204, the sensor system 206, the emitter 208, and the communication connection 210 are shown as being onboard the vehicle system 202. However, in some examples, the vehicle computing device 204, the sensor system 206, the emitter 208, and the communication connection 210 may be implemented external to the actual vehicle (i.e., not onboard the vehicle system 202).

[0049] As shown in FIG. 2, vehicle system 202 is configured to establish a communication link between vehicle system 202 and one or more other devices. For example, network 232 can be configured to enable data exchange between vehicle system 202, other devices coupled to the network, such as other computer systems, other vehicle systems 202 in a group of vehicles, and / or remote operation system 112. For example, network 232 can enable wireless communication between a number of vehicles and / or remote operation system 112. In various embodiments, network 232 can support communication via a general wireless data network, such as a Wi-Fi network. For example, network 232 can support communication via a telecommunications network, such as a cellular communication network, a satellite network, etc.

[0050] The remote operation system 112 can include a processor 234, a memory 236, and an input / output component 238. The remote operation system 112 is configured to receive information (e.g., data) from the vehicle and requests for assistance. The remote operation system 112 is configured to receive data and requests from a group of vehicles of the vehicle system 202. The group of vehicles can transmit data and / or requests for assistance simultaneously or substantially simultaneously. The data enables the remote operation system 112 to discover all the vehicles and their statuses. For example, upon receiving the data, the remote operation system 112 can understand where the vehicles are located, the current state of the vehicles, such as speed, direction of travel, position, whether a remote operator is connected to the vehicle, the health status of the modem, the mission type of the vehicle, etc. In some examples, by knowing the position of the vehicle, the remote operation system 112 can determine the geopreference of the vehicle or which geopreference the vehicle is within. The geopreference can be associated with a geographical area, such as a city, town, municipality, etc., a block, a postal code, etc. in some examples. When discussed herein, the geopreference can be used to match requests for assistance to the remote operator, given the proficiency or preference of the remote operator. In some examples, the data can be received at regular intervals, such as every 100 milliseconds, every second, etc.

[0051] The remote operation system 112 can also store the preferences 240 of the remote operator 108 within the remote operator profile 242. In some examples, the preferences 240 can be added, deleted, modified, or otherwise provided by the remote operator 108. For example, the remote operator 108 can provide indications such as the type of vehicle for which the remote operator desires to provide assistance, the geopence (e.g., geographical area) within which the remote operator 108 desires to provide assistance, any associated specialization or training (such as expertise in HVAC systems or other sub-components, training in interacting with passengers, etc.), the type of mission that the remote operator 108 prefers to provide assistance for, the type of assistance that the remote operator 108 desires to provide, and so on. The remote operator 108 can also provide other preferences.

[0052] In addition, the remote operator profile 242 can store the status 244 of the remote operator. The status 244 can, in some examples, indicate the availability of the remote operator 108, such as whether the remote operator 108 is on duty, off duty, on break, in training mode, already providing assistance, etc. The status 244 of the remote operator 108 can be continuously updated to know the current state of the remote operator and whether the remote operator is able to provide assistance.

[0053] The remote operation system 112 includes a remote operation management component 246, and the remote operation management component 246 manages requests for assistance and selects a remote operator 108 to respond to those requests. For example, the remote operation management component 246 can receive requests for assistance via a queue interface 248. In some examples, the queue interface 248 stores or otherwise records requests received from the vehicle. Those requests can be sorted based on, for example, the time they are received, the level of importance in responding to those requests, and the like. The remote operation management component 246 communicates with one or more remote operators to transmit requests for processing. As part of communicating or sending requests to the remote operator 108, the remote operation management component 246 can utilize preferences 240 and / or status 244 stored in relation to the remote operator profile 242, as well as data and / or details of the requests for assistance received from the vehicle. As described above, such a queue interface 248 can store a number of queues, and accordingly, individual ones of those queues are associated with one or more criteria (which can be a set or subset of criteria associated with the remote operator 108).

[0054] The processor 216 of the vehicle system 202 and the processor 234 of the remote operation system 112 may be any suitable processor capable of processing data and executing instructions to perform the operations described herein. By way of example and not limitation, the processors 216 and 234 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts it into other electronic data that may be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to execute the instructions encoded therein.

[0055] Memories 218 and 236 are examples of non-transitory computer-readable media. Memories 218 and 236 may store an operating system and one or more software applications, instructions, programs, and / or data for performing the methods and functions attributed to the various systems described herein. In various embodiments, memories may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which those shown in the accompanying figures are merely examples relevant to the discussion herein.

[0056] In various embodiments, the parameter values and other data shown herein can be included in one or more data stores, can be combined with other information not described, or can be separately partitioned into more, fewer, or different data structures. In some embodiments, the data store can be physically located in one memory or can be distributed among two or more memories.

[0057] Those skilled in the art will understand that architecture 200 is merely exemplary and is not intended to limit the scope of the present disclosure. Specifically, computing systems and devices can include any combination of hardware or software capable of performing the functions shown, including computers, network devices, Internet appliances, tablet computers, PDAs, wireless telephones, pagers, etc. Architecture 200 can also be connected to other devices not shown or can instead operate as a stand-alone system. Additionally, the functionality provided by the components shown can, in some embodiments, be combined in fewer components or distributed among additional components. Similarly, in some embodiments, some of the functionality of the components shown can be provided or additional functionality may be available.

[0058] While various items are shown as being stored in memory or storage while in use, those skilled in the art will understand that these items or portions thereof can be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments, some or all of the software components can execute in memory on another device and communicate with the illustrated architecture 200. Some or all of the system components or data structures can also be stored (e.g., as instructions or structured data) on a non-transitory, computer-accessible medium or on a portable article that will be read by an appropriate drive, examples of which are described above. In some embodiments, instructions stored on a computer-accessible medium separate from architecture 200 can be transmitted to architecture 200 via a signal such as an electrical signal, an electromagnetic signal, or a digital signal transmitted via a transmission medium or a communication medium such as a wireless link. Various embodiments can further include receiving, transmitting, or storing instructions and / or data according to the foregoing description on a computer-accessible medium. Accordingly, the techniques described herein can be implemented with other control system configurations. Further information regarding the operation of the modules of vehicle system 202 is discussed hereinafter.

[0059] Figure 3 shows an exemplary architecture 300 that includes a group of vehicles 302 including vehicles 302(a), 302(b), …, 302(n) (where n is any integer greater than or equal to 2) and a remote operation system 112. The group of vehicles 302 can include one or more vehicles 302(a), 302(b), …, 302(n), and at least some of those vehicles are communicatively coupled to the remote operation system 112 via the network interface of each vehicle. However, although a group of vehicles 302 is described, it should be understood that only a single vehicle can be included.

[0060] The network proxy 304 of the remote operation system 112 can be communicatively coupled to the network interface of the vehicle. For example, vehicle 302(a) can transmit communication signals via the network interface, and those communication signals are received by the network proxy 304. In some examples, the communication signals can include, for example, sensor data, operating state data, and / or a request for assistance from one or more sensors associated with the vehicle, but any data and / or output from one or more modules of the vehicle system 202 is contemplated.

[0061] As shown in FIG. 3, the queue interface 248 can receive requests from the network proxy 304 and associate those requests with the queue. The queue interface 248 can, for example, remove requests from the queue such as the highest priority, the oldest, the most important, etc., and identify the closest matching criteria and acceptable associated status, such as the first remote operator 108(a) and the second remote operator 108(b) for responding to the requests, to process the requests to select one or more remote operators. In some examples, the queue interface 248 can communicate directly with the vehicle group 302. In this example, only two remote operators 108 are shown, but in reality, any number of remote operators 108 can be associated with the remote operation system 112 to respond to the requests. Additionally, in some examples, the remote operators 108 associated with the remote operation system 112 can be co-located at the same remote operation center, can be located at a number of separately located remote operation centers, and / or can be individually located (e.g., working from home).

[0062] In some examples, the queue interface 248 can be implemented on a device separate from the device that includes the remote operator interface 306. For example, the queue interface 248 can include a gateway device and an application programming interface (“API”) or similar interface. In some examples, the queue interface 248 is configured to receive requests and generate a queue for the requests for processing by the remote operator 108. The queue interface 248 can match requests with the corresponding remote operator 108 based on, for example, sensor data generated by the vehicle, the preferences of the remote operator 108, the status of the remote operator 108, and the like. In such examples, the queue interface 248 can filter available remote operators 108 for assigning requests based on the availability of the remote operator 108. In an example, the queue interface 248 can prioritize requests based on a first-come, first-served basis, and accordingly, requests with earlier timestamps are prioritized higher than more recent requests. The queue interface 248 can also apply filters for adjusting the priority order in which requests are assigned based on safety decisions (e.g., vehicle speed or environmental conditions), the status of the occupants (e.g., occupied or unoccupied), the proximity of the remote operator to the vehicle, and other such filters.

[0063] The queue interface 248 can be configured to load balance requests among the remote operators 108. For example, if multiple remote operators 108 can respond to requests (e.g., based on preferences 240, status 244, etc.), the queue interface 248 can send requests to the remote operators who have responded to fewer requests. In some examples, this can occur over a predetermined amount of time, such as a work shift, day, month, etc. Additionally, as part of sending a request to a remote operator 108, the remote operator 108 can have a predetermined amount of time to respond to or accept the request. If a response is received within the predetermined amount of time, the queue interface 248 can assign the request to the remote operator 108. If no request is received within the predetermined amount of time, the queue interface 248 can determine another remote operator 108 to respond to the request.

[0064] In some examples, the queue interface 248 can hold a number of different or separate queues that operate in parallel with each other. In such examples, the separate parallel queues can correspond to different geographical locations of the vehicle, different vehicle types, different request types (e.g., requests from passengers of the vehicle system as opposed to requests for navigation assistance), and other queues associated with different filters for those queues can also be applied. The number of parallel queues can each be accessible by a separate subset of remote operators. For example, accordingly, the queue for the first vehicle type can be accessible only to remote operators who are qualified to provide navigation or assistance to that class of vehicle. In some examples, those remote operators may be available to process requests from one or more of the parallel queues.

[0065] The queue interface 248 communicates with the remote operator interface 306 of the remote operation system 112. The queue interface 248 generates a queue of requests and communicates those requests to the remote operator 108 via the remote operator interface 306. Each of the remote operators 108 can include their respective remote operation devices (e.g., tablets, laptops, phones, etc.) for communicating with the remote operation system 112 to respond to requests. For example, the first remote operator 108(a) can use the remote operator device 308(a), while the second remote operator 108(b) can use the remote operator device 308(b). In some examples, the network proxy 304 can be communicatively coupled to the remote operator interface 306 via the queue interface 248, and in some examples, the remote operator 108 can access sensor data, operating state data, and / or other data in the communication signals received from the vehicles 302(a), 302(b), …, 302(n) via the remote operator interface 306.

[0066] The remote operator 108 has the ability to accept or reject requests for assistance. For example, when the queue interface 248 selects a remote operator to respond to a request, an indication to that effect can be sent to the remote operator device 308. In response, the remote operator 108 can select to either accept or reject the request. As part of this, the remote operator 108 can selectively access sensor data, operating state data, and / or other data via the remote operator device 308 and view the selected data via one or more of the displays (see FIG. 1). In some examples, the remote operator 108 can have a predetermined amount of time to accept the request, otherwise another remote operator 108 can be determined to respond to the request.

[0067] The remote operation system 112 may provide communication between two or more of the remote operator interfaces 306 and respective remote operators 108 (e.g., via remote operator devices 308) and / or with remote operation data 310. For example, the remote operation system 112 may include multiple remote operator interfaces 306 associated with respective remote operators 108, and the remote operators 108 may communicate with each other to facilitate and / or coordinate guidance provided to vehicles in the fleet of vehicles 302. In some examples, data associated with guidance provided by the remote operators 108 may be stored by the remote operation system 112, for example, in remote operation data 310. In some examples, the remote operation data 310 may be accessible by the remote operators 108, for example, via the remote operator interfaces 306, for use in providing guidance to the vehicles.

[0068] The remote operation data 310 may include sensor data and other operational data from the fleet of vehicles 302 and may be accessed by the remote operator 108 at the remote operator interface 306 without passing the remote operation data 310 through the queue interface 248. In this manner, the queue interface 248 may receive basic information related to the identity of the requesting vehicle and the assistance needed. Upon selecting the remote operator 108, the remote operator 108 establishes a direct communication channel with the vehicle, bypassing the queue interface 248. The bypass may allow for faster communication with reduced latency due to potentially large amounts of traffic (e.g., information) on the queue interface 248.

[0069] In some examples, the remote operation data 310 can include global and / or local map data related to the road network of the vehicle's environment, events associated with road 4, and / or driving conditions associated with the road due to, for example, traffic volume, weather conditions, construction areas, and / or special events. In some examples, the remote operation data 310 can include data associated with another one of the vehicles in the vehicle group 302, such as maintenance and service information, and / or an operation history including, for example, an event history, a route history, a ride history, and other types of data associated with the vehicle.

[0070] Figures 4 and 5 illustrate various processes related to providing remote assistance to a vehicle. The processes described herein are shown as a collection of blocks in a logical flow diagram, which represent a series of operations, some or all of which may be implemented in hardware, software, or a combination thereof. In a software context, the blocks may correspond to computer-executable instructions stored on one or more computer-readable media, which program a processor to perform the recited operations when executed by one or more processors. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the blocks are described should not be construed as limiting, unless otherwise specifically noted. Any number of the described blocks may be combined in any order and / or in parallel to implement the process, or an alternative process, and not all of the blocks need to be executed. For purposes of discussion, the processes are described in the context of the environment, architecture, and system described in the examples herein, e.g., those described with respect to FIGS. 1-3, but the processes may be implemented in a wide variety of other environments, architectures, and systems.

[0071] Figure 4 illustrates an exemplary process 400 for determining a remote operator to provide assistance to a vehicle.

[0072] At 402, process 400 can include receiving data associated with an autonomous vehicle. For example, remote operation system 112 can receive data from vehicle 102 indicating the operating state of the vehicle. In some examples, the data can include or represent the state of the vehicle, such as speed, direction of travel, position, whether a remote operator is connected to or communicating with the vehicle, the mission type of the vehicle (e.g., training mission, recharging, transporting passengers, etc.). Such data can be received on a continuous and / or periodic basis according to a predetermined schedule. In some examples, knowing the position of the vehicle is used to determine the geopence of the vehicle, such as the local area of the city, postal code, area, region, etc. Such data can additionally or alternatively include voice and / or image data from one or more passengers, raw sensor data from one or more sensors, derived data from one or more sensors, log data from one or more messages between various hardware and / or software components, requests from passengers input via an interface mounted on the vehicle and / or from a passenger's personal device, etc.

[0073] At 404, process 400 can include receiving a request associated with an autonomous vehicle that needs assistance. For example, in response to a particular event or the autonomous vehicle facing trouble, the autonomous vehicle can request assistance. In some examples, the request can specify the type of assistance needed, the event experienced by the vehicle, the vehicle type, the status of the passengers, and / or other such information from the vehicle. As part of receiving the request and other requests from vehicles, remote operation system 112 can generate a queue. For example, the queue can be generated by queue interface 248 as described herein by prioritizing requests as received from vehicles.

[0074] In 406, process 400 can include determining, among a plurality of remote operators, the remote operators that are available for providing remote assistance. For example, among the plurality of remote operators providing remote assistance, only a subset of these remote operators may be available. For example, a remote operator may be offline or may not be logged in to the remote operation system 112 to provide assistance. In such cases, these remote operators may not be available for providing assistance. In contrast, remote operators who are online and logged in to the remote system 112 may be available for providing assistance. In some examples, the remote operation system 112 can receive an indication of the availability of the remote operator 108 indicating whether the remote operator 108 is online. Such an indication can be received when the remote operator 108 logs in to the remote operation system 112, sets the availability, etc.

[0075] In 408, process 400 can include determining, among the available remote operators, the first status of a first remote operator. For example, the remote operation system 112 can access the remote operator profile of the first remote operator to determine the status of the first remote operator. Such a status can indicate whether the first remote operator is in training, providing assistance to another vehicle, on break, unavailable for receiving requests, etc. That is, even if the first remote operator is available (e.g., online), the first remote operator may be busy, such as providing assistance to another vehicle.

[0076] At 410, process 400 can include determining whether a first remote operator is available. Whether the first remote operator is available is determined from a first status. For example, if the first remote operator is in the middle of being trained, the first remote operator can be determined to not be available. If the first remote operator is already assisting another vehicle, the first remote operator can be determined to not be available. If the first remote operator is on break, the first remote operator can be determined to not be available. The remote operation system 112 can continuously receive an indication of the first status of the first remote operator to know whether the first remote operator is available (or not). Still, the first remote operator can have the ability to set their status, for example, indicate that they are on break. In contrast, if the first remote operator is not assisting other vehicles, the first remote operator can be determined to be available.

[0077] At 410, if process 400 determines that the first remote operator is available, process 400 can follow the "yes" route and proceed to 412. At 412, process 400 can include determining a first preference of the first remote operator who will assist the autonomous vehicle. For example, the preferences stored in relation to the first remote operator can be accessed. This can indicate, for example, the geographical area where the first remote operator provides assistance, the type of assistance provided by the first remote operator, the type of vehicle for which the first remote operator provides assistance, and so on. The first remote operator has the ability to set or determine the preferences used when allocating requests.

[0078] At 414, process 400 can include determining whether the first preference matches the request and / or the extent to which the preference matches. For example, process 400 can compare the preferences of the first remote operator with the data received from the vehicle and / or the request. For example, knowing the position of the vehicle, process 400 can determine whether the vehicle is within the geographical area served by the first remote operator. Additionally, or alternatively, the type of assistance requested by the vehicle can be compared to the type of assistance provided by the first remote operator. In some examples, any number of preferences can be compared to determine whether the preference matches the request (e.g., experience, location of the remote operator, etc.). In some examples, in order to determine whether the preference matches the request, the preference may have to match exactly the request or exceed a certain threshold. In some examples, the selection can be based on the first remote operator matching more preferences than any other currently available remote operator.

[0079] Generally, preferences can be associated with filters used to filter remote operators for the purpose of selecting a remote operator to provide assistance to the vehicle. Such filters can include the status of the first remote operator and / or the geographical location of the first remote operator, as well as the number of requests processed, the experience associated with providing assistance, the physical proximity to the vehicle that can be selected to reduce latency, the mission type, and other preferences defined by the remote operator, as described above.

[0080] In some examples, the remote operation system 112 may include multiple remote operation centers, whereby the remote operation centers are distributed around the geographic region in which the fleet of vehicles operates. In such examples, a remote operator may be selected based on proximity to the vehicle requesting assistance. In some examples, the remote operation centers may have different bandwidth and / or availability to process the request. To reduce latency, in some examples, the closest remote operation center and / or remote operator may be selected to process the request.

[0081] If the process 400 determines at 414 that the preferences match the request, the process 400 may proceed along the "Yes" route to 416.

[0082] At 416, the process 400 may include sending a first indication of the request to a first device of the first remote operator. For example, the remote operation system 112 may send an indication of the request to the first device of the first remote operator. In some examples, the indication may display information associated with the request and the vehicle, such as the location, the type of assistance needed, etc. The first indication serves as a notification to the first remote operator regarding whether to accept or reject the request to have the vehicle provide assistance.

[0083] At 418, process 400 can include determining whether an acceptance of a request has occurred. For example, the remote operation system 112 can determine whether a first remote operator has accepted a request to provide assistance to a vehicle. In some examples, the first remote operator can have a predetermined amount of time (e.g., 5 seconds) to accept the request in response to a first indication. If acceptance is received before the predetermined amount of time, the request can be assigned to the first remote operator. If acceptance is received after the predetermined amount of time, or if acceptance is not received before the predetermined amount of time has elapsed, the request can be determined to not have been accepted. If acceptance is received at 418, process 400 can follow the "yes" route and proceed to 420.

[0084] At 420, process 400 can include assigning a request to a first remote operator. As part of assigning a request to a remote operator, the first remote operator can be connected to the vehicle to provide assistance and / or control the vehicle. Here, the status of the first remote operator can be updated to indicate that the first remote operator is providing assistance and is therefore not available. This can be used, for example, when assigning additional requests knowing that the first remote operator is busy and unable to accept additional requests. Additionally, if the first remote operator accepts a request, the first remote operator can receive sensor data and look at the events or situations faced by the vehicle and provide an input accordingly. As part of controlling the vehicle, remote operation system 112 can transmit a request to have the vehicle switched manually. In some examples, the switch can be rejected, for example, in cases where a remote operator has been identified and accepted a request, but the vehicle has already resolved the situation and no longer requires assistance. In some examples, after a request is assigned, the request can be removed from the queue interface to avoid the request remaining on the queue interface.

[0085] In some examples, as part of assigning a request to a first remote operator, an indication can be output within the vehicle. For example, the indication can indicate that a remote operator is connecting to the vehicle to provide assistance. The indication can be output to a vehicle speaker, display, etc. In some examples, the first remote operator is enabled to communicate with passengers in the vehicle.

[0086] Following the "no" route from 410, 414, or 418, process 400 indicates that process 400 can proceed to 422. For example, if the first operator is unavailable, if the first preference does not match the request, and / or if the request is not accepted by the first remote operator, process 400 can proceed to 422.

[0087] At 422, process 400 can include determining a second status of a second remote operator. For example, the remote operation system 112 can access a remote operator profile of the second remote operator to determine the status of the second remote operator. Such a status can indicate whether the second remote operator is in training, providing assistance to another vehicle, on break, unavailable to receive requests, etc. That is, even if the second remote operator is available (e.g., online), the first remote operator may be busy, such as providing assistance to another vehicle.

[0088] At 424, process 400 can include determining whether a second remote operator is available. Whether the second remote operator is available is determined from the second status. For example, if the second remote operator is in the middle of being trained, it is possible that the second remote operator is determined to be unavailable. If the second remote operator is already assisting another vehicle, it is possible that the second remote operator is determined to be unavailable. If the second remote operator is on break, it is possible that the second remote operator is determined to be unavailable. The remote operation system 112 can continuously receive an indication of the second status of the second remote operator to know whether the second remote operator is available (or unavailable). Nevertheless, the second remote operator can have the ability to set their status, for example, to indicate that they are on break. In contrast, if the second remote operator is on duty and not assisting other vehicles, it is possible that the second remote operator is determined to be available.

[0089] At 424, if process 400 determines that the second remote operator is available, process 400 can follow the "yes" route and proceed to 426. At 426, process 400 can include determining the second preferences of the second remote operator who will assist the autonomous vehicle. For example, it is possible to access the preferences 240 stored in relation to the second remote operator. This can indicate, for example, the geographical area where the second remote operator provides assistance, the type of assistance provided by the second remote operator, the type of vehicle that the second remote operator provides assistance to, and so on. The second remote operator has the ability to set or determine the preferences used when allocating requests.

[0090] At 428, process 400 can include determining whether the second preference matches the request. For example, process 400 can compare the second remote operator's preference with the data received from the vehicle and / or the request. For example, knowing the vehicle's position, process 400 can determine whether the vehicle is within the geographical area served by the second remote operator. Additionally, or alternatively, the type of assistance requested by the vehicle can be compared to the type of assistance provided by the second remote operator. In some examples, any number of preferences (e.g., experience, remote operator's position, etc.) can be compared to determine whether the preference matches the request.

[0091] At 428, if process 400 determines that the preference matches the request, process 400 can follow the "yes" route and proceed to 430.

[0092] At 430, process 400 can include sending a second indication of the request to the second device of the second remote operator. For example, the remote operation system 112 can send an indication of the request to the second device of the second remote operator. In some examples, the second indication can display information associated with the request and the vehicle, such as the position, the type of assistance needed, etc. The second indication serves as a notification to the second remote operator regarding whether to accept or reject the request to provide assistance to the vehicle.

[0093] At 432, process 400 can include determining whether acceptance of the request has occurred. For example, the remote operation system 112 can determine whether a second remote operator has accepted a request to provide assistance to the vehicle. In some examples, the second remote operator can have a predetermined amount of time (e.g., 5 seconds) to accept the request in response to a second indication. If acceptance is received at 432, process 400 can follow the "yes" route and proceed to 434.

[0094] At 434, process 400 can include assigning the request to a second remote operator. As part of assigning the request to the second remote operator, the second remote operator can be connected to and provide assistance to vehicle 102 and / or control the vehicle. Additionally, if the second remote operator accepts the request, the second remote operator can receive sensor data and look at the events or situations the vehicle is facing and provide an input accordingly.

[0095] Process 400 indicates that when following the "no" route from 424, 428, or 432, process 400 can proceed to 436. For example, if the second operator is not available, if the second preference does not match the request, and / or if the request is not accepted by the second operator, process 400 can proceed to 436.

[0096] In 436, process 400 can include determining to escalate a request. For example, if the request does not match any preference of the remote operator 108, the remote operation system 112 can determine to escalate the request to avoid having the request secure the queue interface 248 as an unprocessed item. Escalating the request can involve transferring the request to a remote operator or other observer, who can analyze the request and determine why no remote operator matched the request. In such an example, the request can be removed (or purged) from the queue interface 248, or the remote operator can process the request manually (e.g., filter it). In other examples, the request can be updated or otherwise modified such that the request matches one or more remote operators. In some examples, requests that do not match a remote operator are recorded to determine that there was no remote operator to satisfy the request.

[0097] Process 400 describes determining whether a request matches the preferences of two remote operators, but the preferences of any number of remote operators can be compared to the request. For example, process 400 can be repeated to see if a third preference of a third remote operator matches the request and if the third remote operator is available. In such cases, the request can be broadcast to any number of remote operators and the first operator to respond can have the request assigned to them. For example, if for any reason, such as the first remote operator being busy or the preferences not matching, the request is not assigned to the first remote operator, the remote operation system 112 can determine other remote operators available to respond to the request. In some examples, assuming other remote operators are available and their preferences match, the request can be sent in parallel to other remote operators and the first remote operator to respond can have the request assigned to them. In this manner, operations 422 - 432 can be performed in parallel for any number of remote operators. For example, the remote operation system 112 can determine a third remote operator among the available remote operations and send the request to the third remote operator in parallel with the second remote operator. Here, it is possible that both remote operators match the request and are suitable to provide assistance.

[0098] In some examples, process 400 can include receiving criteria associated with matching a remote operator to a request for assistance, additionally or alternatively. For example, the criteria can be used to filter available remote operators and can be selected to respond to a request for assistance. In some examples, the criteria can be a configurable (e.g., customizable) strategy for selecting available remote operators that match a filter for routing a request to a remote operator. As an example, a first filter can include filtering remote operators based on the type of vehicle, and a second filter can include filtering remote operators based on the geographic area served by the remote operator. Such filters can then include determining which remote operators are available. However, any number of filters can be used to determine available remote operators, the filters can be applied in any order, and the filters can include filters different from those described herein. In an example, the filter and / or the number of filters can be selected based on the type of assistance requested.

[0099] FIG. 5 shows an exemplary process 500 for determining a remote operator to provide assistance to a vehicle.

[0100] At 502, process 500 can include receiving first data associated with the autonomous vehicle. For example, the remote operation system 112 can receive data from the vehicle indicating the operating state of the vehicle 102. Such data can be received on a continuous and / or periodic basis according to a predetermined schedule.

[0101] At 504, process 500 can include determining one or more characteristics of a vehicle based at least in part on first data. For example, the one or more characteristics can include speed, direction, position, whether a remote operator is connected to / communicating with the vehicle, the vehicle's mission type (e.g., training mission, recharging, transporting passengers, etc.). In some examples, the one or more characteristics can include the geopence where the vehicle is located, such as a local area of a city, a postal code, an area, a region, etc.

[0102] At 506, process 500 can include receiving second data associated with a remote operator. In some examples, the second data can indicate the availability of the remote operator, the status of the remote operator, and / or one or more preferences of the remote operator. For example, the remote operator can provide preferences when providing assistance to the vehicle. In some examples, such preferences can indicate vehicle type, the position of the vehicle (e.g., within a specific geopence), etc. In some examples, the status of the remote operator can be automatically received from the remote operator (or the remote operator's device) according to a predetermined schedule, thereby keeping the status up to date.

[0103] At 508, process 500 can include determining the status of the remote operator based at least in part on the second data. For example, the remote operation system 112 can determine whether the remote operator is available or unavailable for receiving and / or providing assistance to the vehicle, and whether the remote operator is busy or available for providing assistance.

[0104] At 510, process 500 can include determining one or more preferences of a remote operator. In some examples, the one or more preferences can be determined at least in part based on second data and / or the remote operation system 112 can access preferences stored in relation to a remote operator profile of the remote operator. As discussed herein, the preferences can be used to filter requests for the remote vehicle sent to the remote operator for assistance.

[0105] At 512, process 500 can include receiving requests associated with providing assistance to an autonomous vehicle. For example, the remote operation system 112 can receive requests that the vehicle needs assistance, such as navigating through the surroundings of the environment, passing through a construction area, and so on.

[0106] At 514, process 500 can include determining whether the mission type of the vehicle is acceptable. For example, the remote operation system 112 can determine the mission type of the vehicle, and if the mission type is not acceptable or is a mission for which the remote operation system 112 does not provide assistance, the request can be ignored. The mission type can, in some examples, be determined via the first data and / or request. Mission types that may be unacceptable can be associated with a training mission or other test type of mission where the vehicle is in the midst of being trained or is otherwise being tested. In this manner, the remote operation system 112 can understand the state of all vehicles, but only a subset of these vehicles can be provided with remote assistance. Thus, vehicles that are in operation, transporting passengers, and responding to passenger requests (e.g., ridesharing) can be considered appropriate missions for which the remote operation system 112 provides assistance. At 514, if process 500 determines that the mission type is not acceptable, process 500 can follow the "no" route and proceed to 516.

[0107] In 516, process 500 can escalate a request. For example, if the mission type is not an appropriate type of mission for which remote operation system 112 provides assistance, remote operation system 112 can decide to escalate the request for further reconsideration. This can include having a remote operator or other observer analyze the request to determine why no remote operator matched the request. In some instances, from there, the request can be ignored or disregarded to avoid having the request reserve the queue interface as unprocessed. In such instances, the request can be deleted (or removed) from the queue interface. In other instances, the request can be updated or otherwise modified such that the request matches one or more remote operators. Alternatively, if the mission type is acceptable, process 500 can proceed to 518 by following the "yes" route.

[0108] At 518, process 500 may include determining whether the status and / or preferences match the request. For example, to provide assistance, the remote operator may need to be available (e.g., at work, not currently providing assistance to another vehicle, etc.). In some examples, the remote operator's preferences are compared to the data and / or request to determine whether the preferences match the request. For example, knowing the vehicle's location, process 500 may determine whether the vehicle is within a geographic area serviced by the remote operator. Additionally or alternatively, the type of assistance the vehicle is requesting may be compared against the type of assistance the first remote operator provides. In some examples, any number of preferences may be compared (e.g., experience, remote operator location, etc.) to determine whether the preferences match the request. In some examples, the preferences may have to match the request exactly or above a certain threshold to determine whether the preferences match the request.

[0109] If the status and / or preferences match the request at 518, process 500 may proceed along a "yes" route to 520. At 520, process 500 may include sending an indication to the remote operator's device associated with providing assistance to the autonomous vehicle. For example, the remote operation system 112 may send a request to the remote operator asking them to accept the request to provide assistance to the autonomous vehicle. In some examples, the remote operator's device may display an indication of the request and / or present information associated with the vehicle.

[0110] If, at 516, process 500 determines that the status and / or preferences do not match the requirements, then process 500 can follow the "no" route and proceed to 522. At 522, process 500 can include determining one or more additional remote operators to provide assistance to the autonomous vehicle. In some examples, these remote operators can be determined based on the status and / or preferences of the remote operators. In some examples, if multiple remote operators match the requirements, the remote operator who has provided the least amount of assistance can be assigned to the requirement (e.g., for load balancing). In other examples, the first remote operator to respond to the requirement can be assigned the requirement.

[0111] Process 500 has been described as receiving a single request from a vehicle in need of assistance, but process 500 can be configured to process any number of requests in parallel and assign requests to respective operators. As an example, a first request may be received from a first vehicle in a group of autonomous vehicles, and a second request may be received from a second vehicle in that group.

[0112] Exemplary clauses In the following paragraphs, various examples will be described. Any of the examples in this section can be used with any of the other examples in this section, and / or any of the other examples or embodiments described herein.

[0113] A remote operation system for a group of autonomous vehicles, comprising at least one processor and at least one non-transitory memory storing processor-executable instructions, the processor-executable instructions being, when executed by the at least one processor, to receive a request for remote operator assistance from an autonomous vehicle, the request including information indicating one or more of an event type, a mission type associated with the autonomous vehicle, sensor data associated with the autonomous vehicle, the position of the autonomous vehicle, the direction of travel of the autonomous vehicle, or the speed of the autonomous vehicle, and to associate the request with a queue of remote operator requests based at least in part on the information, to determine the availability of remote operators among a plurality of remote operators, to determine the status of a remote operator, to determine criteria associated with a remote operator, to determine to send the request to a remote operator based at least in part on the information, availability, status, and criteria, and to send the request to the device of the remote operator to provide a display of the request to the remote operator, configuring the remote operation system to perform.

[0114] B. The remote operator is the first remote operator, the availability is the first availability, the status is the first status, and the remote operation system further comprises determining the second availability of a second remote operator among a plurality of remote operators, determining the second status of the second remote operator, and determining that the second remote operator is unavailable for providing remote operator assistance based at least in part on the second status, and sending the request to the first remote operator is further based at least in part on the unavailability of the second remote operator, the remote operation system according to paragraph A.

[0115] C. The remote operation system is further configured to receive, from the device, a response associated with receiving the request and to determine that the response has been received within a threshold amount of time, and determining to send the request to the remote operator is at least partially based on determining that the response has been received within a threshold amount of time, the remote operation system according to any one of paragraphs A - B.

[0116] D. The status includes one or more of an indication that the remote operator is in training, an indication that the remote operator is on break, or an indication that the remote operator is providing remote operator assistance to another vehicle, and the criteria includes one or more of the geographical area served by the remote operator, the type of mission the remote operator is capable of serving, or the type of vehicle the remote operator is capable of serving, the remote operation system according to any one of paragraphs A - C.

[0117] E. A method comprising receiving, from a vehicle, a request for remote operator assistance, associating the request with a queue, determining, from a set of available remote operators, the status of a remote operator to resolve the request, determining criteria associated with the remote operator, determining, at least partially based on the request, status, and criteria, to receive remote operator assistance from the remote operator, and sending information associated with the request to the remote operator.

[0118] F. The request includes information indicating at least one of the mission type of the vehicle, the speed of the vehicle, the location of the vehicle, or the direction of travel of the vehicle, the method according to paragraph E.

[0119] G. The mission type indicates at least one of whether the vehicle is transporting passengers, whether the vehicle is traveling to pick up additional passengers, whether the vehicle is charging, or whether the vehicle is performing training, and the status indicates whether the remote operator is not available to receive a request for providing remote operator assistance, in accordance with any one of paragraphs E - F.

[0120] H. Further comprising the step of transmitting remote operator assistance to the vehicle, and the vehicle is configured to be controlled at least partially based on the remote operator assistance, in accordance with any one of paragraphs E - G.

[0121] I. The criteria indicate at least one of a geographic area with which the remote operator is familiar in providing a response to the request, a vehicle type with which the remote operator is familiar in providing a response, the type of remote operator assistance provided by the remote operator, or the mission type associated with the autonomous vehicle, in accordance with any one of paragraphs E - H.

[0122] J. Further comprising, for a plurality of remote operators, the step of determining the availability of each individual remote operator and the step of determining a set of available remote operators at least partially based on the availability of each individual remote operator, in accordance with any one of paragraphs E - I.

[0123] K. The status includes one or more of an indication of whether the remote operator is in training, an indication of whether the remote operator is on break, or an indication of whether the remote operator is providing remote operator assistance to another vehicle, in accordance with any one of paragraphs E - J.

[0124] A step of determining that the remote operator has accepted the request, and a step of updating the status of the remote operator to indicate that the remote operator is unavailable to receive additional requests while providing remote operator assistance to the autonomous vehicle. The method according to any one of paragraphs E to K further includes these steps.

[0125] M. The remote operator is the first remote operator and the status is the first status. The method further includes a step of determining the second status of the second remote operator for resolving the request among the set of available remote operators, and a step of determining that the second remote operator is not available based at least in part on the second status. The step of sending the request to the first remote operator is further based at least in part on the fact that the second remote operator is not available. The method according to any one of paragraphs E to L includes these steps.

[0126] N. The step of determining the remote operator includes a step of filtering the set of available remote operators using a first filter associated with the mission type to determine a first portion, and a step of filtering the first portion of the available remote operators using a second filter associated with the geographical area to determine a second portion of the remote operators. The remote operator is associated with the second portion. The method according to any one of paragraphs E to M includes these steps.

[0127] One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform actions including receiving, from a vehicle, a request for remote operator assistance; associating the request with a queue; determining, among a set of available remote operators, a status of a remote operator for resolving the request; determining criteria associated with the remote operator; determining to request remote operator assistance from the remote operator based at least in part on the request, the status, and the criteria; and transmitting information associated with the request to the remote operator.

[0128] The one or more non-transitory computer-readable media of paragraph O, wherein the actions further include determining that the remote operator has accepted the request and updating the status of the remote operator to indicate that the remote operator is unavailable to receive additional requests while providing remote operator assistance to the autonomous vehicle.

[0129] The one or more non-transitory computer-readable media of any one of paragraphs O - P, wherein the operation further includes receiving remote operator assistance from a remote operator and transmitting the remote operator assistance to the vehicle, and the vehicle is configured to be controlled at least in part based on the remote operator assistance.

[0130] The one or more non-transitory computer-readable media of any one of paragraphs O - Q, wherein the status includes one or more of an indication of whether the remote operator is in training, an indication of whether the remote operator is on break, or an indication of whether the remote operator is providing remote operator assistance to another vehicle.

[0131] S. The one or more non-transitory computer-readable media of any one of paragraphs O-R, wherein the remote operator is a first remote operator, the status is a first status, and the action further includes determining a second status of a second remote operator among the set of available remote operators for resolving the request and determining that the second remote operator is unavailable based at least in part on the second status, and wherein sending the request to the first remote operator is further based at least in part on the second remote operator being unavailable.

[0132] T. The one or more non-transitory computer-readable media of any one of paragraphs O-S, wherein determining the remote operators includes filtering the set of available remote operators using a first filter associated with a mission type to determine a first portion, and filtering the first portion of the available remote operators using a second filter associated with a geographic area to determine a second portion of the remote operators, wherein the remote operators are associated with the second portion.

[0133] Although the example provisions set forth above are described with respect to one particular implementation, it should be understood in the context of this document that the content of those example provisions may also be implemented via a method, device, system, computer-readable medium, and / or other implementation. Additionally, any of Examples A-T may be implemented alone or in combination with any other one or more of Examples A-T.

[0134] Conclusion Although one or more examples of the technology described herein have been described, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0135] In the description of the examples, reference is made to the accompanying drawings that form a part of this specification, and those drawings illustrate specific examples of the claimed subject matter as examples. It should be understood that other examples can be used and that changes or modifications such as structural changes can be made. Such examples, changes, or modifications are not necessarily a departure from the scope of the intended claimed subject matter. The steps in this specification may be presented in a particular order, but in some cases, that ordering can be changed so that specific inputs are provided at different times or in a different order without changing the functionality of the described system and method. The disclosed procedures can also be executed in a different order. Additionally, the various calculations in this specification need not be executed in the disclosed order, and other examples using alternative orderings of those calculations can be readily implemented. Those calculations can be broken down into sub-calculations that, in addition to being reordered, also result in the same outcome.

Claims

1. Receiving a request from a vehicle to provide remote operator assistance; Associating the request with a queue; Determining the status of a remote operator to resolve the request among a set of available remote operators; Determining criteria associated with the remote operator; Determining to receive remote operator assistance from the remote operator based at least in part on the request, the status, and the criteria; Transmitting information associated with the request to the remote operator and including the method.

2. The request is the mission type of the vehicle, the speed of the vehicle, the location of the vehicle, or the direction of travel of the vehicle including information indicating at least one of: The method according to claim 1.

3. The mission type is whether the vehicle is transporting passengers, whether the vehicle is traveling to pick up additional passengers, whether the vehicle is charging, or whether the vehicle is performing training indicating at least one of: The status indicates whether the remote operator is not available to receive a request to provide the remote operator assistance, The method according to claim 1 or 2.

4. Further including transmitting the remote operator assistance to the vehicle, The vehicle is configured to be controlled based at least in part on the remote operator assistance, The method according to any one of claims 1, 2, or 3.

5. The criteria are a geographical area with which the remote operator is familiar to provide a response to the request, a vehicle type with which the remote operator is familiar to provide the response, the type of remote operator assistance provided by the remote operator, or a mission type associated with an autonomous vehicle indicating at least one of: The method according to any one of claims 1, 2, 3, or 4.

6. Determining the availability of individual remote operators for a plurality of remote operators; Determining the set of available remote operators based at least in part on the availability of the individual remote operators and further including: The method according to any one of claims 1, 2, 3, 4, or 5.

7. The status is An indication of whether the remote operator is in training, an indication of whether the remote operator is on break, or an indication of whether the remote operator is providing remote operator assistance to another vehicle including one or more of: The method according to any one of claims 1, 2, 3, 4, 5, or 6.

8. Determining that the remote operator has accepted the request, and updating the status of the remote operator to indicate that it is not possible to receive additional requests while the remote operator is providing the remote operator assistance to the autonomous vehicle further including: The method according to any one of claims 1, 2, 3, 4, 5, 6, or 7.

9. The remote operator is a first remote operator, the status is a first status, and the method includes: determining a second status of a second remote operator for resolving the request among the set of available remote operators, and determining that the second remote operator is not available based at least in part on the second status further including: Sending the request to the first remote operator is further based at least in part on the second remote operator not being available. The method according to any one of claims 1, 2, 3, 4, 5, 6, 7, or 8.

10. Determining the remote operator includes: filtering the set of available remote operators using a first filter associated with the mission type to determine a first subset, and filtering the first subset of the available remote operators using a second filter associated with a geographic area to determine a second subset of the remote operators including: The remote operator is associated with the second subset. The method according to any one of claims 1, 2, 3, 4, 5, 6, 7, 8, or 9.

11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, receive a request from a vehicle to provide remote operator assistance, and associate the request with a queue Determining, among a set of available remote operators, a status of a remote operator to resolve the request; Determining criteria associated with the remote operator; Determining to request the remote operator assistance from the remote operator, based at least in part on the request, the status, and the criteria; Transmitting information associated with the request to the remote operator; One or more non-transitory computer-readable media causing the one or more processors to perform actions including the above. The actions include Determining that the remote operator has accepted the request; Updating the status of the remote operator to indicate that it is not possible to receive an additional request while the remote operator is providing the remote operator assistance to the autonomous vehicle; and further include The one or more non-transitory computer-readable media according to claim 11. The operations include Receiving the remote operator assistance from the remote operator; Transmitting the remote operator assistance to the vehicle; and further include The vehicle is configured to be controlled based at least in part on the remote operator assistance. The one or more non-transitory computer-readable media according to claim 11 or 12. The status includes An indication of whether the remote operator is in training; An indication of whether the remote operator is on break; or An indication of whether the remote operator is providing remote operator assistance to another vehicle. The status includes one or more of the above. The one or more non-transitory computer-readable media according to any one of claims 11, 12, or 13. The remote operator is a first remote operator, the status is a first status, and the actions include Determining a second status of a second remote operator to resolve the request, among the set of available remote operators; Determining that the second remote operator is not available, based at least in part on the second status; and further include Transmitting the request to the first remote operator is further based at least in part on the second remote operator not being available. ​ ​ ​ ​ One or more non-transitory computer-readable media according to any one of claims 11, 12, 13, or 14.

Images