Vehicle control and guidance

Abstract

The vehicle computing system may identify a scenario in the environment that violates an operational constraint. The vehicle computing system may request remote guidance from a guidance system of the service computing device. The vehicle computing system may receive input from the guidance system including one or more waypoints and / or associated headings for the vehicle to navigate through the scenario. The vehicle computing system may be configured to validate the input. The validation may include processing the input to determine whether the waypoint(s) and / or their associated heading(s) cause the vehicle to violate a safety protocol. Based on a determination that the input does not cause the vehicle to violate a safety protocol, the vehicle computing system may control the vehicle in accordance with the input, such as by causing the driveline to operate the vehicle to the respective waypoint at the associated heading.

Description

Technical Field

[0001] Cross - reference to related applications This international application is a continuation application of U.S. Patent Application No. 16 / 457,654, entitled "VEHICLE CONTROL AND GUIDANCE", filed on June 28, 2019, the entire contents of which are incorporated herein by reference, and claims the benefit thereof.

[0002] Vehicles operate in dynamic environments where situations often change. The changing situations are road closures due to construction and accidents, etc. An autonomous vehicle may be programmed to respond to the changing situations and maintain the vehicle within a defined operating protocol. The operating protocol may include defined driving rules, such as not operating the vehicle in a lane defined for oncoming traffic. However, detours around an accident or construction area may indicate a route outside the parameters of the operating protocol. If it is not permitted to operate outside the operating protocol boundary, the autonomous vehicle may be unable to proceed around an obstacle and thus may become stuck at a given location.

Summary of the Invention

[0003] A detailed description is set forth with reference to the accompanying drawings. In the drawings, the left - most digit(s) of the reference numeral identify the figure in which the reference numeral first appears. The use of the same reference numeral in different figures indicates the same or similar components or features.

Brief Description of the Drawings

[0004] [Figure 1] It is a display of an autonomous vehicle in an environment where a remote vehicle guidance system can be used to assist in guiding the vehicle through the environment. [Figure 2]This is a display of an exemplary service computing device having a graphical user interface that allows an operator to input waypoints and / or direction guidance for a vehicle operating in remote guidance mode. [Figure 3] This illustrates an exemplary process for providing step-by-step guidance to a vehicle. [Figure 4] This is an illustrative flowchart showing an operator workflow for providing remote guidance to a vehicle via a graphical user interface. [Figure 5] This is an illustrative flowchart showing vehicle actions in response to remote guidance input. [Figure 6] This illustrates an exemplary process for providing remote guidance to a vehicle computing device. [Figure 7] This illustrates an exemplary process for controlling a vehicle based on guidance input received from a service computing device. [Figure 8] This is a block diagram of an exemplary system for implementing the technology described herein. [Modes for carrying out the invention]

[0005] This disclosure relates to a technique for remotely providing stepwise guidance to a vehicle operating in an environment facing a scenario that is difficult to navigate (e.g., it is impossible for a planner to plan a route according to a set of driving policies). The vehicle may include an autonomous or semi-autonomous vehicle having a vehicle computing system configured to request guidance from a remote vehicle guidance system based on obstacles encountered in the operating environment. The remote vehicle guidance system may include a graphical user interface (GUI) through which an operator can input proposed waypoints and / or headings (e.g., vehicle yaw at corresponding waypoints) for the vehicle to navigate through a scenario (e.g., around one or more obstacles). The vehicle may receive the proposed waypoints and / or headings and check that navigating along the course defined by the proposed waypoints and / or headings satisfies the vehicle's safety protocol. Thus, the technique described herein improves the safety of a vehicle operating in an environment.

[0006] The vehicle computing system may receive sensor data from one or more sensors on the vehicle (e.g., a camera, motion detector, lidar, radar, time of flight, etc.). Based on the sensor data, the vehicle computing system may determine that the vehicle is approaching an obstacle on a path around which the vehicle may not be configured to determine a feasible course, which are referred to herein as remote guidance scenarios (e.g., road closures, construction zones within the road, or drivelines exceeding the vehicle line of sight (e.g., impaired sensor vision due to road elevation and / or bends within the road)). Remote guidance scenarios may include scenarios not programmed for the vehicle computing system to perform (e.g., unknown scenarios) and / or scenarios that appear to violate an operating protocol (e.g., a policy). An operating protocol may include one or more rules governing actions that the vehicle may or may not take, such as not crossing a double yellow line, not driving into an approaching driving lane, or not crossing the boundary of the drivable surface of the road. For example, a vehicle computing system may detect that the road the vehicle is traveling on is blocked, and that the road around the blockage is driving in a designated lane for traffic moving in the opposite direction, thus violating the operating protocol.

[0007] In some embodiments, in response to detecting a remote guidance scenario, the vehicle computing system may automatically connect to a service computing device configured with a GUI. In various embodiments, a vehicle operator and / or safety supervisor may detect a remote guidance scenario and connect to the service computing device. In some embodiments, the vehicle computing system may send a request for guidance to the service computing device. In various embodiments, before receiving a response (e.g., guidance input) from the service computing device, the vehicle computing system may determine a solution for navigating the remote guidance scenario. In such embodiments, the vehicle computing system may control the vehicle according to the solution. In some embodiments, the vehicle computing system may send a second message to the service computing device indicating that no further guidance input is needed (e.g., indicating that the vehicle is proceeding without remote guidance). In some embodiments, the vehicle computing system may continue to control the vehicle without sending a second message to the service computing device. In such embodiments, at least one of the vehicle computing system or the service computing device may determine that no guidance input is needed and disconnect from the other computing device.

[0008] In some embodiments, in response to receiving a request for guidance, the service computing device may determine that one or more conditions for providing guidance are met. These conditions may include connection thresholds (e.g., threshold latency, threshold bandwidth), vehicle speed being below a threshold speed (e.g., 15 miles per hour, 20 kilometers per hour), the number / type of operable threshold sensors (e.g., thresholds for four operable cameras, thresholds for all operable vehicle lidar sensors), the absence of vehicle health associated with faults (e.g., faults associated with inconsistent sensor readings, one or more non-operational sensors, faults associated with collision avoidance systems, planner systems, and / or other vehicle systems), and / or other conditions that may affect the effectiveness of the remote guidance system. In some embodiments, in response to determining that the conditions are not met, the service computing device may not connect to the vehicle computing system. In some embodiments, in response to determining that the conditions are not met, the service computing device may deny the operator access to the GUI and / or deny input to the GUI. In such embodiments, denying access can enhance the safety of vehicle operation, for example, by preventing remote guidance without sufficient latency from ensuring that the vehicle computing system can respond within a reasonable time frame to avoid unsafe consequences. In some embodiments, in response to a determination that a condition is met, the service computing device may launch a GUI.

[0009] In some embodiments, the GUI may include one or more windows representing sensor data from one or more sensors on the vehicle. In at least one embodiment, the GUI may include a window representing streaming images captured by a camera on the vehicle. The windows may represent data to assist the operator in determining the path the vehicle should take. For example, the GUI may include four windows representing a left-front view, a front view, a right-front view, and a rear view associated with the vehicle. The operator may evaluate each of the views to determine a safe path for the vehicle to transition through a remote guidance scenario.

[0010] In various embodiments, the GUI may include a top view of the vehicle and a description of a remote guidance scenario in the environment. In such embodiments, the top view may include a computer-generated view of the environment based on sensor data. In some embodiments, the top view may include the vehicle's position relative to one or more obstacles in the remote guidance scenario. For example, the top view may represent a vehicle approaching several orange cones that obstruct the vehicle's predetermined (e.g., planned) path. The top view may include the vehicle and the cones.

[0011] In various embodiments, the GUI may be configured to receive input from an operator via a top view. In such embodiments, the operator may be able to provide guidance to the vehicle to navigate through remote guidance scenarios, such as around an associated obstacle. The input may include one or more waypoints. A waypoint may include a location through which the vehicle travels. In various embodiments, each of the waypoints may include a stop point. In such embodiments, the vehicle may be configured to stop at each of the waypoints, such as when there is no reception of a subsequent waypoint or when there is no inspection of a subsequent waypoint. In at least some embodiments, if a subsequent waypoint is provided, the vehicle may continue planning and moving between waypoints without stopping.

[0012] In various embodiments, the input may include the vehicle's orientation (e.g., sway) at the relevant waypoint. In such embodiments, the operator may be able to orient the vehicle in a specific direction at the relevant waypoint. In some embodiments, the orientation may be determined based on the previous and / or subsequent waypoints. In such embodiments, the orientation at a particular waypoint may allow the vehicle to transition smoothly between waypoints on the guidance path. In various embodiments, the orientation associated with a waypoint may be based on a preferred orientation in which the vehicle should stop at the waypoint. For example, a preferred orientation associated with a vehicle at least partially located in a lane associated with traffic in the opposite direction may be parallel to the lane so as to minimize the number of vehicles located in the lane.

[0013] In some embodiments, the operator may cause a service computing device to transmit waypoints and / or headings to the vehicle computing system. In some embodiments, the service computing device may automatically transmit waypoints and / or associated headings in response to input from the operator. In such embodiments, the service computing device may automatically transmit waypoints and / or headings based on the determination that a waypoint has been confirmed at a particular location and / or based on the determination that a heading has been confirmed. In some embodiments, waypoints and / or associated headings may be transmitted in response to receiving a transmit command (e.g., selection of a transmit option) from the operator. In such embodiments, the operator may be able to transmit waypoints individually and / or in groups.

[0014] In some embodiments, a service computing device may be configured to validate waypoints and / or headings before transmitting guidance data to a vehicle computing system. In some embodiments, validation may include an initial check that the waypoints and / or headings pass a safety protocol associated with the vehicle. The safety protocol may include a set of predetermined conditions to ensure vehicle safety. The safety protocol may include one or more threshold distances for the vehicle to maintain distance from dynamic and / or static objects, maximum yaw rates, and other criteria associated with the safe navigation of the vehicle through a remote guidance scenario. In various embodiments, a vehicle computing system may receive waypoints and / or headings from a service computing device and validate the waypoints and / or headings. In such embodiments, the vehicle computing system may check that the waypoints and / or headings pass a safety protocol, which may be safely performed by the vehicle.

[0015] In some embodiments, the vehicle computing system may be configured to check that waypoints and / or headings are at least partially based on a remote guidance protocol. In such embodiments, the vehicle computing system may verify that waypoints and / or headings are at least partially based on a determination that they pass the remote guidance protocol. The remote guidance protocol may include one or more restrictions on the movement of the vehicle while operating in remote guidance mode. The remote guidance protocol may include, for example, course deviation (the distance from the course between two waypoints where the vehicle may deviate by half the width of the vehicle or the width of one vehicle, for example, to avoid an obstacle), responses to static object detection (for example, an attempt to avoid a static object to arrive at a follow-on waypoint), responses to dynamic object detection (for example, yielding to an agent that is expected to enter the course corresponding to a waypoint), following a dynamic object moving in the same direction as the vehicle (for example, within the course corresponding to a waypoint), yielding to the right at intersections as necessary (for example, all-direction stops, traffic lights, detours, etc.), or forward stop movements when no further guidance is received (for example, another waypoint, a command to resume navigation, etc.).

[0016] In addition, or instead, the vehicle computing system may verify the waypoints and / or orientations at least in part on kinematic verification (e.g., dynamic verification, kinematic / dynamic check). Kinematic verification may include determining that the vehicle has the ability to perform a transition from a first position (e.g., initial position, position associated with a waypoint) to a second position (e.g., first waypoint, subsequent waypoint). In some embodiments, the ability to perform a transition may include determining whether the vehicle computing system can generate one or more trajectories for the vehicle to perform to arrive at the waypoints within orientations based on physical constraints. Physical constraints may include maximum deflection rate, maximum transverse distance, maximum turning angle, and / or other constraints associated with vehicle operation.

[0017] In response to a determination that a waypoint and / or associated orientation violates safety protocols and / or remote guidance protocols, the vehicle computing system may reject the waypoint. In some embodiments, the vehicle computing system may not proceed to the rejected waypoint, for example, by not proceeding from a stop point (e.g., a stop line, stop sign, traffic light) or by stopping at a waypoint prior to the rejected waypoint. In some embodiments, the vehicle computing system may send a message to a service computing device indicating the rejection of the waypoint. In some embodiments, the message may include the reason for the rejection. For example, the vehicle computing system may determine that the waypoint is less than a threshold distance from a cement barrier. Based on this determination, the vehicle computing system may reject the waypoint and send a message to the service computing device indicating that the distance is less than a threshold distance.

[0018] In response to a determination that waypoints and / or associated orientations pass the safety protocol and / or remote guidance protocol, the vehicle computing system may accept waypoints and / or orientations. The vehicle computing system may process waypoints and / or orientations and generate one or more trajectories for the vehicle to transition between waypoints. For example, the vehicle computing system may receive a first waypoint and a first orientation associated with the first waypoint. The vehicle computing system may determine a first trajectory (or set of trajectories) for transitioning from an initial position to the first waypoint at the first orientation.

[0019] Based on the generated trajectory, the vehicle computing system may control the vehicle to the waypoint(s) in the relevant direction(s), such as via one or more driving systems. In various embodiments, the vehicle computing system may control the vehicle at a predetermined speed (e.g., 2 kilometers per hour, 15 miles per hour, 10 miles per hour, etc.). In some embodiments, the predetermined speed may include a maximum speed associated with remote guidance. In various embodiments, the vehicle computing system may control the vehicle at a speed associated with the environment and / or scenario. In at least one embodiment, the vehicle computing system may control the vehicle at a speed below the predetermined speed or the speed associated with the environment and / or scenario. For example, when operating in remote guidance mode, the vehicle computing system may operate at a predetermined maximum speed of 15 miles per hour. The vehicle computing system may detect a remote guidance scenario and request guidance input from a service computing device. The vehicle computing system and / or operator may determine, based on sensor data, that the associated remote guidance scenario has a speed limit of 5 miles per hour. Based on the fact that a speed limit of 5 miles per hour is below a predetermined maximum speed of 15 miles per hour, the vehicle computing system may limit the vehicle's speed to 5 miles per hour while navigating through a remote guidance scenario.

[0020] In various embodiments, the vehicle computing system may be configured to continuously and / or periodically inspect that the vehicle does not violate safety protocols and / or remote guidance protocols while navigating through a remote guidance scenario (e.g., while moving from waypoint to waypoint). For example, the vehicle computing system may control the vehicle from a first waypoint to a second waypoint. The vehicle computing system may detect a pedestrian having a trajectory that appears to intersect a trajectory associated with the vehicle. The vehicle computing system may determine an action to take based on the pedestrian, such as yielding to the pedestrian.

[0021] In various embodiments, the service computing device may determine that the scenario has completed (e.g., the vehicle has passed an obstacle and can resume normal operation) and may release the vehicle from remote guidance. In some embodiments, scenario completion may be based in part on the generated waypoints being within a threshold distance (e.g., 0.5 meters, 0.5 feet, 10 inches, etc.) of the original route associated with the vehicle. In some embodiments, scenario completion may be based on sensor data input. In such embodiments, an operator may be able to view the sensor data via a GUI to determine that the scenario has completed. For example, the obstacle may include a construction zone. A camera feed from the vehicle may represent the end of a construction zone behind an aircraft. Based on a determination that the construction zone is behind the vehicle, the operator may be able to determine that the scenario has completed. In some embodiments, the determination may be based on the planner system determining an acceptable (e.g., non - policy - violating) trajectory and / or may continue along the determined route.

[0022] In various embodiments, in response to a determination that a scenario has been completed, the service computing device may send a release signal to the vehicle. In such embodiments, in response to the release signal, the vehicle computing system may be configured to resume the vehicle's navigation (e.g., autonomous control). In various embodiments, the service computing device may determine that a scenario has disrupted the vehicle's initial path. In such embodiments, the service computing device may be configured to generate an updated path for the vehicle to arrive at a predetermined destination. In some embodiments, the service computing device may send the updated path to the vehicle computing system before sending the release signal. In some embodiments, the service computing device may send the updated path substantially simultaneously with the release signal (e.g., within 1 second, within 2 seconds, etc.). In response to receiving the updated path, the vehicle computing system may control the vehicle along the updated path to the predetermined destination. In some embodiments, the vehicle computing system may determine that a scenario has disrupted the initial path. In such embodiments, the vehicle computing system may be configured to generate an updated path for navigating to the destination.

[0023] The techniques discussed herein can improve the functionality of a vehicle computing system. Conventionally, in control plans for autonomous vehicles, the vehicle computing system may continuously process sensor data to determine a feasible route for the vehicle until a solution is determined. In an example where the vehicle faces a remote guidance scenario, the solution may not be achievable based on constraints imposed on the vehicle computing system, such as an operation protocol. In such an example, the vehicle computing system may utilize a vast amount of computing resources without reaching a solution. Thus, the vehicle may become stuck at a particular location on the planned route, and the vehicle computing system continues to use processing power and / or memory to identify a route through the remote guidance scenario. <m>0000095 The techniques described herein provide a technical solution to a technical problem faced by a vehicle computing system due to constraints resulting from utilizing remote computing resources to identify a near real-time solution for navigating through a remote guidance scenario. A graphical user interface on a service computing device can process sensor data from the vehicle computing system and provide guidance to the vehicle computing system, thereby reducing the amount of processing power and / or memory used by the vehicle computing system, which in some examples may fall into an infinite loop of processing without identifying a solution.

[0025] In addition, the technologies described herein can enhance the safety of autonomous vehicle operation through the environment. For example, a vehicle computing system may connect to a service computing device for guidance input based on a determination that further movement may violate the operating protocol. To ensure maximum safety, the vehicle computing system may request guidance from a remote operator, which can be configured to assess the situation and determine a safe path forward. Furthermore, the vehicle computing system may receive inputs and inspect them to ensure that the guidance inputs result in safe vehicle operation before controlling the vehicle in accordance with the inputs.

[0026] The techniques described herein may be implemented in several ways. Exemplary implementations are provided below with reference to the following drawings. Although discussed in the context of autonomous vehicles, the methods, apparatus, and systems described herein may be applied to a variety of systems (e.g., sensor systems or robotic platforms) and are not limited to autonomous vehicles. In another embodiment, the techniques may be used in an aerial or nautical context, or in any system using machine vision (e.g., a system using image data).

[0027] Figure 1 shows a display of an autonomous vehicle 102 (vehicle 102) in environment 100, where the remote guidance system may determine a deviation from the planned path 104 (e.g., initial path, planned path, etc.) based on the detection of a remote guidance scenario 106 (e.g., an obstacle on the planned path that the vehicle may not be configured to determine a feasible path around). The remote guidance scenario 106 may include a scenario that the vehicle computing system is not programmed to perform and / or appears to violate an operating protocol (e.g., a trajectory that the vehicle is required to navigate violates an operating protocol (e.g., control policy)). The operating protocol may include one or more rules governing actions that the vehicle may or may not take, such as not crossing a double yellow line, not driving into an approaching driving lane, or not crossing the boundary of the drivable surface of the road. In some embodiments, the remote guidance scenario 106 may include a scenario in which an obstacle extends beyond the sensing distance associated with vehicle 102. For example, as shown in Figure 1, it may not be possible for vehicle 102 to perceive the entire remote guidance scenario 106 from its initial position 108 (e.g., initial position and / or initial orientation). In addition, as shown in Figure 1, the remote guidance scenario 106 may include one or more obstacles blocking vehicle 102's planned path 104 such that navigating around the remote guidance scenario 106 results in vehicle 102 operating in a designated lane relative to traffic in the opposite direction. In a non-limiting embodiment, the remote guidance scenario 106 may include workers signaling the direction for vehicle 102 to proceed, such as cones and barricades defining altered lane boundaries, or otherwise scenarios different from the set of standard driving scenarios.

[0028] In various embodiments, the vehicle computing system may detect a remote guidance scenario 106 based on sensor data from one or more sensors. The sensors may include cameras, motion detectors, lidars, radars, time-of-flight sensors, and / or other sensors mounted on the vehicle 102. As will be discussed in more detail with reference to Figure 8, the planner component of the vehicle computing system may be configured to navigate the vehicle 102 along a planned path 104. The planner component may receive processed sensor data from the sensing component of the vehicle computing system and determine one or more trajectories for the vehicle to travel along the planned path 104 based on the sensor data. The trajectories may include the direction, orientation, and / or velocity in which the vehicle can travel through the environment.

[0029] The trajectory associated with vehicle movement may be determined in part based on a movement protocol (e.g., one or more movement constraints) imposed on the vehicle computing system. The movement protocol may include one or more rules governing actions that the vehicle may or may not take, such as not crossing a double yellow line, not driving into an approaching lane, not crossing the boundary of the drivable surface of the road, or not exceeding the speed limit. For example, the movement protocol may include restrictions on the vehicle remaining on the paved surface of the road. Thus, the vehicle computing system may be configured to generate one or more trajectories that keep the vehicle 102 on the paved surface of the road.

[0030] In various embodiments, the vehicle computing system may receive sensor data and determine that the planned route 104 is blocked by an obstacle, has high uncertainty for proceeding, or cannot proceed without violating the control policy. In some embodiments, the vehicle computing system may process the sensor data and attempt to generate one or more trajectories (satisfying the operating protocol) for the vehicle to navigate around the obstacle. However, in some embodiments, the scenarios faced, such as the remote guidance scenario 106 shown in Figure 1, do not have to include possible trajectories (or routes) that satisfy the operating protocol. In such embodiments, the vehicle computing system may identify the scenario as the remote guidance scenario 106 and cause the vehicle to come to the stopping position at the initial position 108. In various embodiments, based on the detection (e.g., identification) of the remote guidance scenario 106, the vehicle computing system may cause the vehicle to emit one or more hazard signals (e.g., hazard lights, audio signals, etc.).

[0031] Based on the detection of remote guidance scenario 106, the vehicle computing system may automatically connect to a service computing device via one or more networks. The network(s) may include public networks such as the Internet, private networks such as public sector networks and / or personal networks, or several combinations of public and private networks. The network(s) may also include, but are not limited to, any type of wired and / or wireless network, including satellite networks, cable networks, Wi-Fi networks, WiMAX networks, mobile communication networks (e.g., 3G, 4G, 5G, etc.), local area networks (LANs), wide area networks (WANs), or any combination thereof. In some embodiments, the vehicle computing system may connect to the service computing device based on input received from an input / output device, such as by the press of a button. For example, a safety supervisor, vehicle operator, or passenger in the vehicle may press a button to connect to the service computing device for remote guidance, etc.

[0032] In various embodiments, the vehicle computing system may transmit a signal indicating a remote guidance scenario 106. In some embodiments, the signal may include a request for guidance data (e.g., guidance input) from a service computing device. In some embodiments, the service computing device may, in response to being connected to and / or receiving a request for guidance data from the vehicle computing device, check that conditions for guidance are met. The conditions for guidance may include checks by the system to ensure that the service computing device can effectively provide guidance to the vehicle 102. The conditions may include, for example, that the vehicle speed is below a threshold (e.g., less than 15 miles per hour, less than 20 kilometers per hour), that a threshold number and / or type of sensors are available (e.g., a minimum of four cameras providing operational forward, rear, left, and right views, a minimum number of sensors associated with operational atmospheric near and far views), threshold bandwidth (e.g., 10 megabytes per second or more, 15 megabytes per second or more), threshold latency (e.g., 300 milliseconds or less, 250 milliseconds or less), or that there are no vehicle health-related malfunctions (e.g., steering column malfunction).

[0033] In some embodiments, in response to a determination that one or more conditions are not met, the service computing device may refuse to connect to the vehicle computing system (e.g., not connect). In various embodiments, in response to a determination that one or more conditions are not met, the service computing device may refuse operator access to the service computing device's graphical user interface (GUI) and / or refuse input to the GUI (e.g., disable input capability). In some embodiments, the service computing device may send an indication to the vehicle computing device that one or more conditions are not met and / or that the remote guidance system is unavailable.

[0034] In various embodiments, in response to a determination that a condition is met, the service computing device may launch a GUI on a display. In various embodiments, the GUI may include one or more windows representing sensor data from the vehicle 102's sensors. For example, each of the windows may represent a real-time or near real-time (e.g., based on latency) camera image captured by a camera mounted on the vehicle 102. In various embodiments, the data represented within the windows can provide the operator with information to help reflect on the route around the remote guidance scenario 106.

[0035] In various embodiments, the GUI may include a top-down view of the vehicle 102 (or otherwise configurable by a remote operator) and the remote guidance scenario 106 within the environment 100. In such embodiments, the top view may include a computer-generated view of the environment based on sensor data. In some embodiments, the top view may include the position of the vehicle 102, such as an initial position 108 relative to one or more obstacles in the remote guidance scenario 106. As will be discussed in more detail with respect to Figure 2, the GUI may be configured to receive input from the operator via the top view. In such embodiments, the operator may be able to provide the vehicle with guidance for navigating through the remote guidance scenario 106, such as through a driving corridor 110. The driving corridor 110 may represent a drivable surface around the remote guidance scenario 106 through which the operator can input one or more waypoints 112. The waypoint(s) 112 may include a position (e.g., X, Y, Z coordinates) through which the vehicle travels. In various embodiments, each of the waypoints 112, such as waypoint 112(1) to waypoint 112(N), may include a stop point. In such embodiments, the vehicle may be configured to stop at each of the waypoints 112, such as when there is no reception of a subsequent waypoint 112 or when there is no inspection of a subsequent waypoint 112. For example, the operator may input waypoint 112(1) and miss the reception of waypoint 112(2), and the vehicle computing device may stop the vehicle at waypoint 112(1) until further guidance is received.

[0036] In various embodiments, the input may include the orientation 114 (e.g., sway) of the vehicle 102 at the associated waypoint 112. In such embodiments, the operator may be able to orient the vehicle in a particular direction at the associated waypoint 112. In various embodiments, the orientation 114 associated with the waypoint 112 may be based on a preferred orientation in which the vehicle should stop at the waypoint. For example, to minimize exposure to approaching traffic when navigating around a remote guidance scenario 106, the preferred orientation associated with a vehicle 102 at least partially located in a lane associated with traffic in the opposite direction may be substantially parallel to the lane (e.g., within 5 degrees) so as to minimize the number of vehicles 102 located in the lane. Thus, to minimize exposure to traffic in the opposite direction, the orientation 114(2) associated with the waypoint 112(2) may be substantially parallel to the lane. In at least some embodiments, such orientations may be provided to enable the vehicle to discover a trajectory leading to a subsequent waypoint.

[0037] In some embodiments, the orientation 114 may be determined based on the preceding waypoint 114 and / or the subsequent waypoint 114. In such embodiments, the orientation 114 at a particular waypoint 112 can enable the vehicle to smoothly transition between waypoints on the guidance path. For example, a first orientation 114(1) associated with a first waypoint 112(1) may represent a straight line or spline in the direction of a second waypoint 112(2). A second orientation 114(2) associated with a second waypoint 112(2) may represent a spline in the direction of a third waypoint 112(3), and so on. As will be discussed in more detail below, the vehicle computing device may receive waypoints 112 and / or orientations 114 and generate one or more trajectories for smoothly controlling the vehicle 102 between waypoints and / or orientations.

[0038] In various embodiments, the operator may place waypoints 112, such as waypoint 112(1), at locations within the driving corridor 110. The driving corridor 110 may include determined sections of road through which the vehicle 102 can transition to navigate a remote guidance scenario 106. In various embodiments, a service computing device may generate the driving corridor 110 based on sensor data received from sensors on the vehicle. In some embodiments, the service computing device may generate the driving corridor 110 based on one or more detected obstacles in the environment 100. For example, the remote guidance scenario 106 may include obstacles on the right and left sides of the vehicle 102. The driving corridor 110 and / or its width may be defined by the distance between the obstacles and / or a safety buffer (e.g., a predetermined distance for the vehicle 102 to maintain away from the obstacles). In various embodiments, the width of the driving corridor 110 may be determined based on the width of the vehicle. In some embodiments, the width of the driving corridor 110 may also include a safety buffer on either side of the vehicle 102. In various embodiments, the driving corridor 110 may represent an area where the operator can input waypoints 112. In such embodiments, the GUI may not have to allow waypoints 112 to be input outside of the driving corridor 110.

[0039] In various embodiments, the GUI may limit the distance D between waypoints 112 and / or between the initial position 108 and the first waypoint 112(1) (e.g., 10 feet, 15 feet, 7 meters, etc.). In some embodiments, the GUI may not allow waypoints 112 to be entered when they are farther apart than the distance D from each other. In some embodiments, the GUI may activate an area having a radius D around the vehicle 102 where the operator can enter waypoints 112. In some embodiments, the GUI may provide the operator with a notification in a pop-up notification that the input is outside the distance D and cannot be accepted. In various embodiments, limiting the distance between waypoints can enable stepwise guidance of the vehicle 102 through the environment 100 and can ensure that the vehicle 102 can safely operate the remote guidance scenario 106.

[0040] In various embodiments, the operator can input waypoint 112(1) and / or associated direction 114 into the GUI, such as waypoint 112(1) and direction 114(1). As will be discussed in more detail with reference to Figure 2, the operator can input waypoint 112(1) by selecting a location within the environment 100 for waypoint 112(1) and confirming the location. In some embodiments, location confirmation may include clicking the mouse at the location and / or selecting a confirmation option on the GUI. In various embodiments, in response to receiving confirmation of waypoint 112 and / or direction 114, the service computing device may validate waypoint 112 and / or direction 114. Validation of waypoint 112 and / or direction 114 may include checking that waypoint 112 and / or direction 114 meet the vehicle's safety protocols. The safety protocol may include one or more rules associated with the safe operation of the vehicle 102, such as the location of waypoint 112 on the road (e.g., less than half the width of the vehicle 102 from the edge of the road or drivable surface), waypoint 112 within a threshold distance of the initial position 108 or within a threshold distance of each other, one or more threshold distances for the vehicle 102 to maintain away from dynamic and / or static objects, maximum sway rate, and / or other criteria associated with the safe navigation of the vehicle 102 through the environment 100.

[0041] In some embodiments, the operator may cause a service computing device to transmit waypoints 112 and / or headings 114 to the vehicle computing device. In some embodiments, the service computing device may automatically transmit waypoints 112 and / or associated headings 114 in response to input from the operator. In such embodiments, the service computing device may automatically transmit waypoints 112 and / or headings 114 based on a determination that waypoints 112 have been verified at a particular location and / or based on a determination that headings 114 have been verified. In various embodiments, one or more waypoints 112 and / or associated headings 114 may be transmitted based on a determination that the points have been verified (e.g., passed safety protocols, remote guidance protocols, and / or kinematic verification). In some embodiments, waypoints 112 and / or associated headings 114 may be transmitted in response to receiving a transmit command (e.g., selection of a transmit option), such as input from the operator. In some embodiments, the GUI may be configured to transmit waypoints 112 and / or associated orientations 114 individually and / or in groups.

[0042] In various embodiments, the vehicle computing system may receive waypoints 112 and / or headings 114 from a service computing device and verify the waypoints 112 and / or headings 114. In such embodiments, the vehicle computing system may also check that the waypoints 112 and / or headings 114 pass a safety protocol, which may be safely performed by vehicle 102 and / or kinematic verification (e.g., that vehicle 102 can achieve the desired waypoint based on physical constraints). In response to a determination that the waypoints and / or associated headings violate a safety protocol, the vehicle computing system may reject the waypoints. In some embodiments, the vehicle computing system may not proceed to the rejected waypoint, for example, by not proceeding from a stopping point (e.g., initial position 108) or by stopping at a waypoint prior to the rejected waypoint. For example, the vehicle computing device may receive a first waypoint 112(1) and a first heading 114(1), and may determine that the first waypoint 112(1) and the first heading 114(1) are valid (e.g., they pass the safety protocol). The vehicle computing device may navigate the vehicle from the initial position 108 to the first waypoint 112(1). The vehicle computing device may also receive a second waypoint 112(2) and a second heading 114(2), and may determine that at least one of the second waypoint 112(2) or the second heading 114(2) does not pass the safety protocol. The vehicle computing device may reject the second waypoint 112(2) and / or the second heading 114(2), and may stop the vehicle at the first waypoint 112(1) to await updated guidance.

[0043] In some embodiments, the vehicle computing system may send a message to the service computing device indicating the rejection of waypoint 112 and / or associated orientation 114. In some embodiments, the message may include the reason for the rejection. For example, the vehicle computing system may determine that the waypoint is less than a threshold distance from a static object in the environment 100. Based on this determination, the vehicle computing system may reject the waypoint and send a message to the service computing device indicating that the distance is less than a threshold distance.

[0044] In response to a determination that waypoint(s) 112 and / or associated direction(s) 114 pass the safety protocol, the vehicle computing system may accept waypoint(s) 112 and / or direction(s) 114. The vehicle computing system may process waypoint(s) 112 and / or direction(s) 114 and generate one or more trajectories 116 for the vehicle 102 to transition between waypoint(s) 112. For example, the vehicle computing system may receive a first waypoint(1) and a first direction(1). Based on the determination that the first waypoint(1) and the first direction(1) are valid, the vehicle computing system may determine a first set (one or more trajectories) of trajectories 116 for transitioning from an initial position to the first waypoint(1) at the first direction(1) 114. In at least some embodiments, one or more waypoints 112 may be used as constraints in determining such a trajectory (e.g., vehicle 102 should pass through the waypoint at a particularly defined pitch angle). In additional or alternative embodiments, the waypoint may include the cost on which the trajectory is determined based at least in part on how far vehicle 102 is from the waypoint and / or the cost associated with the difference in pitch between vehicle 102 and the corresponding waypoint 112. In one or more of the determined trajectories, the speed may be constrained so as not to meet or not to meet a threshold speed, where the threshold speed may be a fixed speed (e.g., 5 miles per hour, 10 miles per hour, etc.) and / or a percentage of the relevant speed limit on the drivable surface being traversed.

[0045] Based on the generated track 116, the vehicle computing system may control the vehicle 102 to waypoint 112 at the associated direction 114 via one or more driving systems, etc. In various embodiments, the vehicle computing system may receive waypoint 112 and / or associated direction 114 in stages. In such embodiments, the vehicle computing system may control the vehicle from the initial position 108 to the first waypoint 112(1). In some embodiments, the vehicle computing system may reach the first waypoint 112(1) before receiving and / or inspecting subsequent guidance such as a second waypoint 112(2). In such embodiments, the vehicle computing system may stop the vehicle at the first waypoint 112(1) until the second waypoint 112(2) is received and inspected. In some embodiments, the vehicle computing system may receive and inspect the second waypoint 112(2) before reaching the first waypoint 112(1). In such embodiments, the vehicle computing device may control the vehicle to operate past the first waypoint 112(1) toward the second waypoint 112(2) without stopping. In some embodiments, subsequent guidance may be provided while the vehicle decelerates to a stopping position at a specific waypoint, such as the first waypoint 112(1). In such embodiments, the vehicle computing device may accelerate the vehicle again to an appropriate speed and navigate the vehicle toward the second waypoint 112(2).

[0046] In various embodiments, a vehicle computing device may control the vehicle according to a remote guidance protocol. The remote guidance protocol may include one or more restrictions on the movement of the vehicle 102 while operating in remote guidance mode. The remote guidance protocol may include, for example, deviation from the path (e.g., distance from the path between two waypoints where the vehicle 102 may deviate by half the width of the vehicle, or the width of one vehicle, to avoid an obstacle), response to static object detection (e.g., attempt to avoid a static object to arrive at a subsequent waypoint), response to dynamic object detection (e.g., yielding to an agent that is expected to enter the path corresponding to waypoint 112), following a dynamic object moving in the same direction as the vehicle (e.g., within the path corresponding to waypoint 112), yielding to the right of the road at intersections as necessary (e.g., all-direction stop, traffic lights, detours, etc.), or forward stop movements when no further guidance is received (e.g., another waypoint, command to resume navigation, etc.).

[0047] In various embodiments, the vehicle computing system may control vehicle 102 at a speed appropriate to conditions in the environment. Conditions may include several and / or types of dynamic objects detected in the environment (e.g., several other vehicles on the road, several pedestrians, children playing nearby), or zones associated with the environment (e.g., school zones, playgrounds). In some embodiments, the vehicle computing system may be configured to dynamically determine the maximum speed of the vehicle to maximize safety while still allowing for forward movement. For example, the vehicle computing device may determine that it is in a school zone while operating in guidance mode. If the vehicle computing system detects that a student is approaching the path associated with vehicle 102, it may determine that the maximum speed associated with vehicle 102 is 7 miles per hour to allow for an immediate stop of vehicle 102. In another embodiment, vehicle 102 may be following another vehicle in traffic. The vehicle computing device may control vehicle 102 at a speed that maintains a safe distance behind the preceding vehicle.

[0048] In various embodiments, the vehicle computing system may control the vehicle 102 at a speed below a predetermined maximum speed (e.g., 20 kilometers per hour, 15 miles per hour, 10 miles per hour, etc.). In some embodiments, the predetermined maximum speed may include a maximum speed associated with remote guidance. In various embodiments, the vehicle computing system may control the vehicle 102 at a speed associated with the environment and / or a scenario, or at a speed below the speed associated with the environment and / or a scenario. In at least one embodiment, the vehicle computing system may control the vehicle 102 at the lower of the predetermined maximum speed or the maximum speed associated with the environment and / or a scenario. For example, when operating in remote guidance mode, the vehicle computing system may operate the vehicle at a predetermined maximum speed of 15 miles per hour (e.g., a speed of 0 to 15 miles per hour). The vehicle computing system may detect a remote guidance scenario 106 and request guidance input from a service computing device. The vehicle computing system and / or operator may determine, based on sensor data, that the associated remote guidance scenario 106 has a speed limit of 10 miles per hour. Based on the fact that a speed limit of 10 miles per hour is less than a predetermined maximum speed of 15 miles per hour, the vehicle computing system may limit the vehicle's speed to 10 miles per hour or less while navigating through remote guidance scenario 106.

[0049] In various embodiments, the vehicle computing system may be configured to check that the vehicle 102 does not violate safety protocols while navigating through the remote guidance scenario 106 (e.g., while moving from waypoint to waypoint). The vehicle computing device may monitor the surroundings continuously, periodically (e.g., every 0.1 seconds, every 0.5 seconds, every second, etc.), and / or at random intervals (e.g., every 3 seconds, every 5 seconds, etc.) to ensure that the safety protocols are met by the vehicle (e.g., the vehicle does not affect objects 118 in the environment). For example, the vehicle computing device may control the vehicle 102 to ensure that the vehicle maintains a minimum distance from objects 118 in the environment.

[0050] In various embodiments, a vehicle computing device may detect one or more objects 118 in the environment 100 based on sensor data, etc. The object(s) 118 may be static and / or dynamic objects. In various embodiments, the vehicle computing system may be configured to determine the classification associated with each of the detected object(s) 118 (e.g., the type of object, such as a car, a semi-trailer truck, a pedestrian, a cyclist, etc.). In various embodiments, the vehicle computing system may determine one or more predicted object trajectories 120 based on sensor data. The object trajectory 120 may represent any number of potential paths that the object 118 can take from its position associated with the sensing time through the environment 100. The object trajectory 120 may be determined based on various factors such as the classification of the object 118, traffic laws or rules (e.g., road rules), driving etiquette, position in a turning lane / non-turning lane, proximity to an intersection, other stationary and / or dynamic objects, and drivable surfaces.

[0051] In various embodiments, the object trajectory 120 may be determined using one or more machine learning algorithms. In such embodiments, the vehicle computing system may receive sensor data associated with the object 118 via a prediction system or the like, and may predict the behavior of the object 118 based on the sensor data. For example, the vehicle computing system may be trained on captured image data of the object's motion over time so that the vehicle computing system can recognize behaviors that can suggest subsequent actions (e.g., object trajectories) that the object 118 may take. In some embodiments, the vehicle computing system may determine the predicted object trajectory 120 based on a top-down representation of the environment by utilizing techniques such as those described in U.S. Patent Application No. 16 / 151,607, filed October 4, 2018, entitled "Trajectory Prediction on Top-Down Scenes" (the contents of which are incorporated below by reference). In addition, or alternatively, the vehicle computing system may utilize heatmaps, tree search methods, and / or temporal logic formulas, such as the techniques described in U.S. Patent Application No. 15 / 807,521, filed November 8, 2017, entitled “Probabilistic Heat Map for Behavior Prediction” (the contents of which are incorporated below by reference), to determine one or more object trajectories 120 associated with the detected object 118.

[0052] In various embodiments, the vehicle computing system may process object trajectories 120 associated with detected object(s) 118 in the environment 100 and determine whether the vehicle 102 and object(s) 118 may interact (e.g., collision, near miss). For example, the vehicle computing system may control the vehicle 102 from waypoint 112(8) to a subsequent waypoint 112(N). The vehicle computing system may detect an object 118 (e.g., a pedestrian) whose object trajectory 120 appears to intersect with the trajectory associated with the vehicle 102. The vehicle computing device may determine that the object 118 may cause the vehicle 102 to violate a safety protocol (e.g., a pedestrian is within a threshold distance of the vehicle 102 while it is moving) and / or cause the vehicle 102 to violate a remote guidance protocol (e.g., an object crosses the path of a vehicle in remote guidance mode). The vehicle computing system may determine what action to take based on pedestrians, such as yielding to pedestrians, in order to ensure that vehicle 102 maintains safety protocols and / or remote guidance protocols.

[0053] In various embodiments, the service computing device may determine that a scenario is complete (e.g., the vehicle has passed remote guidance scenario 106 and can resume normal operation) and release the vehicle 102 from remote guidance. In some embodiments, scenario completion may be based in part on the determination that the vehicle was able to safely navigate the remainder of remote guidance scenario 106. For example, an occluded object that was not visible to the vehicle computing system at the initial position 108 may be detected by a sensing system, and the vehicle computing system may be configured to circle the occluded object. In another embodiment, a first section of remote guidance scenario 106 may include a violation of the operating protocol, such as moving in a designated lane relative to traffic in the opposite direction. A second section of remote guidance scenario 106 may include moving within the limits of the operating protocol. Thus, the operator may determine that the scenario can be completed based on the completion of the first section (e.g., based on the vehicle transitioning in an appropriate lane relative to the direction of movement).

[0054] In some embodiments, scenario completion may be partially based on the fact that a generated waypoint 112, such as 112(N), is within a threshold distance (e.g., 0.5 meters, 0.5 feet, 10 inches, etc.) of the original path associated with the vehicle 102. In some embodiments, scenario completion may be based on sensor data input. In such embodiments, the operator can view the sensor data via a GUI to determine that the scenario is complete. For example, a remote guidance scenario 106 may include a construction area. A camera feed from the vehicle may represent the edge of the construction area behind the vehicle 102. Based on the determination that the construction area is behind the vehicle 102, the operator may determine that the scenario is complete.

[0055] In various embodiments, in response to a determination that a scenario has been completed, the service computing device may send a release signal to the vehicle 102. In such embodiments, in response to the release signal, the vehicle computing device may be configured to resume the navigation of the vehicle 102. In various embodiments, the service computing device may determine that the scenario has interfered with the vehicle 102's planned route 104 (e.g., the initial route). In such embodiments, the service computing device may generate an updated route for the vehicle to arrive at a predetermined destination. In some embodiments, the service computing device may send the updated route to the vehicle computing device before sending the release signal. In some embodiments, the service computing device may send the updated route substantially simultaneously with the release signal (e.g., within 1 second, within 2 seconds, etc.). In response to receiving the updated route, the vehicle computing system may control the vehicle 102 along the updated route to the predetermined destination. In some embodiments, the vehicle computing device may determine that the scenario has interfered with the initial route. In such embodiments, the vehicle computing device may be configured to generate an updated route for navigating to the destination. In various embodiments, based on the determination that neither the service computing device nor the vehicle computing device has determined an updated route, the vehicle computing device may cause the vehicle to move to the side of the road (e.g., control the vehicle out of the flow of traffic), stop the vehicle to determine an updated route, and / or receive updated route guidance from the service computing device.

[0056] In various embodiments, after connecting to a service computing device but before receiving guidance data (e.g., waypoints 112 and / or headings 114), the vehicle computing system may determine a solution for safely navigating the remote guidance scenario 106. In such embodiments, the vehicle computing system may send a message to the service computing device canceling the request for remote guidance. In some embodiments, the service computing device may launch a GUI to allow the operator to observe the vehicle 102 autonomously navigating the remote guidance scenario 106, such as by acting as a safety monitor. In some embodiments, the service computing device may complete the scenario, disable input controls associated with the GUI, and refuse the operator to provide input for waypoints 112 and / or headings 114.

[0057] Figure 2 shows an exemplary service computing device 200 having a graphical user interface (GUI 202) to which an operator 204 (shown as an input cursor controlled by the operator) can input waypoints 206 and / or heading 208, such as waypoint 112 and / or heading 114, to a vehicle 210, such as vehicle 102, operating in remote guidance mode. As discussed above, the vehicle computing device associated with vehicle 210 may detect one or more obstacles 212, including remote guidance scenarios 214, such as remote guidance scenario 106. In response to the detection of a remote guidance scenario 214, the vehicle computing system may establish a connection with the service computing device 200. In various embodiments, the vehicle computing system may send a request from the service computing device 200 for guidance data (e.g., guidance input). In at least some embodiments, the vehicle computing system can recognize such remote guidance scenarios 214 before making a complete stop (e.g., by the techniques discussed herein, it is possible to recognize in advance scenarios far enough away that the vehicle can continue moving to its destination without stopping).

[0058] In various embodiments, in response to connection with a vehicle computing device and / or receiving a request for guidance data, the service computing device 200 may determine whether one or more conditions for guidance are met. The conditions(s) for guidance may include the system checking to ensure that the service computing device 200 can effectively provide guidance to the vehicle 210. The conditions(s) may include the vehicle speed being below a threshold (e.g., less than 15 miles per hour, less than 20 kilometers per hour), the availability of a threshold number and / or type of sensors (e.g., a minimum of four cameras providing operational forward, rear, left, and right views, a minimum number of sensors associated with operational atmospheric near-field views, etc.), threshold bandwidth (e.g., 10 megabytes per second or more, 15 megabytes per second or more), threshold latency (e.g., 300 milliseconds or less, 250 milliseconds or less), or the absence of vehicle health-related faults (e.g., steering column failure, etc.).

[0059] In some embodiments, based on the determination that a condition is not met, the service computing device 200 may communicate the lack of the condition to the vehicle computing device. In such embodiments, the service computing device 200 may be unable to provide guidance data until the condition is met (e.g., bandwidth increases, latency decreases, there are no vehicle health-related failures, vehicle speed decreases below a threshold, sensor data availability increases, etc.). In some embodiments, based on the determination that a condition is not met, the service computing device 200 may not activate the GUI 202. In some embodiments, based on the determination that a condition is not met, the service computing device 200 may activate the GUI 202, but may not enable input control. In such embodiments, the operator 204 may be unable to provide guidance for waypoint 206 and / or heading 208. In any such embodiment, the vehicle may perform a fallback operation (e.g., maintaining its current position and / or pulling over to a safe stopping area).

[0060] In various embodiments, based on the determination that conditions(s) are met, the service computing device 200 may launch the GUI 202 and not enable input control. In such embodiments, the operator 204 may be able to provide guidance on waypoints 206 and / or bearings 208. The GUI 202 may include one or more windows 216 representing sensor data from one or more sensors on the vehicle 210. In at least one embodiment, the windows 216 may represent streaming images captured by cameras on the vehicle 210. The windows 216 may represent data to assist the operator in determining the path 218 that the vehicle will take. The path may include straight lines between waypoints, which may or may not indicate the actual path 230 that the vehicle can traverse through the environment. In some embodiments, the difference between the route 218 and the actual route 230 may be based at least in part on the bearing 208 or direction the vehicle faces at waypoint 206, such as a first bearing 208(1) associated with the first waypoint 206(1). In such embodiments, the vehicle computing system may generate the actual route 230 which shows one or more trajectories of the vehicle to arrive at the first waypoint 206(1) at the first bearing 208(1). In such embodiments, the trajectory may include one or more states (such as position, bearing, velocity, acceleration, etc.) acquired over a finite time range and / or between subsequent waypoints. As described above, such waypoints may be used to determine such trajectory as cost and / or constraints.

[0061] In an exemplary embodiment, the GUI 202 includes a first window 216(1) representing a left-front view associated with the vehicle 210, a second window 216(2) representing a front view, a third window 216(3) representing a right-front view, and a fourth window 216(4) representing a rear view. The operator may evaluate each of the views to determine a path 218 for the vehicle to transition through the remote guidance scenario 214. The path 218 can represent a straight-line distance between the initial position 222 of the vehicle 210 and a first waypoint 206(1), and straight-line distances between waypoints 206, such as between the first waypoint 206(1) and a second waypoint 206(2).

[0062] In various embodiments, the GUI 202 may include a display of a top view 220 (e.g., a bird's-eye view) of the vehicle 210 and a remote guidance scenario 214, the remote guidance scenario 214 may be adjusted according to the operator 204 to provide different viewpoints in at least some embodiments. In some embodiments, the top view 220 may include a computer-generated view of the environment based on sensor data. In some embodiments, the top view 220 may initially include the position 222 of the vehicle 210 relative to an obstacle(s) 212 in the remote guidance scenario 214. In an exemplary embodiment, the top view 220 represents the vehicle 210 approaching a number of orange cones blocking the vehicle 210's planned path 224 (e.g., initial path, planned path, etc.), such as a planned route (e.g., planned route) 104.

[0063] In various embodiments, the top view 220 may be configured to receive guidance input from the operator 204. In such embodiments, the operator 204 may be able to provide the vehicle with guidance for navigating through a remote guidance scenario 214, such as around an associated obstacle(s) 212. The input may include one or more waypoints 206. The waypoint(s) 206 may include a location where the vehicle is moving. In various embodiments, each of the waypoint(s) 206 may include a stop point, such as no further guidance. In such embodiments, the vehicle 210 may be configured to stop at each of the waypoint(s) 206, such as no reception of a subsequent waypoint(s) 206 or no inspection of a subsequent waypoint(s) 206.

[0064] In various embodiments, operator 204 may select a waypoint entry option 226. In some embodiments, the waypoint entry option 226 can enable input of waypoint 206 to the top view 220. In various embodiments, operator 204 may input waypoints using an input device associated with the service computing device 200. This is shown as a cursor associated with a mouse, but other input devices such as a touchscreen or keyboard are considered. Operator 204 may move the cursor to different potential positions for waypoint 206 on the top view 220. In some embodiments, symbols associated with waypoint 206, such as a second waypoint 206(2), may also be accompanied by a cursor. In an exemplary embodiment, the second waypoint 206(2) is represented as having a light color. In such embodiments, the light color can represent a waypoint 206 that has not yet been input by operator 204. Waypoint 206 may be entered in response to an entry, such as by clicking the mouse on a specific location, and waypoint 206, such as 206(1), may be represented by a different color. In at least some embodiments, GUI 202 may exclude operator 204 from entering a waypoint if it is greater than a certain distance from a previously entered waypoint and / or the last known location of the vehicle.

[0065] In various embodiments, operator 204 may select a heading entry option 228. The heading entry option can allow operator 204 to input a heading 208 (e.g., a sway) associated with waypoint 206. As discussed above, heading 208 can represent the heading in which the vehicle should be facing at a particular waypoint 206. In some embodiments, heading 208 may be determined based on the preceding waypoint 206 and / or subsequent waypoint 206. In such embodiments, the heading at a particular waypoint 206 can allow the vehicle to transition smoothly between waypoints on the guidance path. In various embodiments, heading 208 associated with waypoint 206 may be based on a preferred heading 208 in which the vehicle should stop at waypoint 206.

[0066] In some embodiments, the GUI 202 may be configured to receive an orientation entry (e.g., confirmed placement) on an input waypoint 206. For example, in response to operator 204 clicking (e.g., selecting) a second waypoint 206(2) and thereby changing its color to match an input waypoint 206(1), operator 204 may be able to input an associated orientation 208, such as orientation 208(2). In exemplary embodiments, operator 204 may move a cursor or other input device around the second waypoint 206(2) (e.g., scratch the cursor) to select the orientation in which the vehicle 210 faces at the second waypoint 206(2). The orientation may represent the orientation 208(2) or sway of the vehicle 210 at the second waypoint 206(2). As discussed above, in various embodiments, the orientation may coincide with a route 218 generated by the GUI 202. In various embodiments, cursor movement may be limited by orientation limitation 242. The orientation limitation 242 may include a range of directions in which the orientation 208 can be set. In various embodiments, the GUI 202 may block orientation inputs outside of the orientation limitation 242. The orientation limitation 242 may be based on physical constraints imposed on the vehicle 210 (e.g., maximum roll percentage, maximum roll angle, maximum cross-sectional distance, maximum turn angle, etc.). For example, the top view 220 may not accept orientation inputs that would cause the vehicle to make a large turn at waypoint 206, such as a 90-degree or 180-degree turn.

[0067] In various embodiments, the service computing device 200 may be configured to validate waypoints 206 and / or direction(s) 208 entered via a GUI. As discussed above, the validation of waypoints 206 and / or direction(s) 208 may include checking that the vehicle 210 does not violate (e.g., passes) safety protocols and / or remote guidance protocols associated with the vehicle navigating according to waypoints 206 and / or direction(s) 208. In some examples, such validation may include kinematic / dynamic checks (verification) to ensure that the vehicle has the ability to perform such waypoint transitions.

[0068] In various embodiments, in response to receiving guidance input (e.g., waypoint(multiple) 206 and / or direction(multiple) 208), the service computing device 200 may transmit waypoint(multiple) 206 and / or direction(multiple) 208 (e.g., waypoint data) to the vehicle computing system. In some embodiments, the service computing device 200 may transmit the first waypoint 206(1) and / or the first direction 208(1) before receiving input associated with a second waypoint 206(2) and / or the second direction 208(2). In such embodiments, the vehicle computing system may be configured to receive the first waypoint 206(1) and / or the first direction 208(1) and cause the vehicle to begin moving from the initial position 222. If no data associated with the second waypoint 206(2) is received, the vehicle computing system may cause the vehicle 210 to stop at the first waypoint 206(1). In various embodiments, in response to the reception (and verification) of a second waypoint 206(2) and / or a second direction 208(2), the vehicle computing system may direct the vehicle toward the second waypoint 206(2). In such embodiments, the service computing device 200 can enable stepwise remote guidance to the vehicle computing system. In some embodiments, the service computing device 200 may transmit data associated with two or more waypoints 206 and / or direction 208. In such embodiments, based on the verification of two or more waypoints, the vehicle computing system may be configured to navigate between two or more waypoints 206. In any of the embodiments described herein, the second waypoint 206(2) may be received as the vehicle crosses to the first waypoint 206(1) so that the vehicle can continue through all waypoints without stopping.

[0069] As discussed above, in response to receiving waypoints 206 and / or bearing 208, such as the first waypoint 206(1) and / or the first bearing 208(1), the vehicle computing system may generate one or more trajectories for the vehicle 210. The trajectories may represent the actual course 230 that the vehicle 210 travels between positions while operating in guidance mode. In various embodiments, the trajectories associated with the actual course 230 may include trajectories that result in the vehicle arriving at the first waypoint 206(1) at the first bearing 208(1).

[0070] In various embodiments, the vehicle computing system may send an indication to the service computing device that it is impossible to proceed beyond the current location. The current location may include waypoint 206, such as waypoint 206(1), or another location in the environment. In some embodiments, the vehicle may determine that the vehicle computing device is unable to generate one or more trajectories for the vehicle 210, that subsequent waypoints 206, such as a second waypoint 206(2), have not yet been received, that the path to the second waypoint 206(2) is blocked, or that waypoint 206 has been rejected. In some embodiments, in response to receiving the indication, the service computing device may display a notice on the GUI 202 indicating that it is impossible for the vehicle to proceed. Based on the notice, the operator may input a second (subsequent) waypoint 206(2) and / or a replacement waypoint 206, such as when the second waypoint 206(2) is rejected.

[0071] In various embodiments, the GUI 202 can enable the operator 204 to stop the vehicle at a specific location. In some embodiments, the vehicle may stop at a waypoint such as waypoint 206(1) based on the determination that further guidance has not yet been received (e.g., that data associated with the second waypoint 206(2) has not been received and / or inspected). In some embodiments, the operator 204 may stop the vehicle 210 by selecting a hold option 232. By selecting the hold option 232, the service computing device 200 can send a hold signal to the vehicle 210. In some embodiments, in response to receiving the hold signal, the vehicle computing system may control the vehicle to stop. In some embodiments, in response to receiving the hold signal, the vehicle computing system may maintain the vehicle at the stopped location.

[0072] In various embodiments, the vehicle computing system may maintain the stopped position until a completion signal (e.g., a release signal) is received (e.g., in response to a hold signal). The completion signal may be transmitted by the service computing device in response to the operator 204 inputting a release option 234. In various embodiments, the completion signal allows the vehicle computing device to control the vehicle based on the received guidance input. In some embodiments, hold and release may allow the operator to transmit multiple waypoints 206 and / or headings 208 at once. In some embodiments, waypoints 206 and / or headings 208 entered during the vehicle hold may be transmitted substantially simultaneously with the completion signal.

[0073] In various embodiments, the GUI 202 may include a completion option 236. The completion option 236 may be selected by the operator in response to a determination that the vehicle can navigate the remote guidance scenario 214 and proceed autonomously. In some embodiments, the completion option 236 may be selected in response to a determination that a termination condition has occurred. The termination condition may include determinations that the remote guidance scenario 214 is behind the vehicle, the vehicle 210 is within a threshold distance of the planned route 224, or the vehicle 210 has the ability to proceed autonomously (e.g., by controlling the vehicle according to an operating protocol, safety protocol, etc.). In various embodiments, in response to the selection of the completion option 236, the service computing device 200 may send a completion signal to the vehicle computing device. The completion signal may include an instruction to proceed autonomously. In various embodiments, the service computing device 200 may determine that the remote guidance scenario 214 has interfered with the vehicle 210's planned route 224 and that the vehicle 210 should proceed along an alternative route (e.g., an updated route). In some embodiments, the service computing device 200 may automatically generate an updated route for the vehicle to navigate to its destination. In some embodiments, the service computing device may generate an updated route based on operator input of an updated route option 238.

[0074] In various embodiments, GUI202 may be configured to receive waypoint edits (e.g., delete or modify) in response to the Edit Waypoint option 240 of GUI202. The Edit Waypoint option 240 may allow an operator to delete and / or modify waypoint 206 and / or its associated heading 208 (e.g., change the position for waypoint 206) (e.g., change the sway for heading 208). Waypoint 206 and / or heading 208 may be edited based on a determination that waypoint 206 has not been acknowledged (e.g., not accepted by the vehicle computing system) or that the waypoint has been acknowledged but not completed by the vehicle (e.g., vehicle 210 has not arrived at waypoint 206 and the vehicle has not started along the actual route 230 to waypoint 206). In various embodiments, in response to a determination that the waypoint has been acknowledged but not completed, GUI202 may check that vehicle 210 has stopped before enabling editing of waypoint 206. In embodiments where waypoints are currently used by a vehicle computing system to control the vehicle, waypoints may be locked for editing. In such embodiments, this eliminates the need for operator 204 to adjust the waypoint after it has been used for planning (thus resulting in safe planning processing).

[0075] Figure 3 shows an exemplary process 300 for providing step-by-step guidance to vehicle 102. Some or all of the process 300 may be performed by one or more components described below with respect to Figure 8.

[0076] In Operation 302, the service computing device may receive a request for guidance from the vehicle computing system. The vehicle computing system may send a request for guidance based on the detection of a remote guidance scenario 106 within the operating environment. A remote guidance scenario 106 may include obstacles on the road, road closures, or construction areas. A remote guidance scenario 106 may include scenarios not programmed for the vehicle computing system to perform, and / or scenarios that appear to violate the operating protocols and / or safety protocols associated with the vehicle 102. As described above, such scenarios may be detected in anticipation of the vehicle approaching the scenario so that guidance can be relayed before stopping, so that the vehicle does not need to stop at all.

[0077] In response to receiving a request for guidance, the service computing device may process the request and invoke a graphical user interface (GUI). The GUI may include an interface through which an operator can provide guidance input to the vehicle. The GUI may be configured to receive inputs corresponding to a first waypoint 112(1) (e.g., a first position) for the vehicle to navigate a remote guidance scenario 106 (e.g., around an obstacle). In some embodiments, the GUI may be configured to receive inputs corresponding to a first orientation associated with the first waypoint 112(1). The first orientation may represent a direction that the vehicle 102 can face at the first waypoint 112(1).

[0078] In operation 304, the service computing device may transmit to the vehicle computing system a first waypoint 112(1) (e.g., data associated with a first location and / or a first orientation associated with the first waypoint 112(1)). The vehicle computing system may receive data associated with the first waypoint 112(1) and may control the vehicle to the first waypoint 112(1). In some embodiments, the vehicle computing system may also control the vehicle 102 based on the first orientation, such as to bring the vehicle 102 to the first waypoint 112(1) in the first orientation (and / or within some threshold thereof). In various embodiments, in response to not receiving further guidance, the vehicle may stop at the first waypoint 112(1), represented by a stop bar 306. In some embodiments, the vehicle 102 may stop at the stop bar 306 in response to receiving a hold signal, such as those described above with respect to Figure 2. In such an embodiment, the service computing device may send a hold signal in response to receiving input from the operator to hold the vehicle 102.

[0079] In operation 308, the service computing device may transmit a second waypoint 112(2) and a third waypoint 112(3) to the vehicle computing system. In various embodiments, the second waypoint 112(2) and the third waypoint 112(3) may be determined at least in part based on the obstructed object 310 initially detected at the first waypoint 112(1). The obstructed object 310 may include an object that was not initially visible to the vehicle computing system when the remote guidance scenario 106 is detected (e.g., detected by a sensor system). In some embodiments, the service computing device may be configured to provide stepwise guidance based on real-time or near-real-time detection of obstacles, objects, etc., in the environment.

[0080] In various embodiments, the vehicle computing device may control the vehicle to a second waypoint 112(2) and then to a third waypoint 112(3) without stopping at any of the waypoints 112. In such embodiments, the vehicle may proceed without stopping based on the determination that a safety waypoint is met (for example, that a detected object does not affect movement).

[0081] In Operation 312, the service computing device may determine that a scenario (e.g., remote guidance scenario 106) is complete (e.g., a termination condition has been met). In some embodiments, the service computing device may determine that a scenario is complete based on the determination, using sensor data, etc., that one or more obstacles associated with remote guidance scenario 106 and / or the blocked object 310 are behind the vehicle. In some embodiments, the service computing device may determine that a scenario is complete based on the determination that a third waypoint 112(3) (e.g., the final waypoint) is within a threshold distance (e.g., 4 inches, 0.5 feet, 2 feet, 1 meter, etc.) of the vehicle's planned path 104. In various embodiments, the vehicle control system may determine that it is possible to plan a trajectory for the vehicle 102 that does not violate any control policies (and safely traverses the environment) and may send a signal to the service computing device indicating this. In such embodiments, the service computing device may determine that a scenario is complete based on the signal from the vehicle computing system.

[0082] In operation 314, the service computing device may release the vehicle from remote guidance. In some embodiments, the service computing device may send a completion signal to the vehicle computing system. In some embodiments, in response to receiving the completion signal, the vehicle computing system may autonomously control the vehicle. In some embodiments, the completion signal may include an autonomously proceeding command. In such embodiments, in response to receiving the command, the vehicle computing system may autonomously control the vehicle.

[0083] Figure 4 shows an operator workflow 400 for providing remote guidance to a vehicle via a graphical user interface 402. In operation 404, the vehicle computing system may be connected to a service computing device. The service computing device may determine that one or more conditions for providing remote guidance are not met. In some embodiments, the service computing device may determine that the conditions are not met based on data received from the vehicle computing system and / or data associated with the network connection. The conditions for guidance may include system checks to ensure that the service computing device can effectively provide guidance to the vehicle. The conditions(s) may include, but are not limited to, the current vehicle speed being below a threshold (e.g., less than 15 miles per hour, less than 20 kilometers per hour), the availability of a threshold number and / or type of sensors (e.g., a minimum of four cameras providing operational forward, rear, left, and right views, a minimum number of sensors associated with operational atmospheric near and far views), threshold bandwidth (e.g., 10 megabytes per second, 15 megabytes per second or more), threshold latency (e.g., 300 milliseconds or less, 250 milliseconds or less), or the absence of vehicle health-related malfunctions (e.g., steering column malfunction).

[0084] Based on the determination that at least one of one or more conditions is not met, the service computing device may determine in operation 406 that guidance is not possible. In various embodiments, based on the determination that at least one of the conditions is not met, the service computing device may prevent guidance from being input to the graphical user interface 402 in operation 408. In such embodiments, it may be impossible for the operator to input waypoint and / or orientation data to the graphical user interface 402.

[0085] In Operation 410, the vehicle computing system may connect to a service computing device and determine that the conditions for providing remote guidance are met. Based on the determination that the conditions are met, the service computing device may determine in Operation 412 that guidance is possible. In response to a determination that the conditions are not met while providing guidance, the service computing device may determine that guidance is not possible, such as as expressed in Operation 406. In some embodiments, based on the determination that guidance is not possible, the vehicle may not proceed beyond its initial position (e.g., it may remain stopped). In some embodiments, based on the determination that guidance is not possible, the service computing device may send a message to the vehicle computing system indicating that remote guidance is not possible. In such embodiments, the message may include a justification for not providing remote guidance (e.g., reasons why guidance is not possible). Conversely, in response to a determination that at least one previously unmet condition is now met, such as as determined in Operation 404, the service computing device may determine that guidance is possible (Operation 412).

[0086] Based on the determination that guidance is possible, the workflow may include activating guidance on the graphical user interface in operation 414. In various embodiments, activating guidance can allow the operator to input waypoint and / or direction-corresponding data to assist the vehicle in navigating around obstacles. Thus, the operator can construct a path for the vehicle around obstacles.

[0087] While the operator constructs the route, further guidance may be blocked in Operation 408 based on the determination that at least one of the conditions(s) is not met (the same or different conditions described above as not being met). In some embodiments, based on the determination that at least one condition is not met, the vehicle may not proceed beyond the initial position and / or the most recently received and / or verified waypoint (e.g., it may remain stopped). In such embodiments, the vehicle computing system may be configured to stop the vehicle in response to not receiving further guidance (e.g., subsequent waypoints, completion signals, etc.). Furthermore, in Operation 408, guidance may be reactivated on the graphical user interface 402 in response to the determination that at least one condition is met, which allows for the input of additional waypoint and / or heading data.

[0088] Figure 5 is a flowchart 500 representing vehicle actions in response to remote guidance input. In operation 502, the operator associated with the graphical user interface of the service computing device may connect to the vehicle computing system. In various embodiments, the operator may connect to the vehicle computing system in response to receiving a request for remote guidance. In some embodiments, the connection may be based on the satisfaction of one or more conditions.

[0089] In Operation 504, the vehicle may be stopped in a hold state. In various embodiments, the vehicle may be stopped by a vehicle computing device, etc., based on the detection of a remote guidance scenario. In some embodiments, the vehicle may be stopped based on a hold command, such as one received from a service computing device via a graphical user interface. The vehicle may be stopped in Go (e.g., permitted movement) mode or NoGo (e.g., unauthorized movement) mode. Go mode may represent a mode in which a time delay is required before forward movement. In some embodiments, the time delay may be based on the time required to input and / or validate waypoints. In some embodiments, the time delay may be based on the satisfaction of safety protocols and / or remote guidance protocols. For example, the vehicle computing device may detect that another vehicle ahead of the vehicle has stopped. To maintain a safe distance behind the other vehicle, the vehicle computing device may stop the vehicle in Go mode. NoGo mode may represent a mode in which remote guidance may not be available, based on conditions not being met (e.g., a detected vehicle health failure) or the vehicle performing a fail-safe operation and deviating from nominal driving behavior.

[0090] In operation 506, the vehicle may be moving in a released state. In some embodiments, the vehicle may be moving in a released state based on the reception of a release signal from a service computing device associated with a graphical user interface. In various embodiments, the vehicle may request remote guidance while moving. From operation 504 or operation 506, the operator may initiate guidance mode 508 on the graphical user interface of the service computing device.

[0091] In operation 510, the vehicle may be stopped in a hold state. In some embodiments, guidance mode 508 may be active while the vehicle is hold. In such embodiments, the operator may input waypoints and / or headings for guidance around obstacles (e.g., for navigating a remote guidance scenario), but due to the hold, the vehicle does not need to be controlled to traverse the environment according to the waypoints.

[0092] In Operation 512, the vehicle may continue to operate standard planner guidance (e.g., autonomous control provided by the vehicle computing system) until data associated with the waypoint is received from the service computing device. In Operation 514, in response to the operator adding a waypoint, the vehicle may satisfy one or more received waypoints (e.g., drive to each waypoint). In some embodiments, in response to receiving a waypoint, the vehicle computing system may validate the waypoint to ensure that the waypoint satisfies the vehicle's safety protocol and / or remote guidance protocol.

[0093] In various embodiments, in response to receiving a waypoint, the vehicle may generate one or more trajectories to control the vehicle to the waypoint. In operation 514, the vehicle computing device may receive data from the service computing device corresponding to one or more additional waypoints (e.g., an operator adds additional waypoints to the graphical user interface). The vehicle computing system may continuously generate trajectories and control the vehicle between subsequent waypoints based on the trajectories.

[0094] In various embodiments, a vehicle stopped in a pending state during operation 510 may be released by the operator selecting a release option on a graphical user interface, for example. In such embodiments, during operation 514, the vehicle may begin to move through or fill the remaining waypoints in its path.

[0095] In operation 516, the vehicle may stop and wait for additional guidance. In some embodiments, the vehicle may stop in operation 516 based on the determination that it has not yet received any further guidance (subsequent waypoints). In such embodiments, in response to receiving a waypoint, the vehicle may resume moving (operation 514). In some embodiments, the vehicle may stop based on the determination that the final waypoint has been met (e.g., the vehicle has arrived at the final waypoint). In some embodiments, in response to the completion of guidance, such as in operation 520, the operator may complete the guidance and disconnect from the vehicle computing system. In some embodiments, the operator may cause the vehicle computing system to send a completion message (e.g., a completion signal) to the service computing device indicating that the guidance route is complete. In response to the completion of the guidance route, the vehicle computing system may resume autonomous control of the vehicle.

[0096] In various embodiments, in operation 518, the vehicle may continue moving in operation 514, etc., until the vehicle reaches the final waypoint in its route. In some embodiments, the operator may indicate the completion of guidance before the final waypoint, such as by designating the waypoint as the final waypoint. In some embodiments, in response to the completion of guidance, the operator may complete the guidance and disconnect from the vehicle computing system in operation 520, etc. In response to the completion of the guidance route, the vehicle computing system may resume autonomous control of the vehicle. In various embodiments, a service computing device may receive a second request for remote guidance. In such embodiments, the operator may reconnect to the vehicle computing system in operation 502.

[0097] Figures 6 and 7 illustrate exemplary processes according to embodiments of the present disclosure. These processes are presented as logical flowcharts, each representing a set of operations that can be implemented in hardware, software, or a combination thereof. In a software context, an operation represents a computer-executable instruction stored in one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, and data structures that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations may be combined in any order and / or in parallel to implement a process.

[0098] Figure 6 shows an exemplary process 600 for providing remote guidance to a vehicle computing device. Some or all of the process 600 may be performed by one or more components in Figure 8, as described herein. For example, some or all of the process 600 may be performed by a service computing device 832.

[0099] In Operation 602, processing may include receiving a request for guidance inputs for navigating a scenario from a vehicle computing system associated with the vehicle. Requests for guidance inputs may be received via a network connection between the vehicle computing system and the service computing device. Requests for guidance inputs may include requests for step-by-step guidance through the scenario, such as via one or more waypoints and / or associated orientations.

[0100] In Operation 604, the process may include determining whether one or more conditions are met. These conditions may include the vehicle speed being below a threshold, a threshold number of sensors and / or sensors of a certain type being available, a threshold bandwidth associated with network connectivity, a threshold latency associated with network connectivity, or the absence of any vehicle health-related failures.

[0101] In response to a determination that at least one condition is not met ("No" in operation 604), the process may include disabling the input capability of the service computing device's graphical user interface (GUI) in operation 606. In some embodiments, in response to a determination that at least one condition is not met, the service computing device does not need to start the GUI.

[0102] In response to a determination that the condition is met ("Yes" in operation 604), the process may include receiving an input corresponding to the waypoint in operation 608. The input may be received from an operator associated with the GUI, for example, via an input / output device associated with a service computing device. In various embodiments, the input may include the location and / or orientation of the waypoint (e.g., sway). In such embodiments, the location and / or orientation of the waypoint may represent the location through which the vehicle passes while operating in guidance mode, and the direction in which the vehicle faces at that location.

[0103] In various embodiments, a service computing device may validate waypoints and / or their associated orientations. Waypoint validation may include checking that waypoints and / or their associated orientations satisfy safety protocols and / or remote guidance protocols associated with vehicle operation. The safety protocols may include one or more rules associated with safe vehicle operation, such as the location of a waypoint on a road (e.g., less than half the width of the vehicle from the edge of the road or drivable surface), waypoints within a threshold distance of an initial position or within a threshold distance of another waypoint, one or more threshold distances for the vehicle to maintain distance from dynamic and / or static objects, a maximum deflection rate, and / or other criteria associated with the safe navigation of the vehicle through the environment. In additional or alternative embodiments, such validation may include kinematic / dynamic checks to ensure that the vehicle has the ability to perform a transition from one waypoint to another.

[0104] In operation 610, the process may include transmitting data associated with waypoints to the vehicle computing system. In some embodiments, the data may be transmitted to the vehicle computing system based on the verification of the associated waypoints and / or orientations. In some embodiments, the data may include the location and / or orientation associated with one or more waypoints. In various embodiments, the vehicle computing system may verify the waypoint(s) and control the vehicle in part based on those waypoint(s).

[0105] In Operation 612, processing may include determining whether scenario navigation is complete. In various embodiments, the service computing device may determine that a scenario is complete based on the transmission of the final waypoint associated with the scenario. In some embodiments, the service computing device may identify the final waypoint based on the fact that the waypoint that has passed through the obstacle associated with the scenario is at least a threshold distance (e.g., 5 feet, 10 feet, 3 meters). In some embodiments, the service computing device may identify the final waypoint based on the fact that the waypoint is less than a threshold distance (e.g., 4 inches, 1 foot, 1 meter) from the initial path associated with the vehicle.

[0106] In response to a determination that scenario navigation is not complete (No in operation 612), the service computing device may return to operation 608 and receive inputs corresponding to waypoints. In various embodiments, the service computing device may continue to receive inputs corresponding to waypoints and transmit data associated with waypoints to the vehicle computing system until scenario navigation is complete.

[0107] In response to the determination that scenario navigation is complete (Yes in operation 612), the process may include sending a completion signal to the vehicle computing system in operation 614. In some embodiments, the completion signal may include an instruction to autonomously proceed to a destination. In response to receiving the completion signal, the vehicle computing system may resume standard planner component functionality for autonomously controlling the vehicle.

[0108] In various embodiments, the service computing device may determine that the initial route associated with the vehicle has been blocked due to a scenario. In such embodiments, the service computing device may determine that the vehicle may not be able to follow the initial route to its destination. In some embodiments, the service computing device may generate an updated route. In such embodiments, the service computing device may send the updated route to the vehicle computing device simultaneously with the transmission of a completion signal or within a threshold time (e.g., 0.5 seconds, 2 seconds, etc.).

[0109] Figure 7 shows an exemplary process 700 for controlling a vehicle based on guidance input received from a service computing device. Some or all of the process 700 may be performed by one or more components in Figure 8, as described herein. For example, some or all of the process 700 may be performed by a vehicle computing device 804 (e.g., a vehicle computing system).

[0110] In Operation 702, the process may include receiving sensor data from the vehicle's sensors. The sensors may include cameras, motion detectors, lidars, radars, or time-of-flight sensors. The vehicle computing device may be configured to detect and / or classify one or more dynamic and / or static objects in the environment in which the vehicle operates, by using sensing components, etc.

[0111] In various embodiments, a vehicle computing system may process sensor data to determine one or more actions the vehicle should take while operating in an environment. The actions may be determined based on dynamic and / or static objects detected in the environment. The actions may also be based on vehicle-associated operating protocols and / or safety protocols. The operating protocols may include one or more rules governing actions the vehicle may or may not take, such as not crossing a double yellow line, not driving into an approaching lane, or not crossing the boundary of the drivable surface of the road. The safety protocols may include one or more rules associated with the safe operation of the vehicle, one or more threshold distances for the vehicle to maintain distance from dynamic and / or static objects, a maximum sway rate, and / or other criteria associated with the safe navigation of the vehicle through the environment. In some embodiments, the vehicle computing device may process sensor data and may not be capable of determining the actions to take.

[0112] In Operation 704, processing may include determining, based on sensor data, whether the vehicle has encountered a remote guidance scenario. A remote guidance scenario may include obstacles, road closures, construction zones within the road, or drivelines that exceed the vehicle's line of sight (e.g., impaired sensor vision due to road elevation and / or bends within the road). A remote guidance scenario may also include scenarios not programmed for the vehicle computing system to perform and / or scenarios that appear to violate the operating protocol.

[0113] In response to determining that the vehicle is not facing a remote guidance scenario ("No" in Operation 704), the process may include, in Operation 706, the vehicle computing system autonomously controlling the vehicle in a standard planner mode or the like.

[0114] In response to determining that the vehicle was facing a remote guidance scenario (Yes in Operation 704), the process may include sending a request for guidance to the service computing device in Operation 708. In various embodiments, the vehicle computing system may establish a network connection with the service computing device. In such embodiments, the vehicle computing system may send the request over the network connection. In some embodiments, an observer in the vehicle may manually establish a network connection between the vehicle computing system and the service computing device. In some embodiments, the vehicle computing system may automatically establish a network connection and / or send a request for guidance in response to determining that the vehicle was facing a remote guidance scenario.

[0115] In Operation 710, processing may include receiving data associated with a waypoint. The data may include the location and / or orientation associated with the waypoint.

[0116] In Operation 712, the process may include determining whether the waypoint is valid. Waypoint verification may include checking that the waypoint and / or its associated orientation meet safety protocols, remote guidance protocols, and / or kinematic verification associated with vehicle operation. Safety protocols may include one or more rules associated with safe vehicle operation, such as the location of the waypoint on the road (e.g., less than half the width of the vehicle from the edge of the road or drivable surface), the waypoint being within a threshold distance of its initial position or within a threshold distance of another waypoint, one or more threshold distances for the vehicle to maintain distance from dynamic and / or static objects, the maximum deflection rate, and / or other criteria associated with safe vehicle navigation through the environment. Remote guidance protocols may include one or more restrictions on vehicle movement while operating in remote guidance mode. The remote guidance protocol may include: course deviation (e.g., distance from the course between two waypoints where the vehicle may deviate by half the width of the vehicle, or the width of one vehicle, to avoid an obstacle); responses to static object detection (e.g., attempts to avoid static objects to arrive at subsequent waypoints); responses to dynamic object detection (e.g., yielding to an agent expected to enter the course corresponding to a waypoint); following a dynamic object moving in the same direction as the vehicle (e.g., within the course corresponding to a waypoint); yielding to the right of the road at intersections as necessary (e.g., all-direction stops, traffic lights, detours, etc.); or forward stop movements in the absence of further guidance received (e.g., another waypoint, a command to resume navigation, etc.). Kinematic verification may include checking that the vehicle computing system can generate one or more trajectories for navigating the vehicle to the waypoints in the relevant orientations.

[0117] In response to determining that the waypoint and / or heading is invalid ("No" in Operation 712), the process may include rejecting the waypoint in Operation 714. In various embodiments, the vehicle computing system may send a message to the service computing device indicating the rejection of the waypoint. In various embodiments, the message may include the reason for determining that the waypoint is invalid (e.g., an explanation of the rejection).

[0118] In response to determining that the vehicle was facing a remote guidance scenario ("Yes" in Operation 712), the process may, in Operation 716, control the vehicle at least in part on data associated with a waypoint (e.g., control the vehicle in the inclusion of a waypoint). In a non-limiting embodiment, such waypoints (and / or associated orientations) may be used as one or more costs or controls for planning a trajectory from one waypoint to the next. Vehicle control may include transmitting one or more signals to one or more drive systems associated with the vehicle and causing the vehicle to either hold its position or move to a different position, such as one associated with a waypoint, and to face a particular direction, such as one associated with a determined orientation corresponding to a waypoint.

[0119] In at least some embodiments, if the scenario is not surpassed, such processing may continue 710, thereby receiving and considering additional waypoints. Otherwise, processing 700 may continue 706 to continue autonomously controlling the vehicle.

[0120] Figure 8 is a block diagram of an exemplary system 800 for implementing the technology described herein. In at least one embodiment, the system 800 may include a vehicle 802, such as vehicle 102.

[0121] The vehicle 802 may include one or more vehicle computing devices 804, one or more sensor systems 806, one or more emitters 808, one or more communication connections 810, at least one direct connection 812, and one or more drive systems 814.

[0122] The vehicle computing device(s) 804 may include one or more processors 816 and a memory 818 communicably coupled to the one or more processors 816. In the shown embodiment, the vehicle 802 is an autonomous vehicle, but the vehicle 802 may be any other type of vehicle, such as a semi-autonomous vehicle, or any other system having at least an image capture device (e.g., a camera-enabled smartphone). In the shown embodiment, the memory 818 of the vehicle computing device(s) 804 stores a localization component, a sensing component 822, a planning component 824, one or more system controllers 826, and one or more maps 828. Although shown in Figure 8 as existing in the memory 818 for illustrative purposes, the localization component 820, the sensing component 822, the planning component 824, one or more system controllers 826, and one or more maps 828 may, in addition or instead, be accessible to the vehicle 802 (e.g., stored in or accessible by memory remote from the vehicle 802, such as the memory 834 of the service computing device(s) 832).

[0123] In at least one embodiment, the localization component 820 may include functionality to receive data (e.g., x-position, y-position, z-position, roll, pitch, or yaw) from sensor system 806 to determine the position and / or orientation of vehicle 802. For example, the localization component 820 may include and / or request / receive a map of the environment, such as from map 828, and may continuously determine the position and / or orientation of the autonomous vehicle within the map. In various embodiments, map 828 may include road section identifiers associated with various parts of roads on the map. In some embodiments, the road section identifiers may be associated with a road network at least partially managed by service computing device 832. In some examples, the localization component 820 may utilize methods such as SLAM (Simultaneous Localization and Mapping), CLAMS (Simultaneous Calibration, Localization and Mapping), relative SLAM, bundle adjustment, or nonlinear least squares optimization to receive image data, LiDAR data, radar data, IMU data, GPS data, and wheel encoder data, etc., in order to accurately determine the position of the autonomous vehicle. In some examples, the localization component 820 may provide data to various components of the vehicle 802 for determining the initial position of the autonomous vehicle in order to determine whether an object is related to the vehicle 802, as discussed herein.

[0124] In some embodiments, the sensing component 822 may include functionality to perform object detection, segmentation, and / or classification. In some embodiments, the sensing component 822 may provide processed sensor data indicating the presence of an object (e.g., an entity) in close proximity to the vehicle 802 and / or the classification of the object as an object type (e.g., a car, pedestrian, cyclist, animal, building, tree, road surface, curved surface, sidewalk, unknown, etc.). In some embodiments, the sensing component 822 may provide processed sensor data indicating the presence of a stationary entity in close proximity to the vehicle 802 and / or the classification of the stationary entity as a type (e.g., a building, tree, road surface, curved surface, sidewalk, unknown, etc.). In additional or alternative embodiments, the sensing component 822 may provide processed sensor data indicating a detected object (e.g., a tracked object) and / or one or more characteristics associated with the environment in which the object is located. In some embodiments, properties associated with an object may include, but are not limited to, x-position (global and / or local position), z-position (global and / or local position, e.g., height), orientation (e.g., roll, pitch, yaw), object type (e.g., classification), object velocity, object acceleration, object range (size), etc. Properties associated with the environment may include, but are not limited to, the presence of other objects in the environment, the state of other objects in the environment, zones associated with the environment (e.g., school zone, commercial area), date and time, day of the week, season, weather conditions, darkness / brightness indication, etc.

[0125] In general, the planning component 824 may determine a planned path 104 (e.g., an initial path, a planned route, etc.) for the vehicle 802 to follow as it traverses the environment. For example, the planning component 624 may determine various paths and trajectories as well as various levels of detail. For example, the planning component 824 may determine a path to move from a first position (e.g., current position) to a second position (e.g., target position). In some embodiments, the planning component 824 may generate instructions to guide the autonomous vehicle 802 along at least part of the path from the first position to the second position.

[0126] In various embodiments, the planning component 824 may be configured to receive data associated with waypoints and / or headings for the vehicle 802 to operate in remote guidance mode from a remote guidance platform 830 of one or more service computing devices 832. In some embodiments, the planning component 824 may be configured to validate the waypoints and / or headings. In at least one embodiment, the planning component 824 may determine how to guide the autonomous vehicle 802 from a first waypoint received from a service computing device 832 to a second waypoint received from a service computing device 832. In some embodiments, the instructions may be a trajectory or part of a trajectory. In some embodiments, multiple trajectories may be generated substantially simultaneously (e.g., within technical tolerances) according to Receding Horizon technique, where one of the multiple trajectories is selected for the vehicle 802 to navigate.

[0127] In some embodiments, the planning component 824 may include a prediction component for generating predicted trajectories of objects in the environment (e.g., objects). For example, the prediction component may generate one or more predicted trajectories for a dynamic object detected in the environment. In some embodiments, the predicted trajectory may include any number of potential paths, in which the detected object can move from its current position (e.g., at the time of detection) and / or move based on the direction of movement. In some embodiments, the predicted trajectory may represent the distance and direction in which the object moves over a period of time. In some embodiments, the prediction component may measure the trace of the object and generate trajectories for the object based on observed and predicted behavior. In various embodiments, the prediction component may determine the predicted trajectory based on one or more of the following: machine learning techniques, heatmaps, temporal logic formulas, and / or tree search methods.

[0128] In at least one embodiment, the vehicle computing device(s) 804 may include one or more system controllers 826 that can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 802. The system controller(s) 826 may communicate with and / or control the corresponding systems of the drivetrain(s) 814 and / or other components of the vehicle 802.

[0129] Memory 818 may further include one or more maps 828 that can be used by the vehicle 802 to navigate within the environment. For the purposes of this discussion, the maps may be any number of data structures modeled in two, three, or N dimensions, capable of providing information about the environment, such as topology (intersections, etc.), streets, mountain ranges, roads, terrain, and the environment itself. In some examples, the maps may include, but are not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, and HSV / HSL color information, etc.), intensity information (e.g., Lider information and radar information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfaces" (e.g., polygons associated with individual colors and / or intensities)), reflectivity information (e.g., specular reflectivity information, retroreflectivity information, BRDF information, and BSSRDF information, etc.). In one embodiment, the map may include a three-dimensional mesh of the environment. In this embodiment, the vehicle 802 may be controlled at least in part based on a map(s) 828. That is, the map(s) 828 may be used in conjunction with a positioning component 820, a sensing component 822, and / or a planning component 824 to determine the position of the vehicle 802, detect objects in the environment, generate a route, and determine actions and / or trajectories for navigating the environment. In various embodiments, the map(s) 828 may include a road network. The road network may include one or more different road sections.

[0130] In some embodiments, one or more maps 828 may be stored in one or more service computing devices 832. In some embodiments, multiple maps 828 may be stored based on characteristics, for example, entity type, date and time, day of the week, season, etc. Storing multiple maps 828 has similar memory requirements but increases the speed at which data in the maps can be accessed.

[0131] For ease of understanding, the components discussed herein (e.g., the localization component 820, the sensing component 822, the planning component 824, one or more system controllers 826, and one or more maps 828) are described as being separated for illustrative purposes only. However, operations performed by various components may be combined or performed by any of the other components.

[0132] In some examples, some or all aspects of the components discussed herein may include any model, technique, and / or machine learning technique. For example, in some examples, the components in memory 818 (and memory 834 discussed below) may be implemented as a neural network.

[0133] As described herein, an exemplary neural network is a technique that learns from living organisms, passing input data through a series of connected layers to produce an output. Each layer in the neural network may include another neural network, or any number of layers (whether or not they are convolutional layers). As can be understood in the context of this disclosure, a neural network may utilize machine learning, which can refer to a broad class of such techniques in which an output is produced based on learned parameters.

[0134] While discussed in the context of neural networks, any type of machine learning may be used consistently with this disclosure. For example, machine learning techniques include, but are not limited to, regression techniques (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression spline (MARS), locally estimated scatter plot smoothing (LOESS), instanced techniques (e.g., ridge regression, least absolute contraction and selection operator (LASSO), elastic nets, least angular regression (LARS)), decision tree techniques (e.g., classification and regression tree (CART), iterative binary 3 (ID3), chi-squared automatic reciprocal Action detection (CHAID), decision stocks, conditional decision trees, Bayesian techniques (e.g., Naive Bayes, Gaussian Naive Bayes, Multinomial Naive Bayes, Averaged One-Dependency Estimator (AODE), Bayesian Belief Network (BNN), Bayesian Network), Clustering techniques (e.g., k-means, k-central, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning techniques (e.g., Perceptron, Backpropagation, Hopfield Network, Radial Basis Function Network) This may include RBFN), deep learning techniques (e.g., deep Boltzmann machine (DBM), deep belief network (DBN), convolutional neural network (CNN), stacked autoencoder), dimensionality reduction techniques (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), summon mapping, multidimensional scaling, projection tracking, linear discriminant analysis (LDA), mixture discriminant analysis (MDA), quadratic discriminant analysis (QDA), smooth discriminant analysis (FDA)), ensemble techniques (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (blending), gradient boosting machine (GBM), gradient boosted regression tree, random forest), SVM (support vector machine), supervised learning, unsupervised learning, semi-supervised learning, etc. Additional examples of architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, and PointNet.

[0135] In at least one embodiment, the sensor system(s) 806 may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measuring unit (IMU), accelerometer, magnetometer, gyroscope, etc.), cameras (e.g., RGB, IR, intensity, depth, time of flight, etc.), microphones, wheel encoders, and environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.). The sensor system(s) 806 may include multiple instances of each of those sensors or other types of sensors. For example, the lidar sensors may include individual lidar sensors located at the corners, front, rear, sides, and / or top of the vehicle 802. In another embodiment, the camera sensors may include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle 802. The sensor system(s) 806 can provide input to the vehicle computing device(s) 804.

[0136] Vehicle 802 may also include one or more emitters 808 for emitting light and / or sound. The emitters 808 may include internal voice and visual emitters for communicating with occupants of vehicle 802. Internal emitters may include, but are not limited to, speakers, lights, signs, display screens, touchscreens, tactile emitters (e.g., vibration and / or force feedback), and mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.). The emitters 808 may also include external emitters. External emitters may include, but are not limited to, lights or other indicators of vehicle action for signaling the direction of movement (e.g., indicator lights, signs, light arrays, etc.), and one or more voice emitters for audible communication with pedestrians or other nearby vehicles (e.g., speakers, speaker arrays, horns, etc.), one of which may include acoustic beam steering technology.

[0137] The vehicle 802 and / or service computing device(s) 832 may also include one or more communication connections(s) 810 that enable communication between the vehicle 802 and the service computing device(s) 832 and / or other local computing devices(s) on the vehicle 802 and / or drivetrain(s) 814. The communication connections(s) 810 may also enable the vehicle 802 to communicate with other neighboring computing devices(s) (e.g., other neighboring vehicles).

[0138] The communication connection(s) 810 may include physical and / or logical interfaces for connecting the vehicle computing device(s) 804 to another computing device or network(s) 836 or other network(s). For example, the communication connection(s) 810 may enable Wi-Fi communication via frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth, cellular communication (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol that allows each computing device(s) to interface with other computing devices(s).

[0139] In at least one embodiment, the vehicle 802 may include one or more drive systems 814. In some embodiments, the vehicle 802 may have a single drive system 814. In some embodiments, if the vehicle 802 has multiple drive systems 814, each drive system 814 may be located at opposite ends of the vehicle 802 (e.g., front and rear). In at least one embodiment, the drive system 814 may include one or more sensor systems for detecting the conditions around the drive system 814 and / or the vehicle 802. As an example and not an example, the sensor system 814 may include one or more wheel encoders (e.g., rotary encoders) for detecting the rotation of the drive system's wheels, inertial sensors (e.g., inertial measuring units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive system, cameras or other image sensors, ultrasonic sensors, lidar sensors, radar sensors, etc. for acoustically detecting objects around the drive system. Some sensors, such as wheel encoders, may be unique to the drive system(s) 814. In some cases, the sensor system(s) on the drive system(s) 814 may overlap with or complement the corresponding system(s) on the vehicle 802 (e.g., sensor system(s) 806).

[0140] The drivetrain(s) 814 may include a number of vehicle systems, which may include a high-voltage battery, a motor for propelling the vehicle, an inverter for converting DC current from the battery to AC current for use by other vehicle systems, a steering system including a steering motor and steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stabilization control system for distributing braking force to reduce traction losses and maintain control, an HVAC system, lighting (e.g., headlights / taillights for illuminating the exterior of the vehicle), and one or more other systems (e.g., a cooling system, safety systems, an onboard charging system, other electric components such as DC / DC converters, a high-voltage junction, high-voltage cables, a charging system, a charging port, etc.). In addition, the drivetrain(s) 814 may include a drivetrain controller capable of receiving and preprocessing data from sensor systems(s) and controlling the operation of various vehicle systems. In some embodiments, the drivetrain controller may include one or more processors and memory communicably coupled to one or more processors. The memory may store one or more systems for performing various functionalities of the drive system 814. Furthermore, the drive system 814 may also include one or more communication connections that enable communication between one or more other local computing devices or remote computing devices by their respective drive systems.

[0141] In at least one embodiment, a direct connection(s) 812 can provide a physical interface for connecting the body of the vehicle 802 to one or more drive systems(s) 814. For example, a direct connection(s) 812 can enable the transfer of energy, fluids, gases, data, etc., between the drive system(s) 814 and the vehicle. In some examples, the direct connection(s) 812 may further removably fix the drive system(s) 814 to the body of the vehicle 802.

[0142] In at least one embodiment, the location component 820, the sensing component 822, the planning component 824, one or more system controllers 826, and one or more maps 828 may process sensor data as described above and transmit their outputs to one or more service computing devices 832 through one or more networks 836. In at least one embodiment, the location component 820, the sensing component 822, the planning component 824, one or more system controllers 826, and one or more maps 828 may transmit their respective outputs to the service computing device 832 in near real-time, for example, at a specific frequency after a predetermined period of time has elapsed.

[0143] In some embodiments, the vehicle 802 may transmit sensor data to a service computing device 832 via a network 836. In some embodiments, the vehicle 802 may receive sensor data from a service computing device 832 via a network 636. The sensor data may include raw sensor data and / or processed sensor data, and / or representations of the sensor data. In some embodiments, the sensor data (raw or processed) may be transmitted and / or received as one or more log files.

[0144] The service computing device(s) 832 may include one or more processors 842 and memory 834 storing a remote guidance component 830 having a graphical user interface 838. The graphical user interface 838 may be configured to receive inputs from an operator representing one or more waypoints and / or headings for providing remote guidance to the vehicle 802, such as those described above. The inputs may be received via one or more input / output devices 840. The input / output devices 840 may include one or more of the following: a keyboard, a mouse, a touchscreen display, a haptic device, a microphone, a camera, and / or other devices configured to input and / or output data to / from the service computing device(s) 832.

[0145] The processor(s) 816 of vehicle 802 and the processor(s) 842 of service computing device(s) 832 may be any suitable processor having the ability to process data and execute instructions for performing the operations described herein. For example and not limited to, processor(s) 816 and 842 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or part of a device that processes electronic data to convert it into other electronic data that can be stored in registers and / or memory. In some embodiments, integrated circuits (e.g., ASICs), gate arrays (e.g., FPGAs), and other hardware devices may also be considered processors insofar as they are configured to implement encoded instructions.

[0146] Memory 818 and 834 are examples of non-temporary computer-readable media. Memory 818 and 834 may store an operating system, as well as one or more software applications, instructions, programs, and / or data for implementing the methods and functions belonging to various systems described herein. In various implementations, the memory may be implemented using any suitable memory technology, such as static random-access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash-type memory, or any other type of memory having the ability to store information. The architectures, systems, and individual elements described herein may include a number of other logical, programmatic, and physical components, shown in the accompanying drawings, which are merely examples and relevant to the discussion herein.

[0147] In some examples, memories 818 and 834 may include at least working memory and storage memory. For example, the working memory may be a limited-capacity, high-speed memory (e.g., cache memory) used to store data that will be operated on by processors 816 and 842. In some examples, memories 818 and 834 may also include storage memory, which may be a relatively large-capacity, low-speed memory used for long-term storage of data. In some cases, processors 816 and 842 may not be able to operate directly on data stored in storage memory, and the data may need to be loaded into working memory in order to perform operations on the data, as discussed herein.

[0148] Exemplary clause A: A system comprising one or more processors and a memory storing processor-executable instructions, wherein, when executed by the one or more processors, the processor-executable instructions receive a request from a vehicle computing system associated with a vehicle operating on a path for guidance about navigating around obstacles in an environment, receive sensor data associated with one or more sensors of the vehicle, determine that conditions for providing remote guidance are met, and generate a signal configured to output a representation of the vehicle and the obstacles on a display on a user interface associated with the guidance system, wherein the representation is at least partially based on the sensor data, and via the user interface, a first position and a first The system is configured to receive a first input including a first waypoint associated with a bearing, wherein the first bearing includes a first direction for the vehicle to face at the first waypoint, and transmit the first waypoint to the vehicle computing system; receive a second input via the user interface including a second waypoint associated with a second position and a second bearing, wherein the second bearing includes a second direction for the vehicle to face at the second waypoint, and transmit the second waypoint to the vehicle computing system; determine that termination conditions for providing guidance are met; and transmit commands to the vehicle computing system to autonomously control the vehicle.

[0149] B: The system according to Clause A, wherein the conditions include at least one of the following: latency corresponding to a connection to the vehicle computing system below a threshold latency; bandwidth corresponding to a connection to the vehicle computing system below a threshold bandwidth; speed of the vehicle below a threshold speed; the number of sensors available to provide the sensor data exceeding a threshold number; or a determination that the vehicle does not have any associated health-related malfunctions.

[0150] C: The system according to clause A or B, characterized in that the instruction determines that the first position, the first orientation, the second position, and the second orientation satisfy at least one of a safety protocol or remote guidance protocol associated with the operation of the vehicle, the remote guidance protocol includes one or more restrictions associated with the movement of the vehicle while operating in remote guidance, and the system further configures the system to verify the first position, the first orientation, the second position, and the second orientation at least in part on satisfying the at least one of the safety protocol or the remote guidance protocol.

[0151] D: The system according to any one of clauses A to C, wherein determining that the termination conditions for providing guidance are met includes determining that the second waypoint is within a threshold distance of the path, and determining that the obstacle is behind the vehicle, based at least in part on the threshold distance.

[0152] E: The system according to any one of clauses A to D, characterized in that the instruction determines that it is impossible for the vehicle to proceed along the route to the destination, generates an updated route for vehicle guidance to the destination, transmits the updated route to the vehicle computing system, and the instruction proceeding autonomously includes the updated route.

[0153] F: A method comprising the steps of: receiving a request from a vehicle computing system associated with a vehicle for guidance input for navigating around an obstacle; receiving, as received data and from the vehicle computing device, sensor data or data based at least in part on the sensor data; receiving, via a user interface associated with the guidance system, a first input corresponding to a first waypoint, wherein the first input is based at least in part on the received data; transmitting to the vehicle computing device first data associated with the first waypoint; and causing the vehicle to control at least in part on the first waypoint.

[0154] The method of clause F, further comprising the step of determining that conditions for providing remote guidance are met, wherein the conditions include at least one of: latency corresponding to a connection to the vehicle computing system below a threshold latency; bandwidth corresponding to a connection to the vehicle computing system below a threshold bandwidth; speed of the vehicle below a threshold speed; the number of sensors available to provide the received data being greater than or equal to a threshold number; or determination that the vehicle does not have any associated health-related malfunctions.

[0155] H: The method according to clause F or G, further comprising the steps of: determining in a first time that the conditions for providing remote guidance are met; enabling input to the user interface at least in part on the conditions being met; and determining in a second time that the conditions for providing remote guidance are not met; and disabling the input to the user interface at least in part on the conditions being not met.

[0156] I: The method according to any one of the claims F to H, further comprising: determining that the first waypoint satisfies at least one of a safety protocol or a remote guidance protocol associated with the operation of the vehicle, wherein the remote guidance protocol includes one or more restrictions associated with the movement of the vehicle while operating in remote guidance mode; and verifying the first waypoint at least partially on satisfying at least one of the safety protocol or the remote guidance protocol, wherein transmitting the first data is at least partially based on verifying the first waypoint.

[0157] J: The method according to any one of the claims F to I, further comprising the steps of: receiving a second input corresponding to a second waypoint; transmitting second data associated with the second waypoint to the vehicle computing device; and receiving a rejection message from the vehicle indicating rejection of the second waypoint, at least in part, based on at least one of a safety protocol or remote guidance protocol corresponding to the operation of the vehicle.

[0158] K: The method according to any one of the claims F to J, characterized in that the first input includes a first position associated with the first waypoint, the first position corresponds to a selectable area of ​​the user interface, and the first orientation corresponds to a first direction facing the vehicle at the first waypoint.

[0159] L: The method according to any one of the claims F to K, further comprising the steps of: transmitting a hold signal to the vehicle, the hold signal causing the vehicle computing device to hold at a certain position; and transmitting a release signal to the vehicle, the release signal including an instruction to the vehicle computing device to control the vehicle to the first waypoint.

[0160] The method according to any one of the claims F to L, further comprising: receiving a second input corresponding to a second waypoint; transmitting second data associated with the second waypoint to the vehicle computing device; determining that the second waypoint corresponds to a termination condition associated with the obstacle; and transmitting a command to the vehicle computing device to autonomously control the vehicle after completing an operation associated with the second waypoint.

[0161] N: The method according to any one of the claims F to M, further comprising: receiving a second input corresponding to a second waypoint; transmitting a second data associated with the second waypoint to the vehicle computing device; receiving a third input via the user interface indicating an operator's intention to edit at least one of the first waypoint or the second waypoint; determining that the vehicle is transitioning to the first waypoint; disabling a first edit to the first waypoint; determining at least one of the following: the second waypoint has not been accepted by the vehicle computing device or the vehicle has not moved beyond the first waypoint; disabling a second edit to the second waypoint; and transmitting at least one command to the vehicle computing device to delete or modify the second waypoint.

[0162] O: A system or device comprising a processor and a non-temporary computer-readable medium storing instructions, wherein, when the instructions are executed, the system or device causes the processor to perform the method described in any one of clauses F to N.

[0163] P: A system or device comprising processing means and storage means coupled to the processing means, wherein the storage means includes instructions that configure one or more devices to perform the method described in any one of the clauses F to N.

[0164] Q: A non-temporary computer-readable medium storing instructions, wherein, when executed, the instructions cause one or more processors to perform an operation including: receiving a request from a vehicle computing system associated with a vehicle for guidance input to navigate around obstacles; receiving, as received data and from the vehicle computing device, sensor data or data at least in part based on the sensor data; receiving, via a user interface associated with the guidance system, a first input corresponding to a first waypoint, the first input being at least in part based on the received data; transmitting to the vehicle computing device first data associated with the first waypoint; and causing the vehicle to be controlled at least in part based on the first waypoint.

[0165] R: The non-temporary computer-readable medium according to clause Q, further comprising receiving a second input via the user interface indicating a command for the vehicle to stop, and transmitting a command to the vehicle computing device to stop at a location associated with the current location.

[0166] S: The non-temporary computer-readable medium according to clause Q or R, characterized in that the operation further comprises determining that the vehicle has stopped at a certain position and transmitting a command to release the vehicle from the stopped position, the vehicle being able to proceed at least in part based on the command.

[0167] T: The non-temporary computer-readable medium according to any one of the clauses Q to S, further comprising: receiving a second input corresponding to a second waypoint; transmitting second data associated with the second waypoint to the vehicle computing device; receiving a third input via the user interface indicating the operator's intention to edit at least one of the first waypoint or the second waypoint; determining that the vehicle is transitioning to the first waypoint; disabling a first edit to the first waypoint; determining at least one of the following: that the second waypoint is not accepted by the vehicle computing device or that the vehicle is not moving forward beyond the first waypoint; disabling a second edit to the second waypoint; and transmitting at least one command to the vehicle computing device to delete or modify the second waypoint.

[0168] U: A non-temporary computer-readable medium according to any one of clauses Q to T, further comprising: receiving an indication that the vehicle is unable to proceed from a certain position; presenting the indication that the vehicle is unable to proceed from the position via the user interface; receiving a second input corresponding to a second waypoint via the user interface; and transmitting the second waypoint to the vehicle computing system.

[0169] V: The non-temporary computer-readable medium according to any one of the clauses Q to U, wherein the operation further comprises determining that the first waypoint satisfies a kinematic check associated with vehicle control, and verifying the first waypoint at least in part on satisfying the kinematic check, and causing the vehicle to be controlled is at least in part on verifying the first waypoint.

[0170] W: A vehicle comprising a sensor, one or more processors, and a memory storing processor-executable instructions, wherein the processor-executable instructions, when executed by the one or more processors, determine a path for the vehicle to traverse through the environment to reach a destination, receive sensor data from the sensor, generate a first trajectory for the vehicle to traverse along the path, determine, at least in part, based on the first trajectory, determine that it is impossible for the vehicle to continue along the path based, at least in part, on one or more of the obstacles or control policies, send a request for guidance data to a service computing device, receive waypoint data from the service computing device, including waypoints associated with position and orientation, inspect the waypoints, at least in part, based on a safety protocol, determine a second trajectory for the vehicle to navigate from an initial position and initial vehicle orientation to the position and orientation associated with the waypoints, at least in part, based on the second trajectory, and configure the vehicle to control the vehicle.

[0171] X: The vehicle according to clause W, characterized in that the instruction receives a message indicating the release of the service computing device as a remote guidance platform, determines a third trajectory for controlling the vehicle from the waypoint along the route, and programs one or more processors to control the vehicle based at least in part on the third trajectory.

[0172] Y: The vehicle according to clause W or X, wherein the waypoint data includes first waypoint data including a first waypoint associated with a first position and a first orientation, the instruction receives second waypoint data from the service computing device including a second waypoint associated with a second position and a second orientation, verifies the second waypoint at least in part on the safety protocol, determines a third trajectory for the vehicle to navigate the vehicle from the first waypoint to the second waypoint at least in part on verifying the second waypoint, and controls the vehicle from the first waypoint to the second waypoint at least in part on the third trajectory, the vehicle being controlled from the first waypoint to the second waypoint without stopping its forward movement.

[0173] Z: The vehicle according to any one of the terms W to Z, wherein the command further configures the vehicle to detect a dynamic object in the environment, determine an object trajectory associated with the object, determine an object trajectory that the object trajectory crosses within a threshold distance of the vehicle moving along the trajectory, and control the vehicle, at least in part, based on the object.

[0174] AA: The vehicle according to any one of clauses A to D, characterized in that the command further configures the vehicle to receive second waypoint data associated with a second position and a second vehicle orientation, determine that at least one of the second waypoint or the second vehicle orientation violates the safety protocol, and stop the vehicle at the first waypoint.

[0175] AB: A method comprising the steps of: receiving sensor data from sensors of a vehicle operating along a path in an environment; determining, at least in part, based on a control policy, that it is impossible for the vehicle to continue along the path; sending a request to a service computing device for guidance data, wherein the guidance data includes waypoints to facilitate the control of the vehicle through the scenario; receiving waypoint data associated with the waypoints from the service computing device; determining that the waypoints are valid waypoints; and controlling the vehicle at least in part based on the waypoints.

[0176] AC: The method of clause AB, further comprising the steps of receiving a vehicle heading associated with the waypoint, determining that the vehicle heading is a valid vehicle heading, and controlling the vehicle at least in part based on the vehicle heading.

[0177] AD: The method of clause AB or AC, characterized in that determining that the waypoint is the valid waypoint includes determining that the waypoint is located on a drivable plane, determining that the operation of the vehicle navigating to the waypoint does not violate a remote guidance protocol, wherein the remote guidance protocol includes restrictions associated with the movement of the vehicle while operating in remote guidance mode, or determining that the operation of the vehicle navigating to the waypoint does not violate a safety protocol, wherein the safety protocol includes conditions for ensuring the safety of the vehicle.

[0178] AE: The method according to any one of the provisions AB to AD, wherein the scenario is identified within an environment associated with the vehicle, and the method further comprises: receiving an updated route from the service computing device for navigating through the environment based at least in part on the scenario; receiving a message indicating the release of the service computing device as a remote guidance platform; and controlling the vehicle based at least in part on the updated route.

[0179] AF: The method according to any one of the terms AB to AE, wherein the scenario is identified on a route associated with the vehicle, and the method further comprises: receiving a message from the service computing device indicating the release of the service computing device as a remote guidance platform; controlling the vehicle based at least in part on the route; identifying a second scenario on the route that violates operational constraints; transmitting a second request to the service computing device for guidance data; determining a solution to the second scenario independent of input from the service computing device; and controlling the vehicle based at least in part on the solution.

[0180] AG: The method according to any one of clauses AB to AF, wherein the waypoint data associated with the waypoint includes first data associated with a first waypoint, and the method further comprises: receiving second waypoint data associated with a second position and a second vehicle orientation; determining that at least one of the second waypoint or the corresponding vehicle orientation violates a safety protocol; and stopping the vehicle at the first waypoint at least in part based on the determination that the second waypoint violates the safety protocol.

[0181] AH: A method according to any one of the clauses A to AG, characterized in that determining that it is impossible for the vehicle to continue along the path includes determining the path for the vehicle to traverse through the environment, generating a trajectory for the vehicle to traverse along the path based at least in part on the sensor data, and determining that the trajectory violates the control policy.

[0182] AI: The method according to any one of the terms A to AH, further comprising the steps of: receiving second waypoint data associated with a second waypoint and a corresponding vehicle heading; verifying the second waypoint and the corresponding vehicle heading; and controlling the vehicle from the first waypoint to the second waypoint without stopping its forward movement.

[0183] AJ: The method according to any one of clauses AB to AI, characterized in that the control policy includes at least one of traffic laws or rules, rules of good driving, or obstacles in the path of the vehicle.

[0184] AK: A system or device comprising a processor and a non-temporary computer-readable medium storing instructions, wherein, when the instructions are executed, they cause the processor to perform the method described in any one of the clauses A to AJ.

[0185] A system or device comprising: processing means and storage means coupled to the processing means, wherein the storage means includes instructions that configure one or more devices to perform the method described in any one of the clauses A to AJ.

[0186] AM: A non-temporary computer-readable medium storing instructions, wherein, when executed, the instructions cause one or more processors to perform an operation comprising: receiving sensor data from a sensor; determining, at least in part, based on a control policy, that it is impossible for the vehicle to continue along its path; sending a request to a service computing device for guidance data, the guidance data including waypoints to facilitate the control of the vehicle through the scenario; receiving waypoint data associated with the waypoints from the service computing device; determining that the waypoints are valid waypoints; and controlling the vehicle, at least in part, based on the waypoints.

[0187] AN: The non-temporary computer-readable medium according to Clause AM, further comprising: receiving direction data corresponding to the direction associated with the waypoint; determining that the vehicle direction is a valid direction; and controlling the vehicle at least in part based on the direction.

[0188] AO: A non-temporary computer-readable medium according to clause AM or AN, characterized in that controlling the vehicle at least in part based on the waypoint includes identifying the scenario and determining the vehicle trajectory from an initial position associated with the waypoint, and operating the vehicle in the drivetrain at least in part based on the vehicle trajectory.

[0189] AP: A non-temporary computer-readable medium according to any one of clauses AM to AO, wherein the waypoint data associated with the waypoint includes first waypoint data associated with the first waypoint received in a first time, the operation further includes receiving second waypoint data associated with the second waypoint from the service computing device in a second time, determining that the second waypoint is valid, and controlling the vehicle at least in part based on the second waypoint, the vehicle being configured to transition from the first waypoint to the second waypoint without stopping.

[0190] AQ: A non-temporary computer-readable medium according to any one of clauses AM to AP, wherein the waypoint data associated with the waypoint includes first waypoint data associated with a first waypoint received in a first time, and the operation further includes determining that the vehicle is within a threshold distance of the first waypoint, determining that second waypoint data associated with a second waypoint has not been received, and stopping the vehicle at the first waypoint.

[0191] AR: A non-temporary computer-readable medium according to any one of clauses AM to AQ, further comprising: the scenario being identified within an environment associated with the vehicle; the operation receiving a message from the service computing device indicating the release of the service computing device as a remote guidance platform; controlling the vehicle at least in part on the path; identifying a second scenario on the path that violates operational constraints; transmitting a second request to the service computing device for guidance data; determining a solution to the second scenario independent of input from the service computing device; and controlling the vehicle at least in part on the solution.

[0192] Although the exemplary provisions described above have been described in relation to one particular implementation, it should be understood that in the context of this specification, the content of the exemplary provisions may also be implemented through methods, devices, systems, computer-readable media, and / or other implementations. In addition, any of Examples A to AR may be implemented alone or in combination with one or more other Examples A to AR.

[0193] conclusion While one or more embodiments of the techniques described herein have been described, various alternatives, pairs, substitutions, and equivalents thereof are included within the scope of the techniques described herein.

[0194] The description of the embodiments includes references to accompanying drawings that form part thereof, shown in a manner that illustrates specific examples of the claimed subject matter. It will be understood that other examples may be used, and that changes or modifications, such as structural changes, may be made. In such embodiments, the changes or modifications do not necessarily deviate from the intended scope of the claimed subject matter. While the steps in this specification may be presented in a particular order, in some cases the ordering may be changed so that certain inputs are provided at different times or in different orders without changing the function of the described system and method. The disclosed procedures may also be performed in different orders. In addition, various calculations that do not need to be performed in the order disclosed herein, and other examples using calculations in alternative orders, can be readily implemented. In addition to being rearranged, calculations may be broken down into lower-level calculations having the same results.

Claims

1. Sensors and, One or more processors, A vehicle comprising a memory storing processor-executable instructions, wherein when the processor-executable instructions are executed by one or more processors, The sensor receives sensor data from the aforementioned sensor, Based at least in part on the control policy, it is determined that it is impossible for the vehicle to continue along its path through the environment, Send a request for guidance data to the service computing device. The guidance data includes waypoints to facilitate the control of the vehicle through the scenario. If one or more conditions for provisioning and receiving the guidance data are met, the service computing device receives the waypoint data associated with the waypoint. The waypoint is determined to be a valid waypoint, in part on the determination that the waypoint satisfies a predetermined set of conditions associated with vehicle safety. The vehicle is controlled at least partially based on the waypoints. A vehicle characterized by being configured in such a manner.

2. The aforementioned instruction is, As a remote guidance platform, it receives a message indicating the release of the service computing device. A trajectory for controlling the vehicle from the waypoint along the aforementioned path is determined, The vehicle is controlled, at least partially based on the aforementioned track. The vehicle according to claim 1, characterized in that the vehicle is further configured as described above.

3. The waypoint data includes first waypoint data including a first waypoint associated with a first position and a first orientation, and the instruction is, The service computing device receives second waypoint data, including a second waypoint associated with a second position and a second orientation. The second waypoint is verified based at least in part on the safety protocol, Based at least in part on verifying the second waypoint, the trajectory for the vehicle to navigate from the first waypoint to the second waypoint is determined. The vehicle is further configured to control the vehicle from the first waypoint to the second waypoint based at least partially on the aforementioned track, and the vehicle is controlled from the first waypoint to the second waypoint without stopping its forward movement. The vehicle according to feature 1.

4. The aforementioned instruction is, Based at least partially on the aforementioned sensor data, dynamic objects in the environment are detected. Determine the object trajectory associated with the aforementioned object, The object that the object's trajectory crosses within a threshold distance of the vehicle moving on the vehicle trajectory associated with the waypoint is determined. The vehicle is controlled based at least partially on the aforementioned object. The vehicle according to any one of claims 1 to 3, characterized in that the vehicle is further configured as described above.

5. The waypoint data includes first waypoint data, and the instruction is The system receives second waypoint data associated with a second position and a second vehicle orientation. If it is determined that at least one of the second position or the second vehicle orientation violates the safety protocol, The vehicle is stopped at the first waypoint. The vehicle according to any one of claims 1 to 4, characterized in that the vehicle is further configured as described above.

6. A method performed by a vehicle, comprising the steps of receiving sensor data from sensors of the vehicle operating along a path in an environment, A step of determining, at least in part, that it is impossible for the vehicle to continue along the said path, A step of sending a request for guidance data to a service computing device, wherein the guidance data includes waypoints for facilitating the control of the vehicle through a scenario, If one or more conditions for provisioning and receiving the guidance data are met, the service computing device receives waypoint data associated with the waypoint; The steps include determining that the waypoint is a valid waypoint, in part based on determining that the waypoint satisfies a predetermined set of conditions associated with vehicle safety, A method characterized by comprising the step of controlling the vehicle based at least partially on the waypoint.

7. The steps include receiving the vehicle heading associated with the waypoint, The steps include determining that the vehicle direction is a valid vehicle direction, A step of controlling the vehicle based at least partially on the vehicle orientation, The method according to 6, further comprising the following:

8. The step of determining that the waypoint is the valid waypoint is: Determining that the waypoint is located on the drivable surface, Determining that the vehicle's actions in navigating to the waypoint do not violate the remote guidance protocol, wherein the remote guidance protocol includes restrictions associated with the vehicle's movement while operating in remote guidance mode, or Determining that the actions of the vehicle navigating to the waypoint do not violate a safety protocol, wherein the safety protocol includes conditions for ensuring the safety of the vehicle. The method according to 6 or 7, characterized by including the step of determining at least one of the following.

9. The aforementioned scenario is identified within the environment associated with the vehicle, and the method is The steps include receiving an updated route from the service computing device for navigating through the environment based at least partially on the scenario, The steps include receiving a message indicating the release of the service computing device as a remote guidance platform, A step of controlling the vehicle based at least partially on the updated route, The method according to any one of claims 6 to 8, further comprising the above.

10. The aforementioned scenario is identified on the route associated with the vehicle, and the method is The steps include receiving a message from the service computing device indicating the release of the service computing device as a remote guidance platform, A step of controlling the vehicle based at least partially on the aforementioned route, A step of identifying a second scenario on the path that violates the operational constraints, The steps include sending a second request for guidance data to the service computing device, A step of determining a solution to the second scenario, independent of the input from the service computing device, A step of controlling the vehicle based at least in part on the above solution, The method according to any one of claims 6 to 9, further comprising the above.

11. The waypoint data associated with the waypoint includes first data associated with a first waypoint, and the method is The steps include receiving second waypoint data associated with a second position and a second vehicle orientation, The steps include determining that at least one of the second waypoint or the second vehicle orientation violates the safety protocol, A step of stopping the vehicle at the first waypoint, at least in part on determining that the second waypoint violates the safety protocol, The method according to any one of claims 6 to 10, further comprising the following:

12. The step of determining that it is impossible for the vehicle to continue along the aforementioned path is: A step of determining the path the vehicle will take to traverse the environment, A step of generating a trajectory for the vehicle to cross along the path, based at least partially on the sensor data, The steps include determining that the trajectory violates the control policy, The method according to any one of claims 6 to 11, characterized in that it includes

13. The steps include receiving a second waypoint and second waypoint data associated with the corresponding vehicle heading, A step of verifying the second waypoint and the corresponding vehicle orientation, A step of controlling the vehicle from the first waypoint to the second waypoint without stopping its forward movement, The method according to any one of claims 6 to 12, further comprising the above.

14. The step of controlling the vehicle is: A step of determining the vehicle trajectory from an initial position to the waypoint associated with identifying the aforementioned scenario, The steps include: causing the drive system to operate the vehicle based at least partially on the aforementioned vehicle track; The method according to any one of claims 6 to 13, characterized in that it includes

15. A non-temporary computer-readable medium storing instructions, wherein, when the instructions are executed, they cause one or more processors to perform the method described in any one of claims 6 to 14.

Images