Remote guidance for autonomous vehicles

Abstract

Systems, methods, and computer-readable media are disclosed for a systems and methods for remote guidance for autonomous vehicles. An example method may include capturing, at a first time and by a camera of an autonomous vehicle, at least one of: an image or video feed of a first traffic signal at an intersection. The example method may also include classifying, based on the image or the video feed of the first traffic signal, a state of a color of the first traffic signal as unknown. The example method may also include halting movement of the autonomous vehicle at the intersection based on classifying the state of the color of the first traffic signal as unknown. The example method may also include sending a request for guidance to a remote operator device, the request including the image or video feed of the first traffic signal. The example method may also include receiving, from the remote operator device, a first guidance. The example method may also include performing, based on the first guidance, a first action including at least one of: remaining halted at the intersection or proceeding through the intersection.

Description

REMOTE GUIDANCE FOR AUTONOMOUS VEHICLESBACKGROUND

[0001] An autonomous vehicle (AV) traveling on public roads must be able to stop or proceed as appropriate when encountering traffic signals on the roads (for example, at in tersections). The AV may rely on computer vision to determine the active state of the traf fic signal, but computer vision sometimes fails to accurately read the state of the traffic signal. In these cases, where the state of the traffic signal is unknown, the AV may not know how to proceed at the signal.BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The detailed description is set forth with reference to the accompanying drawings.The drawings are provided for purposes of illustration only and merely depict example embodiments of the disclosure. The drawings are provided to facilitate understanding of the disclosure and shall not be deemed to limit the breadth, scope, or applicability of the disclosure. In the drawings, the left-most digit(s) of a reference numeral may identify the drawing in which the reference numeral first appears. The use of the same reference nu merals indicates similar, but not necessarily the same or identical components. However, different reference numerals may be used to identify similar components as well. Various embodiments may utilize elements or components other than those illustrated in the draw ings, and some elements and / or components may not be present in various embodiments. The use of singular terminology to describe a component or element may, depending on the context, encompass a plural number of such components or elements and vice versa.

[0003] FIG. 1 depicts a schematic illustration of an example system, in accordance with one or more example embodiments of the disclosure.

[0004] FIGs. 2A-2B depict an example use case, in accordance with one or more example embodiments of the disclosure.

[0005] FIGs. 3 A-3B depict an example use case, in accordance with one or more example embodiments of the disclosure.

[0006] FIG. 4 depicts an example flowchart, in accordance with one or more example embodiments of the disclosure.

[0007] FIG. 5 depicts an example flowchart, in accordance with one or more example embodiments of the disclosure.

[0008] FIG. 6 depicts an example method, in accordance with one or more example em bodiments of the disclosure.

[0009] FIG. 7 depicts an example method, in accordance with one or more example em bodiments of the disclosure.

[0010] FIG. 8 depicts a schematic illustration of an example computing device architec ture, in accordance with one or more example embodiments of the disclosure.DETAILED DESCRIPTION

[0011] This disclosure generally relates to, among other things, remote guidance for au tonomous vehicles (AVs). Particularly, the systems and methods describe herein may re late to situations where an AV is unable to determine a state of a traffic signal (the terms “traffic signal” and “traffic signal” may be used interchangeably herein) on a road. How ever, the systems and methods may also be applied to any other situation where an AV requires guidance from a remote operator station as well. The state of a traffic signal may refer to a color that the traffic signal is illuminating. For example, in some geographic lo cations, the traffic signal may illuminate a red color to indicate that vehicles should stop at the intersection or a green light to indicate that vehicles may proceed through the inter section. An AV traveling on a road must be able to stop or proceed as appropriate when encountering traffic signals on the roads (for example, at an intersection on the road). The AV may rely on computer vision to determine the active state of the traffic signal, but computer vision sometimes fails to accurately detect the state of the traffic signal. In these cases, where the state of the traffic signal is unknown to the AV, the AV may not know whether to stop at the intersection or proceed through the intersection. In some embodi ments, the AV may initially treat an unknown traffic signal as if the traffic signal is in a red light state. That is, if the AV reaches the intersection, and the state of the traffic signal is still unknown to the AV, then the AV may stop before the intersection and not proceed through the intersection. The AV may maintain this stopped position indefinitely until ei ther a remote operator station provides guidance to the AV or the sensors of the AV are eventually able to determine the current state of the traffic signal. Additionally, while stopped, the AV may activate its hazard indicators to signal to other vehicles that it isstopped, or may provide this indication to other vehicles on the road through any other mechanism as well (for example, vehicle-to-vehicle (V2V) communications). It should be noted that while reference may be made to AVs herein, these systems and methods and also be applicable to any other type of vehicle as well.

[0012] In some embodiments, when the AV is unable to determine the state of a traffic signal, it may send a request for guidance to a remote operator station. The remote opera tor station may be located remotely from the AV and the intersection, but the remote op erator station may also be integrated in the AV itself or a mobile device (for example, smartphone or other type of device) of a passenger of the AV as well. Upon receiving the request from the AV, a remote operator may, using the remote operator station, determine the state of the unknown traffic signal and may send an appropriate guidance to the AV.In some cases, guidance may refer to a message that may be provided from the remote operator station to the AV. The message may include an indication of an action for the AV to take. One example action may be to treat the intersection as an all-way stop. An other example action may be to treat the intersection as a partial-way stop. A third exam ple action may be to remain halted at the intersection. These are merely examples and any other actions may also be applicable. Upon receiving the guidance from the remote opera tor station, the AV may then negotiate the intersection based on the provided guidance (or may remain halted if the guidance is for the AV to remain halted).

[0013] In more detail, in some embodiments, the AV may be navigating a route through a road network including a number of intersections. The AV may have stored in local memory (or at a remote server) a pre-established map of the road network, which may specify, among other things, the presence of traffic signals at the intersections. When an upcoming section of the AV’s route contains an intersection including one or more traffic signals, the AV may perform one or more checks. First, the AV may determine whether the upcoming traffic signal is in a field of view of its sensors (for example, cameras, LI- DAR, RADAR, etc.). Second, if the traffic signal is in the field of view, the AV may de termine whether it is able to identify the state of the traffic signal. For example, the AV may capture one or more images and / or video feeds of the intersection and may use com puter vision to determine the state of the traffic signal (however, any other sensors may be used to identify the state of the traffic signal as well). If the traffic signal is within thefield of view, but not detected, the state of the signal may be classified as unknown. Like wise, the traffic signal may be in the field of view, and the traffic signal itself may be identified, but a state of the traffic signal may be unknown. When there is an upcoming unknown traffic signal along the AV’s route, the AV may then send a request for guid ance to the remote operator station over a wireless or wired Internet connection. This re quest for guidance may include any information that may assist the remote operator that is managing the remote operator station in providing an instruction to the AV. For example, the information may indicate the reason for the request (for example, an unknown traffic signal), the type of the upcoming intersection (for example, traffic signal), a distance the AV’s is from the intersection, an identifier associated with the traffic signal, which can be used by the remote operator to identify the location of the traffic signal on a map (which, in some instances, may be the same map that is used by the AV to perform navigation), a current location of the AV, as well as any other information that may be used by the re mote operator station to provide guidance to the AV.

[0014] In some embodiments, when the remote operator station receives a request for guidance from the AV, the remote operator may be presented with any of the information received in the request, as well as any other relevant information. For example, the re mote operator station may include a user interface that may be used to display one or more images and / or live video feeds from one or more cameras from the AV, a map of the location of the AV, and / or any other relevant data. The interface may also present any other types of information that may be relevant to the remote operator, including infor mation not received from the AV. For example, the interface may present a visual map that may be used to track the locations of one or more AVs in communication with the remote operator station (for example, the remote operator station may be in communica tion with a fleet of AVs). The remote operator may determine the true state of the traffic signal using any of this information (as well as any other information not listed above). That is, the remote operator may view the one or more images and / or video feeds and may identify that the traffic signal is illuminating a particular color. The remote operator may then provide guidance to the AV based on the color of the traffic signal. This may be accomplished by the remote operator providing an input to the remote operator station, where the remote operator station subsequently provides guidance to the AV based on the input from the remote operator. For example, if the traffic signal is solid red, the operatormay wait for the light state to change before sending the guidance (or may send the guid ance for the AV to remain halted at the intersection). If the traffic signal is solid yellow, the operator may wait for the light state to change before sending the guidance (or may send guidance for the AV to remain halted at the intersection or proceed through the in tersection depending on the length of time the yellow light has been illuminated). If the traffic signal is solid green or flashing yellow, the operator may provide guidance to the AV to proceed as if the intersection is an all-way stop. If the traffic signal is flashing red, the operator may provide guidance to the AV to proceed as if the intersection is a partial way stop (“all-way stop” and “partial-way stop” may be described in more detail below). When the AV receives the guidance, the AV may deactivate its hazard indicators and may proceed based on the guidance received from the remote operator station. It should be noted that although examples of guidance that may be provided when the AV is presented with certain traffic signal colors are provided herein, these are merely exemplary. For ex ample, the remote operator station may provide guidance for the AV to proceed as if the intersection is a partial-way stop when the traffic signal is green as well. Additionally, while it may be described herein that the remote operator may make determinations based on the information presented by the remote operator station, in some cases, the remote op erator station itself may automatically make the same determinations made by the remote operator. That is, in some cases, the remote operator may not be necessary, and guidance may be provided through the use of only the remote operator station. As one non-limiting example, the remote operator station may include artificial intelligence, or the like, to al low the remote operator station to analyze the images and / or video feeds provided by the AV, as well as any other relevant information.

[0015] In some embodiments, a remote operator may be able to indicate to the remote op erator station to provide a number of different types of guidance to the AV. A first exam ple of a type of guidance may include an indication for the AV to proceed as if the inter section is an all-way stop (for example, the AV should proceed as if the intersection in cludes stop signs or flashing red lights at each of the roads in the intersection). A second example of a type of guidance may include an indication for the AV to proceed as if the intersection is a partial-way stop (for example, the AV should proceed as if the road the AV is on includes a stop sign or a red light and the roads in the intersection that are per pendicular to the road that the AV is on have no stop sign or red light). The all-way stopand partial-way stop scenarios may be further illustrated through the use cases depicted in FIGs. 2-3. Based on the guidance provided by the remote operation, the AV may adjust how it approaches the intersection and / or predicts behaviors of other actors in the inter section, such as other vehicles. That is, the AV may adjust how it predicts when other ve hicles at the intersection will be traversing through the intersection or stopping in front of the intersection. The remote guidance provided by the remote operator station to the AV can authorize an AV to perform certain maneuvers, but may not instruct the AV on how or when to perform the maneuvers. Remote guidance may also not override the percep tion stack of the AV. That is, once the AV receives guidance from the remote operator, it may proceed or remain halted, but may proceed using its own sensors and internal pro cessing. For example, if the AV received guidance to proceed as if the intersection were a partial-way stop, then the AV may then use its own processing to identify when it may proceed into the intersection, and also to continuously monitor the environment while proceeding through the intersection. Continuing this example, even though the AV has received guidance to proceed, another vehicle at the intersection may illegally drive through a red light and into the intersection. Although it was instructed to proceed, the AV may still monitor for scenarios like this and adjust its movement correspondingly. This is just one non-limiting example of how the AV may still rely on internal processing even after being instructed to proceed.

[0016] In some embodiments, when provided with guidance to proceed as if the intersec tion is a partial-way stop, the AV may assume that cross-traffic in the intersection has right-of-way. In this mode of operation, when predicting the behavior of other vehicles at the intersection, the AV may assume that every cross-traffic vehicle may proceed through the intersection without stopping. The AV may assume this by default unless there is evi dence to the contrary that a specific vehicle in the intersection is yielding to the AV. For example, if a cross-traffic vehicle stops at the inlet to the intersection for a sufficiently long time, the AV may update its belief about that vehicle to assume that it is yielding right-of-way to the AV. The AV may only proceed into the intersection when there are no non-yielding cross-traffic vehicles approaching the intersection. The reason the remote operator may provide guidance to the AV to proceed as if the intersection is a partial-way stop when the traffic signal is flashing red is because neither the remote operator nor the AV can determine the state of the traffic signals that control the cross-traffic. If the cross-traffic has a traffic signal that is flashing yellow, the cross-traffic has right-of-way over the AV. If the cross-traffic also has a traffic signal that is flashing red, then the intersec tion is properly an all-way stop. Because the AV is operating under uncertainty, the safest action is the most conservative one, which is to treat the intersection as a partial-way stop.

[0017] In some embodiments, when provided with guidance to proceed as if the intersec tion is an all-way stop, the AV may assume that all vehicles approaching the intersection will yield mutually, and that right of way alternates around the intersection, such that ve hicles take turns proceeding. The AV may assume that other vehicles at the intersection will use this right-of-way sharing behavior as a prior assumption and assume that the ve hicle whose turn it is will proceed through the intersection unless there is evidence that the vehicle is yielding. For example, the AV may assume other vehicles stopping at cross- traffic inlets do not have right of way and are yielding, whereas vehicles approaching the intersection and not slowing are assumed not to be stopping. The AV may proceed on its own turn, assuming it is safe to do so and no other vehicles are observed to be proceeding out-of-turn. The reason the remote operator provides guidance to the AV to proceed as if the intersection is an all-way stop when the traffic signal is green is to avoid unsafe situa tions caused by remote operator error or poor network communication between the AV and remote operator station. If the remote operator were allowed to command the AV to proceed as if it has right-of-way, the remote operator could accidentally cause the AV to proceed without caution through a solid red traffic signal. Treating the intersection as an all-way stop causes the AV to proceed with more caution than it would through a solid green traffic signal.

[0018] In some embodiments, if at any time during this process, the AV’s perception of the traffic signal improves, the AV may become confident that it has detected the state of the traffic signal. If the AV confidently detects the traffic signal’s state before a remote operator station connects to the AV and sends a command, the AV may cancel the request for guidance and proceed as normal, acting on the traffic signal as the AV perceives it. If the traffic signal’s state is confidently detected after the AV receives a command from the remote operator station, the AV may act on the traffic signal as-perceived, and the com mand from the remote operator station may be disregarded.

[0019] Turning to the figures, FIG. 1 illustrates an example system 100, in accordance with one or more embodiments of this disclosure. In some embodiments, the system 100may include one or more vehicles (for example, vehicle 112, vehicle 114, vehicle 116, and / or any other vehicle). The system 100 may also include one or more remote operator stations 150. In some instances, any of the one or more vehicles and the one or more re mote operator stations 150 may communicate over a communications network 108.

[0020] In some embodiments, the one or more vehicles may be autonomous vehicles (AVs). The one or more vehicles may navigate one or more routes through a road net work including a number of intersections (for example, intersection 102, as well as any other number of intersections). The one or more vehicles may have stored in local memory 135 (or at a remote server) a pre-established map of the road network, which may specify, among other things, the presence of traffic signals (for example, traffic sig nal 104 and / or traffic signal 106) at the intersections. When an upcoming section of the vehicle’s route contains an intersection 102 including one or more traffic signals, the ve hicle may perform one or more checks. First, the vehicle may determine whether the up coming traffic signal is in the field of view of its sensors (for example, cameras, LIDAR, RADAR, etc.). Second, if the traffic signal is in the field of view, the vehicle may deter mine whether it is able to identify the state of the traffic signal using one or more sen sors) 130. For example, the vehicle may capture one or more images and / or video feeds of the intersection and may use computer vision to determine the state of the traffic sig nal. The vehicle may also use any other sensors 130 (or multiple different types of sen sors) to determine the state of the traffic signal as well. If the traffic signal is within the field of view, but not detected, the state of the signal may be classified unknown. Like wise, the traffic signal may be in the field of view, and the traffic signal itself may be identified, but a state of the traffic signal may be unknown. When there is an upcoming unknown traffic signal along the vehicle’s route, the vehicle may then send a request for guidance to the remote operator station 150 over a communications network 108. This re quest for guidance may include any information that may assist the remote operator in providing an instruction to the vehicle. For example, the information may indicate the reason for the request (for example, an unknown traffic signal), the type of the upcoming intersection (for example, traffic signal), a distance the vehicle is from the intersection, an identifier associated with the traffic signal, which can be used by the remote operator to identify the location of the traffic signal on a map (which, in some instances, may be thesame map that is used by the vehicle to perform navigation). The vehicle may also pro vide any other relevant information to the remote operator station 150. This information may be provided to the remote operator station 150 over the communications network 108 through a communications interface 140 (which may be the same as network inter face 712 depicted in FIG. 7). The vehicle may also include any other elements (for exam ple, any elements depicted in the computing device 700 illustrated in FIG. 7 or other wise).

[0021] In some embodiments, a remote operator station 150 may include at least one or more display interface(s) 154, memory 152, and one or more communication interface(s) 156. In some embodiments, when the remote operator station 150 may receive a request for assistance from the vehicle through the communications interface(s) 156 (which may be the same as network interface 712 depicted in FIG. 7), the remote operator 158 may be presented with any of the information received in the request, as well as any other rele vant information. For example, the display interface(s) 154 that may be used to present one or more images and / or live video feeds from one or more cameras from the vehicle, a map of the location of the vehicle, and / or any other relevant data. The remote operator 158 may determine the true state of the traffic signal using any of this information. That is, the remote operator 158 may view the one or more images and / or video feeds and may identify that the traffic signal is illuminating a particular color. The remote operator 158 may then provide guidance to the vehicle based on the color of the traffic signal. This may be accomplished by the remote operator 158 providing an input to the remote opera tor station 152, where the remote operator station 150 may subsequently provide guidance to the vehicle based on the input from the remote operator 158. For example, if the traffic signal is solid red, the operator 158 may wait for the light state to change before sending the guidance (or may send guidance for the vehicle to remain halted at the intersection). If the traffic signal is solid yellow, the operator 158 may wait for the light state to change before sending guidance (or may send guidance and for the vehicle to remain halted at the intersection or proceed through the intersection depending on the length of time the yel low light has been illuminated). If the traffic signal is solid green or flashing yellow, the operator 158 may provide guidance to the vehicle to proceed as if the intersection is an all-way stop. If the traffic signal is flashing red, the operator 158 may provide guidance tothe vehicle to proceed as if the intersection is a partial-way stop. When the vehicle re ceives the guidance, the vehicle may deactivate its hazard indicators and may proceed based on the action command received from the remote operator station. The remote op erator station 150 may also include any other elements (for example, any elements de picted in the computing device 700 illustrated in FIG. 7 or otherwise). It should be noted that although examples of guidance that may be provided when the AV is presented with certain traffic signal colors are provided herein, these are merely exemplary. For example, the remote operator station may provide guidance for the AV to proceed as if the intersec tion is a partial-way stop when the traffic signal is green as well.

[0022] In some embodiments, the communications network 108 may include any one or a combination of multiple different types of networks, such as cable networks, the Internet, wireless networks, and other private and / or public networks. In some instances, the com munications network 108 may include cellular, Wi-Fi, or Wi-Fi direct. In some embodi ments, the network may involve communications between vehicles in the network and / or between vehicles in the network and elements external to the network. For example, Ve- hi cl e-to- Vehicle (V2V), Vehicle-to-Infrastructure (V2I), Vehicle-to-Everything (V2X), and / or Dedicated Short Range Communications (DSRC), to name a few, may be used.

[0023] FIGs. 2A-2B depict an example use case including a first scene 200 and a second scene 250. Beginning with the first scene 200, the use case may depict a first vehicle 216 and a second vehicle 212. The use case may be described from the perspective of the first vehicle 216. That is, the first vehicle 216 may be an autonomous vehicle (AV) that may require guidance from a remote operator station 230 as described herein. The first vehicle 216 may be approaching an intersection 202 that may include one or more traffic signals (for example, traffic signal 206 and traffic signal 208). In the example scene 200, the traf fic signal 206 may currently be displaying a flashing red color and the traffic signal 208 may currently be displaying a flashing yellow color. In order to proceed through the inter section 202, the first vehicle 216 may need to know a state of the traffic signal 206. For example, the first vehicle 216 may need to know the color of the traffic signal 206 (which may be red in this case). In order to determine the status of the traffic signal 206, the first vehicle 216 may include one or more cameras that may be used to capture one or more images and / or video feeds of the traffic signal 206 and the intersection 202 as a whole. Under normal operation, the first vehicle 216 may analyze the one or more images and / orvideo feeds of the traffic signal 206 to determine the current color of the traffic signal. Based on the color of the traffic signal, the first vehicle 216 may determine whether to proceed through the intersection 202 or to stop in front of the intersection 202.

[0024] In the scene 200, the first vehicle 216 may be unable to determine through analy sis of captured images and / or video feed the color of the traffic signal 206. The first vehi cle 216 may then classify the state of the traffic signal 206 as unknown. Based on the state of the traffic signal 206 being set to unknown, the first vehicle 216 may stop at the intersection 202, and may send a request to a remote operator station 230 for guidance. Upon receiving the request from the first vehicle 216, the remote operator station 230 may present any of the information received in the request, as well as any other relevant information. For example, the remote operator station 230 may include an interface that may be used to present one or more images and / or live video feeds from one or more cameras from the first vehicle 216, a map of the location of the first vehicle 216, and / or any other relevant data. This information may be presented to a remote operator 232 who may be responsible for monitoring the remote operator station 230. The remote operator 232 may determine the true state of the traffic signal using any of this information. That is, the remote operator 232 may view the one or more images and / or video feeds and may identify that the traffic signal 206 is flashing red and the traffic signal 208 is flashing yel low. The remote operator 232 may then provide guidance to the first vehicle 216. In this specific use case, the remote operator 232 may provide guidance to the first vehicle 216 to treat the intersection as a partial-way stop given that the traffic signals are displaying flashing lights. The first vehicle 216 may then wait for all cross-traffic to exit the intersec tion 202 before proceeding through the intersection. For example, as shown in scene 250, the first vehicle 216 may wait for the vehicle 212 to proceed through the intersection 212 before the first vehicle 216 itself proceeds through the intersection.

[0025] FIGs. 3A-3B depict an example use case including a first scene 300 and a second scene 350. Beginning with the first scene 300, the use case may depict a first vehicle 316 and a second vehicle 312. The use case may be described from the perspective of the first vehicle 316 That is, the first vehicle 316 may be an autonomous vehicle (AV) that may require the assistance of a remote operator station 330 as described herein. The first vehi cle 316 may be approaching an intersection 302 that may include one or more traffic sig nals (for example, traffic signal 306 and traffic signal 308). In the example scene 300, thetraffic signal 306 may currently be displaying a green color and the traffic signal 308 may currently be displaying a red color. In order to proceed through the intersection 302, the first vehicle 316 may need to know a state of the traffic signal 306. For example, the first vehicle 316 may need to know the color of the traffic signal 306 (which may be red in this case). In order to determine the status of the traffic signal 306, the first vehicle 316 may include one or more cameras that may be used to capture one or more images and / or video feeds of the traffic signal 306 and the intersection 302 as a whole. Under normal operation, the vehicle 316 may analyze the one or more images and / or video feeds of the traffic signal 306 to determine a current color of the traffic signal. Based on the color of the traffic signal, the first vehicle 316 may determine whether to proceed through the in tersection 302 or to stop in front of the intersection 302.

[0026] In the scene 300, the first vehicle 316 may be unable to determine through analy sis of captured images and / or video feed the color of the traffic signal 306. The first vehi cle 316 may then classify the state of the traffic signal 306 as unknown. Based on the state of the traffic signal 306 being set to unknown, the first vehicle 316 may stop at the intersection 302, and may send a request to a remote operator station 330 for instruction. Upon receiving the request from the first vehicle 316, the remote operator station 330 may present any of the information received in the request, as well as any other relevant information. For example, the remote operator station 330 may include an interface that may be used to present one or more images and / or live video feeds from one or more cameras from the first vehicle 316, a map of the location of the first vehicle 316, and / or any other relevant data. This information may be presented to a remote operator 332 who may be responsible for monitoring the remote operator station 330. The remote operator 332 may determine the true state of the traffic signal using any of this information. That is, the remote operator 332 may view the one or more images and / or video feeds and may identify that the traffic signal 306 is green and the traffic signal 308 is red. The remote operator 332 may then provide guidance to the first vehicle 316. In this specific use case, the remote operator 332 may provide the guidance to the first vehicle 316 to treat the in tersection as an stop given that the traffic signals are displaying flashing lights. The first vehicle 316 may then wait for all cross-traffic to exit the intersection 302 before proceed ing through the intersection. For example, as shown in scene 350, the first vehicle 316 may wait for the second vehicle 312 to proceed through the intersection 302 before thefirst vehicle 316 itself proceeds through the intersection. It should be noted that although examples of guidance that may be provided when the AV is presented with certain traffic signal colors are provided herein, these are merely exemplary. For example, the remote operator station may provide guidance for the AV to proceed as if the intersection is a partial-way stop when the traffic signal is green as well. Additionally, while it may be de scribed herein that the remote operator may make determinations based on the infor mation presented by the remote operator station, in some cases, the remote operator sta tion itself may automatically make the same determinations made by the remote operator. That is, in some cases, the remote operator may not be necessary, and guidance may be provided through the use of only the remote operator station. As one non-limiting exam ple, the remote operator station may include artificial intelligence, machine learning, or the like to allow the remote operator station to analyze the images and / or video feeds pro vided by the AV, as well as any other relevant information.

[0027] FIG. 4 depicts an example flowchart 400. The flowchart 400 may illustrate exam ple high level operations performed by an AV that may encounter a traffic signal that is an unknown state to the AV (for example, the AV may be unable to determine a color the traffic signal as described herein). In some cases, the flowchart 400 may begin at condi tion 414. Condition 414 may involve a determination as to whether the AV has cleared (for example, traversed through) an intersection at its current location. For example, the flowchart 400 may illustrate a scenario starting from a point when the AV is already at an intersection. However, it should be noted that although the flowchart 400 illustrates con dition 414 as being the starting point, in some cases, the operations described with respect to the flowchart 400 may similarly begin before the AV encounters an intersection as well. That is, the flowchart 400 may be modified such that the beginning point may be the condition 404. Returning to the illustrated flowchart 400, if it is determined in condition 414 that the AV has cleared the particular intersection, then the flowchart 400 may pro ceed to operation 402. If it is determined that the AV has not cleared the intersection, then the flowchart may proceed to operation 412. Operation 402 may involve the AV being in communication with a remote operator station, but has not yet received any guidance from the remote operator station. That is, once the AV has cleared a given intersection and continues navigation along a route (potentially towards a subsequent intersection), the AV may be navigating without guidance provided by the remote operator station (hence,the remote operator station may be “not active”). Operation 412 may involve the AV hav ing received guidance from the remote operator station to proceed as if the intersection is an all-way stop or a partial-way stop. Thus, the AV may continue traversing through the intersection unless the condition 410 is met. Condition 410 may involve a determination as to whether guidance has been provided from the remote operator station to the AV in dicating that the AV should disregard prior guidance for the AV to proceed through the intersection. If the guidance to disregard prior guidance is not received, the flowchart 400 may continue to loop back to condition 412, and then AV may continue to traverse the intersection until it ultimately clears the intersection. However, if the guidance to disre gard prior guidance is received, then the flowchart may proceed to operation 406.

[0028] In some embodiments, following operation 402, the flowchart 400 may proceed to condition 404. Condition 404 may involve a determination as to whether the AV is unable to identify the state of a current color of a traffic signal. For example, the AV may be ap proaching an intersection (which may be a subsequent intersection along the AV’s navi gation route beyond the initial intersection that was cleared) including one or more traffic signals, and may be unable to determine if the color of the traffic signal is red, yellow, or green (or any other color that may be used depending on the geographic location of the traffic signal). As mentioned above, the AV may make such determinations based on im ages and / or video feeds captured by one or more cameras included on the AV. If the AV is not able to determine the state of the traffic signal, the AV may halt movement and may send a request to a remote operator station for further guidance. In this case, the flowchart 400 may proceed to operation 406. However, if the AV is able to determine the color of the traffic signal, then the AV may proceed through the intersection without the assistance of the remote operator station. In such a case, the flowchart 400 to operation 402. In some cases, the condition 404 may be continuously monitored as the AV is navigating an environment. For example, the AV may determine at a first time that it is able to identify the color of the traffic signal, but at a second time may be unable to determine the color of the traffic signal. In this example scenario, the AV may proceed to operation 406 at any time the vehicle is unable to determine the status of the traffic signal. This may, how ever, depend on a number of factors, such as the current location of the AV. For example, if the AV detects that the traffic signal is green and proceeds through the intersection, but is unable to determine the color of the traffic signal while already in the intersection, thenthe AV may simply proceed through the intersection so as to prevent the AV from halting in the middle of an intersection to request guidance from a remote operator station. An other example factor may include the amount of time the color of the traffic signal re mains unknown. For example, the color of the traffic signal may only be unknown to the AV for a short duration of time, and after this short duration of time, the traffic signal color may again become known to the AV. Thus, to avoid scenarios where the vehicle halts operations because the color of the traffic signal is unknown for a short duration (and becomes known again), a threshold amount of time may be used. That is, the AV may be required to wait a threshold amount of time without being able to detect the color of the traffic signal before halting movement and requesting guidance from the remote operator station.

[0029] In some embodiments, while in operation 406, the remote operator station may be used to provide instructions to the AV when the AV may be unsure how to proceed through a given intersection. After the message is sent to the remote operator station in operation 406, the flowchart 400 may proceed to condition 408. Condition 408 may in volve a determination as to whether a message including the guidance was transmitted back to the AV from the remote operator station. The remote operator may be able to pro vide several different types guidance to the AV. A first example type of guidance may in clude an indication for the AV to proceed as if the intersection is an all-way stop (for ex ample, the AV should proceed as if the intersection includes stop signs or flashing red lights at each of the roads in the intersection). A second example type of guidance may include an indication for the AV to proceed as if the intersection is a partial-way stop (for example, the AV should proceed as if the road the AV is on includes a stop sign or a red light and the roads in the intersection that are perpendicular to the road that the AV is on have no stop sign or red light). Based on the guidance provided by the remote operation, the AV may adjust how it approaches the intersection and / or predicts behaviors of other actors in the intersection, such as other vehicles. That is, the AV may adjust how it pre dicts when other vehicles at the intersection will be traversing through the intersection or stopping in front of the intersection. The remote guidance provided by the remote opera tor to the AV can authorize the AV to perform certain maneuvers, but may not instruct the AV on how or when to perform the maneuvers. Remote guidance may also not over-ride the perception stack of the AV. That is, once the AV receives guidance from the re mote operator, it may proceed or remain halted, but may proceed using its own sensors and internal processing. For example, if the AV received guidance to proceed as if the in tersection were a partial-way stop, then the AV may then use its own processing to iden tify when it may proceed into the intersection, and also to continuously monitor the envi ronment while proceeding through the intersection. Continuing this example, even though the AV has received guidance to proceed, another vehicle at the intersection may illegally drive through a red light and into the intersection. Although it was instructed to proceed, the AV may still monitor for scenarios like this and adjust its movement correspondingly. This is just one non-limiting example of how the AV may still rely on internal processing even after being instructed to proceed. If the condition 408 is not met, the flowchart 400 may loop through the condition 408 until it is met. Once the condition 408 is met, the flowchart may either proceed back to operation 406 or may proceed to operation 412. The flowchart 400 may proceed back to operation 406 if the guidance provided by the remote operator station indicates that the AV should remain halted (for example, if the remote operator at the remote operator station determines that the color of the traffic signal is currently red or yellow).

[0030] FIG. 5. depicts an example flowchart 500. The example flowchart 500 may in volve similar operations as the flowchart 400, but may be a more detailed illustration than the high-level operations illustrated in the flowchart 400. The flowchart 500 may begin with operation 502, which may involve an AV approaching an intersection that may in clude one or more traffic signals. In some cases, the AV may determine it is approaching an intersection based on a map stored in local memory or at a remote location (for exam ple, the AV may determine its current location with respect to one or more intersections on the map). In some cases, the AV may determine it is approaching the intersection us ing information from one or more sensors on the AV (for example, the AV may use a camera to capture an image or video feed of an external environment to identify any up coming intersections. In other cases, the AV may determine that is approaching an inter section using any other suitable methods. Additionally, a remote system may also provide an indication to the AV that it is approaching an intersection rather than the AV making the determination itself. From operation 502, the flowchart 500 may proceed to condition 504. Condition 504 may involve a determination as to whether the AV is able to identifya color of a traffic signal at the intersection. Condition 504 may be similar to conditions 510, 522, and 542 described below. This condition may also be continuously looped throughout any of the operations being described with respect to the flowchart 500. That is, although the flowchart 500 depicts the condition as being included in specific locations in the flowchart 500, the AV may continuously determine whether it is able to discern the color of the traffic signal at any point as well. If it is determined in condition 504 that the AV is able to identify the color of the traffic signal, then the flowchart 500 may proceed to operation 548. However, if the AV is not able to identify the color of the traffic signal, then the flowchart 500 may proceed to operation 506. At operation 506, the AV may send a request for guidance to a remote operator station. The request may include any images and / or video feed captured by the AV. For example, the AV may transmit any images and / or video feed captured of the traffic signal at the intersection. The request may also include any other information that may be relevant to the remote operator station, such as a current location of the AV, a speed of the AV, and / or any other information relating to the AV itself, the environment external to the AV, or any other information. The flowchart 500 may then proceed to operation 508, which may involve the AV stopping at the intersection. That is, if the AV is unable to determine the color of a traffic signal at the intersection, the AV may halt movement instead of proceeding through the intersection, and may send the request for guidance to a remote operator station as indicated in opera tion 506. In some cases, operations 506 and / or 508 may occur in any order, and may also occur simultaneously. During this time, the AV may continuously loop condition 510. Condition 510 may involve determining if the color of the traffic signal is known by the AV. For example, the AV may determine at a first time that it is able to identify the color of the traffic signal, but at a second time may be unable to determine the color of the traf fic signal. In some cases, condition 510 may only be checked once, or a finite number of times as well (for example, instead of being continuously looped). If at any time the AV is able to identify the color of the traffic signal, the flowchart 500 may proceed to opera tion 544. At operation 544, the AV may send a subsequent message to the remote opera tor station indicating that the AV no longer requires instructions. If the AV has already received instructions from the remote operator station, the AV may also disregard the in structions and proceed using the local processing of the AV. From operation 544, theflowchart 500 may proceed to operation 546 and operation 548, in which the AV may de activate its hazard lights and proceed through the intersection.

[0031] Returning to condition 510, if it is determined that the color of the traffic signal at the intersection is unknown to the AV, then the flowchart 500 may proceed to operation 512. In operation 512, the AV may halt movement at the intersection. At operation 514, the AV may also activate its hazard lights (or provide any other type of indication to other vehicles at the intersection that the AV is stopped). At operation 516, the information pro vided to the remote operator station may be presented on an interface of the remote opera tor station. A remote operator managing the remote operator station may then examine the information presented on the interface of the remote operator station. At condition 518, it may be determined by the remote operator if the traffic signal is a solid red color. If it is determined that the traffic signal is a solid red color, then the remote operator may pro vide an input to the remote operator station to provide guidance to the AV to remain halted at the intersection. Alternatively, the remote operator may simply refrain from providing an input to the remote operator station, as the AV may remain halted at the in tersection until either guidance is received from the remote operator station or the AV is able to subsequently determine the color of the traffic signal. If it is determined that the color of the traffic signal is not red, then, at operation 520, the remote operator may pro vide an input to the remote operator station, which may trigger the remote operator station to send guidance to the vehicle.

[0032] In some embodiments, following operation 520, the flowchart 500 may proceed to condition 522, which may involve a determination as to whether the color of the traffic signal at the intersection is unknown to the AV. If the condition 522 is met (the AV is able to determine the color of the traffic signal), then the flowchart may proceed to opera tion 540. In operation 540, the AV may ignore guidance provided by the remote operator station, and may instead proceed to operation 544 and beyond. However, if the condition 522 is not met (the AV is still unable to determine the color of the traffic signal), then the flowchart 500 may proceed to condition 524. Condition 524 may involve a determination as to what type of guidance was provided by the remote operator station. In the example provided in flowchart 500, the condition 524 may involve a determination as to whether the remote operator station indicated that the light is green or unlit. If the condition 524 is met, then the flowchart 500 may proceed to operation 526. If the condition 528 is not met,then the flowchart 500 may proceed to operation 528. Operation 526 may involve the AV treating the intersection as an all-way stop. To this end, the remote operator station may provide guidance to the AV indicating that the AV should treat the intersection as an all way stop. Operation 528 may involve the remote operator providing an input to the re mote operator station indicating that the light is blinking red. From operation 528, the flowchart 500 may proceed to operation 530. In operation 530, the AV may treat the in tersection as a partial-way stop. To this end, the remote operator station may provide guidance to the AV indicating that the AV should treat the intersection as a partial-way stop. It should be noted that although examples of guidance that may be provided when the AV is presented with certain traffic signal colors are provided herein, these are merely exemplary. For example, the remote operator station may provide guidance for the AV to proceed as if the intersection is a partial-way stop when the traffic signal is green as well. Additionally, while it may be described herein that the remote operator may make deter minations based on the information presented by the remote operator station, in some cases, the remote operator station itself may automatically make the same determinations made by the remote operator. That is, in some cases, the remote operator may not be nec essary, and guidance may be provided through the use of only the remote operator station. As one non-limiting example, the remote operator station may include artificial intelli gence, machine learning, or the like to allow the remote operator station to analyze the images and / or video feeds provided by the AV, as well as any other relevant information.

[0033] In some embodiments, following operations 526 and / or 528, the flowchart may proceed to operation 532. Operation 532 may involve updating the AV’s perception stack (for example, internal processing) to indicate that the AV may proceed through the inter section while yielding to other vehicles at the intersection based on the particular guid ance provided to the AV. Following operation 532, the flowchart 500 may proceed to condition 524. Condition 534 may involve determining if there are any conflict movers in or at the intersection. A conflict mover may be any other actor that may be moving in or at the intersection, such as another vehicle, a pedestrian, or any other moving actor. If it is determined that there are no conflict movers, then the flowchart 500 may proceed to oper ation 550. If it is determined that there are conflict movers, then the flowchart may pro ceed to condition 542. At operation 550, the AV may proceed into the intersection, and then may ultimately clear the intersection at operation 552. At condition 542, the AV mayagain determine if the color of the traffic signal is determined. If the color of the traffic signal is determined, then the flowchart may proceed to operation 544. If the color of the traffic signal is still not determined, then the flowchart 500 may proceed to operation 538. At operation 538, the AV may wait for conflicting movers to leave the intersection (or may wait for a prediction that the conflict movers currently in the intersection will yield to the AV). From operation 538, the flowchart 500 may proceed to condition 536.

[0034] FIG. 6 is an example method 600. At block 602 of the method 600 in FIG. 6, the method may include capturing, at a first time and by a camera of an autonomous vehicle, at least one of: an image or video feed of a first traffic signal at an intersection. At block 604, the method 600 may include classifying, based on the image or the video feed of the first traffic signal, a state of a color of the first traffic signal as unknown. At block 606, the method 600 may include halting movement of the autonomous vehicle at the intersec tion based on classifying the state of the color of the first traffic signal as unknown. At block 608, the method 600 may include sending a request for guidance to a remote opera tor device, the request including the image or video feed of the first traffic signal. At block 610, the method 600 may include receiving, from the remote operator device, a first guidance. At block 612, the method 600 may include performing, based on the first guid ance, a first action including at least one of: remaining halted at the intersection or pro ceeding through the intersection.

[0035] In some embodiments, the method 600 may also include capturing, at a second time and by the camera of an autonomous vehicle, at least one of: a second image or sec ond video feed of the first traffic signal. The method 600 may also include classifying, by the autonomous vehicle and based on the second image or video feed, the state of the color of the first traffic signal as known. The method 600 may also include performing, based on the state of the color of the first traffic signal being known, the first action.

[0036] In some embodiments, the method 600 may also include capturing, at a second time and by the camera of an autonomous vehicle, a second image or video feed of the first traffic signal. The method 600 may also include classifying, by the autonomous vehi cle and based on the second image or video feed, the state of the color of the first traffic signal as being known. The method 600 may also include performing, based on the state of the color of the first traffic signal being known, a second action, the second action be ing different than the first action based on the first guidance.

[0037] In some embodiments, the first guidance includes guidance for the autonomous vehicle to operate in a partial-way stop mode, and wherein operating in the partial-way stop mode involves the autonomous vehicle predicting that cross-traffic at the intersection will proceed without yielding to the autonomous vehicle. In some embodiments, the first guidance includes guidance for the autonomous vehicle to operate in an all-way stop mode, wherein operating in the all-way stop mode involves the autonomous vehicle pre dicting that cross-traffic at the intersection will yield to the autonomous vehicle. In some embodiments, the first guidance includes guidance for the vehicle to remain halted at the intersection. In some embodiments, performing the second action further comprises disre garding the first guidance received from the remote operator device and proceeding through the intersection.

[0038] FIG. 7 is an example method 700. At block 702 of the method 700 in FIG. 7, the method may include receiving, from an autonomous vehicle and by a remote operator de vice, a request for guidance, the request including at least one of: an image or video feed of a first traffic signal at an intersection captured by the autonomous vehicle, the request also indicating that the autonomous vehicle has classified a state of a color of the first traffic signal as unknown. At block 704, the method 700 may include receiving, by the remote operator device, an input selection of a first guidance to provide to the autono mous vehicle. At block 706, the method 700 may include providing, by the remote opera tor device and to the autonomous vehicle, the first guidance.

[0039] The operations described and depicted in the illustrative process flows of FIGs. 4- 7 may be carried out or performed in any suitable order as desired in various example em bodiments of the disclosure. Additionally, in certain example embodiments, at least a por tion of the operations may be carried out in parallel. Furthermore, in certain example em bodiments, less, more, or different operations than those depicted in FIGs. 4-7 may be performed.

[0040] FIG. 8 illustrates an example computing device 800, in accordance with one or more embodiments of this disclosure. The computing 800 device may be representative of any number of elements described herein, such as any of the remote operator station(s), the vehicle(s), and / or any other element described herein. The computing device 800 may include at least one processor 802 that executes instructions that are stored in one or morememory devices (referred to as memory 804). The instructions can be, for instance, in structions for implementing functionality described as being carried out by one or more modules and systems disclosed above or instructions for implementing one or more of the methods disclosed above. The processor(s) 802 can be embodied in, for example, a CPU, multiple CPUs, a GPU, multiple GPUs, a TPU, multiple TPUs, a multi-core processor, a combination thereof, and the like. In some embodiments, the processor(s) 802 can be ar ranged in a single processing device. In other embodiments, the processor(s) 802 can be distributed across two or more processing devices (e.g., multiple CPUs; multiple GPUs; a combination thereof; or the like). A processor can be implemented as a combination of processing circuitry or computing processing units (such as CPUs, GPUs, or a combina tion of both). Therefore, for the sake of illustration, a processor can refer to a single-core processor; a single processor with software multithread execution capability; a multi-core processor; a multi-core processor with software multithread execution capability; a multi core processor with hardware multithread technology; a parallel processing (or compu ting) platform; and parallel computing platforms with distributed shared memory. Addi tionally, or as another example, a processor can refer to an integrated circuit (IC), an ASIC, a digital signal processor (DSP), an FPGA, a PLC, a complex programmable logic device (CPLD), a discrete gate or transistor logic, discrete hardware components, or any combination thereof designed or otherwise configured (e.g., manufactured) to perform the functions described herein.

[0041] The processor(s) 802 can access the memory 804 by means of a communication architecture 806 (e.g., a system bus). The communication architecture 806 may be suita ble for the particular arrangement (localized or distributed) and type of the processor(s) 802. In some embodiments, the communication architecture 806 can include one or many bus architectures, such as a memory bus or a memory controller; a peripheral bus; an ac celerated graphics port; a processor or local bus; a combination thereof, or the like. As an illustration, such architectures can include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electron ics Standards Association (VESA) local bus, an Accelerated Graphics Port (AGP) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express bus, a Personal Computer Memory Card International Association (PCMCIA) bus, a Universal Serial Bus (USB), and / or the like.

[0042] Memory components or memory devices disclosed herein can be embodied in ei ther volatile memory or non-volatile memory or can include both volatile and non-volatile memory. In addition, the memory components or memory devices can be removable or non-removable, and / or internal or external to a computing device or component. Exam ples of various types of non-transitory storage media can include hard-disc drives, zip drives, CD-ROMs, digital versatile disks (DVDs) or other optical storage, magnetic cas settes, magnetic tape, magnetic disk storage or other magnetic storage devices, flash memory cards or other types of memory cards, cartridges, or any other non-transitory me dia suitable to retain the desired information and which can be accessed by a computing device.

[0043] As an illustration, non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), which acts as external cache memory. By way of illustra tion and not limitation, RAM is available in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory devices or memories of the operational or computational environments described herein are intended to include one or more of these and / or any other suitable types of memory. In addition to storing executable instructions, the memory 804 also can retain data.

[0044] Each computing device 800 also can include mass storage 808 that is accessible by the processor(s) 802 by means of the communication architecture 806. The mass stor age 808 can include machine-accessible instructions (e.g., computer-readable instructions and / or computer-executable instructions). In some embodiments, the machine-accessible instructions may be encoded in the mass storage 808 and can be arranged in components that can be built (e.g., linked and compiled) and retained in computer-executable form in the mass storage 808 or in one or more other machine-accessible non-transitory storage media included in the computing device 800. Such components can embody, or can con stitute, one or many of the various modules disclosed herein. Such modules are illustrated as modules 814. In some instances, the modules may also be included within the memory 804 as well.

[0045] Execution of the modules 814, individually or in combination, by at least one of the processor(s) 802, can cause the computing device 800 to perform any of the opera tions described herein (for example, the operations described with respect to FIG. 4, as well as any other operations).

[0046] Each computing device 800 also can include one or more input / output interface devices 810 (referred to as I / O interface 810) that can permit or otherwise facilitate exter nal devices to communicate with the computing device 800. For instance, the I / O inter face 810 may be used to receive and send data and / or instructions from and to an external computing device.

[0047] The computing device 800 also includes one or more network interface devices 812 (referred to as network interface(s) 812) that can permit or otherwise facilitate func tionally coupling the computing device 800 with one or more external devices. Function ally coupling the computing device 800 to an external device can include establishing a wireline connection or a wireless connection between the computing device 800 and the external device. The network interface devices 812 can include one or many antennas and a communication processing device that can permit wireless communication between the computing device 800 and another external device. For example, between a vehicle and a smart infrastructure system, between vehicles, etc. Such a communication processing de vice can process data according to defined protocols of one or several radio technologies. The radio technologies can include, for example, 3G, Long Term Evolution (LTE), LTE- Advanced, 5G, IEEE 802.11, IEEE 802.16, Bluetooth, ZigBee, near-field communication (NFC), and the like. The communication processing device can also process data accord ing to other protocols as well, such as vehicle-to-infrastructure (V2I) communications, vehi cl e-to- vehicle (V2V) communications, and the like. The network interface(s) 812 may also be used to facilitate peer-to-peer ad-hoc network connections as described herein.

[0048] As used in this application, the terms “environment,” “system,” “unit,” “module,” “architecture,” “interface,” “component,” and the like refer to a computer-related entity or an entity related to an operational apparatus with one or more defined functionalities. The terms “environment,” “system,” “module,” “component,” “architecture,” “interface,” and “unit,” can be utilized interchangeably and can be generically referred to functional ele ments. Such entities may be either hardware, a combination of hardware and software,software, or software in execution. As an example, a module can be embodied in a pro cess running on a processor, a processor, an object, an executable portion of software, a thread of execution, a program, and / or a computing device. As another example, both a software application executing on a computing device and the computing device can em body a module. As yet another example, one or more modules may reside within a pro cess and / or thread of execution. A module may be localized on one computing device or distributed between two or more computing devices. As is disclosed herein, a module can execute from various computer-readable non-transitory storage media having various data structures stored thereon. Modules can communicate via local and / or remote processes in accordance, for example, with a signal (either analogic or digital) having one or more data packets (e.g., data from one component interacting with another component in a local sys tem, distributed system, and / or across a network such as a wide area network with other systems via the signal).

[0049] As yet another example, a module can be embodied in or can include an apparatus with a defined functionality provided by mechanical parts operated by electric or elec tronic circuitry that is controlled by a software application or firmware application exe cuted by a processor. Such a processor can be internal or external to the apparatus and can execute at least part of the software or firmware application. Still, in another example, a module can be embodied in or can include an apparatus that provides defined functional ity through electronic components without mechanical parts. The electronic components can include a processor to execute software or firmware that permits or otherwise facili tates, at least in part, the functionality of the electronic components.

[0050] In some embodiments, modules can communicate via local and / or remote pro cesses in accordance, for example, with a signal (either analog or digital) having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and / or across a network such as a wide area network with other systems via the signal). In addition, or in other embodiments, modules can communicate or otherwise be coupled via thermal, mechanical, electrical, and / or electro mechanical coupling mechanisms (such as conduits, connectors, combinations thereof, or the like). An interface can include input / output (I / O) components as well as associated processors, applications, and / or other programming components.

[0051] Further, in the present specification and annexed drawings, terms such as “store,” “storage,” “data store,” “data storage,” “memory,” “repository,” and substantially any other information storage component relevant to the operation and functionality of a com ponent of the disclosure, refer to memory components, entities embodied in one or sev eral memory devices, or components forming a memory device. It is noted that the memory components or memory devices described herein embody or include non-transi- tory computer storage media that can be readable or otherwise accessible by a computing device. Such media can be implemented in any methods or technology for storage of in formation, such as machine-accessible instructions (e.g., computer-readable instructions), information structures, program modules, or other information objects.

[0052] Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain implementations could include, while other im plementations do not include, certain features, elements, and / or operations. Thus, such conditional language generally is not intended to imply that features, elements, and / or op erations are in any way required for one or more implementations or that one or more im plementations necessarily include logic for deciding, with or without user input or prompting, whether these features, elements, and / or operations are included or are to be performed in any particular implementation.

[0053] What has been described herein in the present specification and annexed drawings includes examples of systems, devices, techniques, and computer program products that, individually and in combination, permit the automated provision of an update for a vehi cle profile package. It is, of course, not possible to describe every conceivable combina tion of components and / or methods for purposes of describing the various elements of the disclosure, but it can be recognized that many further combinations and permutations of the disclosed elements are possible. Accordingly, it may be apparent that various modifi cations can be made to the disclosure without departing from the scope or spirit thereof.In addition, or as an alternative, other embodiments of the disclosure may be apparent from consideration of the specification and annexed drawings, and practice of the disclo sure as presented herein. It is intended that the examples put forth in the specification and annexed drawings be considered, in all respects, as illustrative and not limiting. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

THAT WHICH IS CLAIMED IS:

1. A method comprising: capturing, at a first time and by a camera of an autonomous vehicle, at least one of: an image or video feed of a first traffic signal at an intersection; classifying, based on the image or the video feed of the first traffic signal, a state of a color of the first traffic signal as unknown; halting movement of the autonomous vehicle at the intersection based on classify ing the state of the color of the first traffic signal as unknown; sending a request for guidance to a remote operator device, the request including the image or video feed of the first traffic signal; receiving, from the remote operator device, a first guidance; and performing, based on the first guidance, a first action including at least one of: re maining halted at the intersection or proceeding through the intersection.

2. The method of claim 1, wherein the first guidance includes guidance for the autonomous vehicle to operate in a partial-way stop mode, and wherein operating in the partial-way stop mode involves the autonomous vehicle predicting that cross-traffic at the intersection will proceed without yielding to the autonomous vehicle.

3. The method of claim 1, wherein the first guidance includes guidance for the autonomous vehicle to operate in an all-way stop mode, wherein operating in the all-way stop mode involves the autonomous vehicle predicting that cross-traffic at the intersection will yield to the autonomous vehicle.

4. The method of claim 1, wherein the first guidance includes guidance for the vehicle to re main halted at the intersection.

5. The method of claim 1, further comprising: capturing, at a second time and by the camera of an autonomous vehicle, at least one of: a second image or second video feed of the first traffic signal;classifying, by the autonomous vehicle and based on the second image or video feed, the state of the color of the first traffic signal as known; and performing, based on the state of the color of the first traffic signal being known, the first action.

6. The method of claim 1, further comprising: capturing, at a second time and by the camera of an autonomous vehicle, a second image or video feed of the first traffic signal; classifying, by the autonomous vehicle and based on the second image or video feed, the state of the color of the first traffic signal as being known; and performing, based on the state of the color of the first traffic signal being known, a second action, the second action being different than the first action based on the first guidance.

7. The method of claim 6, wherein performing the second action further comprises overrid ing the first guidance received from the remote operator device and proceeding through the intersection.

8. A method comprising: receiving, from an autonomous vehicle and by a remote operator device, a request for guidance, the request including at least one of: an image or video feed of a first traffic signal at an intersection captured by the autonomous vehicle, the request also indicating that the autonomous vehicle has classified a state of a color of the first traffic signal as unknown; receiving, by the remote operator device, an input selection of a first guidance to provide to the autonomous vehicle; and providing, by the remote operator device and to the autonomous vehicle, the first guidance.

9. The method of claim 8, wherein the first guidance includes guidance for the autonomous vehicle to operate in a partial-way stop mode, and wherein operating in the partial-waystop mode involves the autonomous vehicle predicting that cross-traffic at the intersection will proceed without yielding to the autonomous vehicle.

10. The method of claim 8, wherein the first guidance includes guidance for the autonomous vehicle to operate in an all-way stop mode, wherein operating in the all-way stop mode involves the autonomous vehicle predicting that cross-traffic at the intersection will yield to the autonomous vehicle.

11. The method of claim 8, wherein the first guidance includes guidance for the vehicle to re main halted at the intersection.

12. A system comprising: a processor; and a memory storing computer-executable instructions, that when executed by the processor, cause the processor to: capture, at a first time and by a camera of an autonomous vehicle, at least one of: an image or video feed of a first traffic signal at an intersection; classify, based on the image or video feed of the first traffic signal, a state of a color of the first traffic signal as unknown; halt movement of the autonomous vehicle at the intersection based on clas sifying the state of the color of the first traffic signal as unknown; send a request for guidance to a remote operator device, the request includ ing the image or video feed of the first traffic signal; receive, from the remote operator device, a first guidance; and perform, based on the first guidance, a first action including at least one of: remaining halted at the intersection or proceeding through the intersection.

13. The system of claim 12, wherein the first guidance includes guidance for the autonomous vehicle to operate in a partial-way stop mode, and wherein operating in the partial-way stop mode involves the autonomous vehicle predicting that cross-traffic at the intersection will proceed without yielding to the autonomous vehicle.

14. The system of claim 13, wherein predicting that cross-traffic at the intersection will pro ceed without yielding further comprises: capturing data indicating that a second vehicle is at the intersection at a first time; capturing data indicating that the second vehicle has passed through the intersec tion at a second time; and proceeding through the intersection based on the data indicating that the second vehicle has passed through the intersection at the second time.

15. The system of claim 12, wherein the first guidance includes guidance for the autonomous vehicle to operate in an all-way stop mode, wherein operating in the all-way stop mode involves the autonomous vehicle predicting that cross-traffic at the intersection will yield to the autonomous vehicle.

16. The system of claim 12, wherein the guidance includes guidance for the vehicle to remain halted at the intersection.

17. The system of claim 12, wherein the computer-executable instructions further cause the processor to: capture, at a second time and by the camera of an autonomous vehicle, a second image or video feed of the first traffic signal; classify, by the autonomous vehicle and based on the second image or video feed, the state of the color of the first traffic signal as known; and perform, based on the state of the color of the first traffic signal being known, the first action.

18. The system of claim 17, wherein the computer-executable instructions further cause the processor to: send, based on the state of the color of the first traffic signal being known, a mes sage to the remote operator device indicating that the color of the first traffic signal is known.

19. The system of claim 12, wherein the computer-executable instructions further cause the processor to: capture, at a second time and by the camera of an autonomous vehicle, a second image or video feed of the first traffic signal; classify, by the autonomous vehicle and based on the second image or video feed, the state of the color of the first traffic signal as known; and perform, based on the state of the color of the first traffic signal being known, a second action, the second action being different than the first action based on the first guidance, wherein performing the second action further comprises disregarding the first guidance received from the remote operator device and proceeding through the intersec tion.

20. The system of claim 19, wherein performing the second action further comprises disre garding the first guidance received from the remote operator device and proceeding through the intersection.

Images