Teleoperation for exception handling

Abstract

Exception handling, such as an obstruction situation, by an autonomous vehicle (AV) is disclosed. The method includes: identifying an abnormal situation; identifying a risk associated with autonomously resolving the abnormal situation; and in response to the risk exceeding a risk threshold, initiating a request for assistance from a teleoperator, and stopping to wait for the teleoperator to respond to the request; and receiving a response from the teleoperator.

Description

Technical Field

[0001] The present application relates generally to autonomous vehicles and, more particularly, to teleoperation assistance in obstructive situations. Background Art

[0002] Autonomous vehicles (AVs) offer human drivers the convenience of efficient transportation from one location to another without having to focus their attention on road conditions. AVs may be defined as self-propelled (e.g., computer-controlled) vehicles that are capable of driving on roads while obeying traffic rules and regulations. However, even the best autonomous vehicle programming cannot account for and control all conditions and situations that may arise during the operation of an autonomous vehicle. Furthermore, there are times when an autonomous vehicle encounters conditions and situations that may benefit from the assistance of a human operator (e.g., a teleoperator). Summary of the invention

[0003] Disclosed herein are aspects, features, elements, implementations and implementations for generating solutions associated with autonomous operation of vehicles.

[0004] Aspects of the disclosed implementation include: a method for exception handling by an autonomous vehicle, i.e., AV, the method comprising: identifying an abnormal situation; identifying a risk associated with autonomously resolving the abnormal situation; and in response to the risk exceeding a risk threshold: initiating a request for assistance from a teleoperator; and stopping to wait for the teleoperator to respond to the request; and receiving a response from the teleoperator.

[0005] Aspects of the disclosed implementation include: a method for exception handling performed by an autonomous vehicle (AV), the method comprising: identifying a lane obstruction condition; identifying a first trajectory for avoiding the lane obstruction condition; and in response to identifying the lane obstruction condition as an exception condition based on the first trajectory: determining a risk associated with the first trajectory; and in response to determining that the risk exceeds a risk threshold: initiating a request to a teleoperator, wherein the request includes the first trajectory; and waiting for a response from the teleoperator.

[0006] Aspects of the disclosed implementations include a device comprising a memory and a processor. The memory comprises instructions for exception handling by an autonomous vehicle (AV). The processor is configured to execute the instructions to perform the following operations: identify an abnormal situation; identify a risk associated with autonomously resolving the abnormal situation; and in response to the risk exceeding a risk threshold, initiate a request to a teleoperator; wait for the teleoperator to respond to the request; and receive a response from the teleoperator.

[0007] These and other aspects of the invention are disclosed in the following detailed description of the embodiments, the appended claims and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The disclosed technology can be best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be emphasized that, according to common practice, the various features of the drawings are not to scale. Instead, the size of the various features is arbitrarily expanded or reduced for clarity.

[0009] Figure 1 is a diagram of an example of a portion of a vehicle in which the aspects, features, and elements disclosed herein may be implemented.

[0010] Figure 2 is a diagram of an example of a portion of a vehicle transportation and communication system in which the aspects, features, and elements disclosed herein may be implemented.

[0011] Figure 3 is a block diagram illustrating a remote vehicle assistance center according to an implementation of the present invention.

[0012] Figure 4 is an example of modules of a system for exception handling according to implementation of the present invention.

[0013] Figure 5 is a flow chart of a technique for exception handling by an autonomous vehicle implemented in accordance with the present invention.

[0014] Figures 6A to 6B is a block diagram illustrating an example of an obstruction scenario according to an implementation of the present invention.

[0015] Figure 7 is a diagram of a teleoperator user interface 700 according to an implementation of the present invention.

[0016] Figure 8 is a diagrammatic representation of a ticket according to an implementation of the present invention.

[0017] Fig. 9 is a diagram of a menu of teleoperator responses according to an implementation of the present invention.

[0018] Fig.10 is a flow chart of a technique for exception handling by an autonomous vehicle implemented in accordance with the present invention. DETAILED DESCRIPTION

[0019] Autonomous driving vehicles (AVs) may not be able to cope with many road conditions. Some situations may be outside the programming of AV. For example, AV may not be able to fully evaluate, classify and / or understand the intentions of other road users. Other road users may include pedestrians, construction workers, police, construction equipment, vehicles, cyclists or other static objects (e.g., buildings, road signs, etc.). For example, evaluating road conditions may be outside the sensing capabilities of AV, such as in identifying whether other road users are present near corners blocked by buildings or whether other road users are present on the other side of the mountain, etc. In addition, parsing some other situations (e.g., obstruction situations) may require AV to deviate from normal (e.g., legal, socially acceptable) driving rules in an unacceptable manner without human supervision. Road conditions that AV cannot cope with are referred to as abnormal conditions here. Abnormal conditions are driving conditions that require AV to suspend some driving specifications to move forward (such as towards the final destination of AV, etc.).

[0020] When an AV encounters an unusual situation, the AV may stop and request assistance from a teleoperator. For example, when an AV encounters an obstruction in a travel lane (e.g., a construction site, a stopped vehicle, etc.), if circumventing the obstruction means that the AV will travel through an area that is physically safe but restricted by traffic regulations, the AV may not do so. Therefore, a teleoperator (e.g., a human operator, a vehicle manager) may be tasked with assisting the AV in negotiating its problematic situation by, for example, drawing a path (i.e., a trajectory) for the AV around the obstruction. The teleoperator may be one of many teleoperators available at a teleoperation center (i.e., a remote vehicle assistance center), each of which may monitor the status or condition of one or more AVs.

[0021] However, a stopped AV (such as in the middle of a road) may cause a public nuisance or lead to dangerous and unsafe road conditions. Thus, even when teleoperator assistance is requested and / or provided, the teleoperator must respond quickly to the assistance request.

[0022] In the experiment, during a three-hour drive in the San Francisco area, the AV encountered 19 abnormal situations that required a teleoperator to resolve. On average, the teleoperator spent one minute per abnormal situation to evaluate the abnormal situation and draw the path of the AV (such as based on information received from the AV). Thus, on average, 11% of the daily routine (i.e., 360 minutes / 19 minutes ≈ 11%) is spent on resolving an abnormal situation for an AV. In San Francisco, there are 450,000 vehicles registered to provide taxi services. It is expected that in the near future, taxi services may be provided only by autonomous vehicles. In this way, and assuming that one teleoperator can manage 10 AVs exclusively, 450 teleoperators will be needed in the city of San Francisco alone. This model (i.e., a model that relies heavily on human intervention) is not practical, sustainable, profitable, or efficient.

[0023] In the above description, upon encountering an abnormal situation, the AV stops and requests teleoperator assistance. The teleoperator assistance model may require many teleoperators and a lot of time to resolve each abnormal situation.

[0024] Implementations according to the present invention may reduce the need for teleoperation support (e.g., intervention). The need for teleoperation support may be reduced by reducing the time required for a teleoperator to resolve (e.g., respond to) an abnormal situation. The need for teleoperation support may be reduced by reducing the number of abnormal situations requiring teleoperator support (such as through AV trajectories, etc.).

[0025] When the AV encounters an obstructing situation, the AV may classify the obstructing situation as a normal situation or an abnormal situation. If the obstructing situation is classified as a normal situation, the AV may autonomously negotiate (e.g., bypass) the obstructing situation. On the other hand, if the obstructing situation is classified as an abnormal situation, the AV may determine (e.g., select, calculate, draw, etc.) a trajectory (i.e., a path) around the abnormal situation and determine the risk of the path. In some cases, the trajectory around the abnormal situation may simply be that the AV stops completely until the abnormal situation resolves itself.

[0026] The risk may be a numerical number (such as a percentage, a number between 0 and 1, a number between 0 and 10, or other numbers, etc.). In another example, the risk may be an ordered label (such as, "normal," "low," or "high," "green," "yellow," or "red," etc.). If the risk is below a risk threshold (e.g., less than "high," less than 75%), the AV may proceed autonomously along the path. On the other hand, if the risk is not below the risk threshold, the AV may issue a request to the teleoperator. The request may include a path. The teleoperator may only approve the path. Thus, if the AV receives approval for the path, the AV may proceed autonomously along the path.

[0027] In implementation, the systems and techniques disclosed herein can be summarized as follows. When an AV is driving autonomously without human supervision, the AV may encounter an obstacle. The obstacle may be a new object, which may be another road user, a static object, or another object. For example, the obstacle may be a stopped vehicle ahead (i.e., a vehicle driving in front of the AV). In response to the obstacle, the AV determines a path around the obstacle. If there may not be a path, the AV may stop and wait for the obstacle to clear. This is referred to as a waiting response in this article. If there may be a path, the AV may evaluate the risk associated with autonomously bypassing the obstacle using the path. Based on the risk, the AV may respond, whereby (a) if the risk is less than a risk threshold, the AV may autonomously bypass the obstacle in a normal manner; or (b) if the risk is not less than the risk threshold, the AV may contact the teleoperator for assistance. For ease of reference, responses (a) to (b) may be referred to as normal responses and auxiliary responses, respectively. In this way, possible responses of the AV to the obstacle may include a waiting response, a normal response, and an auxiliary response.

[0028] In the case of waiting for a response, the AV stops completely and can re-evaluate the situation over time. Depending on how the situation changes over time, the AV can continue to wait, autonomously circumvent the obstruction, or contact the teleoperator. In an example, if the obstruction situation does not resolve itself within a predefined period of time, the AV can perform an assisted response. In an example, the obstruction situation can change so that the AV can perform a normal response or an assisted response.

[0029] The AV may be programmed to stay in a single lane (e.g., a single lane constraint). However, when encountering an obstruction, the AV may relax (e.g., release) the single lane constraint and autonomously steer around the obstruction when a gap in other traffic allows. If there is no gap in traffic, the AV stops and waits, or remains waiting, until the situation changes (e.g., the obstruction clears, a gap in traffic exists, etc.).

[0030] In the case of an assisted response (ie, the AV contacts the teleoperator for assistance), the AV contacts the teleoperation center for assistance and makes a controlled stop behind the obstruction while awaiting a teleoperator response from the teleoperation center.

[0031] The request for assistance may include an AV suggested action. In an example, the AV suggested action may be to bypass an obstruction using a trajectory (e.g., a path) drawn by the AV. In an example, the trajectory may be a new route to the AV's destination. The trajectory drawn by the AV may be included in the request for assistance. The response from the teleoperator may be an authorization for the AV to perform the AV suggested action, also referred to herein as a "forward response" or "forward response"; an authorization for the AV to continue waiting, also referred to herein as a "wait response" or "waiting response"; an authorization for the AV to bypass the obstruction using the trajectory provided by the teleoperator, also referred to herein as a "trajectory response" or "response including a trajectory"; or other responses. If the teleoperator approves the AV suggested action to bypass the obstruction, the AV may temporarily enable itself to enter the opposite traffic lane and continue to autonomously bypass the obstruction.

[0032] In an implementation, the assistance response may be in the form of a ticket issued by the AV. The ticket may be assigned to a teleoperator at a teleoperation center. The ticket may be assigned to a specific teleoperator who may be selected based on the teleoperator's expertise, the AV's geographic location, the AV's status information, information about the obstruction situation, or some other criteria. In another implementation, the ticket may be placed in a first-in, first-out queue and assigned to the next available teleoperator.

[0033] Of the 19 exception situations described above, and using the techniques described herein, approximately 50% were judged to have a risk below the risk threshold. Thus, by resolving such blocking situations autonomously (i.e., without teleoperator intervention), the overall teleoperation time can be reduced by 50%. In addition, additional reductions in teleoperation time can be achieved by reducing the required teleoperator time to only approve the AV proposed path (i.e., the teleoperator only needs to approve the action most of the time, without having to closely guide the AV, such as by drawing a path for the AV), at least in some cases. Thus, implementations according to the present invention can reduce the teleoperation time of an AV fleet. For example, the ratio of AVs to required teleoperators can be reduced from 10:1 to 40:1 (e.g., from 1 teleoperator configured for every 10 AVs to 1 teleoperator configured for every 40 AVs). In this way, the teleoperation cost of exception handling can be reduced by 75%.

[0034] To describe some of the implementations in more detail, refer to the following figure.

[0035] Figure 11 is a diagram of an example of a vehicle 1050 that may implement aspects, features, and elements disclosed herein. The vehicle 1050 includes a chassis 1100, a powertrain 1200, a controller 1300, wheels 1400 / 1410 / 1420 / 1430, or any other element or combination of elements of the vehicle. Although the vehicle 1050 is shown as including four wheels 1400 / 1410 / 1420 / 1430 for simplicity, any other propulsion device or devices such as pushers or treads may be used. Figure 1 1400 / 1410 / 1420 / 1430, etc., indicate that information such as data or control signals, power such as electricity or torque, or both information and power can be communicated between the respective elements. For example, the controller 1300 can receive power from the powertrain 1200 and communicate with the powertrain 1200, the wheels 1400 / 1410 / 1420 / 1430, or both to control the vehicle 1050, which can include accelerating, decelerating, steering, or otherwise controlling the vehicle 1050.

[0036] The powertrain 1200 includes a power source 1210, a transmission 1220, a steering unit 1230, a vehicle actuator 1240, or any other element or combination of elements of the powertrain (such as a suspension, a drive shaft, an axle, or an exhaust system, etc.). Although shown separately, the wheels 1400 / 1410 / 1420 / 1430 can be included in the powertrain 1200.

[0037] The power source 1210 may be any device or combination of devices operable to provide energy such as electrical energy, thermal energy, or kinetic energy. For example, the power source 1210 includes an engine such as an internal combustion engine, an electric motor, or a combination of an internal combustion engine and an electric motor, and is operable to provide kinetic energy as a motive force to one or more of the wheels 1400 / 1410 / 1420 / 1430. In some embodiments, the power source 1210 includes a potential energy unit, such as one or more dry cells (e.g., nickel-cadmium (NiCd) batteries, nickel-zinc (NiZn) batteries, nickel-metal hydride (NiMH) batteries, lithium-ion (Li-ion) batteries, etc.); solar cells; fuel cells; or any other device capable of providing energy, etc.

[0038] The transmission 1220 receives energy such as kinetic energy from the power source 1210 and sends the energy to the wheels 1400 / 1410 / 1420 / 1430 to provide motive force. The transmission 1220 may be controlled by the controller 1300, the vehicle actuator 1240, or both. The steering unit 1230 may be controlled by the controller 1300, the vehicle actuator 1240, or both, and controls the wheels 1400 / 1410 / 1420 / 1430 to steer the vehicle. The vehicle actuator 1240 may receive a signal from the controller 1300 and may actuate or control the power source 1210, the transmission 1220, the steering unit 1230, or any combination thereof to operate the vehicle 1050.

[0039] In some embodiments, controller 1300 includes positioning unit 1310, electronic communication unit 1320, processor 1330, memory 1340, user interface 1350, sensor 1360, electronic communication interface 1370, or any combination thereof. Although shown as a single unit, any one or more elements of controller 1300 may be integrated into any number of separate physical units. For example, user interface 1350 and processor 1330 may be integrated into a first physical unit, and memory 1340 may be integrated into a second physical unit. Although in Figure 1 Although not shown, the controller 1300 may include a power source such as a battery, etc. Although shown as separate elements, the positioning unit 1310, the electronic communication unit 1320, the processor 1330, the memory 1340, the user interface 1350, the sensor 1360, the electronic communication interface 1370, or any combination thereof may be integrated into one or more electronic units, circuits, or chips.

[0040] In some embodiments, the processor 1330 includes any device or combination of devices that are currently available or subsequently developed and can manipulate or process signals or other information, including optical processors, quantum processors, molecular processors, or combinations thereof. For example, the processor 1330 may include one or more dedicated processors, one or more digital signal processors, one or more microprocessors, one or more controllers, one or more microcontrollers, one or more integrated circuits, one or more application-specific integrated circuits, one or more field programmable gate arrays, one or more programmable logic arrays, one or more programmable logic controllers, one or more state machines, or any combination thereof. The processor 1330 may be operably coupled to the positioning unit 1310, the memory 1340, the electronic communication interface 1370, the electronic communication unit 1320, the user interface 1350, the sensor 1360, the powertrain 1200, or any combination thereof. For example, the processor may be operably coupled to the memory 1340 via the communication bus 1380.

[0041] In some embodiments, processor 1330 may be configured to execute instructions including remote operation instructions that may be used to operate vehicle 1050 from a remote location including an operations center. The remote operation instructions may be stored in vehicle 1050, or received from an external source such as a traffic management center or a server computing device that may include a cloud-based server computing device. Processor 1330 may be configured to execute instructions for exception handling as described herein.

[0042] The memory 1340 may include any tangible, non-transitory computer-usable or computer-readable medium that, for example, is capable of containing, storing, communicating, or transporting machine-readable instructions or any information associated therewith for use by or in conjunction with the processor 1330. For example, the memory 1340 is one or more solid-state drives, one or more memory cards, one or more removable media, one or more read-only memories, one or more random access memories, one or more disks (including hard disks, floppy disks, optical disks, magnetic cards, or optical cards), or any type of non-transitory medium suitable for storing electronic information, or any combination thereof.

[0043] The electronic communication interface 1370 may be a wireless antenna (as shown), a wired communication port, an optical communication port, or any other wired or wireless means capable of interacting with a wired or wireless electronic communication medium 1500 .

[0044] Electronic communication unit 1320 may be configured to send or receive signals via a wired or wireless electronic communication medium 1500 (such as via electronic communication interface 1370 or the like). Figure 1 Although not explicitly shown, the electronic communication unit 1320 is configured to transmit, receive, or both via any wired or wireless communication medium, such as radio frequency (RF), ultraviolet (UV), visible light, optical fiber, wireline, or a combination thereof. Figure 1 A single electronic communication unit 1320 and a single electronic communication interface 1370 are shown, but any number of communication units and any number of communication interfaces may be used. In some embodiments, the electronic communication unit 1320 may include a dedicated short-range communication (DSRC) unit, a wireless security unit (WSU), IEEE802.11p (Wifi-P), or a combination thereof.

[0045] The positioning unit 1310 can determine geographic location information of the vehicle 1050, including but not limited to longitude, latitude, altitude, direction of travel, or speed. For example, the positioning unit includes a global positioning system (GPS) unit, such as a National Marine Electronics Association (NMEA) unit enabled with a wide area augmentation system (WAAS), a radio triangulation unit, or a combination thereof. The positioning unit 1310 can be used to obtain information representing, for example, the current heading of the vehicle 1050, the current position of the vehicle 1050 in two or three dimensions, the current angular orientation of the vehicle 1050, or a combination thereof.

[0046] User interface 1350 can comprise any unit that can be used as interface by people, comprise any one in virtual keyboard, physical keyboard, touch pad, display, touch screen, loud speaker, microphone, video camera, sensor and printer.As shown in the figure, user interface 1350 can be operably coupled with processor 1330 or with any other element of controller 1300.Although being shown as single unit, user interface 1350 can comprise one or more physical units.For example, user interface 1350 comprises the audio interface for carrying out audio communication with people and the touch display for carrying out the communication based on vision and touch with people.

[0047] Sensors 1360 may include one or more sensors (such as a sensor array, etc.) that may be operable to provide information that may be used to control the vehicle. Sensors 1360 may provide information related to the current operating characteristics of the vehicle or its surroundings. Sensors 1360 may include, for example, rate sensors, acceleration sensors, steering angle sensors, traction-related sensors, brake-related sensors, or any sensor or sensor combination that may be operable to report information related to certain aspects of the current dynamic conditions of the vehicle 1050.

[0048] In some embodiments, sensor 1360 may include sensors operable to obtain information related to the physical environment surrounding vehicle 1050. For example, one or more sensors detect road geometry and obstacles (such as fixed obstacles, vehicles, cyclists and pedestrians, etc.). In some embodiments, sensor 1360 may be or include one or more cameras, laser sensing systems, infrared sensing systems, acoustic sensing systems, or any other suitable type of vehicle-mounted environmental sensing device or device combination now known or later developed. In some embodiments, sensor 1360 and positioning unit 1310 are combined.

[0049] Although not shown separately, the vehicle 1050 may include a trajectory controller. For example, the controller 1300 may include a trajectory controller. The trajectory controller may be operable to obtain information describing the current state of the vehicle 1050 and the route planned for the vehicle 1050, and determine and optimize the trajectory of the vehicle 1050 based on the information. In some embodiments, the trajectory controller outputs a signal that is operable to control the vehicle 1050 so that the vehicle 1050 follows the trajectory determined by the trajectory controller. For example, the output of the trajectory controller may be an optimized trajectory, which may be supplied to the powertrain 1200, the wheel 1400 / 1410 / 1420 / 1430, or both. In some embodiments, the optimized trajectory may be a control input such as a set of steering angles, wherein each steering angle corresponds to a point in time or a position. In some embodiments, the optimized trajectory may be one or more paths, routes, curves, or a combination thereof.

[0050] One or more of wheels 1400 / 1410 / 1420 / 1430 may be steering wheels that pivot to a steering angle under the control of steering unit 1230, propulsion wheels that twist under the control of transmission 1220 to propel vehicle 1050, or steering propulsion wheels for steering and propelling vehicle 1050.

[0051] The vehicles may include Figure 1 Units or components not shown, such as housing, Bluetooth module, a frequency modulation (FM) radio unit, a near field communication (NFC) module, a liquid crystal display (LCD) display unit, an organic light emitting diode (OLED) display unit, a speaker, or any combination thereof.

[0052] Figure 2 2100 (e.g., a vehicle 2100, such as a vehicle 2100, a vehicle 2100, and a vehicle 2100 that can implement aspects, features, and elements disclosed herein. Figure 1 The vehicle 1050 shown in the figure, etc.) and one or more external objects (such as, external object 2110, etc.), which external object can include any form of transportation (such as, Figure 1 The vehicle 2100 may travel via one or more portions of the transportation network 2200 and may communicate with external objects 2110 via one or more of the electronic communication networks 2300. Figure 2Although not explicitly shown, vehicles may traverse areas that are not explicitly or fully included in the transportation network (such as off-road areas, etc.). In some embodiments, the transportation network 2200 may include one or more vehicle detection sensors 2202 (such as inductive loop sensors, etc.) that may be used to detect the movement of vehicles on the transportation network 2200.

[0053] The electronic communication network 2300 may be a multi-access system for providing communications such as voice communication, data communication, video communication, messaging communication, or a combination thereof, between the vehicle 2100, the external object 2110, and the operation center 2400. For example, the vehicle 2100 or the external object 2110 may receive information such as information indicating the transportation network 2200 from the operation center 2400 via the electronic communication network 2300.

[0054] The operation center 2400 includes a controller device 2410, which includes Figure 1 2100 and external objects 2110. Controller device 2410 may monitor and coordinate the movement of vehicles, including autonomous vehicles. Controller device 2410 may monitor the state or condition of vehicles such as vehicle 2100 and external objects such as external object 2110. Controller device 2410 may receive vehicle data and infrastructure data including any of the following: vehicle speed; vehicle position; vehicle operating status; vehicle destination; vehicle route; vehicle sensor data; external object speed; external object position; external object operating status; external object destination; external object route; and external object sensor data.

[0055] In addition, the controller device 2410 can establish remote control of one or more vehicles such as the vehicle 2100 or external objects such as the external object 2110. In this way, the controller device 2410 can remotely operate the vehicle or external object from a remote location. The controller device 2410 can exchange (send or receive) state data with a vehicle (such as the vehicle 2100, etc.), an external object (such as the external object 2110, etc.) or a computing device (such as the server computing device 2500, etc.) via a wireless communication link (such as the wireless communication link 2380, etc.) or a wired communication link (such as the wired communication link 2390, etc.).

[0056] The server computing device 2500 may include one or more server computing devices that may exchange (send or receive) status signal data with one or more vehicles or computing devices including the vehicle 2100 , the external object 2110 , or the operation center 2400 via the electronic communication network 2300 .

[0057] In some embodiments, the vehicle 2100 or external object 2110 communicates via a wired communication link 2390, a wireless communication link 2310 / 2320 / 2370, or a combination of any number or type of wired or wireless communication links. For example, as shown, the vehicle 2100 or external object 2110 communicates via a ground wireless communication link 2310, via a non-ground wireless communication link 2320, or via a combination thereof. In some implementations, the ground wireless communication link 2310 includes an Ethernet link, a serial link, a Bluetooth link, an infrared (IR) link, an ultraviolet (UV) link, or any link capable of providing electronic communication.

[0058] A vehicle such as vehicle 2100 or an external object such as external object 2110 may communicate with another vehicle, external object, or operation center 2400. For example, the host or subject vehicle 2100 may receive one or more automated inter-vehicle messages such as basic safety messages (BSMs) from the operation center 2400 via a direct communication link 2370 or via an electronic communication network 2300. For example, the operation center 2400 may broadcast the message to host vehicles within a defined broadcast range (such as 300 meters, etc.), or to a defined geographic area. In some embodiments, the vehicle 2100 receives the message via a third party such as a signal repeater (not shown) or another remote vehicle (not shown). In some embodiments, the vehicle 2100 or external object 2110 periodically sends one or more automated inter-vehicle messages based on a defined interval such as 100 milliseconds.

[0059] Automated inter-vehicle messages may include vehicle identification information, geospatial status information (such as longitude, latitude, or altitude information, etc.), geospatial position accuracy information, dynamic status information (such as vehicle acceleration information, yaw rate information, velocity information, vehicle heading information, brake system status data, throttle information, steering wheel angle information, or vehicle route information, etc.), or vehicle operating status information (such as vehicle size information, headlight status information, turn signal information, wiper status data, transmission information, etc.), or any other information or combination of information related to the status of the sending vehicle. For example, transmission status information indicates whether the transmission of the sending vehicle is in a neutral state, a parked state, a forward state, or a reverse state.

[0060] In some embodiments, the vehicle 2100 communicates with the electronic communication network 2300 via an access point 2330. The access point 2330, which may include a computing device, may be configured to communicate with the vehicle 2100, the electronic communication network 2300, the operation center 2400, or a combination thereof via a wired or wireless communication link 2310 / 2340. For example, the access point 2330 is a base station, a base transceiver station (BTS), a node B, an enhanced node B (eNode-B), a home node B (HNode-B), a wireless router, a wired router, a hub, a repeater, a switch, or any similar wired or wireless device. Although shown as a single unit, the access point may include any number of interconnected elements.

[0061] Vehicle 2100 may communicate with electronic communication network 2300 via satellite 2350 or other non-terrestrial communication means. Satellite 2350, which may include a computing device, may be configured to communicate with vehicle 2100, with electronic communication network 2300, with operations center 2400, or a combination thereof via one or more communication links 2320 / 2360. Although shown as a single unit, a satellite may include any number of interconnected elements.

[0062] The electronic communication network 2300 can be any type of network configured to provide voice, data, or any other type of electronic communication. For example, the electronic communication network 2300 includes a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a mobile or cellular telephone network, the Internet, or any other electronic communication system. The electronic communication network 2300 can use communication protocols such as the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), the Internet Protocol (IP), the Real-time Transport Protocol (RTP), the Hypertext Transfer Protocol (HTTP), or a combination thereof. Although shown as a single unit, the electronic communication network can include any number of interconnected elements.

[0063] In some embodiments, vehicle 2100 communicates with operations center 2400 via electronic communications network 2300, access point 2330, or satellite 2350. Operation center 2400 may include one or more computing devices capable of exchanging (sending or receiving) data from a vehicle such as vehicle 2100, an external object including external object 2110, or a computing device such as server computing device 2500.

[0064] In some embodiments, the vehicle 2100 identifies a portion or condition of the transportation network 2200. For example, the vehicle 2100 may include one or more onboard sensors 2102 (such as, Figure 1The one or more vehicle-mounted sensors 2102 include a rate sensor, a wheel speed sensor, a camera, a gyroscope, an optical sensor, a laser sensor, a radar sensor, an acoustic sensor, or any other sensor or device or combination thereof capable of determining or identifying a portion or condition of the transportation network 2200.

[0065] Vehicle 2100 may use information communicated via electronic communication network 2300 (such as information representative of transportation network 2200, information identified by one or more onboard sensors 2102, or a combination thereof) to traverse one or more portions of transportation network 2200. External object 2110 may be capable of all or some of the communications and actions described above for vehicle 2100.

[0066] To keep it simple, Figure 2 Vehicle 2100 is shown as a host vehicle, external object 2110, transportation network 2200, electronic communication network 2300, and operation center 2400. However, any number of vehicles, networks, or computing devices may be used. In some embodiments, vehicle transportation and communication system 2000 includes Figure 2 Although vehicle 2100 or external object 2110 is shown as a single unit, the vehicle may include any number of interconnected elements.

[0067] Although the vehicle 2100 is shown as communicating with the operations center 2400 via the electronic communications network 2300, the vehicle 2100 (and the external object 2110) may communicate with the operations center 2400 via any number of direct or indirect communications links. For example, the vehicle 2100 or the external object 2110 may communicate with the operations center 2400 via a direct communications link such as a Bluetooth communications link. Although for simplicity, Figure 2 One transportation network 2200 and one electronic communications network 2300 are shown, but any number of networks or communications devices may be used.

[0068] Figure 3 300 is a block diagram illustrating a remote vehicle assistance center 300 according to an implementation of the present invention. The remote vehicle assistance center 300 may also be referred to as a remote system or a teleoperation center. The remote vehicle assistance center 300 includes a queue manager 310, a plurality of vehicle managers (i.e., teleoperators) including but not limited to teleoperator 320 and teleoperator 330, and a plurality of vehicles including but not limited to vehicles 340, 350, 360, and 370.

[0069] The queue manager 310 may include Figure 1The controller 1300 shown or Figure 2 A device that can monitor some or all of the features of a controller device 2410 of a teleoperator. A queue manager 310 can monitor and coordinate the movement of teleoperators, including teleoperators 320 / 330, and vehicles, including autonomous vehicles and vehicles 340 / 350 / 360 / 370. Monitoring and coordinating teleoperators can include any of the following operations: assigning, allocating, or unassigning vehicles to teleoperators; reviewing and monitoring teleoperator performance data; and assigning teleoperators to geographic areas. In an implementation, there can be multiple queue managers, which can in turn be managed or authorized by other queue managers.

[0070] Teleoperator 320 may monitor the status or condition of vehicles (including vehicle 340 and vehicle 350). Figure 3 As shown, teleoperator 320 has been assigned vehicle 340 and vehicle 350. The assignment of vehicles to teleoperators may be made by a queue manager, such as queue manager 310.

[0071] Teleoperator 330 may monitor the status or condition of vehicles (including vehicle 360 ​​and vehicle 370). Figure 3 As shown, teleoperator 330 has been assigned vehicle 360 ​​and vehicle 370. The assignment of vehicles to teleoperators may be made by a queue manager, such as queue manager 310. The assignment of vehicles to teleoperators may also be automated using machine learning techniques.

[0072] In implementations, teleoperators may cluster or group vehicles, establish communications with occupants in vehicles, remotely operate vehicles, and coordinate the movement of vehicles through transportation networks or around various obstacles such as traffic jams, etc. Teleoperators may interact with other teleoperators to assist in the monitoring and management of vehicles.

[0073] The vehicles including vehicles 340 / 350 / 360 / 370 include vehicles such as Figure 2 The vehicle 2100 shown is a vehicle such as the vehicle 2100 shown. The vehicle can be operated autonomously or by a human driver, and can exchange (send and receive) vehicle data related to the status or condition of the vehicle and its surroundings, including any of the following: vehicle speed; vehicle position; vehicle operating status; vehicle destination; vehicle route; vehicle sensor data; external object speed; and external object position.

[0074] In the description herein, sentences describing that an autonomous vehicle takes an action (such as performing, determining, initiating, receiving, calculating, deciding, etc.) should be understood as some appropriate module of the AV taking the action. Such a module may be stored in a memory of the AV (such as Figure 1 1340, etc.), and by a processor (such as, Figure 1 Such a module may be partially or completely included in a processor such as Figure 2 The controller device 2410 of the AV and the like may be partially or completely executed by a processor of the AV, a processor of the operation center, or a combination thereof. For example, the statement "the AV determines the trajectory" may be understood to mean "a module of the AV determines the trajectory" or "a trajectory planning module of the AV determines the trajectory".

[0075] Figure 4 is an example of a module of a system 400 for exception handling according to an implementation of the present invention. The system 400 may include Figure 1 The vehicle 1050 shown, Figure 2 The vehicle 2100 shown, Figure 3 The system 400 may be stored in a vehicle such as a Figure 1 The memory 1340 or the like may be used by a memory such as Figure 1 Computer-executable instructions executed by processor 1330 or the like.

[0076] System 400 includes world model module 402, trajectory planning module 404, obstacle manager module 406, and risk assessment module 408. Other examples of system 400 may include more, fewer, or other modules. In some examples, some modules may be combined; in other examples, modules may be divided into one or more other modules. For example, risk assessment module 408 may be combined with obstacle manager module 406. The modules of system 400 or a subset thereof may be referred to as exception handling modules.

[0077] The world model module 402 may be, for example, Figure 1 The sensor 126 of the present invention receives sensor data and determines (e.g., converts to, detects, etc.) objects based on the sensor data. That is, for example, the world model module 402 can determine the road user (e.g., one or more external objects or such as those related to the road user) based on the received sensor data. Figure 2The external objects 2110 described above. For example, the world model module 402 can convert a point cloud received from a light detection and ranging (LiDAR) sensor (i.e., a sensor of the sensor 126) into an object. Sensor data from several sensors can be fused together to determine an object (e.g., guess the identity of an object). Examples of objects include bicycles, pedestrians, vehicles, structures, etc. The objects maintained by the world model module 402 can include static objects and / or dynamic objects. A dynamic object is a world object that is currently moving. A static object can be any non-moving object. A building or a traffic sign can be an example of a static object. A static object can be an object that was previously classified as a dynamic object; or vice versa.

[0078] The world model module 402 may receive sensor information that enables the world model module 402 to compute and maintain additional information for at least some of the detected objects. For example, the world model module 402 may maintain the state of at least some of the determined objects. For example, the state of the object may include zero or more of a speed, a pose, a geometry (such as width, height, and depth, etc.), a classification (e.g., a bicycle, a large truck, a pedestrian, a road sign, etc.), and a position. Thus, the state of the object includes discrete state information (e.g., a classification) and continuous state information (e.g., a pose and a speed).

[0079] The world model module 402 can fuse sensor information, can track objects, can maintain a list of hypotheses for at least some dynamic objects (e.g., object A may be going straight, turning right, or turning left), can create and maintain a predicted trajectory for each hypothesis, and can maintain a likelihood estimate for each hypothesis (e.g., given object A's pose / velocity and trajectory pose / velocity, there is a 90% probability that object A will go straight).

[0080] In an example, the world model module 402 generates predicted trajectories for each object hypothesis for at least some dynamic objects using one or more instances of the trajectory planning module 404. For example, an instance of the trajectory planner can be used to generate predicted trajectories for vehicles, bicycles, and pedestrians. In another example, an instance of the trajectory planner can be used to generate predicted trajectories for vehicles and bicycles, and a different method can be used to generate predicted trajectories for pedestrians.

[0081] The trajectory planning module 404 can generate a trajectory (i.e., a path) of the vehicle by detecting (e.g., sensing, observing, etc.) the presence of static objects and anticipating (i.e., predicting) the trajectories of other users of the vehicle transportation network (e.g., road users, dynamic objects). The trajectory planning module 404 can generate a trajectory of the AV from the source location to the destination location by, for example, the following operations: receiving map data, teleoperation data, and other input data; longitudinally stitching (e.g., fusing, connecting, etc.) the input data to determine a rate distribution of the path from the source location to the destination location (e.g., a rate distribution that specifies how fast the AV can drive along different segments of the path from the source location to the destination location); and having a trajectory planner process constraints related to static and dynamic objects observed based on the AV's sensor data at discrete time points (e.g., every few milliseconds) to generate a smooth trajectory for the next time window (e.g., 6 seconds of look-ahead time) of the AV.

[0082] The trajectory planning module 404 can determine the trajectory for the next few seconds. Thus, and in the example where the next few seconds are the next 6 seconds (i.e., 6 seconds of look-ahead time), the trajectory planning module 404 can determine the trajectory and position of the AV in the next 6 seconds. For example, the trajectory planning module 404 can determine (e.g., predict, calculate, etc.) the expected position of the AV at several time intervals (e.g., every quarter second or some other time interval). The trajectory planning module 404 can determine the detailed planned trajectory based on the predictable responses of other road users.

[0083] In an example, the world model module 402 can generate predicted trajectories for object hypotheses of at least some dynamic objects using an instance of the trajectory planning module 404. For example, predicted trajectories for vehicles, bicycles, and pedestrians can be generated using an instance of the trajectory planning module 404. In another example, predicted trajectories for vehicles and bicycles can be generated using an instance of the trajectory planning module 404, and predicted trajectories for pedestrians can be generated using a different method.

[0084] Trajectory planning module 404 can receive expected (i.e., predicted) trajectories of other users (also referred to as real-world objects) of the vehicle transportation network from world model module 402. For each detected dynamic object (e.g., real-world object such as a vehicle, pedestrian, and bicycle, etc.), the world model module can maintain (e.g., predict and update) one or more hypotheses related to the possible intent of the real-world object. Examples of intents (e.g., hypotheses) include stop, turn right, turn left, go straight, pass, and park. A likelihood is associated with each hypothesis. The likelihood is updated based on observations received from sensor data.

[0085] In an example, in order to determine the path of the next time window (such as bypassing obstacles, etc.), the trajectory planning module 404 can determine the drivable area and discrete time rate planning of the next time window. The drivable area can be, for example, an area of ​​the vehicle transportation network where the AV can be driven. Initially, the drivable area can include an area where the AV cannot be predicted to be safely driven. The trajectory planning module 404 cuts out the area where the AV cannot be predicted to be safely driven from the drivable area. This process obtains an adjusted drivable area. The adjusted drivable area can include an area that violates traffic regulations.

[0086] The trajectory planning module 404 can identify objects in the vicinity of the AV. In an example, the nearby objects can be at least some of the external objects maintained by the world model module 402. For example, the nearby objects can be objects within a predetermined distance from the AV, objects within a predicted arrival time of the AV, or objects that meet other criteria for identifying a subset of objects, such as objects that constitute an obstruction situation.

[0087] The trajectory planning module 404 determines an adjusted drivable area. The "adjusted drivable area" is the drivable area after removing the area that takes into account static and / or dynamic objects from the drivable area. For example, in determining the adjusted drivable area, the trajectory planning module 404 may take into account any objects that cause obstruction situations, oncoming vehicles, and other road objects that may be near the AV (from behind, to the left, to the right, etc.).

[0088] The trajectory planning module 404 adjusts the drivable area of ​​the static object. That is, the trajectory planning module 404 removes (e.g., cuts out, etc.) the portion of the drivable area where the static object is located from the drivable area. This is because the AV is to be controlled to navigate (e.g., drive) around the static object. Note that an object currently classified as a static object may be classified as a dynamic object at a previous moment, and may become a dynamic object at a future moment. The size of the cut-out area may be determined based on an estimate of the size of the static object. The size of the cut-out area may include a gap area so that the AV does not drive too close to the static object.

[0089] The trajectory planning module 404 can adjust the discrete time rate plan for static objects. For example, in the absence of obstacles or other road users, the discrete time rate plan can follow a predetermined rate distribution. For example, when the adjusted drivable area includes a narrow passage to take into account the static object, instead of following the strategic distribution (i.e., using the strategically distributed rate) literally (i.e., as set in the strategic distribution), the trajectory planning module 404 adjusts the discrete time rate plan to reduce the rate of the AV to a comfortable rate. For example, when the adjusted drivable area includes a static obstruction (e.g., an obstruction situation) to take into account the static object, the trajectory planning module 404 adjusts the discrete time rate plan so that the AV stops at a specified distance before the static obstruction.

[0090] The trajectory planning module 404 can then adjust the drivable area for the dynamic object. That is, the trajectory planning module 404 cuts out portions of the drivable area based on the corresponding predicted trajectories of each dynamic object. The trajectory planning module 404 can use timing information related to the position of each dynamic object to cut out additional portions of the drivable area. The cut-out portion in the drivable area of ​​the dynamic object is generated by comparing the predicted timing for the dynamic object with the timing generated by the discrete time rate planning that now takes into account the static object. That is, the trajectory planning module 404 can predict where the dynamic object is at different discrete time points relative to the position of the AV at the same discrete time point for the dynamic object and based on the predicted trajectory of the dynamic object. The position of the dynamic object is matched to the predicted position of the AV to determine the cut-out portion.

[0091] The trajectory planning module 404 may then perform optimization operations (one or more) such as constraint operations to determine an optimal trajectory for the AV. The trajectory planning module 404 may calculate (e.g., determine, generate, etc.) an optimal trajectory for the AV using (i.e., as input to the optimization operations) a motion model of the AV (e.g., a dynamic motion model), a coarse driving line (e.g., a line in the road that coincides with the longitudinal axis of the AV as the AV moves along the road), and the adjusted drivable area, more, less, or other inputs.

[0092] The obstruction manager module 406 can identify obstruction situations. For example, if the obstruction manager module 406 detects that the trajectory of the AV as determined by the trajectory planning module 404 (i.e., the first trajectory) is blocked, then after a predetermined time period (e.g., a few seconds), the obstruction manager module 406 can cause the trajectory planning module 404 to generate a second trajectory by relaxing traffic regulations. The first trajectory may have been generated without violating traffic regulations. For example, when deriving the drivable area, the trajectory planning module 404 may have excluded areas outside of road lanes and / or roads in the opposite direction. When generating the second trajectory, the obstruction manager module 406 can direct the trajectory planning module 404 to use areas that were considered non-drivable areas by the trajectory planning module 404 when generating the first trajectory.

[0093] The obstruction manager module 406 may cause the risk assessment module 408 to evaluate the risk associated with the obstruction situation, the second trajectory, or a combination thereof. In the event that the risk associated with the second trajectory is greater than a risk threshold, the obstruction manager module 406 may initiate a request to the teleoperator, as further described below. The obstruction manager module 406 may receive a response from the teleoperator and cause the AV to operate according to the response. As described above, the teleoperator's response may be a wait response, a go response, a response that includes the trajectory, or some other response, as further described below.

[0094] The risk assessment module 408 determines a risk associated with an obstructing situation, with a trajectory, or a combination thereof. In a first example, the risk may be considered a measure of the AV's understanding of an obstructing scenario (such as based on received sensor data, high definition (HD) map data, and other data available to the AV). For ease of reference, the measure of the AV's understanding of an obstructing scenario may be referred to as a situation likelihood value, or simply situation likelihood. In a second example, the risk may be based on the situation likelihood and a weighting of the result(s) of following the second trajectory generated by the trajectory planning module 404 as described above. Other ways of measuring risk are possible.

[0095] In a first example, the risk assessment module 408 determines risk based on the values ​​of features that may be important and / or relevant to resolving the abnormal situation. Features may include the presence of any other world objects in the scene, state information of other world objects, the location of the AV on the HD map, the speed of the AV, the visibility of the AV, specific features related to the stopped vehicle, less information, additional information, other information, or a combination thereof.

[0096] For example, features related to other world objects may include the presence or absence of any of vehicles, bicycles, pedestrians and their corresponding numbers, whether the world object was previously detected or a new object, and / or whether the previously detected object was previously judged to be moving (e.g., whether the classification of the detected object changed from dynamic to static). For example, a previous observation that a vehicle was moving in the past but is now stopped may indicate that the vehicle may be moving again. The location of the AV on the HD map may include the type of road, the presence of traffic signs (e.g., traffic lights, stop signs, etc.) and intersections, etc. The visibility of the AV may include whether the field of view of the AV is obstructed (such as by a stopped vehicle, etc.), the presence of mountains, etc. Other features related to stopped vehicles may include, to the extent that they are perceived and detected using sensor information: whether the hazard lights are on, whether the doors are open and which doors are open, and whether pedestrians are loading or unloading, etc. The determined risk value may be a weighted combination of the feature values.

[0097] The presence of pedestrians in the scene may increase the risk. The unobstructed field of view of the AV may reduce the risk. The uncertainty (or degree of uncertainty) associated with the classification of the object may increase the risk. For example, when a vehicle can be parked, it may not be possible to determine whether the vehicle is permanently stopped (such as because the vehicle is damaged, etc.) or whether the vehicle is temporarily stopped (such as at an intersection, etc.). The degree of certainty can be associated with whether the object is permanently stopped. Other markers can be used to determine the risk of whether the vehicle is permanently stopped or temporarily stopped. For example, detecting a pylon and / or flashing hazard lights can increase the certainty that the vehicle is permanently stopped. For example, the longer the vehicle stops, the higher the probability that the vehicle stops permanently. Bad weather conditions (snow, fog, etc.) may increase the risk. For example, observing (e.g., determining, evaluating, etc.) that a vehicle stops at an intersection, such as based on HD map data, may indicate that the vehicle is more likely to give way to other traffic. In this way, the vehicle is likely to move again; unless it is observed that it has been stationary at the intersection for a duration exceeding the normal give way time.

[0098] In the example, a vehicle ahead that was previously judged to be moving (and therefore likely to continue moving) may cause the AV to select a strategy of waiting rather than bypassing the vehicle. However, if the vehicle ahead is not judged to have been moving for a predetermined period of time (e.g., 2 minutes, 3 minutes, or some other stop time), a risk associated with the second trajectory may be determined.

[0099] In a second example, and as described above, the risk assessment module 408 can determine the risk based on a weighting (e.g., a product) of the situation likelihood value and the result of following the second trajectory generated by the trajectory planning module 404 as described above. The risk assessment module 408 can be thought of as calculating the risk of enabling the AV to circumvent the obstacle without remote human supervision. The risk can be a measure of the likelihood of the outcome weighted by the social cost of the outcome. Mathematically speaking, given a possible outcome {Yi} (e.g., the obstacle moves while the AV is circling), using the likelihood {Pi} and the "social cost" {Ci}, the risk can be calculated as risk=max_i{Pi*Ci}.

[0100] For example, suppose a person is detected in an obstruction scene. A person in the scene (not a normal pedestrian crossing the road) increases risk, at least because i) it is unknown whether the person is a legitimate traffic controller (e.g., a traffic policeman or a construction worker); and ii) ignoring a legitimate traffic controller or ignoring his instructions may have a high cost. If the person is actually a policeman, but has not been classified as such with complete certainty (such as by the world model module 402), then the consequences of ignoring the policeman's instructions are high.

[0101] In a second example, risk may be described as the risk of the AV making a decision (e.g., a decision to follow a trajectory) based on the AV's own understanding of the obstructing situation and without any further input (such as from a human teleoperator, etc.). In other words, the calculation of risk is based on the action selected by the AV. The action selected by the AV is essentially the action of proceeding according to the second trajectory. In yet other words, risk may be calculated based on the AV's understanding of the situation. Lack of knowledge (i.e., understanding) may mean that unacceptable outcomes (e.g., disobeying a traffic policeman) cannot be eliminated and therefore must be considered possible.

[0102] The risk assessment module 408 can estimate one or more situation likelihood values, each of which is associated with a combination of possible feature values. Features in the scene can be used to inform possible outcome likelihoods and costs. For example, if there is a clear view and enough space ahead, the cost of exceeding the obstacle to movement may be low. For example, if the hazard lights of the obstacle are turned on, the likelihood of its movement may also be low. Therefore, the risk is very low (e.g., risk = low * low = (very low)). For example, if a person is identified in the scene, a situation likelihood value can be associated with the person being a policeman, and a second situation likelihood value can be associated with the person being a construction worker. In addition, although a second trajectory is described above, the trajectory planning module 404 can generate more than one second trajectory and evaluate the results associated with each second trajectory. The final risk assessment (e.g., the risk value provided by the risk assessment module 408) can be the worst-case risk associated with the set of results considered. That is, the risk can be the maximum value (i.e., the most risky) of all the risk values ​​calculated.

[0103] As shown in Table 1, the product of the situation likelihood value and the result value can be used as a risk value. In Table 1, which is reminiscent of a risk assessment matrix, the situation likelihood value is shown as having three values ​​(i.e., low, medium, high), and the result value is also shown as having three values ​​(i.e., low, medium, high). However, as will be appreciated by those skilled in the art, more or fewer values ​​(labels or numerical values) may be assigned to the situation likelihood value and / or the result value. In the case of using numerical values, the risk may be determined as the situation likelihood value multiplied by the result value. An example of a risk assessment matrix is ​​shown in Table 1.

[0104]

[0105] Table I

[0106] Table I shows that when the situation likelihood / result value combination is judged to be low / medium, low / low, and medium / low, the risk is identified as "green"; when the situation likelihood / result value combination is judged to be low / high, medium / medium, and high / low, the risk is identified as "yellow"; when the situation likelihood / result value combination is judged to be medium / high, high / high, and high / medium, the risk is identified as "red". In the example, in the green and yellow risk situations, the AV can autonomously (i.e., without the intervention or help of the teleoperator) resolve the obstruction situation (e.g., bypass the obstruction situation, deal with the obstruction situation, navigate around the obstruction situation, etc.). However, when the risk is red, the AV (i.e., the obstruction manager module 406) initiates a request for assistance from the teleoperator.

[0107] Figure 5is a flow chart of a technique 500 for exception handling by an autonomous vehicle (AV) according to an implementation of the present invention. Figure 1 The vehicle 1050 shown, Figure 2 The vehicle 2100 shown, Figure 3 One of the vehicles 340 / 350 / 360 / 370 (e.g., an autonomous vehicle) or a vehicle including a vehicle such as Figure 2 Some or all of the technique 500 for exception handling may be implemented in a computing device of a controller device such as the controller device 2410 shown. The controller device may include an exception handling module (such as Figure 4 In an implementation, some or all aspects of the technique 500 for exception handling may be implemented in a system incorporating some or all of the features described in the present invention. The technique 500 may be implemented by Figure 4 At least some modules of the system 400 are partially or completely executed.

[0108] At 502, technique 500 identifies an abnormal situation. In an implementation, identifying the abnormal situation may include identifying an obstruction to a first trajectory of the AV, determining a second trajectory that avoids the obstruction, and determining that the second trajectory violates a driving specification.

[0109] Figures 6A to 6B 6 is a block diagram illustrating examples of obstruction situations 610, 630, 650, 670 according to implementations of the present invention. The obstruction situations 610, 630, 650 are depicted from the perspective of, for example, a front-facing camera and / or sensor of an AV (not shown). That is, in each obstruction situation 610, 630, 650, the illustrated example depicts a view (e.g., an obstruction scene) as perceived by one or more sensors of the AV.

[0110] In an obstruction situation 610, an AV is depicted in the right lane of a two-lane road including a left lane 612. The AV stops behind a vehicle 614, which in turn stops behind a truck 616. The vehicle 614, the truck 616, and any other road users depicted with respect to any other (one or more) figures may be maintained by the AV's world model module 402. The AV may detect an obstruction situation based on at least one or more criteria. For example, an obstruction situation may be detected if the AV's path remains obstructed for more than a predefined period of time.

[0111] The predefined time period may be 1 minute, 2 minutes, or some other predefined time period. In an example, the predefined time period may depend on an average traffic signal time close to the AV. For example, the AV (or a module thereof) may include (or may utilize) high definition (HD) map information that may include traffic signal information. Similarly, the AV may include a global positioning sensor (GPS). Thus, using the location of the AV (using the GPS information) and the map information, the AV may determine the average traffic signal time.

[0112] In another example, the AV may detect an obstruction because the AV's path remains obstructed due to the traffic signal 618 having turned green, the vehicle 620 in the left lane 612 being moved forward, some other criteria, or a combination thereof. The HD map information may include accurate information about the vehicle transportation network to within a few centimeters. For example, the HD map information may include details about road lanes, road dividers, traffic signals, traffic signs, speed limits, and the like.

[0113] In the example, you can use Figure 4 The trajectory planning module 404 or the like of the AV may determine that the path (e.g., trajectory) of the AV is obstructed. For example, the trajectory planning module may determine that if the AV continues along its current trajectory, the AV will encounter (e.g., collide with) a static obstruction (i.e., vehicle 614) within 0.5 seconds. As described above, an instance of the trajectory planning module may be used to determine the trajectory of vehicle 614. If vehicle 614 has not moved for a certain period of time, the state information in world model module 402 may indicate that vehicle 614 is a static object.

[0114] Obstruction situation 630 shows an AV traveling on a single lane road where lane 632 is used for oncoming traffic. Vehicle 634 is stopped. Status information of vehicle 634 (such as may be obtained from, for example, Figure 4 The AV may detect that a road cone 636 is placed behind the vehicle 634.

[0115] Obstruction situation 650 shows that the AV is approaching an intersection 652. The AV can identify the intersection based on map information available to the AV. Vehicles 654 and 656 are stopped. However, an obstruction module (such as Figure 4The AV's world model module 402 may be unable to determine with certainty whether at least one of the vehicles 654, 656 is stopped at the intersection or stopped for some other reason. A pedestrian 658 and a vehicle 660 are also detected in the scene. As described above, the AV's world model module 402 may maintain state information associated with each of the vehicles 654, 656, 660 and the pedestrian 658.

[0116] Obstruction situation 670 shows that the AV is approaching intersection 672. Intersection 672 is at the top of a hill. As such, the AV's sensors cannot observe other world objects in opposing lane 674 that may be on the other side of the hill. Vehicle 676 is stopped. Vehicle 676 is obstructing the AV's path. In some examples, the AV is able to detect that vehicle 676 is a minivan. The AV's obstruction module (such as Figure 4 The obstruction module 406, etc.) can identify the vehicle 676 as being loaded / unloaded because the rear door of the vehicle 676 is open.

[0117] As described above, the AV can determine a trajectory that avoids obstacles (e.g., navigates around obstacles). In an example, a trajectory planning module (such as Figure 4 The obstacle module may instruct the trajectory planning module to generate trajectories such that operating the AV according to these trajectories will violate traffic regulations.

[0118] Regarding the obstruction situation 610, the AV may determine a trajectory whereby the AV uses the left lane 612. The direction of traffic in the left lane 612 is the same as that of the AV. Thus, by moving to the left lane 612, the AV does not violate any traffic regulations. In another example, the AV may decide to continue to park behind the vehicle 614 (e.g., because the left lane 612 is not clear). In this case (e.g., continuing to park behind the vehicle 614), traffic regulations are not violated either. Since no traffic regulations are violated, the AV may classify the obstruction situation 610 as a normal obstruction situation.

[0119] On the other hand, obstruction situations 630 , 650 , 670 are classified as abnormal situations because in order to bypass the obstruction, the AV must cross the center line of the road and travel on the opposite traffic lane (eg, lane 632 or reverse lane 674 ).

[0120] exist Figure 5At 504, the technique 500 identifies a risk associated with the abnormal situation. That is, the technique 500 identifies a risk associated with autonomously resolving the abnormal situation, such as resolving the abnormal situation without human (e.g., teleoperator) supervision, etc. In other words, the technique 500 may identify a risk associated with a determined trajectory for navigating around an obstacle. In an example, the risk may be determined by a risk assessment module (such as Figure 4 In an example, the risk can be considered as a measure of the AV's understanding. In another example, the risk can be based on the AV's understanding of the obstruction scene and the weighting of the results of following the second trajectory that avoids the obstruction.

[0121] about Fig. 6A In the case of an obstacle 630, the risk associated with autonomous navigation around the obstacle is not high. However, since navigating around the obstacle requires violating traffic regulations (e.g., traveling in lane 632), the risk is medium ("yellow" according to Table I). Lane 632 is the oncoming (i.e., reverse) traffic lane. In the example, driving in the oncoming traffic lane can be considered at least a medium (e.g., yellow) risk.

[0122] Vehicle 634 is most likely to be permanently stopped because other signs in the scene (e.g., road cone 636 or flashing lights) indicate so. Thus, the risk associated with suspending traffic regulations is not high. The risk is medium risk (e.g., "yellow" according to Table I). However, even if it cannot be determined with high certainty whether vehicle 634 is permanently stopped, the risk is still not high because the AV's sensors can sense (e.g., see) far ahead.

[0123] The consequence of misclassifying vehicle 634 as permanently stopped is also low. For example, if vehicle 634 begins to move while the AV is in the process of maneuvering the second trajectory to avoid the obstacle so as to pass vehicle 634, technique 500 will have to determine whether to accelerate the AV further to completely pass vehicle 634 or to decelerate the AV so that vehicle 634 can pass ahead of the AV. In either case, the severity of the consequence is low because there is ample road ahead (i.e., clear view ahead).

[0124] about Fig. 6AFor obstruction situation 650, the risk is high (e.g., "red" per Table I). This is because it is impossible to determine whether vehicles 654 and 656 are yielding to something in front of them. Additionally, the HD map data may indicate the presence of intersection 652. As such, vehicles 654 and 656 may be lawfully stopped for some other reason; i.e., vehicles 654 and 656 may be yielding at the intersection. The risk is also high because a person (i.e., pedestrian 658) is detected in the scene. The risk may be considered high at least because i) the person may not behave as a typical pedestrian might be expected to behave and / or ii) vehicles 654 and 656 may be lawfully stopped at the stop intersection ahead. For example, the person may be nearly stationary, rather than walking steadily from one side of the road to the other.

[0125] Additionally, the identity (e.g., role) of pedestrian 658 may not be determined with certainty. For example, pedestrian 658 may be a vehicle driver who has stopped to talk with another vehicle driver. If this is the case, the risk may be lower than if pedestrian 658 is classified as a traffic controller or police officer. In the example, whenever a pedestrian (e.g., human) is detected in the scene, the risk may be set to high (e.g., "red" according to Table I).

[0126] With respect to the obstruction situation 670, the risk is high (e.g., "red" per Table I) because to navigate around the obstruction, the AV will have to travel in the reverse lane 674, but the view of what is on the other side of the hill is obstructed. The risk (e.g., risk of passing) may be determined to be high because the vehicle 676 may be yielding to another road user at a stop-and-go intersection.

[0127] At 506, the technique 500 determines whether the risk exceeds a threshold. If the risk does exceed the threshold, the technique 500 proceeds to 508. In implementations, and as shown by dashed line 516, if the risk does not exceed the threshold, the technique 500 may proceed to 514.

[0128] The threshold value may be such that it may be compared to the risk. For example, if the risk is a percentage, the threshold value may also be specified as (or converted to) a percentage (e.g., 60%, 70%, or some other percentage). For example, if the risk is an ordinal value, the threshold value may be (or may be converted to) one of the ordinal values. For example, if the risk may be one of the values ​​"low" / "green", "medium" / "yellow", and "high" / "red" in order of increasing severity, the threshold value may be "medium" / "yellow". Thus, if the risk is determined to be "high" / "red" and the threshold value is selected to be "medium" / "yellow", the technique 500 proceeds to 508. In the example, the risk may correspond to a classification. For example, an obstruction situation may be classified as one of the categories normal, low, high, or some other classification.

[0129] At 508, technique 500 initiates a request for assistance from a teleoperator. Initiating the request may include sending a request to a plurality of remote vehicle assistance centers such as Figure 3 Thus, the teleoperator (such as a remote vehicle assistance center 300) initiates the assistance request. Figure 3 The request may be responded to by a remote operator 320, 330, etc. The request may be responded to by an electronic communication interface such as Figure 1 At 510, while the technology 500 waits for instructions from the teleoperator, the technology 500 stops (e.g., waits) the AV in place behind the obstacle. That is, if the teleoperator has not yet responded, the AV safely stops and waits for the teleoperator to respond.

[0130] The request may include information (e.g., status data) that may be useful to the teleoperator in responding to the request. When the request is initiated, the information may be included in the request. In some examples, in response to a request for information from one or more systems used by the teleoperator, the technique 500 may send at least some of the information. In this way, some of the information may be pushed to the teleoperator's system. Regardless of whether the information is pushed from the vehicle or pulled from the vehicle, such information is collectively referred to as sent information.

[0131] The state data may include, but is not limited to, data indicating the state or condition of the vehicle that is performing technique 500, including any of the following: dynamic state data related to any of the vehicle's speed and acceleration, position data including the vehicle's geographic location (e.g., the vehicle's latitude and longitude) or the vehicle's position relative to another object, the vehicle's position including the vehicle's orientation and inclination (e.g., the slope of the vehicle on a slope), the vehicle's operating state including the vehicle's electrical state or mechanical state (e.g., the health of electrical vehicle systems, mechanical vehicle systems, tire pressure, etc.), maintenance data related to maintenance of the vehicle, energy source data including the amount of remaining fuel or remaining battery charge, sensor data based on outputs from sensors including optical sensors, audio sensors, motion sensors, internal state data including the temperature and humidity within the vehicle's passenger compartment, and the vehicle's current mission (e.g., picking up passengers).

[0132] The information sent may also include information maintained on such Figure 4 The status information of other road users in the world model module 402 of the world model module of the present invention can be included in the information sent. For example, the status of objects that are obstacles or part of obstacles and / or the status of other objects close to the vehicle can be included in the information sent. For example, speed, posture, geometry (such as width, height and depth, etc.), classification (such as bicycles, large trucks, pedestrians, road signs, etc.), location, other information or zero or more of a subset thereof can be sent from the vehicle so that the teleoperator can have enough information to determine a response.

[0133] Figure 7 700 is an illustration of a user interface 700 for a teleoperator according to an implementation of the present invention. User interface 700 is merely an illustrative example of visual information that a teleoperator may need in order to respond to a request. One or more of user interfaces 700 may be displayed (or available for display) to a teleoperator. User interface 700 includes a map view 702, an enhanced view 704, and a dashcam view 706. However, user interface 700 may include more, fewer, or other user interfaces and / or information that may help the teleoperator respond to the received request.

[0134] Map view 702 depicts one or more vehicles managed by a teleoperation center. The one or more vehicles are overlaid on a street map. Map view 702 shows that vehicle 722 is approaching intersection 724. Map view 702 may be generated using location information received from vehicle 722. That is, vehicle 722 may periodically (e.g., every few milliseconds) send its location information to a system that enables creation of view 704. Map view 702 shows information about Figure 8 Further depicted is ticket view 726. Timer 728 may indicate how long vehicle 722 has been stopped.

[0135] Augmented view 704 shows a satellite view of an empty street overlaid with vehicle 722 and other road users. Other road users include any road users that pose an obstruction and / or other nearside road users. Status information sent via technique 500 may be used to generate augmented view 704. For example, posture and size status information of other road users may be used to create augmented view 704. Vehicle 732, vehicle 734, and vehicle 736 are respectively associated with Fig. 6A The augmented information corresponding to the vehicles 654, 656, and 660 of FIG. 704 is provided. Since the augmented view 704 is generated at a different time point than the time point of the obstruction situation 650, different road users have different postures, positions, and orientations. That is, the augmented view 704 can be the latest (e.g., near real-time) view of the obstruction situation.

[0136] As described above, technique 500 may determine a second trajectory that avoids the obstruction. The second trajectory may be sent to the teleoperation center in the request. In this case, enhanced view 704 may include a trajectory indicator 738 showing the second trajectory. The second trajectory may be sent in the form of a series of locations that are connected as trajectory indicator 738 in enhanced view 704. Enhanced view 704 may also include an indicator of the risk as identified by technique 500 (i.e., risk indicator 740). Risk indicator 740 shows that technique 500 determined the risk to be 7 (e.g., 7 out of 10 for the maximum risk). The risk as identified by technique 500 may be included in the information sent.

[0137] In an example, technique 500 may send one or more dashcam images (e.g., in near real time). That is, the vehicle may be equipped with a camera facing forward of the vehicle so that the camera can capture images of the obstructing scene. In some examples, the camera may be a video camera. The dashcam view(s) may be displayed to the teleoperator as dashcam view 706.

[0138] Figure 8 8 is an illustration of a ticket 800 according to an implementation of the present invention. Ticket 800 shows an example of content that may be presented to a teleoperator at a teleoperation center in response to an AV initiating a request for teleoperator assistance. Ticket 800 includes an exception description 802, occupancy information 804, destination information 806, feature information 808, and action buttons 810. In some implementations, more, less, other information or actions, or a combination thereof, may be displayed in the ticket.

[0139] Exception description 802 may be a summary of the nature (e.g., type) of the exception. Figure 6B For an obstruction situation 670, the exception description 802 may be "The view of oncoming traffic is obstructed by the hill." Occupancy information 804 may describe whether there are any occupants in the AV and / or the number of occupants. For example, when the exception situation occurs, the AV may be performing a self-driving taxi (i.e., autonomous taxi) service. In this case, there will be occupants in the AV. In another example, after the occupants exit the vehicle, the AV may be on its way to a service and maintenance center. In this case, there may not be occupants in the AV. The extent to which the AV can be caused to pause (e.g., violate, deviate) traffic regulations to circumvent the obstruction may depend on whether there are occupants in the AV.

[0140] The destination information 806 may indicate the next destination of the AV. Alternatively or additionally, the destination information 806 may indicate the final destination of the AV. For example, the AV may be scheduled to stop along the route to the final destination. Thus, any of the stop locations and / or the final destination may be listed in the destination information 806.

[0141] The feature information 808 may include the values ​​of features that are important and / or relevant to parsing the abnormal situation. As described above, the features may include whether there are any other world objects in the scene, the location of the AV on the HD map, the speed of the AV, the visibility of the AV, specific features related to the stopped vehicle, less information, additional information, or a combination thereof. For example, features related to other world objects may include whether there are any of vehicles, bicycles, pedestrians, and their corresponding numbers. The location of the AV on the HD map may include the type of road, the presence of traffic signs (e.g., traffic lights, stop signs, etc.) and intersections, etc. The visibility of the AV may include whether the field of view of the AV is blocked by the stopped vehicle, the presence of mountains, etc. Other features related to the stopped vehicle may include whether the hazard lights are on, whether the doors are open and which doors are open, and whether pedestrians are loading or unloading, etc.

[0142] When the teleoperator is ready to resolve the ticket, the teleoperator may select (e.g., click, activate, press, etc.) action button 810. When action button 810 is selected, the teleoperator may evaluate the obstruction situation, such as by viewing augmented view 704, dashcam view 706, or any other view in real-time (or near real-time). The teleoperator may also evaluate track indicator 738. The teleoperator may be presented with information such as information about the obstruction situation. Fig. 9 A menu described for selecting a response.

[0143] Fig. 9900 is an illustration of a menu of teleoperator responses in accordance with an implementation of the present invention. In an implementation, menu 900 may be displayed in augmented view 704 proximate to track indicator 738. For example, in response to the teleoperator clicking on or near track indicator 738, menu 900 may overlay track indicator 738. Menu 900 includes three options, each corresponding to a response of the teleoperator. However, there may be more, fewer, or other response options. By selecting a menu option from menu 900, the teleoperator causes indication data to be sent to the AV. Such as information about Figure 2 As mentioned above, the indicator data may be sent via a network including a wireless network (eg, a cellular network) or any other communication link, system, or platform between the remote support system and the vehicle being monitored / supported.

[0144] The instruction data may include, but is not limited to, instructions to be performed by an occupant of the AV or vehicle (e.g., a driver, a passenger). The instruction data may include data associated with any of the following: controlling the movement of the vehicle (including changing the acceleration, speed, or direction (e.g., steering) of the vehicle), activating or deactivating (e.g., turning on or off some or all components of the vehicle) control systems in the vehicle (including mechanical control systems and electrical control systems), activating or deactivating sensors in the vehicle (e.g., activating a camera to view the interior of the vehicle or an area outside the vehicle), activating or deactivating a communication system including any of an internal communication system (e.g., an internal speaker directed toward the occupants of the vehicle) and an external communication system (e.g., an external speaker directed toward an object or individual outside the vehicle).

[0145] In an implementation, sending the instruction data to the AV may include sending driving route data (i.e., a third trajectory) for an implementation of an autonomous device or system of the vehicle. The third trajectory may be a short-term path around an obstruction. The third trajectory may be a new route toward a destination such that the new route circumvents the obstruction. In this way, the driving route data may include instructions for modifying an existing driving route of the vehicle.

[0146] In another implementation, sending the instruction data to the autonomous vehicle can be associated with ignoring at least one operational constraint of the autonomous operation to enable the first vehicle to traverse the driving route. As an example, the autonomous operation of the vehicle can include an operational constraint that prohibits a specific type of action of the vehicle, wherein the specific type of action includes any of violating a speed limit (including driving too fast or driving too slow), violating a traffic law (including driving in the opposite direction of traffic and driving off the road). By ignoring the operational constraint, the vehicle is able to perform the prohibited action according to the instructions in the instruction data. For example, detouring a construction zone can include driving in a portion of a road that was previously restricted (such as when a portion of the road is not a paved road, but a dirt road, etc.).

[0147] By selecting menu option 902 (i.e., "Authorize"), the teleoperator authorizes the AV to proceed along the second trajectory suggested by the AV. That is, the technique 500 receives a response to proceed. By selecting menu option 904 (i.e., "Reject"), the teleoperator instructs the AV to continue to wait (e.g., stop at an appropriate location to await further instructions). That is, the technique 500 receives a response to wait. By selecting menu option 906, the teleoperator can select (e.g., draw, define, etc.) a new route for the AV to follow (i.e., the third trajectory).

[0148] Reference again Figure 5 At 512, technique 500 receives a response (ie, a received response) from the teleoperator. As already mentioned, the response may be one of a first response to wait, a second response to go ahead, and a third response to include a trajectory.

[0149] Technique 500 may also include, in response to the response being a forward response, operating the AV according to a second trajectory. Fig. 9 The menu option 902 described above authorizes the second track, which is Figure 7 In this way, the technique 500 can operate the AV or cause the AV to be operated according to the track indicator 738. The technique 500 can also include: in the case where the response is a response including a track (such as a third track, etc.), the technique 500 can operate the AV or cause the AV to be operated according to the third track.

[0150] At 514 , that is, in response to the risk not exceeding the risk threshold, technique 500 operates the AV according to a second trajectory.

[0151] In implementations, technique 500 may include revoking a request to a teleoperator. For example, while technique 500 is stopping at 510, the obstruction may be cleared. Fig. 6AIn an obstruction situation 650, while the AV is stopping, the vehicle 654 moves forward, and the pedestrian 658 leaves the scene (such as by entering the vehicle 656, etc.). The technology 500 can determine (such as using a trajectory planning module, etc.) that there is enough clearance in the road for the vehicle 656 to pass, and the path of the AV is no longer obstructed. In this way, the AV can proceed normally. Therefore, teleoperator assistance is no longer required. In this way, in response to determining that the abnormal situation is cleared, the technology 500 can revoke (e.g., cancel, withdraw, etc.) the request to the teleoperator. In another example, even if the obstruction is cleared, the AV continues to stop until the technology 500 receives a response to the request. That is, the technology 500 cannot revoke the ticket.

[0152] Fig.10 is a flow chart of a technique 1000 for exception handling by an autonomous vehicle (AV) according to an implementation of the present invention. Figure 1 The vehicle 1050 shown, Figure 2 The vehicle 2100 shown may be a vehicle (e.g., an autonomous vehicle) or may include a vehicle such as Figure 2 Some or all of the technique 1000 for exception handling may be implemented in a computing device of a controller device such as the controller device 2410 shown. The controller device may include an exception handling module (such as Figure 4 In an implementation, some or all aspects of the technique 1000 for exception handling may be implemented in a system (e.g., Figure 2 The technique 1000 may be implemented in a controller device 2410 of the embodiment of the present invention. Figure 4 At least some modules of the system 400 are partially or completely executed.

[0153] At 1002, technique 1000 identifies a lane obstruction condition. For example, an obstruction module (such as Figure 4 The obstruction manager module 406 of the AV may determine that the AV's trajectory is obstructed because if the AV continues along its current trajectory, the AV will collide with the obstruction. For example, the identified lane obstruction may be Fig. 6A At 1004, the technique 1000 identifies a first trajectory for avoiding the lane obstruction situation. For example, a trajectory planning module (such as Figure 4 The trajectory planning module 404, etc.) can identify Figure 7 The trajectory indicator 738 indicates the trajectory.

[0154] At 1006, the technique 1000 determines whether the obstruction situation is an abnormal situation. For example, if the obstruction module determines that the first trajectory violates one or more traffic regulations, the obstruction situation is identified as an abnormal situation and the technique 1000 proceeds to 1008; otherwise, in an implementation, the technique 1000 proceeds to 1018. At 1018, the technique 1000 operates the AV or causes the AV to be operated according to the first trajectory. Again, referring to Figures 6A to 6B Although the obstruction situation 610 will not be judged as an abnormal situation, the obstruction situations 630, 650, and 670 will be judged as abnormal situations.

[0155] At 1008, technique 1000 determines a risk associated with the first trajectory. The risk may be determined by a risk assessment module such as Figure 4 If the risk exceeds the risk threshold, the technique 1000 proceeds to 1012; otherwise, the technique 1000 may proceed to 1018.

[0156] At 1012, the technique 1000 initiates a request to the teleoperator. Figure 5 The request may be initiated as described in 508 of . Thus, the request may include the first trajectory. At 1014, the technique 1000 waits for a response from the teleoperator. That is, the technique 1000 may stop the AV or cause the AV to stop while waiting for a response from the teleoperator.

[0157] In implementations, technique 1000 may include: in response to receiving a go response from a teleoperator, operating the AV according to a first trajectory. Technique 1000 may include: receiving a second trajectory from the teleoperator, such as where the response includes the trajectory; and operating the AV according to the second trajectory.

[0158] The steps or operations of any method, process, technique, or algorithm described in conjunction with the implementation of the technology disclosed herein may be implemented in hardware, firmware, software executed by hardware, circuits, or any combination thereof. For ease of illustration, Figure 5 and 10 The techniques 500 and 1000 shown in the drawings are depicted and described as a series of operations. However, operations according to the present invention may occur in various orders or simultaneously. In addition, operations according to the present invention may occur together with other operations not presented and described herein.

[0159] As used herein, the terms "driver" or "operator" may be used interchangeably. As used herein, the terms "braking" or "deceleration" may be used interchangeably. As used herein, the terms "computer" or "computing device" include any unit or combination of units capable of performing any method disclosed herein or any of its multiple parts.

[0160] As used herein, the term "processor" refers to one or more processors, such as one or more special-purpose processors, one or more digital signal processors, one or more microprocessors, one or more controllers, one or more microcontrollers, one or more application processors, one or more application-specific integrated circuits, one or more application-specific standard products; one or more field programmable gate arrays, any other type or combination of integrated circuits, one or more state machines, or any combination thereof, etc.

[0161] As used herein, the term "memory" indicates any computer-usable or computer-readable medium or device that can tangibly contain, store, communicate, or transmit any signal or information that can be used by or in conjunction with any processor. For example, the memory can be one or more read-only memories (ROMs), one or more random access memories (RAMs), one or more registers, low-power double data rate (LPDDR) memories, one or more cache memories, one or more semiconductor memory devices, one or more magnetic media, one or more optical media, one or more magneto-optical media, or any combination thereof.

[0162] As used herein, the term "instruction" may include instructions or representations for performing any method disclosed herein or any one or more parts thereof, and may be implemented in hardware, software, or any combination thereof. For example, an instruction may be implemented as information such as a computer program stored in a memory, which may be executed by a processor to perform any of the corresponding methods, algorithms, aspects, or combinations thereof as described herein. In some implementations, an instruction or a portion thereof may be implemented as a dedicated processor or circuit, which may include dedicated hardware for performing any of the methods, algorithms, aspects, or combinations thereof as described herein. In some implementations, a portion of an instruction may be distributed across multiple processors on a single device, multiple devices that may communicate directly or across a network such as a local area network, a wide area network, the Internet, or a combination thereof.

[0163] As used herein, the terms "example," "embodiment," "implementation," "aspect," "feature," or "element" indicate use as an example, instance, or illustration. Unless explicitly indicated otherwise, any example, embodiment, implementation, aspect, feature, or element is an independent example, embodiment, implementation, aspect, feature, or element and can be used in combination with any other example, embodiment, implementation, aspect, feature, or element.

[0164] As used herein, the terms "determine" and "identify" or any variations thereof include selecting, ascertaining, calculating, searching, receiving, determining, establishing, obtaining, or otherwise identifying or determining in any manner using one or more of the devices shown and described herein.

[0165] As used herein, the term "or" means an inclusive "or" rather than an exclusive "or". That is, unless specified otherwise or clear from the context, "X includes A or B" is intended to indicate any of the natural inclusive permutations. If X includes A; X includes B; or X includes both A and B, then "X includes A or B" is satisfied under any of the above instances. In addition, the articles "a" and "an" used in this application and the appended claims should generally be construed to mean "one or more" unless specified otherwise or clear from the context to be directed to the singular.

[0166] In addition, for the sake of simplicity of explanation, although the figure and description here may include the sequence or a series of steps or the stage of step or stage, the elements of the method disclosed here may occur in various orders or simultaneously. In addition, the elements of the method disclosed here may occur with other elements that are not clearly presented and described here. In addition, all elements of the method described here may not be needed to realize the method according to the present invention. Although aspects, features and elements are described here with specific combinations, various aspects, features or elements can be used alone, or can be used in various combinations with or without other aspects, features and elements.

[0167] Implementations of the present invention provide technical improvements, particularly to computer networks and autonomous vehicle management, for example, related to extending computer network components to generate solutions associated with the operation and monitoring of autonomous vehicles. Developing new methods for generating solution data, for example, to generate solution data for coping with mechanical and electronic failures in autonomous vehicles, or to overcome limitations in autonomous vehicle programming, is fundamentally related to computer networks associated with autonomous vehicles.

[0168] The implementation of the present invention at least provides a system and method for generating solution data and sending the solution data to one or more vehicles for execution (i.e., for using the solution data to solve the problems / problems encountered). The system includes a controller device for generating state data of a vehicle using the received sensor data. The state data may include any one of the location, destination, and operating state of the vehicle. For example, the location of the vehicle may be generated based on map data associated with the vehicle that has been received from a global positioning satellite (GPS). In response to determining that the generated state data meets the state criteria, solution configuration data matching the state data is determined based on a comparison of the state data with the solution configuration data. The solution configuration data may be based on a solution in an aggregation of solution data from past events that provide assistance to an autonomous vehicle. Solution data may be generated based on matching solution configuration data and subsequently sent to the vehicle for execution. For example, solution data may include a solution to a request for assistance from a driver of an autonomous vehicle.

[0169] Although the disclosed technology has been described in conjunction with certain embodiments, it should be understood that the disclosed technology is not limited to the disclosed embodiments, but is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope should be given the broadest interpretation to cover all such modifications and equivalent structures permitted by law.

Claims

1. A method for exception handling by an autonomous vehicle (AV), the method comprising: identifying an abnormal condition in response to detecting a vehicle stopped in front of the AV; identifying a risk associated with autonomously resolving the abnormal situation, wherein the risk is identified as a weighting of a situation likelihood and a result of autonomously resolving the abnormal situation, the situation likelihood relating to whether the stopped vehicle is permanently stopped or temporarily stopped, and the situation likelihood is determined based on at least two of the following: whether the stopped vehicle's hazard lights are flashing, how long the stopped vehicle has been stopped, whether the stopped vehicle is at an intersection, whether there are road cones, and whether the stopped vehicle was classified as a moving vehicle at a previous moment; wherein the result relates to a cost of passing the stopped vehicle, and the result is determined based on whether the AV has a clear view ahead and sufficient space to pass the stopped vehicle; and In response to the risk exceeding a risk threshold: Initiating a request for assistance from a teleoperator; Stopping to wait for the teleoperator to respond to the request; and receiving a response from the teleoperator; and In response to determining that the abnormal condition is cleared before receiving a response from the teleoperator, revoking the request to the teleoperator.

2. The method according to claim 1, wherein: Identifying the abnormal situation includes: identifying an obstruction to a first trajectory of the AV; determining a second trajectory avoiding the obstacle; and It is determined that the second trajectory violates driving regulations.

3. The method according to claim 2, further comprising: In response to the risk not exceeding the risk threshold, operating the AV according to the second trajectory.

4. The method according to claim 3, further comprising: In response to the response being a forward response, the AV is operated according to the second trajectory.

5. The method according to claim 2, wherein: The received response includes a third trace, the method further comprising: The AV is autonomously operated according to the third trajectory.

6. The method according to claim 1, wherein: A response from the teleoperator is selected from a set consisting of: a first response to wait, a second response to go ahead, and a third response to include a trajectory.

7. A method for exception handling by an autonomous vehicle (AV), the method comprising: identifying a lane obstruction condition, the lane obstruction condition being a stopped vehicle; identifying a first trajectory for avoiding the lane obstruction condition; as well as In response to identifying that the lane obstruction situation is an abnormal situation based on the first trajectory: determining a risk associated with the first trajectory, wherein the risk is identified as a weighted result of autonomously parsing the abnormal situation and a situation likelihood related to whether the stopped vehicle is permanently stopped or temporarily stopped, and the risk is determined based on at least two of the following: whether the stopped vehicle is yielding or permanently stopped; whether sensors of the AV can detect oncoming other world objects; whether a door of the stopped vehicle is open; and whether the AV must travel in an oncoming lane in order to go around the stopped vehicle; and In response to determining that the risk exceeds the risk threshold: Initiating a request to a teleoperator, wherein the request includes the first trajectory; and awaiting a response from the teleoperator; and In response to determining that the risk associated with the first trajectory does not exceed the risk threshold: The AV is autonomously operated according to the first trajectory.

8. The method according to claim 7, further comprising: In response to receiving a go response from the teleoperator, the AV is operated according to the first trajectory.

9. The method according to claim 7, further comprising: receiving a second trajectory from the teleoperator; as well as The AV is operated according to the second trajectory.

10. An apparatus for exception handling by an autonomous vehicle (AV), comprising: a memory in which instructions for exception handling by an autonomous vehicle (AV) are stored; as well as A processor configured to execute the instructions to perform the following operations: Identify anomalies; identifying the risks associated with autonomously resolving the anomaly, wherein the risk is identified as a weighted combination of the likelihood of the situation and the result of autonomously parsing the abnormal situation, and wherein the situation likelihood is a measure of understanding of the abnormal situation using sensor data from sensors of the AV, and the situation likelihood relates to whether the identified person is a police officer or a traffic controller, wherein the risk weighs the situation likelihood that the identified person is the police officer or the traffic controller and a result of autonomously resolving the abnormal situation by ignoring instructions from the police officer or the traffic controller; and In response to the risk exceeding a risk threshold: Initiate a request to the teleoperator; Waiting for the teleoperator to respond to the request; and A response is received from the teleoperator.

11. The device according to claim 10, wherein: Identifying the abnormal situation includes: identifying an obstruction to a first trajectory of the AV; determining a second trajectory avoiding the obstacle; and It is determined that the second trajectory violates driving regulations.

12. The device according to claim 11, wherein The instructions also include instructions for performing the following operations: In response to the risk not exceeding the risk threshold, the AV is autonomously operated according to the second trajectory.

13. The device according to claim 12, wherein: The instructions also include instructions for performing the following operations: In response to the response being a forward response, the AV is autonomously operated according to the second trajectory.

14. The device according to claim 11, wherein: The received response includes the third trace, and the instructions further include instructions to: The AV is autonomously operated according to the third trajectory.

15. The apparatus according to claim 10, wherein: A response from the teleoperator is selected from a set consisting of: a first response to wait, a second response to go ahead, and a third response to include a trajectory.

16. The apparatus according to claim 10, wherein: The instructions also include instructions for performing the following operations: Identify other anomalies; initiating another request for assistance from the teleoperator in response to the other abnormal condition; and Cancel other requests to the teleoperator.

17. The device according to claim 16, wherein: Revoking a request to the teleoperator comprises: In response to determining that the other abnormal condition is cleared before receiving a response to the other request, revoking the other request to the teleoperator.

Images