Competence recognition system and method for autonomous vehicles

Abstract

The first method includes detecting an environmental state based on sensor data, selecting an action based on the environmental state, determining an autonomy level associated with the environmental state and the action, and executing the action according to the autonomy level. The autonomy level can be selected based on at least an autonomy model and a feedback model. The second method includes calculating a policy for solving a task by solving an extended stochastic shortest path (SSP) problem. The policy can map the environmental state and the autonomy level to actions and autonomy levels. Calculating the policy can include generating a plan that operates across multiple levels of autonomy.

Description

Technical Field

[0001] The present disclosure generally relates to the operation management and autonomous driving of autonomous vehicles, and more specifically, to determining the level of autonomy of the operation of an autonomous vehicle according to a competence model.

Background Art

[0002] Vehicles such as autonomous vehicles can cross a part of a vehicle traffic network (e.g., a road). Driving on a part of a vehicle traffic network may include generating or capturing data such as data representing the operating environment of the vehicle or a part thereof by sensors of the vehicle. In some cases, some data may be unavailable due to obstructions.

Summary of the Invention

[0003] A first aspect of the disclosed embodiments is a method of autonomous driving by an autonomous vehicle (AV). The method includes detecting an environmental state based on sensor data, selecting an action based on the environmental state, identifying a set of currently non-discriminable states, identifying discriminators from the set of currently non-discriminable states, training a feedback model for the discriminators, determining an autonomy level associated with the environmental state and the action, and executing the action according to the autonomy level. The autonomy level can be selected based at least on an autonomy model and a feedback model. The feedback model can be trained regardless of the presence or absence of discriminators and an iterative state space refinement technique.

[0004] A second aspect of the disclosed embodiments is a system for autonomy that includes a memory and a processor. The processor is configured to execute instructions stored in the memory to calculate a policy for solving a task by solving an extended stochastic shortest path (SSP) problem, identify discriminators from a current set of discrimination states, and train a feedback model for the discriminators. The policy can map environmental states and autonomy levels to actions and autonomy levels. Calculating the policy can include generating a plan that operates across multiple levels of autonomy. The feedback model can be trained regardless of the presence or absence of discriminators and an iterative state space refinement technique.

[0005] A third aspect of the disclosed embodiments is a method for autonomous driving. The method includes calculating a policy for solving a task by solving an extended stochastic shortest path (SSP) problem, identifying discriminators from a current set of discrimination states, and training a feedback model for the discriminators. The policy can map environmental states and autonomy levels to actions and autonomy levels. Calculating the policy can include generating a plan that operates across multiple levels of autonomy. The feedback model can be trained regardless of the presence or absence of discriminators and an iterative state space refinement technique.

[0006] These and other aspects, features, elements, implementations, and variations of the methods, apparatuses, procedures, and algorithms disclosed herein will be described in more detail below.

[0007] Various aspects of the methods and apparatuses disclosed herein will become more apparent by reference to the examples provided in the following description and drawings, where like reference numerals refer to like elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] [Figure 1]This figure shows an example of a vehicle in which the embodiments, features, and elements disclosed herein may be implemented.

[0009] [Figure 2] This figure shows an example of a part of a vehicle traffic and communication system in which embodiments, features, and elements disclosed herein may be implemented.

[0010] [Figure 3] This diagram shows a portion of the vehicle traffic network as disclosed in this disclosure.

[0011] [Figure 4] This is a diagram showing an example of an autonomous vehicle operation management system according to an embodiment of the present disclosure.

[0012] [Figure 5] This is a diagram showing an example of a competence recognition system (CAS) according to an embodiment of the present disclosure.

[0013] [Figure 6] This is a flowchart illustrating an example of a method for providing CAS with the ability to improve its competence over time.

[0014] [Figure 7] This is a diagram illustrating an example of the method shown in Figure 6.

[0015] [Figure 8] This figure shows an example of a single-step state-space refinement algorithm.

[0016] [Figure 9] This is an example of a flowchart for a competence recognition decision-making technology according to an embodiment of the present disclosure.

[0017] [Figure 10] This is a diagram illustrating the autonomous driving scenario used to explain the technology in Figure 7.

[0018] [Figure 11] This is a flowchart illustrating an example of technology for autonomous driving by an autonomous vehicle (AV) according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0019] Vehicles such as AVs or semi-autonomous vehicles may traverse a portion of a vehicle traffic network. A vehicle may include one or more sensors, and traversing a vehicle traffic network may involve the sensors generating or capturing sensor data, such as sensor data corresponding to the vehicle's operating environment or a portion thereof. For example, the sensor data may include information corresponding to one or more external objects, such as pedestrians, remote vehicles, other objects in the vehicle's operating environment, vehicle traffic network geometry, or a combination thereof.

[0020] The AV may include an autonomous vehicle operation management system, which may include one or more operating environment monitors capable of processing operating environment information, such as sensor data for autonomous vehicles.

[0021] The AV motion management system may include an autonomous vehicle motion management controller, which may detect one or more motion scenarios, such as a pedestrian scenario, an intersection scenario, a lane change scenario, or any other vehicle motion scenario or combination of vehicle motion scenarios corresponding to an external object. The above description, namely the pedestrian scenario, intersection scenario, lane change scenario, or any other vehicle operation scenario or combination of vehicle operation scenarios, is any A set of behavioral scenarios or related behavioral scenarios together In this specification, this may be referred to as the environmental condition.

[0022] The AV operation management system may include one or more scenario-specific operation control evaluation modules (SSOCEMs). Each scenario-specific operation control evaluation module may be a model, such as a partially observed Markov decision process (POMDP) ​​model, for each operation scenario. That is, each model is configured to handle a specific scenario. The AV operation management controller may instantiate each instance of the scenario-specific operation control evaluation module in response to detecting a corresponding operation scenario.

[0023] The AV operation management controller may receive candidate vehicle control actions from each instantiated scenario-specific operation control evaluation module (SSOCEM) instance, identify vehicle control actions from the candidate vehicle control actions, and control the AV to traverse a portion of the vehicle traffic network according to the identified vehicle control actions.

[0024] In some implementations, SSOCEM can be configured to autonomously complete certain tasks while requiring human intervention to complete other tasks. That is, SSOCEM can operate autonomously under certain conditions, but may require human intervention or assistance to achieve its goal (e.g., crossing an intersection). Thus, SSOCEM can be one of two binary autonomous states or levels.

[0025] For example, in response to detecting an obstacle on a road (i.e., on a portion of a vehicle traffic network), SSOCEM may issue a request for assistance to a remote operator. The remote operator may be a human operator responsible for remotely monitoring and assisting one or more autonomous vehicles. The remote operator may have sensor data (e.g., camera images and / or video) streamed to the remote operator, as a result, the remote operator can gain situational awareness, plot a navigation path for the AV around obstacles, and / or remotely control the AV's behavior.

[0026] In another example, a lane-crossing SSOCEM may use lane markings to maintain the AV within the lane. In some situations, such as when the sensors are dirty or malfunctioning, or when the lane markings are covered with snow or mud, the lane-crossing SSOCEM may no longer be able to define the lane boundary. In such situations, the vehicle-crossing SSOCEM may require a human (e.g., the AV's driver occupant or a remote operator) to take control of the AV.

[0027] The reliance on human assistance (i.e., intervention) can demonstrate the limited competence of SSOCEM in its autonomy model. Human intervention can be costly. For example, it may take a relatively long time for a remote operator to respond to a request for assistance from an AV. During that time, the AV may be causing traffic congestion. For example, as the number of AVs requesting assistance from remote operators increases, the number of available remote operators will inevitably need to increase as well.

[0028] In the implementation provided by this disclosure, SSOCEM may be or include an Autonomous Cognitive Agent (ACA), which selects the next action to perform and the level of autonomy required to perform that action based on an autonomy model that the ACA maintains and evolves. For example, initially, the autonomy model may indicate that the ACA seeks human assistance (e.g., feedback) for an action, taking into account the detected environmental state. As the ACA receives more feedback from humans, etc., its dependence on human assistance may gradually decrease, and the ACA learns the appropriate timing to perform actions with a lower level of assistance, which means higher autonomy (i.e., competence). Thus, it can be said, or can be considered, that the ACA is aware of its level of competence.

[0029] In the implementation provided in this disclosure, the ACA can consider all available levels of autonomy when generating a plan (as opposed to adjusting the level of autonomy during plan execution, for example). Thus, the ACA can create a plan that more effectively utilizes its knowledge of its own levels of autonomy.

[0030] Furthermore, in the implementation described herein, ACA can model feedback from multiple forms of human interaction, thereby enabling ACA to proactively plan in a way that also considers the likelihood of feedback from each form of human interaction. Thus, ACA can proactively avoid situations that may result in negative feedback.

[0031] Furthermore, through experience that allows the ACA to reduce its reliance on humans over time, it can maintain predictive models for human feedback and intervention by adjusting the ACA's level of autonomy over time, as well as avoiding situations that are more likely to require human intervention. Thus, the ACA can operate at the lowest cost level of autonomy for any situation (i.e., environmental state) it encounters (i.e., detected based on sensor data).

[0032] Subsequently, ACA can use an autonomous model to traverse the vehicle traffic network.

[0033] Autonomous systems are increasingly being deployed in open worlds that include highly complex and unstructured domains. Examples of these systems include space exploration rovers, autonomous underwater vehicles, service robots, and AVs. Because it is difficult to fully model the open world, these systems must rely on approximate models of those domains that may not be sufficient to handle all situations, potentially leading to dangerous behavior. Nevertheless, these systems are expected to maintain safe and reliable operation over the course of potentially long-term deployment. To achieve this, these systems rely on various forms of human supervision, assistance, and intervention. In that sense, the most sophisticated AI systems currently under development are, at best, semi-autonomous, in that they operate autonomously only under specific conditions and otherwise require human intervention to complete assigned tasks.

[0034] For example, a space exploration rover may pause its operation and await a new plan from ground control when it encounters an unexpected obstacle or when system parameters such as wheel resistance torque fall outside acceptable limits. Similarly, in the event of adverse weather conditions or when dense, intersecting traffic exists and there is no safe gap for an extended period when attempting to merge, the AV may request that control be returned to a human driver. Human assistance and feedback may be available in different forms or modalities corresponding to the behavior. For example, enabling a system to operate autonomously under human supervision represents a higher level of competence compared to requiring explicit approval from a human before performing each action, while representing a lower level of competence compared to autonomous operation without supervision.

[0035] The embodiments disclosed herein are described in the context of semi-autonomous systems, where competence may be a measure of the optimal range of autonomous operation of the system in any given situation, taking into account the available human involvement, i.e., knowledge of the various forms of assistance that humans can provide and their ability to communicate with the semi-autonomous system. Competence may be a measure of the performance of a system or algorithm. Competence of automated decision-making in a semi-autonomous system may be based on the level of autonomy. The level of autonomy may be a paradigm for modeling stages of autonomous behavior in safety-critical domains, where each level of autonomy corresponds to some set of constraints, limitations, or requirements for autonomous operation. Thus, competence recognition may be defined as the ability of an agent to predict the correct level of autonomy to operate in any given situation.

[0036] Embodiments disclosed herein include a planning model called a competence-recognized system (CAS) for operating at multiple levels of autonomy, each level associated with different forms of human assistance that compensate for the system's constrained capabilities. The system may associate each type of human assistance with a unique set of feedback signals that the system can receive from humans, the likelihood of which can be learned over time. This model enables the system to operate more reliably in the open world, reduce inappropriate reliance on humans, and ultimately optimize the system's autonomous behavior. To address situations where initial domain models have insufficient representations to properly model human feedback, an iterative technique is introduced to refine the system's state space to better discriminate human feedback, generating finer divisions of the state-action space with different levels of competence, enabling the system to better learn and act in its true competence.

[0037] Embodiments disclosed herein may include methods for iterative state-space refinement. Iterative state-space refinement may enable a CAS to refine the granularity of its state representation online. The CAS models multiple forms of human feedback and uses this feedback to enable a semi-autonomous system to learn its competence over time. Furthermore, the CAS enables the semi-autonomous system to learn a predictive model of human feedback, thereby enabling the semi-autonomous system to converge to an optimal level of autonomy over time. The CAS may be configured to learn from human interactions, and in particular from human input and from empirical evidence.

[0038] The decision process associated with CAS may be based on an unspecified Markov decision process (UMDP). Embodiments disclosed herein may use two sources of prior models that are poorly or incompletely specified, the first of which may be used for transition dynamics in the form of human feedback and subcompetencies, and the second of which may be used for feature space in the form of missing features.

[0039] Figure 1 is a diagram of an example vehicle in which the embodiments, features, and elements disclosed herein may be implemented. In the illustrated embodiment, vehicle 1000 includes various vehicle systems. The vehicle system includes a chassis 1100, a powertrain 1200, a controller 1300, and wheels 1400. Additional or different combinations of vehicle systems may be used. For simplicity, vehicle 1000 is shown as including four wheels 1400, but one or more of any other propulsion devices, such as a propeller or tread, may be used. In Figure 1, lines interconnecting elements such as the powertrain 1200, the controller 1300, and the wheels 1400 indicate that information, such as data or control signals, force, such as power or torque, or both information and power, may be communicated between the respective elements. For example, the controller 1300 may receive power from the powertrain 1200 and communicate with the powertrain 1200, the wheels 1400, or both to control the vehicle 1000, which may include controlling the vehicle 1000 by accelerating, decelerating, steering, or in other ways.

[0040] The powertrain 1200 shown in the example of Figure 1 includes a power source 1210, a transmission 1220, a steering unit 1230, and an actuator 1240. Any other elements or combinations of elements of the powertrain may be included, such as a suspension, drive shafts, axles, or exhaust system. Wheels 1400 may also be included in the powertrain 1200, as shown separately.

[0041] The power source 1210 includes an engine, a battery, or a combination thereof. The power source 1210 may be any device or combination of devices that operate to supply energy such as electrical energy, thermal energy, or kinetic energy. For example, the power source 1210 includes an engine such as an internal combustion engine, an electric motor, or a combination of an internal combustion engine and an electric motor, which operates to supply kinetic energy as power to one or more of the wheels 1400. Alternatively or additionally, the power source 1210 may include a potential energy unit such as one or more dry cell batteries, solar cells, fuel cells, or any other device that can supply energy, such as nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), or lithium-ion (Li-ion) batteries.

[0042] The transmission 1220 receives energy, such as kinetic energy, from the power source 1210 and transmits that energy to the wheels 1400 to supply power. The transmission 1220 may be controlled by the controller 1300, the actuator 1240, or both. The steering unit 1230 may be controlled by the controller 1300, the actuator 1240, or both, and may control the wheels 1400 to steer the vehicle. The actuator 1240 may receive signals from the controller 1300 and may actuate or control the power source 1210, the transmission 1220, the steering unit 1230, or any combination thereof, in order to operate the vehicle 1000.

[0043] In the illustrated embodiment, the controller 1300 includes a location unit 1310, an electronic communication unit 1320, a processor 1330, a memory 1340, a user interface 1350, a sensor 1360, and an electronic communication interface 1370. Fewer of these elements may exist as part of the controller 1300. Although shown as a single unit, any one or more elements of the controller 1300 may be integrated into any number of separate physical units. For example, the user interface 1350 and the processor 1330 may be integrated into a first physical unit, and the memory 1340 may be integrated into a second physical unit. Although not shown in Figure 1, the controller 1300 may include a power source such as a battery. Although shown as separate elements, the positioning unit 1310, the electronic communication unit 1320, the processor 1330, the memory 1340, the user interface 1350, the sensor 1360, the electronic communication interface 1370, or any combination thereof, may be integrated into one or more electronic units, circuits, or chips.

[0044] The processor 1330 may include any device or combination of devices that can manipulate or process signals or other information, whether currently existing or to be developed, including optical processors, quantum processors, molecular processors, or combinations thereof. For example, the processor 1330 may include one or more dedicated processors, one or more digital signal processors, one or more microprocessors, one or more controllers, one or more microcontrollers, one or more integrated circuits, one or more application-specific integrated circuits, one or more field-programmable gate arrays, one or more programmable logic arrays, one or more programmable logic controllers, one or more state machines, or any combination thereof. The processor 1330 is operably coupled to one or more of the following: location unit 1310, memory 1340, electronic communication interface 1370, electronic communication unit 1320, user interface 1350, sensor 1360, and powertrain 1200. For example, the processor may be operably coupled to memory 1340 via communication bus 1380.

[0045] Memory 1340 includes any tangible, non-temporary, computer-usable or computer-readable medium capable of storing, storing, communicating, or transporting machine-readable instructions or any information associated therewith, for example, by or in conjunction with any processor such as processor 1330. Memory 1340 may be, for example, one or more solid-state drives, one or more memory cards, one or more removable media, one or more read-only memories, one or more random-access memories, one or more disks including hard disks, floppy disks, optical disks, magnetic or optical cards, or any type of non-temporary medium suitable for storing electronic information, or any combination thereof. For example, memory may be one or more read-only memories (ROMs), one or more random-access memories (RAMs), one or more registers, low-power double data-rate (LPDDR) memories, one or more cache memories, one or more semiconductor memory devices, one or more magnetic media, one or more optical media, one or more magneto-optical media, or any combination thereof.

[0046] The communication interface 1370 may be a wireless antenna, a wired communication port, an optical communication port, or any other wired or wireless unit that can interface with a wired or wireless electronic communication medium 1500, as shown in the figure. Although Figure 1 shows a communication interface 1370 communicating over a single communication link, the communication interface may be configured to communicate over multiple communication links. Although Figure 1 shows a single communication interface 1370, the vehicle may include any number of communication interfaces.

[0047] The communication unit 1320 is configured to transmit or receive signals via a wired or wireless electronic communication medium 1500, such as a communication interface 1370. Although not explicitly shown in Figure 1, the communication unit 1320 may be configured to transmit, receive, or both, via any wired or wireless communication medium, such as radio frequency (RF), ultraviolet (UV), visible light, optical fiber, wireline, or a combination thereof. Figure 1 shows a single communication unit 1320 and a single communication interface 1370, but any number of communication units and any number of communication interfaces may be used. In some embodiments, the communication unit 1320 includes a dedicated narrow-range communication (DSRC) unit, an on-board unit (OBU), or a combination thereof.

[0048] The positioning unit 1310 can determine geographical location information of the vehicle 1000, such as its longitude, latitude, altitude, direction of travel, or speed. For example, the positioning unit 1310 may include a GPS unit such as a Wide Area Augmentation System (WAAS)-enabled National Marine Electronics Association (NMEA) unit, a radio triangulation unit, or a combination thereof. The positioning unit 1310 may be used, for example, to obtain information representing the vehicle 1000's current direction of travel, its current position in two or three dimensions, its current angular direction, or a combination thereof.

[0049] The user interface 1350 includes any units that can interface with a person, such as virtual or physical keypads, touchpads, displays, touch displays, head-up displays, virtual displays, augmented reality displays, haptic displays, feature tracking devices such as eye-tracking devices, speakers, microphones, video cameras, sensors, printers, or any combination thereof. The user interface 1350 may be operably coupled with any other elements of the processor 1330 or controller 1300 as shown in the figure. Although shown as a single unit, the user interface 1350 may include one or more physical units. For example, the user interface 1350 may include both an audio interface for performing audio communication with a person and a touch display for performing visual and touch-based communication with a person. The user interface 1350 may include multiple displays, such as multiple physically separate units, multiple defined parts within a single physical unit, or a combination thereof.

[0050] Sensor 1360 is operable to provide information that can be used to control the vehicle. Sensor 1360 may be an array of sensors. Sensor 1360 may provide information about the current operating characteristics of vehicle 1000, including vehicle operation information. Sensor 1360 may include, for example, a speed sensor, an acceleration sensor, a steering angle sensor, a traction-related sensor, a braking-related sensor, a steering wheel position sensor, an eye-tracking sensor, a seat position sensor, or any sensor or combination of sensors, which are operable to report information about several aspects of the current dynamic situation of vehicle 1000.

[0051] Sensor 1360 includes one or more sensors 1360 that are capable of acquiring information about the physical environment surrounding the vehicle 1000, such as operating environment information. For example, one or more sensors may detect road geometry such as lane boundary lines, as well as obstacles such as fixed obstacles, vehicles, and pedestrians. Sensor 1360 may be or include one or more video cameras, laser detection systems, infrared detection systems, acoustic detection systems, or any other suitable type of in-vehicle environment sensing device, or a combination of such devices, which are currently known or may be developed in the future. In some embodiments, sensor 1360 is combined with a positioning unit 1310.

[0052] Although not shown otherwise, the vehicle 1000 may include a track controller. For example, controller 1300 may include a track controller. The track controller may be operable to acquire information describing the current state of the vehicle 1000 and a planned route for the vehicle 1000, and to determine and optimize the track of the vehicle 1000 based on this information. In some embodiments, the track controller may output a signal that can be operated to control the vehicle 1000 so that it follows the track determined by the track controller. For example, the output of the track controller may be an optimized track that can be supplied to the powertrain 1200, the wheels 1400, or both. In some embodiments, the optimized track may be a control input such as a set of steering angles, where each steering angle corresponds to a time or position. In some embodiments, the optimized track may be one or more paths, lines, curves, or a combination thereof.

[0053] One or more of the wheels 1400 may be a steering wheel pivoted to a steering angle under the control of a steering unit 1230, a drive wheel to which torque is supplied to propel the vehicle 1000 under the control of a transmission 1220, or a steering and drive wheel capable of steering and propelling the vehicle 1000.

[0054] Although not shown in Figure 1, the vehicle may include additional units or elements not shown in Figure 1, such as an enclosure, a Bluetooth® module, a frequency modulation (FM) radio unit, a near-field communication (NFC) module, a liquid crystal display (LCD) display unit, an organic light-emitting diode (OLED) display unit, a speaker, or any combination thereof.

[0055] Vehicle 1000 may be an autonomous vehicle that is autonomously controlled and travels through a portion of the vehicle traffic network without direct human intervention. Although not shown separately in Figure 1, the autonomous vehicle may include an autonomous vehicle control unit that performs routing, navigation, and control of the autonomous vehicle. The autonomous vehicle control unit may be integrated with another unit of the vehicle. For example, the controller 1300 may include an autonomous vehicle control unit.

[0056] If present, the autonomous vehicle control unit may control or operate vehicle 1000 to travel through a portion of the traffic network according to current vehicle operation parameters. The autonomous vehicle control unit may control or operate vehicle 1000 to perform a defined action or operation, such as parking the vehicle. Based on vehicle information, environmental information, traffic network information representing the traffic network, or a combination thereof, the autonomous vehicle control unit may generate a route from a starting point, such as the current location of vehicle 1000, to a destination, and may control or operate vehicle 1000 to travel through the traffic network according to the route. For example, the autonomous vehicle control unit may output the route to a trajectory controller and operate vehicle 1000 to travel from the starting point to the destination using the generated route.

[0057] Figure 2 is a diagram illustrating an example of a portion of a vehicle traffic and communication system in which embodiments, features, and elements disclosed herein may be implemented. The vehicle traffic and communication system 2000 may include one or more vehicles 2100 / 2110, such as the vehicle 1000 shown in Figure 1, which travel through one or more portions of the vehicle traffic network 2200 and communicate via one or more electronic communication networks 2300. Although not expressly shown in Figure 2, vehicles may travel in off-road areas.

[0058] The electronic communication network 2300 may be a multiple access system that provides communication, such as voice communication, data communication, video communication, messaging communication, or a combination thereof, between a vehicle 2100 / 2110 and one or more communication devices 2400. For example, a vehicle 2100 / 2110 may receive information from the communication devices 2400 via the network 2300, such as information representing the vehicle traffic network 2200.

[0059] In some embodiments, the vehicles 2100 / 2110 may communicate via wired communication links (not shown), wireless communication links 2310 / 2320 / 2370, or any combination of any number of wired or wireless communication links. As shown, the vehicles 2100 / 2110 communicate via terrestrial wireless communication link 2310, via non-terrestrial wireless communication link 2320, or a combination thereof. The terrestrial wireless communication link 2310 may include an Ethernet link, a serial link, a Bluetooth link, an infrared (IR) link, an ultraviolet (UV) link, or any link capable of providing electronic communication.

[0060] Vehicle 2100 / 2110 can communicate with other vehicles 2100 / 2110. For example, the local vehicle, i.e., the target vehicle (HV) 2100, can receive one or more automated inter-vehicle messages, such as basic safety messages (BSMs), from a remote vehicle, i.e., the target vehicle (RV) 2110, either via the direct communication link 2370 or via the network 2300. The remote vehicle 2110 can broadcast messages to the local vehicle within a defined broadcast range, such as 300 meters. In some embodiments, the local vehicle 2100 can receive messages via a third party, such as a signal repeater (not shown) or another remote vehicle (not shown). Vehicle 2100 / 2110 can periodically transmit one or more automated inter-vehicle messages based on a defined interval, such as 100 milliseconds.

[0061] Automated vehicle-to-vehicle messages may include vehicle identification information, geospatial state information such as longitude, latitude, or altitude information, geospatial position accuracy information, vehicle acceleration information, yaw rate information, speed information, vehicle direction of travel information, braking system status information, throttle information, steering wheel angle information, or vehicle routing information, or vehicle operation state information such as vehicle size information, headlight status information, turn signal information, wiper status information, transmission information, or any other information, or a combination of information related to the transmitting vehicle state. For example, transmission status information may indicate whether the transmitting vehicle's transmission is in neutral, parked, forward, or reverse.

[0062] Vehicle 2100 may communicate with the communication network 2300 via access point 2330. Access point 2330, which may include computing devices, is configured to communicate with vehicle 2100, the communication network 2300, one or more communication devices 2400, or a combination thereof, via wired or wireless links 2310 / 2340. For example, access point 2330 could be a base station, base transceiver station (BTS), node B, extension node B (eNode-B), home node B (HNode-B), wireless router, wired router, hub, relay, switch, or any similar wired or wireless device. Although shown here as a single unit, the access point may include any number of interconnected elements.

[0063] The vehicle 2100 may communicate with the communication network 2300 via a satellite 2350 or other non-terrestrial communication device. The satellite 2350, which may include computing devices, is configured to communicate with the vehicle 2100, the communication network 2300, one or more communication devices 2400, or a combination thereof, via one or more communication links 2320 / 2360. Although shown here as a single unit, the satellite may include any number of interconnected elements.

[0064] The electronic communication network 2300 is any type of network configured to provide voice, data, or any other type of electronic communication. For example, the electronic communication network 2300 may include a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a mobile or cell phone network, the internet, or any other electronic communication system. The electronic communication network 2300 uses communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Internet Protocol (IP), Real-time Transport Protocol (RTP), Hypertext Transport Protocol (HTTP), or a combination thereof. Although shown here as a single unit, the electronic communication network may include any number of interconnected elements.

[0065] Vehicle 2100 can identify a portion or situation of the vehicle traffic network 2200. For example, the vehicle includes at least one on-board sensor 2105, such as the sensor 1360 shown in Figure 1, which may be or include any other sensor or device or combination thereof that can determine or identify a portion or situation of the vehicle traffic network 2200.

[0066] The vehicle 2100 may traverse part or more of the vehicle traffic network 2200 using information communicated via the network 2300, such as information representing the vehicle traffic network 2200, information identified by one or more onboard sensors 2105, or a combination thereof.

[0067] Figure 2 shows, for simplification, one vehicle traffic network 2200, one electronic communication network 2300, and one communication device 2400, but any number of networks or communication devices may be used. The vehicle traffic and communication system 2000 may include devices, units, or elements not shown in Figure 2. The vehicle 2100 is shown as a single unit, but the vehicle may include any number of interconnected elements.

[0068] Although the vehicle 2100 is shown to communicate with the communication device 2400 via the network 2300, the vehicle 2100 may communicate with the communication device 2400 via any number of direct or indirect communication links. For example, the vehicle 2100 may communicate with the communication device 2400 via a direct communication link such as a Bluetooth communication link.

[0069] Figure 3 is a diagram of a portion of the vehicle traffic network according to this disclosure. The vehicle traffic network 3000 may include one or more unnavigable areas 3100 such as buildings, one or more partially navigable areas such as parking areas 3200, one or more navigable areas such as roads 3300 / 3400, or a combination thereof. In some embodiments, an autonomous vehicle, such as the vehicle 1000 shown in Figure 1, one of the vehicles 2100 / 2110 shown in Figure 2, a semi-autonomous vehicle, or any other vehicle implementing autonomous driving, may traverse a portion or more of the vehicle traffic network 3000.

[0070] The vehicle traffic network 3000 may include one or more interchanges 3210 between one or more navigable or partially navigable areas 3200 / 3300 / 3400. For example, a portion of the vehicle traffic network 3000 shown in Figure 3 includes an interchange 3210 between a parking area 3200 and a road 3400.

[0071] A portion of the vehicle traffic network 3000, such as roads 3300 / 3400, may include one or more lanes 3320 / 3340 / 3360 / 3420 / 3440, which may be associated with one or more directions of travel as indicated by the arrows in Figure 3.

[0072] A portion of a vehicle traffic network, such as the vehicle traffic network 3000 shown in Figure 3, or a portion thereof, may be represented as vehicle traffic network information. For example, vehicle traffic network information may be represented as a hierarchy of elements, such as markup language elements that can be stored in a database or file. For simplicity, the figures herein depict vehicle traffic network information where a portion of a vehicle traffic network is represented as a figure or map, but vehicle traffic network information may be represented in any computer-readable form that can represent a vehicle traffic network or a portion thereof. In some embodiments, vehicle traffic network information may include vehicle traffic network control information such as direction of travel information, speed limit information, toll information, gradient information such as incline or angle information, surface material information, landscape information, number of lanes, known hazards, or a combination thereof.

[0073] Vehicle traffic networks may be associated with or include pedestrian traffic networks. For example, Figure 3 includes a portion of a pedestrian traffic network, which may be a pedestrian walkway. Although not shown separately in Figure 3, pedestrian-navigable areas such as pedestrian walkways or crosswalks may correspond to navigable or partially navigable areas of a vehicle traffic network.

[0074] In some embodiments, a portion or combination of portions of a vehicle traffic network may be identified as a point of interest or destination. For example, vehicle traffic network information may identify a building such as an unnavigable area 3100 and an adjacent partially navigable parking area 3200 as a point of interest, a vehicle may identify the point of interest as a destination, and the vehicle may travel from the starting point to the destination by traversing the vehicle traffic network. In Figure 3, the parking area 3200 associated with the unnavigable area 3100 is shown as being adjacent to the unnavigable area 3100, but the destination may include, for example, a building and a parking area that is not physically or geographically adjacent to the building.

[0075] Traversing a portion of a vehicle network can proceed from the topological location estimation of a vehicle to its destination. The destination may be an individual, uniquely identifiable geographical location. For example, the vehicle network may include defined locations such as a destination's physical address, postal address, vehicle network address, GPS address, or a combination thereof. The destination may be associated with one or more entrances, such as entrance 3500 shown in Figure 3. The destination may be associated with one or more docking locations, such as docking location 3700 shown in Figure 3. Docking location 3700 may be a designated or undesignated location or area adjacent to the destination where a vehicle can stop, wait, or park so that docking operations, such as passenger boarding or alighting, can be performed.

[0076] Figure 4 shows an example of an autonomous vehicle operation management system 4000 according to an embodiment of the present disclosure. The autonomous vehicle operation management system 4000 may be implemented in an autonomous vehicle such as the vehicle 1000 shown in Figure 1, one of the vehicles 2100 / 2110 shown in Figure 2, a semi-autonomous vehicle, or any other vehicle that implements autonomous driving.

[0077] An autonomous vehicle may traverse a vehicle traffic network or a portion thereof, which may include traversing individual vehicle operation scenarios. An individual vehicle operation scenario (also referred to herein as a scenario) may include any clearly identifiable set of operating conditions that may affect the operation of the autonomous vehicle within a defined spatiotemporal area or operating environment. An individual vehicle operation scenario may be based on the number or density of roads, road sections, or lanes that the autonomous vehicle may traverse within a defined spatiotemporal distance. An individual vehicle operation scenario may be based on one or more traffic control devices that may affect the operation of the autonomous vehicle within a defined spatiotemporal area or operating environment. An individual vehicle operation scenario may be based on one or more identifiable rules, regulations, or laws that may affect the operation of the autonomous vehicle within a defined spatiotemporal area or operating environment. An individual vehicle operation scenario may be based on one or more identifiable external objects that may affect the operation of the autonomous vehicle within a defined spatiotemporal area or operating environment.

[0078] Examples of individual vehicle operation scenarios include individual vehicle operation scenarios in which an autonomous vehicle is crossing an intersection, individual vehicle operation scenarios in which a pedestrian is on or approaching the predicted path of an autonomous vehicle, and individual vehicle operation scenarios in which an autonomous vehicle is changing lanes. Individual vehicle operation scenarios may separately include merging lanes, or individual vehicle operation scenarios in which an autonomous vehicle is changing lanes may also include merging lanes.

[0079] For simplification and clarity, similar vehicle operation scenarios may be described herein by reference to a type or class of vehicle operation scenario. A type or class of vehicle operation scenario may refer to a specific pattern or set of patterns in a scenario. For example, a vehicle operation scenario involving pedestrians may be referred herein to as a pedestrian scenario, referring to a type or class of vehicle operation scenario involving pedestrians. As an example, a first pedestrian vehicle operation scenario may include a pedestrian crossing the road at a crosswalk, and a second pedestrian vehicle operation scenario may include a pedestrian crossing the road in violation of traffic regulations. While pedestrian vehicle operation scenarios, intersection vehicle operation scenarios, and lane change vehicle operation scenarios are described herein, any other vehicle operation scenario or type of vehicle operation scenario may be used.

[0080] The characteristics of the autonomous vehicle's operating environment may be represented within each individual vehicle operation scenario. For example, the relative orientation, trajectory, and predicted path of external objects may be represented within each individual vehicle operation scenario. In another example, the relative geometry of a vehicle traffic network may be represented within each individual vehicle operation scenario.

[0081] For example, the first individual vehicle operation scenario may correspond to a pedestrian crossing the road at a crosswalk, and the relative orientation of the pedestrian, such as crossing from left to right rather than from right to left, and the predicted path may be represented within the first individual vehicle operation scenario. The second individual vehicle operation scenario may correspond to a pedestrian crossing the road in violation of traffic regulations, and the relative orientation of the pedestrian, such as crossing from left to right rather than from right to left, and the predicted path may be represented within the second individual vehicle operation scenario.

[0082] An autonomous vehicle may traverse multiple different individual vehicle operation scenarios within an operating environment that may constitute a composite vehicle operation scenario. For example, a pedestrian may approach the predicted path of an autonomous vehicle crossing an intersection.

[0083] The autonomous vehicle operation management system 4000 may operate or control the autonomous vehicle to traverse individual vehicle operation scenarios in accordance with defined constraints, such as safety constraints, legal constraints, physical constraints, user acceptability constraints, or any other constraints or combinations of constraints that may be defined or derived for the operation of the autonomous vehicle.

[0084] Controlling an autonomous vehicle to traverse individual vehicle operation scenarios may include identifying or detecting individual vehicle operation scenarios, identifying candidate vehicle control actions based on individual vehicle operation scenarios, controlling the autonomous vehicle to traverse a portion of a vehicle traffic network according to one or more of the candidate vehicle control actions, or a combination thereof.

[0085] Vehicle control actions may include accelerating, decelerating, changing direction, stopping, inching forward, or any other vehicle actions or combinations of vehicle actions that can be performed by an autonomous vehicle in connection with crossing a portion of a vehicle traffic network.

[0086] The autonomous vehicle operation management controller 4100 or another unit of the autonomous vehicle may control the autonomous vehicle to traverse a vehicle traffic network or a portion thereof in accordance with vehicle control actions. Examples of vehicle control actions include “stop” vehicle control actions that bring the autonomous vehicle to a stop or remain stationary or control it in any other way; “forward” vehicle control actions that move the autonomous vehicle slowly forward for a short distance such as a few inches or a foot; “accelerate” vehicle control actions that accelerate the autonomous vehicle (e.g., at a defined acceleration or within a defined range); “decelerate” vehicle control actions that decelerate the autonomous vehicle (e.g., at a defined deceleration rate or within a defined range); “maintain” vehicle control actions that maintain current operating parameters (e.g., current speed, current path or route, current lane direction, etc.); “turn around” vehicle control actions (which may include a turning angle); “move forward” vehicle control actions that start or restart a previously identified set of operating parameters; or any other standard vehicle actions.

[0087] Vehicle control actions may be complex vehicle control actions that include a sequence, combination, or both of vehicle control actions. For example, a “forward” or “inching” vehicle control action may involve controlling the autonomous vehicle according to the “forward” vehicle control action, which involves controlling the autonomous vehicle to move slowly and gradually forward a short distance such as a few inches or a foot, such that the “stop” vehicle control action includes controlling the autonomous vehicle to move slowly and gradually forward a short distance such as a few inches or a foot, a “stop” vehicle control action, a subsequent “accelerate” vehicle control action associated with a defined acceleration, and a subsequent “stop” vehicle control action associated with a defined deceleration rate.

[0088] The autonomous vehicle operation management system 4000 may include an autonomous vehicle operation management controller 4100, a blocking monitor 4200, an operating environment monitor 4300, an SSOCEM 4400, or a combination thereof. As will be described separately, the blocking monitor 4200 may be one or more instances of the operating environment monitor 4300.

[0089] The autonomous vehicle operation management controller 4100 may receive, identify, or otherwise access operation environment information representing the autonomous vehicle's operation environment, such as the current operation environment, a predicted operation environment, or one or more of these. The autonomous vehicle's operation environment may include a clearly identifiable set of operation conditions that may affect the vehicle's operation within a defined spatiotemporal area.

[0090] Operating environment information may include vehicle information for autonomous vehicles, such as information indicating the vehicle's geospatial location, information correlating the geospatial location with information representing the vehicle traffic network, the vehicle's route, the vehicle's speed, the vehicle's acceleration status, the vehicle's occupant information, or any other information relating to the vehicle or its operation.

[0091] The operating environment information may include information representing the vehicle traffic network in close proximity to the autonomous vehicle, such as within a defined spatial distance of the vehicle (e.g., 300 meters), information indicating the geometry of one or more aspects of the vehicle traffic network, information indicating the state of the vehicle traffic network, such as the surface state, or any combination thereof.

[0092] The operating environment information may include information representing external objects within the operating environment of the autonomous vehicle, such as pedestrians, non-human animals, non-motorized means of transportation such as bicycles or skateboards, motorized means of transportation such as remote vehicles, or any other external objects or entities that may affect the operation of the vehicle.

[0093] The autonomous vehicle operation management controller 4100 may monitor the operating environment of the autonomous vehicle, or a defined mode thereof. Monitoring the operating environment may include identifying and tracking external objects, identifying individual vehicle operation scenarios, or a combination thereof.

[0094] For example, the autonomous vehicle operation management controller 4100 may identify and track external objects using the operating environment of the autonomous vehicle. Identifying and tracking external objects may include identifying the spatiotemporal position of each external object, which may be relative to the vehicle, and identifying one or more predicted paths of each external object, which may include identifying the velocity, trajectory, or both of the external objects. Descriptions of position, predicted position, path, and predicted path as used herein may omit explicit indication that the corresponding position and path refer to geospatial and temporal components, but unless expressly indicated herein or otherwise unambiguously apparent from the context, the positions, predicted positions, paths, and predicted paths as used herein may include geospatial components, temporal components, or both.

[0095] The operating environment monitor 4300 may include a pedestrian operating environment monitor 4310, an intersection operating environment monitor 4320, a lane change operating environment monitor 4330, or a combination thereof. The operating environment monitor 4340 is indicated using a dashed line to show that the autonomous vehicle operation management system 4000 may include any number of operating environment monitors 4300.

[0096] One or more individual vehicle operation scenarios may be monitored by their respective operating environment monitors 4300. For example, a pedestrian operating environment monitor 4310 may monitor operating environment information corresponding to multiple pedestrian vehicle operation scenarios, an intersection operating environment monitor 4320 may monitor operating environment information corresponding to multiple intersection vehicle operation scenarios, and a lane change operating environment monitor 4330 may monitor operating environment information corresponding to multiple lane change vehicle operation scenarios.

[0097] The operating environment monitor 4300 may receive, or otherwise access, operating environment information such as operating environment information, vehicle traffic network information, vehicle traffic network geometry information, or a combination thereof, generated or captured by one or more sensors of the autonomous vehicle. For example, the pedestrian operating environment monitor 4310 may receive, or otherwise access, information such as sensor data that may indicate, correspond to, or otherwise associate with one or more pedestrians in the operating environment of the autonomous vehicle.

[0098] The operating environment monitor 4300 may associate the operating environment information or a portion thereof with the operating environment or its characteristics, such as external objects like pedestrians, remote vehicles, or vehicle traffic network geometry.

[0099] The operating environment monitor 4300 may generate, or otherwise identify, information representing one or more aspects of the operating environment, such as external objects including pedestrians, remote vehicles, or aspects of vehicle traffic network geometry, which may include filtering, abstracting, or otherwise processing the operating environment information.

[0100] The operating environment monitor 4300 may output information representing one or more modes of the operating environment to the autonomous vehicle operation management controller 4100, or for access by the autonomous vehicle operation management controller 4100, by storing information representing one or more modes of the operating environment in a memory such as the memory 1340 shown in Figure 1 of the autonomous vehicle accessible by the autonomous vehicle operation management controller 4100, transmitting information representing one or more modes of the operating environment to the autonomous vehicle operation management controller 4100, or a combination thereof. The operating environment monitor 4300 may also output information representing one or more modes of the operating environment to one or more elements of the autonomous vehicle operation management system 4000, such as the blocking monitor 4200.

[0101] The pedestrian environment monitor 4310 can identify, track, or predict the behavior of one or more pedestrians by correlating, associating, or otherwise processing environment information. For example, the pedestrian environment monitor 4310 may receive information such as sensor data that can correspond to one or more pedestrians from one or more sensors. The pedestrian environment monitor 4310 may associate the sensor data with one or more identified pedestrians, which may include identifying for one or more of each identified pedestrian the direction of travel, the predicted path, the current or predicted speed, the current or predicted acceleration, or a combination thereof, and the pedestrian environment monitor 4310 may output the identified, associated, or generated pedestrian information to or for access by the autonomous vehicle operation management controller 4100.

[0102] The intersection operating environment monitor 4320 may correlate, associate, or otherwise process operating environment information to identify, track, or predict the behavior of one or more remote vehicles within the operating environment of an autonomous vehicle, thereby identifying an intersection or its configuration within the operating environment, and identifying a vehicle traffic network geometry or a combination thereof. For example, the intersection operating environment monitor 4320 may receive information from one or more sensors, such as sensor data, that may correspond to one or more remote vehicles within an operating environment, an intersection, or one or more configurations thereof within an operating environment, a vehicle traffic network geometry, or a combination thereof. The intersection operating environment monitor 4320 may associate sensor data with one or more identified remote vehicles in the operating environment, intersection, or one or more aspects thereof in the operating environment, vehicle traffic network geometry, or a combination thereof, which may include identifying for one or more of the identified remote vehicles the current or predicted direction of travel, predicted path, current or predicted speed, current or predicted acceleration, or a combination thereof, and the intersection operating environment monitor 4320 may output the identified, associated, or generated intersection information to or for access by the autonomous vehicle operation management controller 4100.

[0103] The lane change operation environment monitor 4330 may correlate, associate, or otherwise process operation environment information to identify, track, or predict the behavior of one or more remote vehicles within the autonomous vehicle's operation environment, such as information indicating slow-moving or stationary remote vehicles along the vehicle's predicted path, thereby identifying one or more aspects of the operation environment, such as the vehicle traffic network geometry in the operation environment, or a combination thereof that geospatially corresponds to current or predicted lane change operations. For example, the lane change operation environment monitor 4330 may receive information from one or more sensors, such as sensor data, that can correspond to one or more remote vehicles within the autonomous vehicle's operation environment, one or more aspects of the operation environment, or a combination thereof that geospatially corresponds to current or predicted lane change operations. The lane change operation environment monitor 4330 may associate sensor data with one or more identified remote vehicles in the operating environment of the autonomous vehicle, one or more aspects of the operating environment, or a combination thereof that geospatially corresponds to the current or predicted lane change operation, which may include identifying the current or predicted direction of travel, predicted route, current or predicted speed, current or predicted acceleration, or a combination thereof for one or more of the identified remote vehicles. The lane change operation environment monitor 4330 may output the identified, associated, or generated lane change information to or for access by the autonomous vehicle operation management controller 4100.

[0104] The autonomous vehicle operation management controller 4100 may identify one or more individual vehicle operation scenarios based on one or more aspects of the operating environment represented by the operating environment information. The autonomous vehicle operation management controller 4100 may identify individual vehicle operation scenarios in response to, or based on, the identification of operating environment information indicated by one or more of the operating environment monitors 4300. For example, the operating environment information may include information representing pedestrians approaching an intersection along the predicted path of the autonomous vehicle, and the autonomous vehicle operation management controller 4100 may identify a pedestrian vehicle operation scenario, an intersection vehicle operation scenario, or both.

[0105] The autonomous vehicle operation management controller 4100 may instantiate one or more instances of SSOCEM4400 based on one or more aspects of the operating environment represented by the operating environment information. For example, the autonomous vehicle operation management controller 4100 may instantiate each instance of SSOCEM4400 in response to identifying an upcoming scenario. An upcoming scenario may be a specific vehicle operation scenario that the autonomous vehicle operation management controller 4100 determines is likely to be encountered if the autonomous vehicle continues along its route. An upcoming scenario may be predictable (e.g., can be determined from the autonomous vehicle's route) or unexpected. An unexpected upcoming scenario may be one that can be detected by the vehicle's sensors and cannot be determined without sensor data.

[0106] When instantiated, the SSOCEM4400 receives operating environment information, including sensor data, and can determine and output candidate vehicle control actions, also referred to herein as candidate actions. A candidate action is a vehicle control action identified by a particular SSOCEM4400 as the most likely optimal action for a vehicle to perform in a given scenario. For example, an SSOCEM4400 configured to handle intersections (e.g., intersection SSOCEM4420) may output "proceed," a candidate action suggesting that the vehicle should pass through the intersection. Simultaneously, an SSOCEM4400 for handling lane changes (e.g., lane change SSOCEM4430) may output a "turn left" candidate action indicating that the vehicle should merge two steps to the left. In some implementations, each SSOCEM4400 outputs a confidence score indicating the confidence level of the candidate action determined by the SSOCEM4400. For example, a confidence score greater than 0.95 may indicate very high confidence in the candidate action, while a confidence score less than 0.5 may indicate relatively low confidence in the candidate action. Further details about the SSOCEM4400 will be provided later.

[0107] The autonomous vehicle operation management controller 4100 receives candidate actions and determines a vehicle control action based on the received candidate actions. In some implementations, the autonomous vehicle operation management controller 4100 determines a vehicle control action using hardcoded logic. For example, the autonomous vehicle operation management controller 4100 may select the candidate action with the highest confidence score. In other implementations, the autonomous vehicle operation management controller 4100 may select the candidate action that is least likely to result in a collision. In other implementations, the autonomous vehicle operation management controller 4100 may generate a composite action based on two or more non-conflicting candidate actions (for example, combining "proceed" and "turn left in two stages" to result in a vehicle control action that changes the vehicle's course to the left and passes through an intersection). In some implementations, the autonomous vehicle operation management controller 4100 may use a machine learning algorithm to determine a vehicle control action based on two or more different candidate actions.

[0108] For example, identifying vehicle control actions from candidate actions may involve implementing machine learning components, such as supervised learning for a classification problem, and training these machine learning components using examples, such as 1000 examples of corresponding vehicle operation scenarios. In another example, identifying vehicle control actions from candidate actions may involve implementing a Markov decision process (MDP) or a partially observed Markov decision process (POMDP), which may include a reward function that describes how each candidate action influences subsequent candidate actions and outputs a positive or negative reward for each vehicle control action.

[0109] The autonomous vehicle operation management controller 4100 may de-instantiate instances of SSOCEM4400. For example, the autonomous vehicle operation management controller 4100 may identify a separate set of operation statuses as representing individual vehicle operation scenarios of the autonomous vehicle, instantiate instances of SSOCEM4400 for the individual vehicle operation scenarios, monitor the operation statuses, and then determine that one or more of the operation statuses are expired or have a probability of affecting the operation of the autonomous vehicle below a defined threshold, and the autonomous vehicle operation management controller 4100 may de-instantiate instances of SSOCEM4400.

[0110] The blocking monitor 4200 can receive operating environment information that represents the vehicle's operating environment or its characteristics. For example, the blocking monitor 4200 may receive operating environment information from the autonomous vehicle operation management controller 4100, from the vehicle's sensors, from external devices such as remote vehicles or infrastructure devices, or a combination thereof. The blocking monitor 4200 may also read operating environment information or a portion thereof from memory, such as the memory of the autonomous vehicle, such as the memory 1340 shown in Figure 1.

[0111] The blocking monitor 4200 may determine the probability of usefulness for each of the parts of the vehicle traffic network, or the corresponding blocking probability, for one or more parts of the vehicle traffic network. The parts may include those parts of the vehicle traffic network that correspond to the predicted route of the autonomous vehicle.

[0112] The probability of usefulness, or the corresponding blocking probability, may indicate the probability or likelihood that the autonomous vehicle will safely traverse a portion of the traffic network or a spatial location within it, such as not being obstructed by external objects such as remote vehicles or pedestrians. For example, a portion of the traffic network may include obstacles such as stationary objects, and the probability of usefulness for that portion of the traffic network may be low, such as 0%, which may be expressed as a high blocking probability, such as 100%, for that portion of the traffic network. The blocking monitor 4200 may identify the respective probabilities of usefulness for each of multiple portions of the traffic network within the operating environment, such as within 300 meters of the autonomous vehicle.

[0113] The probability of usefulness may be indicated by a blocking monitor 4200 corresponding to each external object in the operating environment of the autonomous vehicle, and the geospatial area may be associated with multiple usefulness probabilities corresponding to multiple external objects. The total probability of usefulness may be indicated by a blocking monitor 4200 corresponding to each type of external object in the operating environment of the autonomous vehicle, such as the usefulness probability of pedestrians and the usefulness probability of remote vehicles, and the geospatial area may be associated with multiple usefulness probabilities corresponding to multiple external object types.

[0114] The blocking monitor 4200 may identify external objects, track external objects, and display location information, path information, or both, or a combination thereof, of the external objects. For example, the blocking monitor 4200 may identify external objects and identify their predicted paths based on operating environment information (e.g., the current location of the external object), information indicating the current trajectory and / or velocity of the external object, information indicating the classification type of the external object (e.g., pedestrian or remote vehicle), vehicle traffic network information (e.g., crosswalks near the external object), previously identified or tracked information associated with the external object, or any combination thereof. The predicted paths may represent a predicted spatial location, a predicted temporal location, and a corresponding sequence of probabilities.

[0115] The blocking monitor 4200 may communicate the probability of usefulness, or the corresponding blocking probability, to the autonomous vehicle operation management controller 4100. The autonomous vehicle operation management controller 4100 may communicate the probability of usefulness, or the corresponding blocking probability, to each instantiated instance of the scenario-specific operation control evaluation module 4400.

[0116] Although not explicitly shown in Figure 4, the autonomous vehicle operation management system 4000 may include a prediction module capable of generating prediction information and transmitting it to the blocking monitor 4200, and the blocking monitor 4200 may output the probability of usefulness information to one or more of the operating environment monitors 4300.

[0117] Each SSOCEM4400 may model its own individual vehicle operation scenario. The autonomous vehicle operation management system 4000 includes any number of SSOCEM4400s, each modeling its own individual vehicle operation scenario. Modeling an individual vehicle operation scenario may include generating and / or maintaining state information representing aspects of the vehicle's operating environment corresponding to the individual vehicle operation scenario, identifying potential interactions between the respective modeled aspects of the corresponding state, and determining candidate actions to resolve the model. More simply, an SSOCEM4400 may include one or more models configured to determine one or more vehicle control actions for handling a given set of inputs. Models may include, but are not limited to, partially observed Markov decision process (POMDP) ​​models, Markov decision process (MDP) models, classical planning (CP) models, partially observed probabilistic game (POSG) models, distributed partially observed Markov decision process (Dec-POMDP) ​​models, reinforcement learning (RL) models, artificial neural networks, hardcoded expert logic, or any other suitable type of model. Examples of different types of models are provided below. Each SSOCEM4400 contains computer executable instructions that define how the model operates and how the model is utilized.

[0118] SSOCEM4400 may implement a CP model, which may be a single-agent model that models individual vehicle operation scenarios based on defined input states. The defined input states may represent the non-stochastic states of each element of the autonomous vehicle's operating environment for individual vehicle operation scenarios. In the CP model, one or more aspects (e.g., geospatial location) of a modeled element (e.g., an external object) associated with a temporal location may differ non-stochastically from a corresponding aspect associated with another temporal location, such as the immediate next temporal location, by only a defined or fixed amount. For example, at a first temporal location, the remote vehicle may have a first geospatial location, and at the immediate next second temporal location, the remote vehicle may have a second geospatial location that differs from the first geospatial location by only a defined geospatial distance, such as a defined number of meters, along the remote vehicle's predicted path.

[0119] SSOCEM4400 may implement discrete-time stochastic control processes such as MDP models, which may be single-agent models that model individual vehicle operation scenarios based on defined input states. Changes in the operating environment of the autonomous vehicle, such as changes in the position of external objects, may be modeled as stochastic changes. MDP models may utilize more processing resources and may model individual vehicle operation scenarios more accurately than CP models.

[0120] The MDP model may model individual vehicle behavior scenarios using a set of states, a set of actions, a set of state transition probabilities, a reward function, or a combination thereof. In some embodiments, modeling individual vehicle behavior scenarios may include using a discount factor that can adjust or discount the output of the reward function applied to subsequent transient periods.

[0121] The set of states may include the current state of the MDP model, one or more possible subsequent states of the MDP model, or a combination thereof. A state represents an identified situation that may be a predictable situation of each defined aspect of the vehicle's operating environment, such as external objects and traffic control devices, which may probabilistically influence the vehicle's movement at each individual temporal location. For example, a remote vehicle operating in the vicinity of a vehicle may influence the vehicle's movement and may be represented in the MDP model. The MDP model may include representing the following identified or predictable information of the remote vehicle: its geospatial location corresponding to each temporal location, its path, direction of travel, or both, its speed, its acceleration or deceleration rate, or a combination thereof. At instantiation, the current state of the MDP model may correspond to concurrently occurring states or situations in the operating environment.

[0122] Any number or concentration of states may be used, but the number or concentration of states included in the model may be limited to a defined maximum number of states. For example, the model may include the 300 most likely states for the corresponding scenarios.

[0123] The set of actions may include vehicle control actions available to the MDP model in each state within the set of states. Each set of actions may be defined for each individual vehicle operation scenario.

[0124] The set of state transition probabilities may probabilistically represent potential or predicted changes to the vehicle's operating environment in response to an action, as represented by the state. For example, the state transition probabilities may represent the probability that the operating environment corresponds to each state at each temporal position immediately following the current temporal position corresponding to the current state, in response to the vehicle traversing the vehicle network from the current state according to each action.

[0125] The set of state transition probabilities may be identified based on operating environment information. For example, the operating environment information may include traffic conditions, such as area type (e.g., urban or rural), time of day, ambient light level, weather conditions, rush hour conditions, event-related traffic congestion, or holiday-related driver behavior; road conditions; jurisdictional conditions, such as national, state, or municipal conditions; or any other conditions or combinations of conditions that may affect vehicle operation.

[0126] Examples of state transition probabilities associated with pedestrian vehicle action scenarios may include a defined probability that a pedestrian crosses the road in violation of traffic regulations (e.g., based on the geospatial distance between the pedestrian and each section of road), a defined probability that a pedestrian stops at an intersection, a defined probability that a pedestrian crosses a crosswalk, a defined probability that a pedestrian yields to an autonomous vehicle at a crosswalk, and any other probabilities associated with pedestrian vehicle action scenarios.

[0127] Examples of state transition probabilities associated with intersection vehicle behavior scenarios may include a defined probability that a remote vehicle reaches the intersection, a defined probability that a remote vehicle obstructs the progress of an autonomous vehicle, a defined probability that a remote vehicle crosses the intersection in very close proximity to a second remote vehicle crossing the intersection (piggybacking), such as when no right of way exists, a defined probability that a remote vehicle stops adjacent to the intersection before crossing it in accordance with traffic control devices, regulations, or other instructions regarding the right of way, a defined probability that a remote vehicle crosses the intersection, a defined probability that a remote vehicle deviates from a predicted path near the intersection, a defined probability that a remote vehicle deviates from a predicted right of way, and any other probabilities associated with intersection vehicle behavior scenarios.

[0128] Examples of state transition probabilities associated with a lane change vehicle action scenario may include a defined probability that a remote vehicle changes speed, such as a defined probability that a remote vehicle behind the vehicle increases speed or a defined probability that a remote vehicle in front of the vehicle decreases speed; a defined probability that a remote vehicle in front of the vehicle changes lanes; a defined probability that a remote vehicle is close to the vehicle's speed change in order to allow the vehicle to merge into the lane; or any other probability associated with a lane change vehicle action scenario.

[0129] The reward function can determine the positive or negative (cost) values ​​that occur for each combination of state and action. These occurrences represent the expected value for vehicles traversing the vehicle network from a corresponding state to a subsequent state, according to the corresponding vehicle control actions.

[0130] For example, a POMDP model may include an autonomous vehicle at a first geospatial and temporal location corresponding to a first state. The model may indicate that the vehicle identifies and performs, or attempts to perform, vehicle control actions to traverse a vehicle traffic network from the first geospatial location to a second geospatial location at a second temporal location immediately following the first temporal location. The set of observations corresponding to the second temporal location may include operating environment information identified in relation to the second temporal location, such as the vehicle's geospatial location information, the geospatial location information of one or more external objects, the probability of usefulness, or predicted route information.

[0131] The set of conditional observation probabilities may include the probability of making each observation based on the operating environment of the autonomous vehicle. For example, the autonomous vehicle may approach an intersection by crossing a first road, and at the same time, the remote vehicle may approach the intersection by crossing a second road. The autonomous vehicle may identify and evaluate operating environment information, such as sensor data corresponding to the intersection, which may include operating environment information corresponding to the remote vehicle. The operating environment information may be inaccurate, incomplete, or incorrect. In the MDP model, the autonomous vehicle may non-probabilistically identify the remote vehicle, which may include identifying its location or predicted route, and the identified information, such as the location identified based on inaccurate operating environment information, may be inaccurate or incorrect. In the POMDP model, the autonomous vehicle may identify information that probabilistically identifies the remote vehicle, such as probabilistically identifying the remote vehicle's location information. The conditional observation probability corresponding to observing or probabilistically identifying the remote vehicle's location represents the probability that the identified operating environment information accurately represents the remote vehicle's location.

[0132] The set of conditional observation probabilities may be identified based on operating environment information, such as operating environment information described with respect to the reward function.

[0133] SSOCEM4400 may implement a Dec-POMDP model, which may be a multi-agent model that models individual vehicle operation scenarios. The Dec-POMDP model may be similar to the POMDP model, except that the POMDP model models a suitable subset, such as a vehicle and one of the external objects, while Dec-POMDP models an autonomous vehicle and a set of external objects.

[0134] SSOCEM4400 may implement a POSG model, which may be a multi-agent model that models individual vehicle operation scenarios. The POSG model may be similar to Dec-POMDP, except that the Dec-POMDP model includes a reward function for the vehicle, while the POSG model includes a reward function for the vehicle and its respective reward functions for each external object.

[0135] SSOCEM4400 may implement an RL model, which may be a learning model that models individual vehicle operation scenarios. The RL model may be similar to an MDP model or POMDP model, except that defined state transition probabilities, observation probabilities, reward functions, or any combination thereof may be omitted from the model. Alternatively, for example, the RL model may be a model-based RL model that generates state transition probabilities, observation probabilities, reward functions, or any combination thereof based on one or more modeled or observed events.

[0136] In an RL model, the model may evaluate one or more events or interactions that may include simulated events, and in response to each event, it may generate or modify a corresponding model or solution. Simulated events may include, for example, crossing an intersection, crossing a traffic network near pedestrians, or changing lanes. An example of using an RL model to cross an intersection includes an RL model that shows candidate actions for crossing an intersection. The autonomous vehicle then crosses the intersection using the candidate actions as vehicle control actions in relation to its temporal position. The result of crossing the intersection using the candidate actions may be determined, and the RL model may be updated based on that result.

[0137] The autonomous vehicle operation management system 4000 may include any number or combination of models. For example, the pedestrian SSOCEM4410, intersection SSOCEM4420, and lane change SSOCEM4430 may implement the POMDP model. In another example, the pedestrian SSOCEM4410 may implement the MDP model, and the intersection SSOCEM4420 and lane change SSOCEM4430 may implement the POMDP model. Furthermore, the autonomous vehicle operation management controller 4100 may instantiate any number of instances of SSOCEM4400 based on operating environment information. Module 4440 is shown using dashed lines to indicate that the autonomous vehicle operation management system 4000 may include any number or additional types of SSOCEM4400.

[0138] One or more of the autonomous vehicle operation management controller 4100, blocking monitor 4200, operating environment monitor 4300, or SSOCEM 4400 may operate continuously or periodically at a frequency such as 10 Hz. For example, the autonomous vehicle operation management controller 4100 may identify vehicle control actions many times, such as 10 times per second. The operating frequencies of each component of the autonomous vehicle operation management system 4000 may or may not be synchronized, and the operating speed of one or more of the autonomous vehicle operation management controller 4100, blocking monitor 4200, operating environment monitor 4300, or SSOCEM 4400 may be independent of the operating speeds of the others.

[0139] Figure 5 shows an example of a competence recognition system (CAS) 5000 according to an embodiment of the present disclosure. The CAS 5000 includes an autonomous cognitive agent (ACA) 5002. Any of the SSOCEMs, such as one of SSOCEMs 4410, 4420, 4430, and 4440 in Figure 4, can be an ACA as described below with respect to the ACA 5002. The ACA 5002 can have multiple levels of autonomy. The CAS 5000 can be the autonomous vehicle operation management system 4000 in Figure 4.

[0140] The ACA5002 operates in multiple levels of autonomous operation (i.e., levels of autonomy), and multiple levels of autonomous operation can be planned / used. When the AV performs a vehicle control action at an autonomous level, the AV (more specifically, the ACA) may receive a feedback signal (or simply feedback) in response to the AV's performance of the vehicle control action. For example, the AV may perform a vehicle control action while being monitored by a human. The vehicle may be remotely monitored by a remote operator. The vehicle may be monitored by a person inside the vehicle. The person may cause the AV (or a module, component, or circuit within the AV, such as the ACA5002) to send or receive a signal indicating feedback.

[0141] The ACA5002 can proactively decide what level of autonomy to implement. Therefore, the ACA5002 can have a gradient of autonomy levels. Consequently, the ACA5002 can maintain robustness without requiring the same effort as all-or-nothing (i.e., binary) autonomy. Therefore, the ACA5002 can proactively avoid situations where it cannot autonomously navigate and proactively issue requests for help before the AV gets stuck. By proactively requesting help, wasted operational time and resources can be reduced.

[0142] As will be further explained below, the ACA5002 can learn from its experience traversing a portion of the vehicle traffic network the level of autonomy it will enter when traversing the same or similar portion of the vehicle traffic network. In other words, the ACA5002 can learn from its experience to anticipate the feedback it will receive from human operators, such as remote operators or people on board the vehicle.

[0143] ACA5002 can modify its autonomy level selection based on new knowledge gained while traversing a portion of the vehicle traffic network. Learned predictions can be used to modify how ACA5002 traverses a portion of the vehicle traffic network. For example, if ACA5002 has high confidence in responses received from humans, a request for human assistance should not be issued. Therefore, ACA5002 will only request assistance in situations where it is confident that it cannot handle the situation autonomously. Furthermore, ACA5002 can proactively avoid situations in which it determines (based on its competence level) that it will require human assistance. Thus, ACA5002 can prioritize situations in which it can navigate autonomously. For example, ACA5002 can replan to avoid a situation (e.g., decide on a new route).

[0144] ACA5002 uses the Domain Model (DM) 5004, the Human Model (HM) 5006, and the Autonomy Model (AM) 5008 as inputs. HM5006 may also be called the Feedback Model. ACA5002 updates HM5006, indicated by arrow 5012, as will be further described below. ACA5002 updates AM5008, indicated by arrow 5010, as will be further described below.

[0145] As described above, DM5004 can model the environment in which ACA5002 operates (i.e., the operating environment).

[0146] AM5008 can model the levels of autonomy in which ACA5002 can operate, when (e.g., under what circumstances or constraints) ACA5002 may be able to operate at each level of autonomy, and what the utility is for each level of autonomy. The utility can indicate the expected value of performing vehicle control actions at each level of autonomy. Utility values ​​can be used to represent preferences between levels of autonomy.

[0147] HM5006 can describe a feedback model that models the feedback that ACA5002 can receive from humans (e.g., remote operators), how much each type of feedback costs, and how likely ACA5002 is to receive each type of feedback.

[0148] As described above, DM5004 can model the environment in which ACA5002 operates (i.e., the operating environment). For example, DM5004 can describe (e.g., include) the environmental transition dynamics and / or cost dynamics for ACA5002. In one example, DM5004 can be modeled as a stochastic shortest path (SSP) problem. As is known, an SSP is a formal decision-making model for reasoning in a fully observable stochastic environment, which can aim to find the minimum cost path from a starting state to a target state. For example, the goal of ACA5002 may be to successfully cross an intersection. Thus, the starting state may be a first temporal position before the intersection (e.g., 50 meters before the intersection), and the target state may be a second temporal position after passing the intersection (e.g., 50 meters after the intersection). As ACA5002 approaches the intersection, ACA5002 plans a trajectory, which includes a set of actions, to achieve its goal. As a result of the plan, ACA5002 selects the following actions to be performed at the level of autonomy, in accordance with HM5006 and AM5008.

[0149] DM5004 is a tuple <S,A,T,C,s o It can be formally modeled as G〉. S can be a finite set of states (i.e., a set of the set of operating environment information). A can be a finite set of actions (i.e., a set of vehicle control actions). T (i.e., T:S × A → Δ |S| C(i.e., C) can represent a transition function that describes the probability distribution over subsequent states when an action a∈A is taken in a state s∈S.

number

[0150] The solution to SSP in DM5004 can be expressed as policy π: S → A. That is, under policy π, action a (i.e., π(s)) is selected for state s. In other words, policy π can be shown to indicate that action π(s) ∈ A should be taken in state s. Policy π is the predicted cumulative cost V to reach the target state G from state s following policy π. π Value function V that can represent (s) π :S→C can be included. That is, the value function is defined by each intermediate state s from the starting state to the target state. i Provides the cost (i.e., value) of π. Optimal policy π * This minimizes the predicted cumulative cost.

[0151] Therefore, for all state configurations of interest, policies can be used to determine the action the AV should take in that state. An example of a state description might be "There is a pedestrian in front of the AV" and "The AV is at a stop sign." Such a state can be associated with the action "Stop" to avoid hitting the pedestrian. Another example of a state description might be "The AV is in an intersection" and "There is another vehicle at a stop sign." Such a state can be associated with the action "Go" so that the AV can complete crossing the intersection.

[0152] Intuitively speaking, DM5004 can include descriptors for all combinations of states that describe how the world (i.e., the operating environment) changes in a single time step. DM5004 can include the concepts of good and bad states; that is, DM5004 can include descriptors for which configuration of the world is good. For example, in a crossing scenario, a good state might be that the AV has successfully crossed the intersection, while a bad state might correspond to the AV colliding with another vehicle or the AV violating the law. Negative rewards might be associated with bad states, and positive rewards with good states. Given such descriptors and rewards, the optimal policy π is the set of actions (in other words, the paths through the operating environment) that maximize the reward. * However, it can be calculated as a function of how the operating environment evolves over time.

[0153] Other fully observable probabilistic models, such as MDPs, may be used with minimal variation, but partially observable MDPs (POMDPs) can introduce additional sources of uncertainty, particularly with respect to human interactions, making them difficult to use. AMs describe the levels of autonomy an agent can operate on, the constraints on the circumstances under which each level is permitted, the utility of each level, a set of system subcompetencies, or any combination thereof.

[0154] As described above, AM5010 can model the range of autonomous operations (i.e., autonomy levels) in which ACA5002 can operate. Autonomy levels can indicate both the actual different forms or ranges of autonomous operations (as described below with respect to the set of autonomy levels L) and when each of the autonomy levels may be permitted under certain external constraints (as described below with respect to the autonomy profile κ).

[0155] AM5008 can be formally modeled as a tuple ⟨L, κ, μ, T⟩, where L represents a finite set of autonomy levels, and each level l ∈ L corresponds to some set of constraints on the autonomous behavior of ACA5002. κ (i.e., κ: S × A → P(L)) is an autonomy profile that describes which autonomy levels l ∈ L are permitted when performing an action a ∈ A in a state s ∈ S. μ (i.e.,

Number

[0156] L = {l0,..., l n} can be a set of autonomy levels. Each autonomy level l i can correspond to a set of constraints on the autonomous behavior of ACA5002. In one example, the set of actions L can be a partially ordered set (i.e., a poset). That is, the actions in the set L can have an order or sequence that, for example, indicates an increase in the level of autonomy.

[0157] In one example, the autonomy levels of the set L can include four autonomy levels, namely, the "no autonomy" level (i.e., l0), the "autonomy with verification" level (i.e., l1), the "autonomy with supervision" level (i.e., l2), and the "autonomy without supervision" level (i.e., l4). The disclosure herein is not limited to such a set of autonomy levels L. That is, other autonomy levels with different semantics are possible.

[0158] The "no autonomy" level 10 can indicate that the ACA5002 requires a human to perform actions on its behalf. The no autonomy level can be summarized as the AV requiring complete control from the human so that the human can get the AV out of a certain situation (e.g., an obstacle scenario).

[0159] "Verified autonomy" level 11 indicates that the ACA5002 must seek and receive explicit approval from a human operator even before attempting a selected (e.g., identified, decided, etc.) action. For example, in a set of actions (i.e., a plan) decided by the ACA5002, the ACA must seek explicit approval for each action before it is performed.

[0160] "Supervised Autonomy" level 12 indicates that the ACA5002 can perform actions autonomously as long as there is a human monitoring it (e.g., remotely or otherwise). "Supervised Autonomy" level 12 allows for human intervention if something goes wrong while an action is being performed autonomously. For example, a series of actions may be performed as long as a human is supervising the AV. If a failure is detected before or after the execution of one of these actions, the ACA may request assistance from a human (e.g., a remote operator).

[0161] To clarify the distinction between "verified autonomy" level l1 and "supervised autonomy" level l2, an example is given here. At "supervised autonomy" level l2, supervision does not need to be remote. For example, an AV test procedure may be considered "supervised autonomy" because the AV can operate autonomously, but only if there is a supervising human inside the AV who is ready to override and control the AV in dangerous situations. As a further distinction, "verified autonomy" level l1 may require the ACA5002 to receive explicit permission from a human (which may be inside the AV or remotely) before performing a desired action. In particular, receiving explicit permission may mean that the ACA5002 should stop until permission is received. On the other hand, at "supervised autonomy" level l2, there is no such requirement as long as there is a supervising human. That is, the ACA5002 does not need to stop and can continue to perform its desired action, relying on a supervising human authority to override in case of potential danger.

[0162] Level 13 of "unsupervised autonomy" can indicate that the ACA5002 can be fully autonomous without requiring human approval, supervision, or monitoring.

[0163] An autonomy profile κ (i.e., κ: S × A → P(L)) can map a state s ∈ S and an action a ∈ A to a subset of the set of autonomy levels L, where P(L) represents the power set of the set of autonomy levels L. The autonomy profile κ can define constraints on the allowed level of autonomy for any given situation (i.e., a state in DM5004). Given the current state of the environment and the next action to be performed, the autonomy profile κ defines a set of acceptable levels of autonomy.

[0164] Constraints can be hard constraints or may include hard constraints. For example, constraints can be technical, legal, or ethical constraints. For example, an unrestricted example of a legal constraint might be that an autonomous vehicle cannot operate autonomously (i.e., at unsupervised autonomy level 14) within a school zone. An unrestricted example of a social constraint might be a road rule that when a traffic light turns green, oncoming vehicles yield to the first vehicle turning left. Thus, the constraint is that the first vehicle turning left must proceed without waiting for traffic to clear.

[0165] The constraints may be, may include, or may be used as temporary conservative constraints that can be updated over time as ACA5002 improves. The autonomy profile κ can constrain the space of all policies (π), so that ACA5002 can only follow policies that never violate the autonomy profile κ.

[0166] Utility μ represents the following: Given that an action at time step t was performed at autonomy level l, what is the utility of performing another action at time step t+1 at a different autonomy level l'? The action at time step t+1 does not have to be the same as the action taken at time step t, but it can be. In some situations, there may be negative utilities associated with swapping autonomy levels. For example, in dynamic situations (e.g., complex intersections), ACA5002 may swap "supervised autonomy" and "unsupervised autonomy" at each time step if utility μ is not given. Constant swapping of autonomy levels can actually be more unpleasant for a human being who has to constantly switch attention than simply remaining in "supervised" mode for the entire duration of time.

[0167] As will be further explained below, the AM5008 can evolve; that is, the AM5008 can be trained based on the experience of the ACA5002. For example, suppose a first AV is deployed in a first market (e.g., Japan) and a second AV is deployed in a second market (e.g., France). The first and second AVs may initially contain the same autonomy model operating in binary autonomy mode; that is, the AV can either secretly handle situations that it has been programmed to recognize and traverse in advance, or the AV may request assistance from a human (e.g., a remote operator). Since each of the first and second markets may have different (e.g., social) road rules, the autonomy model of the first AV will evolve differently from the autonomy model of the second AV based on the feedback each receives from a human. Once the autonomy model learns the situations it is capable of handling in its respective market, the ACA no longer needs to request assistance from a human (e.g., a remote operator) for the learned situations (i.e., scenarios).

[0168] Without losing universality, L can be assumed to be a perfectly ordered set, and can be extended to any graph where two levels are connected, if the level of autonomy can change from one level to another. The constraints corresponding to each level of autonomy can be internally imposed constraints, such as being technical in nature, i.e., requiring human supervision during bad weather that is known in advance to be error-causing, and externally imposed constraints, such as being ethical or legal in nature. Furthermore, κ can be defined to reflect not only fixed constraints but also transient constraints that can be updated over time and help enable more conservative autonomous behavior while the system is still learning. Each constraint may be associated with a corresponding form of human assistance or involvement. For example, at the level of "supervised autonomy," the agent can operate fully autonomously, provided that there is a human presence that actively monitors the agent's execution and can override any actions deemed unsafe or undesirable. Thus, the higher the level of autonomy, the lower the cost of human involvement, but this is not a requirement of the model.

[0169] A system's competence may depend on the behavior of subsystems that inform the main process the system is planning. For example, a perceptual system that is highly likely to produce incorrect state updates in certain situations may contribute to lower competence in those situations. Subcompetencies can be mathematical expressions of subsystem behavior that inform the main process at an abstract level what the planning model is inferring. A perceptual subcompetency may be defined as the likelihood that perception will fail in a given state. In this case, perception may be a subsystem that informs the main process by providing state update information. Subcompetencies that are fully known, or assumed to be fully known, may be included in the domain model as part of the transition function and therefore may be omitted from the autonomy model. Unknown subcompetencies may be calculated based on observations and data collected while operating online.

number

[0170] As described above, HM5006 (i.e., the feedback model) can model the reliability of ACA5002 in relation to its interaction with a human operator. HM5006 is a tuple <Σ,λ,ρ,τ> H It can be formally expressed as , where Σ represents a finite set of feedback signals that the agent can receive from the human, λ represents the feedback profile and the probability distribution of feedback signals that the agent will receive when performing action a∈A at level l∈L in state s∈S, assuming the agent has just performed action at level l'∈L, ρ represents the human cost function and the cost to the human when performing action a∈A at level l∈L in state s∈S, assuming the agent has just performed action at level l'∈L, and τ H This represents the human state transition function and the probability distribution of the subsequent state s'∈S when a human controls the system as an agent attempts to perform an action a∈A in state s∈S.

[0171] Σ={σ0,...,σ n} can be the set of possible feedback signals that ACA5002 can receive from a human operator. Non-restrictive examples of feedback signals are described below with respect to Figure 6. The feedback profile λ can represent the probability that ACA5002 receives a signal σ∈Σ when performing action a∈A at autonomy level l'∈L, assuming that ACA5002 is in state s∈S and has just operated at autonomy level l∈L. Thus, the feedback profile λ is λ:S×L×A×L→Δ |Σ| It can be represented symbolically as follows:

[0172] "Having just acted at ~" can mean, in the following time step t, "Having just acted at ~" can mean the level of autonomy at which the action taken by the ACA in time step t-1 was performed. As an example, suppose that in time step t, the ACA performed action a at autonomy level l2 (i.e., "supervised autonomy"). Thus, the human is already involved and observing the ACA's behavior. If the ACA performs action a' again at autonomy level l2 in time step t+1, the probability that the human will override action a' may be lower than if the ACA performed action a at autonomy level l3 (i.e., "unsupervised autonomy"), where the human is more likely to be surprised and therefore more likely to override the action.

[0173] The human cost function ρ can return a positive cost for a human to perform action a∈A at autonomy level l'∈L, assuming that ACA5002 is in state s∈S and has just acted at autonomy level l∈L. The human cost function ρ is symbolically

number

[0174] The human state transition function τ can represent the probability that a human (e.g., a remote operator) will transition ACA5002 to state s'∈S when ACA5002 chooses to perform action a∈A in state s∈S and a human controls AV. "Transitioning ACA5002 to state s" means that a human manipulates AV so that state s is realized. Human state transition function τ H is, τ H :S×A→Δ |S| It can be symbolically represented as follows. For example, suppose the state is s (e.g., s = "at an intersection") and ACA is about to perform action a (e.g., a = "turn left"), but a human overrides ACA and takes over control. In this case, the human state transition function τ H This represents the probability that a human would transition the ACA to any state (e.g., complete the left turn or go straight instead), given the state in which the ACA was (i.e., state s) and the action the agent attempted to take (i.e., turn left).

[0175] In practice, the feedback profile λ and the human state transition function τ H It should be noted that this is not known in advance. Therefore, ACA5002 uses the feedback profile λ and the human state transition function τ based on previous data collected by ACA5002 under the same or similar circumstances. H Each of the estimated values ​​can be maintained. The update of HM5006 is indicated by arrow 5012. Thus, as will be further explained with respect to Figure 6, after ACA5002 has performed an action (in the action execution phase), system 5000 can record any feedback received by ACA5002 from a human operator, and use the feedback to create a feedback profile λ or a human state transition function τ. H At least one of them can be updated.

[0176] System 5000 (i.e., the Competence Recognition System (CAS)), more specifically ACA5002, can be considered a solution to problems (e.g., definition, decision, etc.) that combine DM5004, HM5006, and AM5008 in the context of automated planning and decision-making.

[0177] While DM5004 can represent the underlying SSP on which ACA5002 finds a solution, ACA5002 can proactively generate plans that operate across multiple levels of autonomy (e.g., for different levels of autonomy and using different levels of autonomy) using AM5008. This is in contrast to autonomous agents that can adjust their plans during plan execution. Proactively generated plans can be subject to a set of constraints κ. ACA5002 can predict the likelihood of each feedback signal in advance using HM5006, thereby enabling ACA5002 to avoid situations where it is unlikely to be able to operate autonomously.

[0178] System 5000 can combine all three components—DM5004, HM5006, and AM5008—to form a single decision-making framework. System 5000 (more specifically, ACA5002) is used to solve the problem of generating policies to accomplish its task (e.g., successfully crossing an intersection).

[0179] The problem can be formally defined as an extended SSP, the details of which are presented here. The competence recognition system (CAS) is a tuple

number

number

number

number

number

number

number

number

number

number

number

number

[0180] CAS status

number

number

number

number

[0181] The solution for a given CAS is state and level

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0182] The policy is Π κ Because it is restricted to being selected from, if the autonomy profile κ changes, the space of permitted policies also changes. This, in turn, is the optimal policy π. * This intuitively means that it is only as good as the function κ. Therefore, there is a trade-off when determining the initial constraint (κ) on the allowed autonomy.

[0183] In implementations, a conservative approach that significantly constrains the system can be chosen, such as setting |κ(s,a)|=1 for all (s,a)∈S×A, thereby reducing the complexity of the problem for solving the underlying domain model, which has a deterministic level. However, this carries the risk of resulting in a policy that is not overall optimal with respect to the autonomy level L, and it may become impossible to reach an overall optimal policy depending on the initial autonomy profile κ.

[0184] In another implementation, a high-risk approach is chosen by leaving the system completely unconstrained beforehand, thereby allowing the system to make all decisions regarding the level of autonomy. This approach inevitably includes optimal policies (following the ACA model), but is naturally slower due to the wider policy space, inherently less safe because the ACA may act at undesirable levels, and in a sense, renders the entire purpose of the model meaningless.

[0185] In yet another implementation, in most domains, the ideal initialization may be closer to the center of the above extrema. The autonomy profile κ may be less constrained in situations where the predicted cost of failure is relatively low, and more constrained in situations where the cost of failure is high. For example, in AV, the autonomy profile κ may initially be more constrained in situations involving cluttered environments such as pedestrians, poor visibility, or large intersections with multiple vehicles, but driving along a highway is generally low-risk, and the benefits from a constrained autonomy profile may be far less.

[0186] A key component of the CAS model is the system's ability to adjust its autonomy profile over time using what it has learned, in order to optimize autonomy by reducing unnecessary reliance on human assistance, regardless of how the autonomy profile is initialized. However, before operating at a new level of autonomy, the system may know nothing about how humans will interact with the system at that level; that is, the feedback profile at that new level may be initialized by default to some baseline distribution. As a result, the system may need to explore levels of autonomy for which there may be reasons to consider them more cost-effective than its current level, and consequently, the system may generate the data necessary to improve the accuracy and reliability of its feedback profile at those levels.

[0187] However, allowing a system to change its own autonomy profile can have serious consequences in the real world if not done carefully. Therefore, embodiments disclosed herein perform gated exploration and are configured to obtain permission from a human before exploring new (i.e., unauthorized) levels of autonomy. Thus, the system must first query a human to update its autonomy profile to enable such exploration and gate the exploration of unauthorized levels by a human authority to prevent the agent from randomly performing dangerous actions.

[0188] Embodiments disclosed herein may use a variation of the ε-greedy search-exploit strategy in which ε is not fixed but instead proportional to the relatively predicted cost of performing a given action at each level of autonomy. The probability of exploring level l' adjacent to the current level l in L is proportional to the softmax of the negative q value operating at level l' across all levels adjacent to l.

number

number

[0189] Various properties of CAS can be used to demonstrate important outcomes of competence recognition systems. In the example provided below, it can be assumed that there is a single human authority with which semi-autonomous systems within CAS interact. The single human authority can be represented as H, where H is a tuple <F H ,λH ,κ H It can be expressed by >, and here, -F H This is a set of features used by H when providing feedback, -

number

number

[0190] Feedback consistency is a characteristic of how consistently an authority figure provides inappropriate feedback when the same query is given to the acting agent. For example, F H Let ⊂F be the set of features H used by human authorities.

number

number

number

number

[0191] λ H be the stationary distribution of the feedback signal that human authorities follow. X S The competence of CAS S shown by X is λ H and each τ i ∈ T given complete knowledge of,

Number

Number

Number

Number

Number

[0192] Basically, the state

Number

Number

[0193] This definition of competence is λ H It depends on and therefore defines the competence of the entire human-agent system, and is clearly not merely a measure of the underlying agent's technical capabilities (i.e., D and T). As a consequence of this fact, CAS may result in a system with lower competence if the human authority figure is only as capable as the human authority figure perceives it and does not fully understand the system's capabilities, compared to a human authority figure who is aware of the system's limitations and capabilities. One reason for modeling competence in this way may be to avoid relying on arbitrary thresholding based on evaluation metrics to determine whether a system is competent or not.

[0194] CAS S is the true distribution λ H Any new feedback derived from any

number

number

[0195] Let S be CAS. All states

number

Mathematics

[0196] In the first proposition,

Mathematics

Mathematics

Mathematics

Mathematics

Mathematics

Mathematics

Mathematics

Mathematics

[0197] In the second proposition,

Mathematics

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0198] Let S be the CAS. S is in the following state

number

number

number

[0199] Under certain conditions, a competence recognition system S can be guaranteed to reach level optimality. Therefore, it can be guaranteed that the system will reach a point where it operates at its true competence in all situations. If the set T of subcompetencies in A is not empty (i.e., there are subcompetencies assumed to be completely unknown beforehand), it can be observed that the existence of a mechanism for learning those subcompetencies online may be necessary. If there are some subcompetencies that are completely unknown beforehand and for which the system has no means of learning, the system's true competence cannot be learned online, and level optimality cannot be reached. Therefore, if sufficient data is available, all subcompetencies τ i It can be assumed that ∈T can be learned online. In other words, each estimator

number

[0200] The concept of gate search can be relied upon to prove that a competence recognition system will reach level optimality. However, the following application is possible, given

number

number

number

[0201] Let S be CAS, κ t Let κ represent the autonomy profile at time t.

number

number

number

number

[0202] In the first theorem, S is defined as a CAS that follows a gate-controlled search strategy and performs utilization under stationarity, and here,

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0203] Many problems in open worlds are too complex to fully specify all the features that will be relevant in the course of system deployment, even with domain expertise. Therefore, CAS models are limited by the fixed features within their model, which may appear insufficient to fully model human feedback, although they allow semi-autonomous systems to optimize their autonomy over time.

[0204] For example, consider a robot deployed on a campus with the task of delivering packages to various offices within different buildings. Initially, the robot knows it may detect doors and must seek approval or supervision before opening them, but its initial model does not use additional information about the doors because their properties differ from building to building and from campus to campus. The supervising human authority figure may use additional features of the doors, such as their size, shape, and whether they can be pushed open. Since the CAS model does not represent these features, human feedback may appear inconsistent in situations involving doors because the feedback is effectively normalized across all these additional features. This, in turn, can lead to decreased competence, insufficient performance, and unnecessary further reliance on humans due to a lack of information. To address this drawback, methods such as the one shown in Figure 6 may be implemented.

[0205] Figure 6 is a flowchart of an example of method 6000 for providing CAS with the ability to improve its competence over time. In 6002, method 6000 includes increasing the granularity of its state representation by online model updates. In 6004, method 6000 includes identifying states that are considered indistinguishable under the system's current feedback profile (i.e., states where human feedback cannot be predicted with high confidence). In 6006, method 6000 includes determining a feature or set of features that is available to the system but not currently used, and that best discriminates human feedback. This can result in a finer depiction of the boundaries between domains of state space having different levels of competence.

[0206] Figure 7 illustrates an exemplary example of the method shown in Figure 6. This example illustrates one embodiment in a navigation task for a robotic system (e.g., a robot) that can leverage two key properties of the CAS model. The first property that can be leveraged may be existing information available in a standard CAS model in the form of human feedback to identify where new features should be added, without requiring any additional work from the human. The second property that can be leveraged may include properties of human-agent interaction that avoid the need to directly modify the transition function or reward function, and instead directly modify only the state space. As a result, the entire process can be performed online and completely autonomously.

[0207] Figure 7 includes a navigation environment 7000 containing several corridors and doors. The doors are shown as red (R) doors, blue (B) doors, and green (G) doors, each color representing a different type of door. The navigation environment 7000 contains several paths corresponding to optimal paths under different granularities of state-space representation. For example, path 7002 (shown as a solid line) may correspond to the optimal path for the robot to cross using door R. Path 7004 (shown as a large dashed line) may correspond to the optimal path for the robot to cross using door B. Path 7006 (shown as a small dashed line) may correspond to the optimal path for the robot to cross using door G. As features are identified and added to the state representation, the system can better learn to leverage human assistance and crossing paths that better fit the system's competence.

[0208] Embodiments described herein can determine indistinguishable states. In one example, let S be a competence recognition system. When a robotic system is deployed in an open world, both the exact environment in which the system will operate and the human authority with which the system will interact may not be known in advance. Simply including all possible features available to the system from perception or external sources in its planning model may be unhelpful and make the plan unwieldy, as many features serve only to increase the number of states without adding useful information. Therefore, we assume that a complete feature space is available, which can be divided into an active feature space used by S and an inactive feature space not yet used by S in its planning model. However, as S receives additional feedback over time, S will learn to leverage some of the inactive features and add them to its state representation to more effectively align its features with those used by the human authority.

[0209] For example, the complete feature space available to S from the sensor or other external source can be divided into an active feature space used by S and an inactive feature space not yet used by S. As S receives additional feedback over time, S will learn to leverage the inactive features to more effectively align with the features used by human authorities.

[0210] S is the complete feature space F={F1,F2,...,F n Given}, the active feature space is

number

number

number

number

[0211] For example, a human authority figure H

number

number

number

number

[0212] Here, δ is called discrimination slack and is used to determine the required predictive confidence for states that are declared undiscriminable. For example, the lower the discrimination slack is set, the higher the required confidence. Discrimination slack helps provide a formal trade-off mechanism between increasing the complexity of the underlying planning model and the completeness of the competence recognition model. The decision of how to set δ can be made through expertise, offline evaluation, and dynamic online adjustments.

[0213] Given the concept of indiscriminate states, the central idea of ​​this method can be defined. A discriminator is any subset of the inactive feature space that can help the agent better discriminate feedback from H for indiscriminate states. As an example, consider an agent that initially considers only the presence of doors in its active feature space, but has additional features representing the size of the doors in its inactive feature space. Without these additional features, the agent might perceive that it has received equal approval and rejection from a human authority figure, leading to a feedback profile with a low probability of any feedback signal for all doors, while H consistently did not allow the robot to open doors of a certain size in order to prevent damage to itself. By including these features representing door sizes in the active feature space, the agent's new feedback profile may be able to predict with a high probability the correct feedback signals for both small and large doors.

[0214] Embodiments disclosed herein can perform iterative state-space refinement. In one example, the discriminator is an arbitrary subset

number

number

[0215] Figure 8 shows an example of the single-step state-space refinement algorithm 8000. The single-step state-space refinement algorithm 8000 represents pseudocode for improving the competence of CAS by iterative partitioning of the state space by adding new features to the state representation over time. As shown in Figure 8, the single-step state-space refinement algorithm first identifies a set of current indistinguishable states 8002. To avoid randomly and indiscriminately labeling sparsely sampled state-action pairs,

number

[0216] Next, the single-step state-space refinement algorithm 8000 samples an indistinguishable state 8006 and identifies the most likely discriminator for that state using any standard feature selection technique such as least redundancy maximum relevance (mRMR) 8008. For each potential discriminator, a new feedback profile is trained using a portion of the entire dataset with the discriminator temporarily added to the active feature set 8010. The discriminator that yields the best-performing feedback profile is selected 8012. In one example, the discriminator that yields the best-performing feedback profile may be the one with the highest Matthews coefficient. If validation is successful, the discriminator is added to the active feature set and the system is updated 8014.

[0217] Two assumptions can be made in the design and use of the single-step state-space refinement algorithm 8000. First, the initial transition function provided in the domain model is such that the agent is κ HIt can be assumed to be sufficiently correct for any scenario under which autonomous action is possible. In some examples, competence can be improved by iteratively refining the state space. It may also be possible to enhance competence by directly updating and redesigning the transition functions as a human authority figure gains a deeper understanding of the agent's capabilities.

[0218] Secondly, it can be assumed that a human authority figure has a good understanding of the agent's capabilities, both to prevent the agent from performing actions it is unable to perform well and to provide consistent feedback. This assumption can be made for two reasons. Firstly, there can be various ways to deepen the authority figure's understanding of the system's capabilities so that the authority figure can have appropriate trust or reliance on the system. These may include pre-deployment training, standardized feedback criteria, and expertise in the system. Secondly, the recognition of potential failures and handling of failure recovery are separate areas of proactive investigation orthogonal to the examples described herein.

[0219] Under these assumptions, directly updating the transition or reward function of the domain model may not be necessary at any point. It is sufficient for the agent to be able to distinguish between actions for which it has competence to perform autonomously and actions that require human intervention.

[0220] Adding discriminators does not make a discriminant state indistinguishable. Any given discriminant state will either be affected by the discriminator or not. If a state is not affected, the feedback profile for that state does not change. If a state is affected, the initial state in the problem no longer exists by definition. More importantly, given a sufficient set of features, it can be guaranteed that all states will ultimately be properly discriminated.

[0221] The following theorem states that if all the features that humans use to determine those feedbacks are available to the robot, then there must be a point in time when the robot can completely discriminate all states, and beyond that point, there are no states that it cannot discriminate. t Let this be the number of indistinguishable states at time t.

number

number

number

number

number

[0222] To prove the above theorem, first, F H As ⊆F,

number

number

number

number

number

number

number

number

number

number

number

[0223] Figure 9 is an example flowchart of Technique 9000 for competence recognition decision-making according to an embodiment of the present disclosure. Technique 9000 may be implemented by a competence recognition system (CAS), such as the competence recognition system (CAS) 5000 in Figure 5. Technique 9000 may be implemented by an autonomous cognitive agent, such as the ACA 5002 in Figure 5. Some or all of the operations of Technique 9000 may be implemented by SSOCEM 4400 or by another component of the autonomous vehicle operation management system 4000 in Figure 4.

[0224] Flowchart 9000 illustrates how CAS, or CAS's ACA, can select actions, receive feedback, and update the model(s) that are the subject of the experience(s). Figure 7 illustrates Flowchart 9000.

[0225] Figure 10 is a diagram of autonomous driving scenario 10000 used to illustrate technology 9000 of Figure 9. Scenario 10000 includes intersection 10002. Vehicle 10004 is approaching intersection 10000. Stop line 10006 (i.e., stop sign) is the line on which vehicle 10004 is supposed to stop to ensure it can proceed safely along track 10008 (i.e., route). Vehicle 10004 can be vehicle 1000 in Figure 1. Vehicle 10004 can be one of vehicles 2100 / 2110 in Figure 2. Vehicle 10004 may include an autonomous vehicle operation management system, such as the autonomous vehicle operation management system 4000 in Figure 4. Vehicle 10004 may include a competence recognition system (CAS), such as the system 5000 in Figure 5. Thus, vehicle 10004 can be an autonomous vehicle or a semi-autonomous vehicle. A human operator (e.g., a remote operator or an in-vehicle operator) can play a role in (remotely) monitoring and supporting vehicle 10004, such as by responding to remote requests issued by vehicle 10004 seeking assistance from a remote operator.

[0226] Flowchart 9000 describes domain models (such as DM5004 in Figure 5) relating to intersection scenarios, such as intersection 10002 in Figure 10.

[0227] The domain model for the intersection scenario in Figure 10 may include abstract (e.g., symbolic) information extracted based on sensor (Lidar, radar, camera, etc.) information. For example, the domain model may include information such as the position of vehicle 10004 (e.g., "approaching the stop line at the intersection"), other relevant world objects (e.g., "vehicle on the left," which is vehicle 10010), road configuration (e.g., "east-west road is through traffic," meaning there are no traffic signals, stop signs, yield signs, etc.), and other relevant information for the scenario (e.g., "object on the left has not moved for a long time," "sensor on the left is obstructed," etc.).

[0228] As can be understood, an identified obstacle or occlusion (such as vehicle 2010) can be identified as such when vehicle 10004 is at a certain distance from the obstacle. However, as vehicle 10004 approaches the occlusion or obstacle, vehicle 10004 may determine that the occlusion or obstacle is not. This may be due to noise in the sensor data. Therefore, the condition associated with the scenario is corrected in 9002 of Figure 9, and a new plan is calculated.

[0229] More generally, a scenario can be described as an abstract state space and / or a combination of objects associated with the detected scenario. The state can include all the objects necessary for the success of autonomy or at least semi-autonomy. For each of these objects, a state space can be maintained for the AV (e.g., vehicle 10004) with respect to that object (e.g., vehicle 10010). The combination of all state spaces together can form a domain space in which competence modeling is performed (e.g., can constitute a domain space, can be a domain space, etc.). Thus, competence is modeled in terms of entity-entity pairings for each pair of AV-another entity associated with the scenario. For example, if scenario 10000 includes an associated pedestrian, then competency modeling (i.e., determining the level of autonomy) is determined separately for the pairing of vehicle 10004 and the pedestrian.

[0230] The states used in a domain model, or the states used to evolve a system, can be specified at different levels of granularity. For illustrative and non-limiting examples, a domain model may be for a specific intersection 10002, a domain model may be for all similarly configured intersections within any geographic area of ​​interest, a domain model may be for crossing one or more intersections at a given time of day, and a domain model may include additional metadata information such as the presence of nearby venues and the fact that a concert has just finished, and so on.

[0231] Flowchart 9000 describes a set of actions A. The set of actions relates to a domain. With respect to the domain of an intersection, the actions may include the actions "go," "stop," and "inching forward." However, the set of actions may include more, fewer, other actions, or combinations thereof. In a different scenario, the set of actions may be a different set of actions. For example, with respect to a passing scenario, the actions may include the actions "follow," "stop," "overtake left," and "overtake right."

[0232] The action "go" can mean moving along track 10008. The action "stop" can mean that the vehicle should remain stationary in the next step / action step, or should stop if it was moving. The action "inching forward" can mean that vehicle 10004 should inching forward slightly from a stopping position such as stop line 10006. The "inching forward" action may be useful, for example, when the sensor of vehicle 10004 is obstructed at vehicle 10004's current position. For example, in scenario 10000, the view of the left sensor of AV10004 is obstructed by vehicle 10010. Therefore, the "inching forward" action causes vehicle 10004 to move slightly forward into the intersection to attempt to see beyond vehicle 10010, just as a human driver might do.

[0233] Flowchart 9000 describes a set of autonomy levels L that includes four autonomy levels L = {l0, l1, l2, l3}.

[0234] Autonomy level 10 can indicate "no autonomy," which means that direct human assistance is required and that the ACA of vehicle 10004 has determined that it is incapable of performing any decided actions, if any.

[0235] Autonomy level l1 can indicate "verified autonomy," which may mean that the vehicle 10004's ACA seeks and receives approval from a human (e.g., a remote operator) before performing a selected action. The vehicle 10004's ACA has determined that it does not have the capacity to perform the decided (e.g., selected) action entirely autonomously and therefore requires assistance.

[0236] Autonomy level 12 can indicate "supervised autonomy," which requires that a human presence (e.g., remote monitoring) be able to intervene (e.g., override the system) if a failure occurs while a selected action is being performed.

[0237] Autonomy level 13 can indicate "unsupervised autonomy," which can mean that selected actions can be performed (i.e., can be performed) without any human intervention, supervision, or monitoring. In other words, autonomy level 13 indicates that the ACA has sufficient capacity to perform actions.

[0238] Flowchart 9000 describes a set of feedback signals Σ containing four feedback signals, namely, the set Σ = {no feedback, approved, denied, override}. For ease of reference, the "no feedback", "approved", "denied", and "override" signals are respectively

number

number

number

number

number

number

number

number

number

[0239] Furthermore, flowchart 9000 assumes that "approval" and "denial" feedback can only be received at autonomy level l1 (i.e., "verified autonomy"), and that "override" feedback can only be received at autonomy level l2 (i.e., "supervised autonomy").

[0240] Given L and Σ, the state transition function T of this CAS can be specified.

number

number

number

number

[0241] In equation (7), [·] represents Iverson brackets. Equation (7) states the following: When the ACA operates at autonomy level l0 (i.e., "no autonomy"), the ACA can follow the transition dynamics of a controlling human (e.g., remote operator); when the ACA operates at autonomy level l1 (i.e., "verified autonomy"), the probability that vehicle 10004 arrives at state s' is the probability that the ACA is authorized to take action, multiplied by the probability that the ACA succeeds in following T, plus the probability that the action is denied and the state remains the same; when the agent operates at autonomy level l2 (i.e., "supervised autonomy"), the probability that vehicle 5004 reaches state s' is the probability that the ACA succeeds in following T without any human intervention, plus the probability that a human overrides the action selected by the ACA and the human puts the vehicle into that state (i.e., τ H This can be summarized as follows: ) is added, and when ACA operates at autonomy level l3 (i.e., "unsupervised autonomy"), ACA follows the transition dynamics of its own model (e.g., domain model).

[0242] situation

number

number

number

number

[0243] Equation (8) shows the state in the extended SSP.

number

number

number

number

number

number

[0244] While we have described a competence recognition system or autonomous cognitive agent model therein, technology 9000 can be summarized as comprising three stages: an action selection stage, an action execution stage, and a model update stage. In the action selection stage, an action and a level of autonomy are selected. In the action execution stage, the action is performed according to the level of autonomy, and feedback is received from a human operator or the like. In the model update stage, technology 9000 updates the parameters of the feedback model (e.g., HM5006 in Figure 5) according to the new experience. More specifically, technology 9000 updates the feedback profile λ and / or the human state transition function τ H Update.

[0245] In 9002, technology 9000 can detect the state s of the world around a vehicle, such as vehicle 10004 in Figure 10. The state s is a state of a domain model (e.g., DM5004 in Figure 5). The state s can be detected using sensors on the vehicle, such as one or more of the sensors 1360 in Figure 1.

[0246] In 9004, the detected state s can be used to select policy π. As described above, policy π can be limited by autonomy profile κ, which may include one or more constraints. For example, if the autonomy level selected by the ACA is not within autonomy profile κ, the ACA can replan to select a new behavior and a new autonomy level permitted by the autonomy profile.

[0247] The corresponding competence probabilities can be associated with each level of autonomy; that is, competence probabilities can be associated with taking action in specific situations.

[0248] For example, a 10% competence probability associated with choosing to perform an action (e.g., "go") autonomously (i.e., with "unsupervised autonomy") in a particular state can mean that there is a 10% chance of autonomously performing the action in that state at the correct level of autonomy. Thus, if the action was performed autonomously but the resulting state was a bad state, there may be a 0.1 probability of a large penalty being imposed. Conversely, there may be a 0.9 probability of correctly choosing the level of competence. Thus, competence is incorporated into the probability of transitioning states, i.e., from s at time T to s' at time T+1, and since we know it is incorporated there, when rewards are evaluated over time, the system will autonomously choose whether to perform all calculations, just as actuarial science calculates similar risks. It essentially uses all these probabilities over time to perform a large-scale risk assessment and decide when to be autonomous, when to ask for help, and when to ask for approval.

[0249] The competence probability can be considered (for example, used) when calculating policy π. The competence probability can be a factor in the state transition from the first state s1 at time T to the second state s2 at time T+1.

[0250] The corresponding cost can be associated with the selected level of autonomy. For example, the cost (e.g., reward) can be measured in units of time. However, other units of measurement can also be used. For instance, a remote operator needs to first gain situational awareness, analyze the situation, control the AV, and then actually control the AV to take action, which can take a long time before they can ask a human for help. Such an interaction could take, for example, 60 seconds and could be considered to incur a significant cost. On the other hand, at autonomy level l3 (i.e., "unsupervised autonomy"), no delay occurs. Therefore, when calculating policy π, the cost in units of time or some other unit can be considered (e.g., can be used) as the choice of different levels of autonomy that involve different amounts and different degrees of human involvement.

[0251] In 9006, technology 9000 can output action a and autonomy level l. Therefore, in 9004, it can output the tuple (a,l). Autonomy level l determines how the vehicle 10004's ACA performs action a. Examples of how to interpret the tuple (a,l) include "ACA chooses the stop action, but ACA will seek help to determine if stopping is the correct action" or "ACA chooses the go action because ACA is fully capable, and will go."

[0252] If the level of autonomy is l0 (i.e., "no autonomy"), technology 9000 proceeds to 9014. For example, a human operator can remotely control vehicle 10004 or send commands to vehicle 10004 to perform an action. After the manual operation in 9014 is completed, technology 9000 proceeds to 9016.

[0253] If the level of autonomy is l1 (i.e., "verified autonomy"), technology 9000 proceeds to 9008. In a simple example, the ACA of vehicle 10004 may decide that the action to take is to "inching forward" toward the center of intersection 10002 because its field of view is obstructed by vehicle 10010. However, before completing the action, vehicle 10004 may transfer the action to a human (e.g., a remote operator) for approval. The remote operator may send a "denial" feedback signal (and the ACA may receive the feedback signal), which will cause the level of autonomy to switch to "no autonomy" at 9014. The ACA may receive an "approval" feedback signal. In the case of an "approval" feedback signal, technology 9000 proceeds to 9010 to perform the chosen action.

[0254] If the level of autonomy is l2 (i.e., "supervised autonomy"), technology 9000 proceeds to 9010. In 9010, before attempting the selected action, the ACA of vehicle 10004 can ensure that a remote operator is monitoring vehicle 10004 while the action is being performed. While the ACA is performing the selected action, in 9012, the ACA can complete the action without receiving feedback from a human. Thus, after the action is completed, technology 9000 proceeds to 9016. Meanwhile, in 9014, while the action is being performed, the ACA can receive an override feedback signal, which causes the level of autonomy to transition to the "no autonomy" level.

[0255] If the level of autonomy is l3 (i.e., "unsupervised autonomy"), the action is performed autonomously by vehicle 5004, and technology 9000 proceeds to 9016.

[0256] In 9016, technique 9000 uses feedback, which may or may not be received, to determine the feedback profile λ and the human state transition function τ. H Update.

[0257] How the feedback profile λ is updated may depend on the classifier being used. However, generally, the feedback profile λ can be updated by augmenting the data set of received human feedback signals and retraining the classifier (decision-maker) on the new data.

[0258] Human state transition function τ H This can be updated by observing what new state the ACA enters when a human takes over control, and generally by taking the maximum aposterior probability estimate. In practice, this is done when the ACA enters some state

number

number

[0259] The model can be updated using model-free reinforcement learning, model-based reinforcement learning, or some other learning technique. In model-free reinforcement learning, probability values ​​can be adjusted up or down. In model-based reinforcement learning, probability values ​​can be calculated based on the number of times the target state in a scenario was successfully reached compared to the total number of times the scenario was encountered.

[0260] In one example of reinforcement learning, an estimate of how long (e.g., cost) it will take for the ACA to reach its target state (e.g., crossing the intersection) is maintained. Such an estimate can be maintained by different levels of autonomy. If the target state is not reached (e.g., not crossing the intersection), if human assistance is requested, or if a human assists or rejects the ACA, the probability can be adjusted downwards. In the case of "unsupervised autonomy" or "no feedback," the probability can be adjusted upwards; that is, for every different combination of states, the probability can always be adjusted up or down.

[0261] In some implementations, from 9016, technique 9000 can proceed to 9018, while in other implementations, technique 9000 returns to 9002. That is, block 9018 can be an optional block.

[0262] In 9018, technology 9000 can engage in gate-controlled search (GE).

[0263] As mentioned above, a fundamental component of CAS is that it can optimize its autonomy by reducing unnecessary reliance on humans, using what the system has learned to adjust its autonomy profile κ over time. However, before operating at a new level of autonomy, ACA may not have knowledge of how humans will interact with ACA at that level of autonomy. Therefore, the feedback profile at the new level of autonomy can be uniformly random because no data has been received. Consequently, CAS needs to "explore" levels of autonomy that it believes may be more cost-effective than its current level. Thus, CAS can generate the data necessary to improve the accuracy and reliability of its feedback profile at those levels.

[0264] However, any kind of random or pseudo-random search (e.g., random search, ε-greedy search) can lead to frequent failures and, in the real world, can have serious consequences (e.g., collisions, traffic jams, etc.). For this purpose, a simple extension of conventional search methods called gate-controlled search can be used herein.

[0265] In gate-controlled search, the ACA can still follow a random or pseudo-random search policy to attempt to act at a level of autonomy not permitted by autonomy profile κ. When this happens, instead of simply performing the action, the ACA can first request permission from the human to change the ACA's autonomy profile κ, thereby granting permission for the level of autonomy the ACA is attempting to act at.

[0266] In reinforcement learning materials with an unknown domain, the agent must make a trade-off between using the information it has to simply take the action that performed best in the past, or to explore new actions and states (or those that were merely suboptimal in the past) that may yield better results.

[0267] This concept is used herein. The model described herein assumes that the optimal level of autonomy is not known in advance; otherwise, the ACA could simply be designed to operate at that level. Therefore, the ACA must learn over time what its optimal level is. However, since the ACA is also operating in the real world while learning, it must make this same trade-off: it must either operate at the current optimal level of autonomy, taking into account the history of human feedback signals it receives, or try a new, possibly higher, level of autonomy where there is no or limited feedback signals. This is the "search" part of gate-controlled search (GE).

[0268] However, allowing ACA to operate at a level of autonomy it is not permitted to operate at, simply to test whether it is better, would defeat the purpose of the model described herein. Therefore, if we want to change the highest level of autonomy at which ACA can operate, we must first query a human to change that permitted level of autonomy. This is the "gate-controlled" part of gate-controlled search (GE).

[0269] Generally, ACAs are expected to explore higher levels of autonomy, but it's worth noting that, assuming a conservative initial model, this isn't a requirement, and the agent can actually explore downwards. That is, an ACA can query a human to actually lower the highest level of autonomy it can act at depending on the situation. By doing so, the ACA is forced to act at a level of autonomy it wouldn't normally, allowing it to acquire more data and improve the model's quality.

[0270] From 9018, technology 9000 returns to 9002 and repeats the above operation, thereby technology 9000 detects the current state of the world, selects the actions and levels of autonomy to be performed, and so on.

[0271] To test the competence recognition system, the CAS model can be implemented in two simulated autonomous vehicle domains at different levels of abstraction. The first domain may be a high-level navigation problem in which the autonomous vehicle must plan and execute to take the optimal route between two locations, given its knowledge of different intersections and roads and its competence in performing different operations that may be performed in the previous domain, namely, passing obstacles blocking its lane.

[0272] In the autonomous vehicle navigation domain, the autonomous vehicle operates within a known map represented by a directed graph G=(V,E), where each vertex v∈V represents an intersection and each edge e∈E represents a road. The autonomous vehicle is simply tasked with navigating the map from a starting node to a target node. The state v∈S of each vertex is represented by a tuple〈ID,p,o,v,θ〉, where ID is the ID of the vertex, p is a Boolean value indicating the presence of a pedestrian, o is a Boolean value indicating the presence of an obstruction, v is the number of other vehicles, if any, and θ is the direction of travel. The state e∈S of each edge is represented by a tuple〈u,v,l,θ,o〉, where u and v are the IDs of the starting and ending states of the edge, respectively, l is the number of lanes along the edge, θ is the direction of travel, and o is a Boolean value indicating the presence of an obstacle blocking the lanes of A and B. Furthermore, each edge is associated with the length and speed of travel. Model parameters, such as the probability of encountering a pedestrian at a vertex or obstacles on an edge, are given as part of the model input.

[0273] In the vertex state, the agent can choose to go straight, turn right, turn left, make a U-turn, or wait. All maneuvers are assumed to be deterministically successful. In the edge state, the agent can choose to continue, overtake an obstacle, or wait. Overtaking is assumed to succeed with a probability of [0.2, 0.5, 0.8] depending on the number of lanes, and continuing fails deterministically in the presence of an obstacle, otherwise the agent transitions to the edge vertex with probability p∝speed / length. Each action has a unit cost in the domain model.

[0274] The autonomy profile κ is initialized to L in an obstacle-free edge state, and in a node state without pedestrians, occluders, or other vehicles, or when in action standby mode. In all other cases, κ is initialized to {l0, l1}. The feedback profile λ is initialized to be uniformly random across possible feedback signals. Because a human is required for manual control of the vehicle, the cost for a human to operate at l0 is 8.0, the cost for operating at l1 is 3.0, the cost at level l2 is 1.0, and there is no additional cost for a human when operating at l3. The system incurs a cost of 1.0 when receiving a negative response at l1 and a cost of 3.0 when receiving an override at l2.

[0275] In the obstacle passage domain for autonomous vehicles, the autonomous vehicle must overtake an obstacle blocking a lane on a single-lane road, and importantly, to do so, it must enter the oncoming lane. A state s∈S is represented by a tuple <p,o,t,d,w>, where p is the vehicle's position (0-4), o is the position of the nearest oncoming vehicle (1-3, 0 if none, -1 if unknown), t indicates the presence of a following vehicle, d indicates whether the obstacle is dynamic (e.g., a slow-moving tractor) or static (i.e., debris or a car with a stalled engine), and w indicates whether the nearest oncoming vehicle has stopped.

[0276] The autonomous vehicle can perform the following actions: wait, incline, and go. Incline moves AV's position by 1 with a probability of 0.5 unless it is at position 0, in which case it deterministically moves to edging position 1. Go deterministically moves AV's position by 1 at all positions except position 0, at position 0, moves AV to position 2 (i.e., skips edging positions). All actions have a unit cost unless AV and the oncoming vehicle share or cross a position where a high-cost crash occurs.

[0277] The autonomy profile κ is initialized to {l2} in all cases, meaning that in such safety-critical domains, it is initially expected that a human will always be aware of the system and ready to override it. Thus, the feedback profile λ is uniformly initialized randomly. If the CAS operates at l0 but is assumed to successfully complete the operation (i.e., the human does not relinquish control midway through passing an obstacle), the human incurs a large cost of 10.0, while supervising at l2 incurs a cost of 1.0, and there is no cost at l3. The system receives a large penalty of 12.0 if it is overridden by a human.

[0278] To evaluate iterative state-space elaboration techniques, a simulated domain can be implemented involving a mobile robot tasked with delivering packages to various rooms in various buildings across a small campus. To achieve this goal, the robot must navigate two main obstacles: doors and crosswalks. Improper handling of these obstacles by the agent can lead to costly failures. In this case, the initial domain model of CAS may lack certain features used by humans in determining their feedback.

[0279] In the delivery robot domain, robots operate within a known map and are tasked with delivering packages from one office to another within a campus environment. The robots must safely navigate an environment that includes closed doors and crosswalks over major roads. A state s∈S is represented by a tuple <x, y, θ, o>, where x, y, and θ are the robot's orientations, and o represents the presence of a door, traffic conditions (if at a crosswalk), or none if there are no obstacles at the robot's current location. Additional information is available from sensor data for each obstacle, but is not used beforehand in the domain model. For doors, this information includes the door's color, height, width, and opening type (i.e., pull or push). For crosswalks, this includes whether there are obstacles blocking the view and whether the road is one-way or two-way. Furthermore, the time is also always known to the robot.

[0280] The robot can perform the following actions: move, open, wait, cross, and hand over. Move moves the agent in the direction they are moving. Opening does not occur if the robot is not at the door, but otherwise, the robot deterministically opens the door if it can, and otherwise imposes a penalty on the robot (Note: the initial model of the robot does not distinguish and assumes that it can open all doors). Waiting does not occur unless the robot is at a crosswalk where traffic conditions may change. Crossing does not occur unless the robot is at a crosswalk, and if it is at a crosswalk, the robot deterministically crosses if there is no traffic (empty), has a 50% chance of successfully crossing if there is little traffic (otherwise stays in the same place), and if there is a heavy traffic, there is a 10% chance of crossing, a 10% chance of crashing and getting stuck, and an 80% chance of staying in the same place.

[0281] A negative reward of a unit is incurred at each time step while the robot is working before handing over the package. If the robot attempts to open certain types of doors that it is not capable of opening, it may be damaged and incur a small penalty (-10). If the robot collides with a vehicle while crossing the road, it incurs a very large penalty (-100).

[0282] The autonomy profile κ is initialized to L when there are no obstacles or when waiting for an action, and to {l0,l1} otherwise. Thus, the feedback profile λ is uniformly initialized randomly. The cost to operate at level l0 is 10.0, the cost to operate at level l1 is 2.0, the cost to operate at level l2 is 1.0, and there is no additional cost at level l3. In this domain, there is no additional cost for the feedback received.

[0283] Figure 11 is a flowchart illustrating an example of technology 11000 for autonomous driving by an autonomous vehicle (AV) according to an embodiment of the present disclosure. Technology 11000 in Figure 11 may be implemented by the competence recognition system or autonomy recognition agent shown in Figure 5. Technology 11000 may be implemented in an AV which may be vehicle 1000 shown in Figure 1, one of the vehicles 2100 / 2110 shown in Figure 2, a semi-autonomous vehicle, or any other vehicle that performs autonomous driving.

[0284] In 11110, technology 11000 detects environmental conditions based on sensor data. For example, the environmental condition may be a contributing state, as described above with respect to the extended SSP problem. For example, the condition may be a state s of a set of states S. The condition may be detected as described with respect to 9002 in Figure 9.

[0285] In 11120, technology 11000 selects an action based on the environmental conditions. The action may be selected as described with respect to 9006 in Figure 9.

[0286] In 11130, technique 11000 identifies a set of current indistinguishable states. To avoid randomly labeling sparsely sampled state-action pairs, and assuming that there exists a true correct feedback signal returned by a human with a probability of at least ε for all state-action pairs, the probability of observing all labeled instances of that element in an existing dataset D is at least one threshold p. ε Only in this case can the process be restricted to considering state-action pairs.

[0287] In 11140, technique 11000 samples an indistinguishable state from a set of indistinguishable states and identifies one or more possible discriminators for that state using a feature selection technique such as mMMR.

[0288] In 11150, for each potential discriminator, a new feedback profile can be trained using a portion of the entire dataset, with the discriminator temporarily added to the active feature set. If validation is successful, the discriminator is added to the active feature set and the system is updated.

[0289] In 11160, technique 11000 determines the level of autonomy associated with the environmental state and the action. The level of autonomy can be selected based on at least an autonomy model and a feedback model, as described with respect to 9004 in Figure 9. The autonomy model can be as described with respect to AM5008 in Figure 5. Thus, in one example, the autonomy model can include a utility model and an autonomy profile. The utility model can describe a utility that performs a first action at the first level of autonomy for a first environmental state, given that the AV has just been operating at the second level of autonomy. The autonomy profile can map each environmental state to each action and can define constraints on the level of autonomy allowed for a particular environmental state. The feedback model can be as described with respect to HM5006 in Figure 5.

[0290] For example, the level of autonomy may be selected from a set of autonomy levels that includes a first level of autonomy indicating "no autonomy," a second level of autonomy indicating "verified autonomy," a third level of autonomy indicating "supervised autonomy," and a fourth level of autonomy indicating "unsupervised autonomy."

[0291] In 11170, technology 11000 performs actions according to its level of autonomy.

[0292] In example 11170, the autonomy level can be a second autonomy level indicating verified autonomy, and performing an action according to the autonomy level may include receiving an approval or denial feedback signal for the action. The feedback signal may be received from a human. In one example, the human may be a person inside the vehicle. In another example, the human may be a remote operator. In another example, performing an action according to the autonomy level may include querying for an approval feedback signal before receiving one.

[0293] In example 11170, the autonomy level could be a third level of autonomy indicating “supervised autonomy,” and performing an action according to the autonomy level could include determining that the AV is being monitored by a human before performing the action. For example, technology 11000 could receive a signal from a human indicating that a human is monitoring the AV.

[0294] In example 11170, as illustrated with respect to Figure 9, an acknowledgment feedback signal may be received, and taking action according to the autonomy level may include determining that the AV is being monitored, taking action in response to determining that the AV is being monitored, receiving an override signal, and stopping the action and switching the AV to manual operation mode in response to receiving the override signal.

[0295] In example 11170, performing an action according to the level of autonomy could include deciding, based on the level of autonomy, whether to seek human approval for the action before performing it.

[0296] For example, as explained with respect to 9014 in Figure 9, the autonomy level can be set to "no autonomy," meaning the AV is not permitted to perform autonomous actions, and performing actions according to the autonomy level may include allowing the AV to be manually controlled by a human.

[0297] For example, technique 11000 may include updating at least one of the following in response to performing an action: an autonomy profile, a feedback profile, or a human transition function. Updating each of the autonomy profile, feedback profile, or human transition function for an AV may be based on feedback signals received by the AV itself. However, the update may be based on feedback signals received by many AVs (such as AVs in a fleet). For example, all signals sent to an AV may be aggregated at a central location, the model(s) may be updated, and the updated model may be redistributed to the AVs. However, other methods are also possible for updating the model based on feedback signals received by multiple AVs (i.e., sent to multiple AVs).

[0298] As described above, one embodiment of the disclosed implementation may include a system comprising a memory such as memory 1340 in Figure 1 and a processor such as processor 1330 in Figure 1. The memory may include instructions that can be executed by the processor to compute a policy for solving a task, as described above with respect to the augmented stochastic shortest path (SSP) problem, by solving the augmented SSP problem. The policy may map environmental states and autonomy levels to actions and autonomy levels. Computing the policy may include generating a plan that operates across multiple autonomy levels.

[0299] For example, generating a plan that operates across multiple levels of autonomy could involve generating a plan constrained by the allowed level of autonomy in each state, as described above with respect to the autonomy profile κ.

[0300] For example, an instruction may include an instruction to update a feedback profile. The feedback profile may represent a first probability that the system will receive a first signal when it performs a first action at a second level of autonomy, given that the system is in a first state and has just operated at a first level of autonomy.

[0301] For example, the instructions could include instructions to update the human state transition function. The human state transition function could represent a second probability that the human transitions to the second state of the environment model, given that the system chooses to perform a second action in a first state and the human has performed manual control.

[0302] For example, an instruction may include an instruction to update the autonomy profile. The autonomy profile can define a set of acceptable levels of autonomy, given the current state and the next action to be performed.

[0303] As used herein, the term “instruction” may include instructions or expressions for performing any method, or any part or more part thereof, disclosed herein, and may be implemented in hardware, software, or any combination thereof. For example, an instruction may be implemented as information such as a computer program stored in memory, which can be executed by a processor to perform any of the methods, algorithms, embodiments, or combinations thereof as described herein. An instruction or a part thereof may be implemented as a dedicated processor or circuit which may include dedicated hardware for performing any of the methods, algorithms, embodiments, or combinations thereof as described herein. In some implementations, parts of an instruction may be distributed across multiple processors on multiple devices, on a single device, or on multiple devices, which may communicate directly or over a network such as a local area network, wide area network, the Internet, or a combination thereof.

[0304] As used herein, the terms “example,” “embodiment,” “implementation,” “aspect,” “feature,” or “element” indicate that they serve as examples, cases, or illustrations. Unless expressly indicated otherwise, any example, embodiment, implementation, aspect, feature, or element is independent of each other and may be used in combination with any other example, embodiment, implementation, aspect, feature, or element.

[0305] As used herein, the terms “determine,” “identify,” or any variation thereof include selecting, verifying, calculating, searching, receiving, determining, establishing, obtaining, or otherwise identifying or determining in any way using one or more of the devices shown and described herein.

[0306] As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or,” unless otherwise specified or as is evident from the context. Furthermore, the articles “a” and “an” as used in this application and the attached claims should generally be interpreted as meaning “one or more,” unless otherwise specified or as is evident from the context that they refer to a singular noun.

[0307] Furthermore, for the sake of simplicity, the figures and descriptions herein may include a series or sequence of steps or stages, but the elements of the methods disclosed herein may be performed in various orders or simultaneously. Furthermore, the elements of the methods disclosed herein may occur together with other elements not expressly presented and described herein. Furthermore, not all elements of the methods described herein are required to carry out the methods disclosed herein. While embodiments, features, and elements are described herein in specific combinations, each embodiment, feature, or element may be used independently or in various combinations, with or without other embodiments, features, and elements.

[0308] The above-described embodiments, examples, and implementations are provided for the purpose of facilitating the understanding of this disclosure and are not limiting. Rather, this disclosure covers various modifications and equivalent structures that fall within the scope of the appended claims, and its scope should be given the broadest possible interpretation to encompass all such modifications and equivalent structures as permitted under the law.

Claims

1. A method of autonomous driving using an AV (autonomous vehicle), Based on sensor data, the system detects environmental conditions, Selecting an action based on the aforementioned environmental conditions, Identifying the set of currently indistinguishable states, Identifying a discriminator from the aforementioned set of indistinguishable states, Training a feedback model for the aforementioned discriminator, Determining the level of autonomy associated with the environmental state and the behavior, wherein the level of autonomy is selected and determined based on at least an autonomy model and a feedback model. To perform the aforementioned actions according to the aforementioned level of autonomy Methods that include...

2. The method according to claim 1, wherein the autonomy level is selected from a set including a first autonomy level indicating "no autonomy," a second autonomy level indicating "verified autonomy," a third autonomy level indicating "supervised autonomy," and a fourth autonomy level indicating "unsupervised autonomy."

3. The aforementioned level of autonomy is the second level of autonomy that indicates "verified autonomy," and performing the aforementioned action in accordance with the second level of autonomy is Receiving an approval feedback signal or a denial feedback signal in response to the aforementioned action. including, The method according to claim 2.

4. Performing the aforementioned actions according to the aforementioned level of autonomy, Inquiring about the approval feedback signal before receiving the approval feedback signal. The method according to claim 3, including the method described in claim 3.

5. The aforementioned level of autonomy is the third level of autonomy that indicates "supervised autonomy," Performing the aforementioned actions according to the third level of autonomy is Before performing the aforementioned action, determine that the AV is being monitored by a human. including, The method according to claim 2.

6. When the aforementioned approval feedback signal is received, Performing the aforementioned actions according to the second level of autonomy is Determining that the aforementioned AV is being monitored, In response to the determination that the AV is being monitored, the aforementioned action is performed, Receiving an override signal, In response to receiving the override signal, the action is stopped and the AV is switched to manual operation mode. including, The method according to claim 3.

7. Performing the aforementioned actions according to the aforementioned level of autonomy, Based on the aforementioned level of autonomy, a decision is made as to whether to seek human approval for the aforementioned action before performing the aforementioned action. The method according to claim 1, including the method described in claim 1.

8. The autonomy level is "no autonomy," such that the AV is not permitted to perform autonomous actions. Performing the aforementioned actions according to the aforementioned level of autonomy, To enable the aforementioned AV to be manually controlled by a human. including, The method according to claim 1.

9. The autonomy model includes a utility model and an autonomy profile, The utility model describes a utility that performs a first action at the first level of autonomy for a first environmental state, when the AV has just started operating at the second level of autonomy. The autonomy profile maps each environmental state to each action and defines constraints on the permitted level of autonomy for a particular environmental state. The method according to claim 1.

10. Updating at least one of the autonomy profile, feedback profile, or human transition function in response to performing the aforementioned action. The method according to claim 1, further comprising:

11. A system for autonomous driving by an AV (autonomous vehicle), Memory and A processor is provided, and the processor executes instructions stored in the memory. By solving the extended stochastic shortest path (SSP) problem, we calculate a policy for solving the task. Identify a discriminator from the current set of indistinguishable states, and Train a feedback model for the aforementioned discriminator. It is configured in such a way, The aforementioned policy maps environmental conditions and autonomy levels to behaviors and autonomy levels, Calculating the aforementioned policy Generating plans that operate across multiple levels of autonomy including, system.

12. To generate a plan that operates across the aforementioned multiple levels of autonomy, To generate plans under constraints on the permitted level of autonomy in each state, including pedestrian scenarios, intersection scenarios, or lane change scenarios. The system according to claim 11, including the following:

13. The system according to claim 12, wherein constraints map each state and action to a subset of autonomy levels.

14. The aforementioned instruction, The feedback model, which represents the first probability of receiving a first signal when the system performs a first action at a second level of autonomy, is updated when the system is in a first state and has just started operating at a first level of autonomy. The system according to claim 11, further comprising instructions.

15. The aforementioned instruction, When the system selects to perform a second action in a first state, and a human performs manual control, the human updates the human state transition function, which represents the second probability of the human transitioning to the second state of the environmental model. The system according to claim 11, further comprising instructions.

16. The aforementioned instruction, The system according to claim 11, further comprising an instruction to update an autonomy profile, wherein the autonomy profile defines a set of acceptable autonomy levels given the current state and the next action to be performed.

17. A method for autonomous driving by an AV (autonomous vehicle), Solving the extended stochastic shortest path (SSP) problem involves calculating a policy for solving the task, Identifying discriminants from a set of currently indistinguishable states, Training a feedback model for the aforementioned discriminator Includes, The aforementioned policy maps environmental conditions and autonomy levels to behaviors and autonomy levels, Calculating the aforementioned policy Generating plans that operate across multiple levels of autonomy including, method.

18. To generate a plan that operates across the aforementioned multiple levels of autonomy, To generate plans under constraints on the permitted level of autonomy in each state, including pedestrian scenarios, intersection scenarios, or lane change scenarios. The method according to claim 17, including the method described in claim 17.

19. The method according to claim 18, wherein constraints map each state and action to a subset of autonomy levels.

20. Updating the feedback model that represents the first probability of receiving a first signal when the agent performs a first action at a second level of autonomy, when the agent is in a first state and has just started operating at a first level of autonomy, When the agent chooses to perform a second action in the first state, and a human performs manual control, the human updates the state transition function, which represents the second probability of the human transitioning to the second state of the environmental model. Further including updating the autonomy profile, wherein the autonomy profile defines a set of acceptable autonomy levels given the current state and the next action to be performed. The method according to claim 17.

Images