Method and system for autonomous driving

Abstract

A method of autonomous driving of an autonomous vehicle (AV) includes: detecting (11110) an environmental condition based on sensor data; selecting (11120) an action based on the environmental state; identifying (11130) a current set of distinct-free states; identifying (11140) a discriminator from the current set of distinct-free states; training (11150) the feedback model for the discriminator; determining (11160) an autonomy level associated with the environmental state and action, where the autonomy level is selected based on at least an autonomy model and a feedback model; and performing (11170) the action according to the autonomy level.

Description

Technical Field

[0001] The present disclosure relates generally to autonomous vehicle operations management and autonomous driving, and more particularly to determining an operational autonomy level of an autonomous vehicle based on a capability model. Background Art

[0002] A vehicle, such as an autonomous vehicle, may traverse a portion of a vehicle transportation network (e.g., a road). Traversing a portion of the vehicle transportation network may include generating or capturing data, such as data representing the operating environment of the vehicle or a portion thereof, such as by sensors of the vehicle. Sometimes, obstructions make some data unavailable. Summary of the invention

[0003] A first aspect of the disclosed embodiments is a method of autonomous driving of an autonomous vehicle (AV). The method includes: detecting an environmental state based on sensor data; selecting an action based on the environmental state; identifying a current set of indistinguishable states; identifying a distinguisher from the current set of indistinguishable states; training a feedback model for the distinguisher; determining an autonomy level associated with the environmental state and the action; and performing the action according to the autonomy level. The autonomy level may be selected based on at least an autonomy model and the feedback model. The feedback model may be trained with or without the distinguisher and an iterative state space refinement method.

[0004] A second aspect of the disclosed embodiment is a system for autonomous driving, comprising a memory and a processor. The processor is configured to execute instructions stored in the memory to: compute a policy for solving a task by solving an extended stochastic shortest path (SSP) problem; identify a distinguisher from a current set of indistinguishable states; and train a feedback model for the distinguisher. The policy may map environmental states and autonomy levels to actions and autonomy levels. Calculating the policy may include generating a plan for operating across multiple autonomy levels. The feedback model may be trained with or without a distinguisher and an iterative state space refinement method.

[0005] A third aspect of the disclosed embodiment is a method for autonomous driving. The method includes: computing a policy for solving a task by solving an extended stochastic shortest path (SSP) problem; identifying a distinguisher from a current set of indistinguishable states; and training a feedback model for the distinguisher. The policy may map environmental states and autonomy levels to actions and autonomy levels. Calculating the policy may include generating a plan for operating across multiple autonomy levels. The feedback model may be trained with or without a distinguisher and an iterative state space refinement method.

[0006] Variations of these and other aspects, features, elements, implementations, and embodiments of the methods, devices, processes, and algorithms disclosed herein are described in further detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Various aspects of the methods and apparatus disclosed herein will become more apparent by reference to the examples provided in the following description and accompanying drawings, in which like reference numerals refer to like elements.

[0008] Figure 1 is a diagram of an example of a vehicle in which the aspects, features, and elements disclosed herein may be implemented.

[0009] Figure 2 is a diagram of an example of a portion of a vehicle transportation and communication system in which aspects, features, and elements disclosed herein may be implemented.

[0010] Figure 3 is a diagram of a portion of a vehicle transportation network according to the present disclosure.

[0011] Figure 4 is a diagram of an example of an autonomous vehicle operations management system according to an embodiment of the present disclosure.

[0012] Figure 5 is a diagram of an example of a capability awareness system (CAS) according to an embodiment of the present disclosure.

[0013] Figure 6 is a flow chart of an example of a method for providing a CAS with skills that improve its capabilities over time.

[0014] Figure 7 yes Figure 6 FIG. 1 is a diagram of an illustrative example of the method.

[0015] Figure 8 is a diagram of an example of a single-step state-space refinement algorithm.

[0016] Fig. 9 is an example of a flow diagram of a technique for capability-aware decision making according to an embodiment of the present disclosure.

[0017] Fig.10 It is used to illustrate Figure 7 Figure 1. Illustration of an autonomous driving scenario using the technology.

[0018] Fig.11 is a flow chart of an example of a technique for autonomous driving of an autonomous vehicle (AV) according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] A vehicle, such as an AV or semi-autonomous vehicle, may traverse a portion of a vehicle transportation network. The vehicle may include one or more sensors, and traversing the vehicle transportation network may include the sensor generating or capturing sensor data, such as sensor data corresponding to the vehicle's operating environment or a portion thereof. For example, the sensor data may include information corresponding to one or more external objects, such as pedestrians, remote vehicles, other objects within the vehicle's operating environment, vehicle transportation network geometry, or a combination thereof.

[0020] The AV may include an autonomous vehicle operations management system that may include one or more operating environment monitors that may process operating environment information (such as sensor data, etc.) of the autonomous vehicle.

[0021] The AV operation management system may include an autonomous vehicle operation management controller that can detect one or more operation scenarios corresponding to external objects, such as pedestrian scenarios, intersection scenarios, lane change scenarios, or any other vehicle operation scenario or combination of vehicle operation scenarios, etc. An operation scenario or a collection of operation scenarios may be referred to herein as an environmental state.

[0022] The AV operation management system may include one or more scenario-specific operation control evaluation modules (SSOCEM). Each scenario-specific operation control evaluation module may be a model of a corresponding operation scenario, such as a partially observable Markov decision process (POMDP) ​​model, etc. That is, each model is configured to cope with a specific scenario. The AV operation management controller may instantiate a corresponding instance of the scenario-specific operation control evaluation module in response to detecting a corresponding operation scenario.

[0023] The AV operation management controller can receive candidate vehicle control actions from a corresponding instantiated scenario-specific operation control evaluation module (SSOCEM) instance, can identify a vehicle control action from the candidate vehicle control actions, and can control the AV to traverse a portion of the vehicle transportation network based on the identified vehicle control action.

[0024] In some implementations, the SSOCEM can be configured to complete some tasks autonomously while requiring human intervention to complete other tasks. That is, the SSOCEM can operate autonomously under certain conditions, but may require human intervention or assistance to achieve its goals (e.g., crossing an intersection). Thus, the SSOCEM can be in one of two binary autonomous states or levels.

[0025] For example, in response to detecting an obstacle on a road (i.e., on a portion of a vehicle transportation network), the SSOCEM may issue an assistance request to a teleoperator. A teleoperator may be a human operator responsible for remotely monitoring and assisting one or more autonomous vehicles. The teleoperator may cause sensor data (e.g., camera images and / or video) to be streamed to the teleoperator so that the teleoperator may gain situational awareness and plot a navigation path for the AV around the obstacle and / or remotely control the actions of the AV.

[0026] In another example, the lane crossing SSOCEM may use lane markings to maintain the AV within the lane. In some scenarios, the lane crossing SSOCEM may no longer be able to define lane boundaries, such as when sensors may be dirty or malfunctioning or lane markings are covered with snow or mud, etc. In such scenarios, the lane crossing SSOCEM may request a human (e.g., a driver occupant of the AV or a teleoperator) to assume control of the AV.

[0027] The reliance on human assistance (i.e., intervention) may indicate the limited capabilities of the SSOCEM in its autonomy model. Human intervention may be expensive. For example, a teleoperator may take a relatively long period of time to respond to a request for assistance from an AV. On the other hand, the AV may be holding up traffic. For example, as the number of teleoperator requests for assistance from an AV increases, the number of available teleoperators needs to necessarily increase.

[0028] In an implementation according to the present disclosure, the SSOCEM may be or may include an autonomous cognitive agent (ACA) that selects the next action to be performed and the level of autonomy at which to perform the action based on an autonomy model maintained and evolved by the ACA. For example, initially, the autonomy model may indicate that the ACA is to seek human assistance (e.g., feedback) for the action given a detected environmental state. As the ACA receives more and more feedback (such as from humans), the ACA may become less and less dependent on human assistance because the ACA learns when it is appropriate to perform an action with a lower level of assistance, which means higher autonomy (i.e., capability). Therefore, the ACA may be said or considered to be aware of the level of its capabilities.

[0029] In implementations according to the present disclosure, the ACA may consider all levels of autonomy available to it during plan generation (as opposed to, for example, adjusting the level of autonomy during plan execution). Thus, the ACA may create plans that more efficiently exploit the ACA's knowledge of its own level of autonomy.

[0030] In addition, in implementations according to the present disclosure, ACA can model multiple forms of human feedback, thereby enabling ACA to proactively plan in a manner that also considers the likelihood of various forms of human feedback. Thus, ACA can preemptively avoid scenarios where negative feedback may occur.

[0031] Furthermore, the ACA can maintain a predictive model of human feedback and intervention through experience, thereby enabling the ACA to reduce its reliance on humans over time and adjust the ACA's autonomy level over time by avoiding scenarios that are more likely to require human intervention. Thus, the ACA can operate at a minimum cost autonomy level for any scenario (i.e., environmental state) that the ACA encounters (i.e., detects based on sensor data).

[0032] The ACA can then use the autonomy model to traverse the delivery vehicle transportation network.

[0033] Autonomous systems are increasingly being deployed in open worlds, involving highly complex and unstructured domains. Examples of these systems include space exploration rovers, autonomous underwater vehicles, service robots, and AVs. Since it is challenging to fully model an open world, these systems must rely on approximate models of their domain that may not be adequate for every scenario, introducing the potential for risky behavior. However, these systems are expected to maintain safe and reliable operation over potentially long-term deployments. To achieve this, these systems rely on various forms of human supervision, assistance, and intervention. In this sense, the most complex AI systems being developed today are semi-autonomous at best, in that they operate autonomously only under certain conditions and require human intervention to complete their assigned tasks under other conditions.

[0034] For example, a space exploration rover may suspend operation and await new countermeasure planning from ground control when encountering an unexpected obstacle or when system parameters such as wheel drag torque fall outside of an allowable range. Similarly, an AV may request to transfer control back to a human driver in difficult weather conditions or when there is high density cross traffic and no safe gap exists for a long time when attempting to merge. Human assistance and feedback may be available in different forms or modalities corresponding to the behavior. For example, allowing a system to operate autonomously with human supervision indicates a higher level of capability relative to requiring explicit approval from a human before performing individual actions, while indicating a lower level of capability compared to autonomous operation without supervision.

[0035] Embodiments disclosed herein are described in the context of semi-autonomous systems, where capability can be a measure of the optimal degree of autonomous operation of the system in any given scenario, given an understanding of available human involvement (i.e., the various forms of assistance that humans can provide and the ability of humans to communicate with the semi-autonomous system). Capability can be a measure of the performance of a system or algorithm. The capability for automatic decision making in a semi-autonomous system can be based on the level of autonomy. The level of autonomy can be a paradigm for modeling a hierarchy of autonomous behavior in a safety-critical domain, where each level of autonomy corresponds to a set of some constraints, restrictions, or requirements for autonomous operation. Thus, capability awareness can be defined as the skill of an agent in predicting the correct level of autonomy to operate in any given scenario.

[0036] Embodiments disclosed herein include a planning model referred to as a Capability Awareness System (CAS) that operates at multiple levels of autonomy, where each level is associated with a different form of human assistance that compensates for the system's constrained skills. The system can associate each type of human assistance with a unique set of feedback signals that the system can receive from humans, where the likelihood of the set of feedback signals can be learned over time. The model enables the system to operate more reliably in an open world, reduce inappropriate reliance on humans, and ultimately optimize the system's autonomous behavior. In order to cope with scenarios where the representation of the initial domain model is insufficient to correctly model human feedback, an iterative method is introduced to refine the system's state space to better distinguish human feedback, thereby producing a more subtle partitioning of the state-action space with different levels of capability, allowing the system to better learn and act at its true capabilities.

[0037] Embodiments disclosed herein may include methods for iterative state space refinement. Iterative state space refinement may enable a CAS to refine the granularity of its state representation online. The CAS models multiple forms of human feedback and uses the feedback to enable a semi-autonomous system to learn its capabilities over time. In addition, the CAS enables a semi-autonomous system to learn a predictive model of human feedback, thereby allowing the semi-autonomous system to converge to an optimal autonomy level over time. The CAS may be configured to learn from human interactions, and in particular, to learn from human input and to learn from demonstrations.

[0038] The decision process associated with CAS can be based on an uncertain Markov decision process (UMDP). Embodiments disclosed herein can use two sources: under-specified model priors or incompletely specified model priors, the first for human feedback and transition dynamics in the form of sub-capabilities, and the second for feature space in the form of missing features.

[0039] Figure 11000 is a diagram of an example of a vehicle that may implement aspects, features, and elements disclosed herein. In the illustrated embodiment, vehicle 1000 includes various vehicle systems. The vehicle systems include chassis 1100, powertrain 1200, controller 1300, and wheels 1400. Additional or different combinations of vehicle systems may be used. Although vehicle 1000 is shown as including four wheels 1400 for simplicity, one or more of any other propulsion devices (such as pushers or running boards, etc.) may be used. Figure 1 1000, which may include accelerating, decelerating, steering, or otherwise controlling the vehicle 1000.

[0040] Figure 1 The powertrain 1200 shown by way of example in the figure includes a power source 1210, a transmission 1220, a steering unit 1230, and an actuator 1240. Any other element of the powertrain (such as a suspension, a drive shaft, an axle, or an exhaust system, etc.) or a combination of elements may also be included. Although shown separately, the wheel 1400 may be included in the powertrain 1200.

[0041] The power source 1210 includes an engine, a battery, or a combination thereof. The power source 1210 may be any device or combination of devices operable to provide energy (such as electrical energy, thermal energy, or kinetic energy, etc.). In an example, the power source 1210 includes an engine (such as an internal combustion engine, an electric motor, or a combination of an internal combustion engine and an electric motor, etc.), and is operable to provide kinetic energy as a motive force to one or more wheels in the wheels 1400. Alternatively or additionally, the power source 1210 includes a potential energy unit, such as one or more dry cells (such as nickel-cadmium (NiCd) batteries, nickel-zinc (NiZn) batteries, nickel-metal hydride (NiMH) batteries, lithium-ion (Li-ion) batteries, etc.), solar cells, fuel cells, or any other device capable of providing energy, etc.

[0042] The transmission 1220 receives energy (such as kinetic energy, etc.) from the power source 1210 and transmits the energy to the wheel 1400 to provide motive force. The transmission 1220 may be controlled by the controller 1300, the actuator 1240, or both. The steering unit 1230 may be controlled by the controller 1300, the actuator 1240, or both, and the steering unit 1230 controls the wheel 1400 to steer the vehicle. The actuator 1240 may receive a signal from the controller 1300 and actuate or control the power source 1210, the transmission 1220, the steering unit 1230, or any combination thereof to operate the vehicle 1000.

[0043] In the illustrated embodiment, the controller 1300 includes a positioning unit 1310, an electronic communication unit 1320, a processor 1330, a memory 1340, a user interface 1350, a sensor 1360, and an electronic communication interface 1370. A smaller portion of these components may exist as part of the controller 1300. Although shown as a single unit, any one or more of the components of the controller 1300 may be integrated into any number of separate physical units. For example, the user interface 1350 and the processor 1330 may be integrated into a first physical unit, and the memory 1340 may be integrated into a second physical unit. Although in Figure 1 Although not shown, the controller 1300 may include a power source such as a battery, etc. Although shown as separate elements, the positioning unit 1310, the electronic communication unit 1320, the processor 1330, the memory 1340, the user interface 1350, the sensor 1360, the electronic communication interface 1370, or any combination thereof may be integrated into one or more electronic units, circuits, or chips.

[0044] The processor 1330 may include any device or combination of devices that are currently available or subsequently developed and that can manipulate or process signals or other information, including an optical processor, a quantum processor, a molecular processor, or a combination thereof. For example, the processor 1330 may include one or more dedicated processors, one or more digital signal processors, one or more microprocessors, one or more controllers, one or more microcontrollers, one or more integrated circuits, one or more dedicated integrated circuits, one or more field programmable gate arrays, one or more programmable logic arrays, one or more programmable logic controllers, one or more state machines, or any combination thereof. The processor 1330 is operably coupled to one or more of the positioning unit 1310, the memory 1340, the electronic communication interface 1370, the electronic communication unit 1320, the user interface 1350, the sensor 1360, and the powertrain 1200. For example, the processor may be operably coupled to the memory 1340 via a communication bus 1380.

[0045] The memory 1340 includes any tangible, non-transitory computer-usable or computer-readable medium capable of, for example, containing, storing, communicating, or transporting machine-readable instructions or any information associated therewith, for use by or in connection with any processor, such as the processor 1330. The memory 1340 may be, for example, one or more solid-state drives, one or more memory cards, one or more removable media, one or more read-only memories, one or more random access memories, one or more disks (including hard disks, floppy disks, optical disks), magnetic or optical cards, or any type of non-transitory media suitable for storing electronic information, or any combination thereof. For example, the memory may be one or more read-only memories (ROMs), one or more random access memories (RAMs), one or more registers, a low-power double data rate (LPDDR) memory, one or more cache memories, one or more semiconductor memory devices, one or more magnetic media, one or more optical media, one or more magneto-optical media, or any combination thereof.

[0046] The communication interface 1370 may be a wireless antenna as shown, a wired communication port, an optical communication port, or any other wired or wireless means capable of interacting with the wired or wireless electronic communication medium 1500. Figure 1 The communication interface 1370 is shown communicating via a single communication link, but the communication interface may be configured to communicate via multiple communication links. Figure 1 A single communication interface 1370 is shown, but the vehicle may include any number of communication interfaces.

[0047] The communication unit 1320 is configured to send or receive signals via a wired or wireless electronic communication medium 1500 (such as via a communication interface 1370). Figure 1 Although not explicitly shown, the communication unit 1320 may be configured to transmit, receive, or both via any wired or wireless communication medium, such as radio frequency (RF), ultraviolet (UV), visible light, optical fiber, wireline, or a combination thereof. Figure 1 A single communication unit 1320 and a single communication interface 1370 are shown, but any number of communication units and any number of communication interfaces may be used. In some embodiments, the communication unit 1320 includes a dedicated short range communication (DSRC) unit, an on-board unit (OBU), or a combination thereof.

[0048] The positioning unit 1310 can determine geographic location information, such as longitude, latitude, altitude, direction of travel, or speed of the vehicle 1000. In an example, the positioning unit 1310 includes a GPS unit, such as a National Marine Electronics Association (NMEA) unit enabled with a wide area augmentation system (WAAS), a radio triangulation unit, or a combination thereof. The positioning unit 1310 can be used to obtain information indicating, for example, the current heading of the vehicle 1000, the current position of the vehicle 1000 in two or three dimensions, the current angular heading of the vehicle 1000, or a combination thereof.

[0049] The user interface 1350 includes any unit that can interact with a person, such as a virtual or physical keyboard, a touch pad, a display, a touch display, a head-up display (heads-up display), a virtual display, an augmented reality display, a tactile display, a feature tracking device (such as an eye tracking device, etc.), a speaker, a microphone, a camera, a sensor, a printer, or any combination thereof. As shown, the user interface 1350 can be operably coupled to the processor 1330, or operably coupled to any other element of the controller 1300. Although shown as a single unit, the user interface 1350 can include one or more than one physical unit. For example, the user interface 1350 can include an audio interface for audio communication with a person and a touch display for visual and touch-based communication with a person. The user interface 1350 can include multiple displays, such as physically separated multiple units, multiple definition parts in a single physical unit, or a combination thereof.

[0050] Sensor 1360 is operable to provide information that can be used to control the vehicle. Sensor 1360 can be a sensor array. Sensor 1360 can provide information related to the current operating characteristics of vehicle 1000 (including vehicle operation information). Sensor 1360 can include, for example, a speed sensor, an acceleration sensor, a steering angle sensor, a traction-related sensor, a brake-related sensor, a steering wheel position sensor, an eye tracking sensor, a seat position sensor, or any sensor or combination of sensors that are operable to report information related to certain aspects of the current dynamic scenario of vehicle 1000.

[0051] Sensor 1360 includes one or more sensors 1360 operable to obtain information related to the physical environment around vehicle 1000 (such as operating environment information, etc.). For example, one or more sensors can detect road geometry (such as lane lines, etc.) and obstacles (such as fixed obstacles, vehicles and pedestrians, etc.). Sensor 1360 can be or can include one or more cameras, laser sensing systems, infrared sensing systems, acoustic sensing systems, or any other suitable type of vehicle-mounted environment sensing device, or combination of devices, now known or subsequently developed. In some embodiments, sensor 1360 and positioning unit 1310 are combined.

[0052] Although not shown separately, the vehicle 1000 may include a trajectory controller. For example, the controller 1300 may include a trajectory controller. The trajectory controller is operable to obtain information describing the current state of the vehicle 1000 and the route planned for the vehicle 1000, and to determine and optimize the trajectory of the vehicle 1000 based on the information. In some embodiments, the trajectory controller may output a signal operable to control the vehicle 1000 so that the vehicle 1000 follows the trajectory determined by the trajectory controller. For example, the output of the trajectory controller may be an optimized trajectory, which may be supplied to the powertrain 1200, the wheel 1400, or both. In some embodiments, the optimized trajectory may be a control input such as a set of steering angles, wherein each steering angle corresponds to a point in time or a position. In some embodiments, the optimized trajectory may be one or more than one path, route, curve, or a combination thereof.

[0053] One or more of the wheels 1400 may be: a steering wheel that pivots to a steering angle under the control of the steering unit 1230; a propulsion wheel that twists under the control of the transmission 1220 to propel the vehicle 1000; or a steering propulsion wheel that can steer and propel the vehicle 1000.

[0054] although Figure 1 Not shown, but the vehicle may include Figure 1 Additional units or elements shown in (Bluetooth) module, a frequency modulation (FM) radio unit, a near field communication (NFC) module, a liquid crystal display (LCD) display unit, an organic light emitting diode (OLED) display unit, a speaker, or any combination thereof.

[0055] Vehicle 1000 may be an autonomous vehicle that is autonomously controlled to traverse a portion of a vehicle transportation network without direct human intervention. Figure 11300, but the autonomous vehicle may include an autonomous vehicle control unit that performs autonomous vehicle routing, navigation, and control. The autonomous vehicle control unit may be integrated with other units of the vehicle. For example, controller 1300 may include an autonomous vehicle control unit.

[0056] The autonomous vehicle control unit (when present) can control or operate the vehicle 1000 to traverse a portion of the vehicle transportation network based on the current vehicle operating parameters. The autonomous vehicle control unit can control or operate the vehicle 1000 to perform defined operations or maneuvers, such as parking the vehicle, etc. The autonomous vehicle control unit can generate a driving route from a starting point, such as the current location of the vehicle 1000, to a destination based on vehicle information, environmental information, vehicle transportation network information representing the vehicle transportation network, or a combination thereof, and can control or operate the vehicle 1000 to traverse the vehicle transportation network based on the route. For example, the autonomous vehicle control unit can output the driving route to a trajectory controller to operate the vehicle 1000 to travel from the starting point to the destination using the generated route.

[0057] Figure 2 is a diagram of an example of a portion of a vehicle transportation and communication system that may implement aspects, features, and elements disclosed herein. The vehicle transportation and communication system 2000 may include, for example, Figure 1 One or more vehicles 2100 / 2110, such as the vehicle 1000 shown, travel via one or more portions of the vehicle transportation network 2200 and communicate via one or more electronic communication networks 2300. Figure 2 It is not explicitly shown, but the vehicle can traverse off-road areas.

[0058] The electronic communication network 2300 may be, for example, a multiple access system for providing communication between the vehicle 2100 / 2110 and one or more communication devices 2400, such as voice communication, data communication, video communication, messaging communication, or a combination thereof. For example, the vehicle 2100 / 2110 may receive information such as information representing the vehicle transportation network 2200 from the communication device 2400 via the network 2300.

[0059] In some embodiments, the vehicle 2100 / 2110 can communicate via a wired communication link (not shown), a wireless communication link 2310 / 2320 / 2370, or a combination of any number of wired or wireless communication links. As shown, the vehicle 2100 / 2110 communicates via a ground wireless communication link 2310, via a non-ground wireless communication link 2320, or via a combination thereof. The ground wireless communication link 2310 may include an Ethernet link, a serial link, a Bluetooth link, an infrared (IR) link, an ultraviolet (UV) link, or any link capable of providing electronic communication.

[0060] The vehicle 2100 / 2110 can communicate with another vehicle 2100 / 2110. For example, the host vehicle or main vehicle (HV) 2100 can receive one or more automated inter-vehicle messages (such as basic safety messages (BSM)) from the remote or target vehicle (RV) 2110 via a direct communication link 2370 or via the network 2300. The remote vehicle 2110 can broadcast the message to the host vehicle within a defined broadcast range (such as 300 meters, etc.). In some embodiments, the host vehicle 210 can receive the message via a third party such as a signal repeater (not shown) or other remote vehicles (not shown). The vehicle 2100 / 2110 can periodically send one or more automated inter-vehicle messages based on, for example, a defined interval (such as 100 milliseconds, etc.).

[0061] The automated inter-vehicle message may include vehicle identification information, geospatial status information (such as longitude, latitude, or altitude information, etc.), geospatial position accuracy information, kinematic status information (such as vehicle acceleration information, yaw rate information, speed information, vehicle heading information, brake system status information, throttle information, steering wheel angle information, etc.), or vehicle routing information, or vehicle operating status information (such as vehicle size information, headlight status information, turn signal information, wiper status information, transmission information, or any other information or combination of information related to shifting vehicle status, etc.). For example, the transmission status information may indicate whether the transmission that shifts the vehicle is in a neutral state, a parking state, a forward state, or a reverse state.

[0062] The vehicle 2100 can communicate with the communication network 2300 via the access point 2330. The access point 2330, which may include a computing device, is configured to communicate with the vehicle 2100, the communication network 2300, one or more than one communication device 2400, or a combination thereof via a wired or wireless communication link 2310 / 2340. For example, the access point 2330 can be a base station, a base transceiver station (BTS), a node B (Node-B), an enhanced node B (eNode-B), a home node B (Home Node-B, HNode-B), a wireless router, a wired router, a hub, a repeater, a switch, or any similar wired or wireless device. Although shown here as a single unit, the access point can include any number of interconnected elements.

[0063] Vehicle 2100 may communicate with communication network 2300 via satellite 2350 or other non-terrestrial communication means. Satellite 2350, which may include a computing device, is configured to communicate with vehicle 2100, with communication network 2300, with one or more communication devices 2400, or a combination thereof via one or more communication links 2320 / 2360. Although shown here as a single unit, the satellite may include any number of interconnected elements.

[0064] The electronic communication network 2300 is any type of network configured to provide voice communication, data communication, or any other type of electronic communication. For example, the electronic communication network 2300 may include a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a mobile or cellular telephone network, the Internet, or any other electronic communication system. The electronic communication network 2300 uses a communication protocol such as the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), the Internet Protocol (IP), the Real-time Transport Protocol (RTP), the Hypertext Transfer Protocol (HTTP), or a combination thereof. Although shown here as a single unit, the electronic communication network may include any number of interconnected elements.

[0065] The vehicle 2100 can identify a portion or condition of the vehicle transportation network 2200. For example, the vehicle includes at least one onboard sensor 2105 (e.g., Figure 1 ), the at least one vehicle-mounted sensor 2105 may be or may include a speed sensor, a wheel speed sensor, a camera, a gyroscope, an optical sensor, a laser sensor, a radar sensor, an acoustic wave sensor, or any other sensor or device or combination thereof capable of determining or identifying a portion or condition of the vehicle transportation network 2200.

[0066] Vehicle 2100 may traverse one or more portions of vehicle transportation network 2200 using information communicated via network 2300, such as information representative of vehicle transportation network 2200, information identified by one or more onboard sensors 2105, or a combination thereof.

[0067] Although for simplicity, Figure 2 A vehicle transport network 2200, an electronic communications network 2300, and a communications device 2400 are shown, but any number of networks or communications devices may be used. The vehicle transport and communications system 2000 may include Figure 2 Although vehicle 2100 is shown as a single unit, the vehicle may include any number of interconnected elements.

[0068] Although vehicle 2100 is shown communicating with communication device 2400 via network 2300, vehicle 2100 may communicate with communication device 2400 via any number of direct or indirect communication links. For example, vehicle 2100 may communicate with communication device 2400 via a direct communication link such as a Bluetooth communication link.

[0069] Figure 3 3000 is a diagram of a portion of a vehicle transportation network according to the present disclosure. Vehicle transportation network 3000 may include one or more non-navigable areas 3100 (such as buildings, etc.), one or more partially navigable areas (such as parking areas 3200, etc.), one or more navigable areas (such as roads 3300 / 3400, etc.), or a combination thereof. In some embodiments, autonomous vehicles (such as Figure 1 The vehicle 1000 shown, Figure 2 One of the vehicles 2100 / 2110 shown, a semi-autonomous vehicle, or any other vehicle that achieves autonomous driving, etc.) can traverse one or more portions of the vehicle transportation network 3000.

[0070] The vehicle transportation network 3000 may include one or more junctions 3210 between one or more navigable or partially navigable areas 3200 / 3300 / 3400. For example, Figure 3 The portion of the vehicle transportation network 3000 shown includes an intersection 3210 between a parking area 3200 and a road 3400.

[0071] A portion of the vehicle transportation network 3000, such as a road 3300 / 3400, etc., may include one or more lanes 3320 / 3340 / 3360 / 3420 / 3440 and may be connected to ( Figure 3is associated with one or more directions of travel (indicated by the arrows in ).

[0072] A vehicle transport network or a portion thereof (such as Figure 3 The vehicle transportation network information may be represented as a portion of the vehicle transportation network 3000 shown, etc.) can be represented as vehicle transportation network information. For example, the vehicle transportation network information can be represented as a hierarchy of elements (such as markup language elements, etc.) that can be stored in a database or file. For simplicity, the figures herein depict vehicle transportation network information representing portions of the vehicle transportation network as graphs or maps; however, the vehicle transportation network information can be represented in any computer-usable form capable of representing the vehicle transportation network or a portion thereof. In some implementations, the vehicle transportation network information may include vehicle transportation network control information, such as driving direction information, speed limit information, toll information, grade information (such as inclination or angle information, etc.), surface material information, aesthetic information, number of lanes, known hazards, or a combination thereof, etc.

[0073] The vehicle transportation network may be associated with or may include the pedestrian transportation network. Figure 3 A portion 3600 of a pedestrian transport network may be a pedestrian walkway. Figure 3 Not separately shown in FIG. 1 , a pedestrian navigable area, such as a sidewalk or crosswalk, may correspond to a navigable area or portion of a navigable area of ​​a vehicle transportation network.

[0074] In some embodiments, a portion or combination of portions of the vehicle transportation network may be identified as a point of interest or a destination. For example, the vehicle transportation network information may identify a building (such as the non-navigable area 3100) and an adjacent portion of the navigable parking area 3200 as a point of interest, the vehicle may identify the point of interest as a destination, and the vehicle may travel from the origin to the destination by traversing the vehicle transportation network. Although the parking area 3200 associated with the non-navigable area 3100 is not located in the vehicle transportation network, the vehicle may be located in the vehicle transportation network. Figure 3 3100, but the destination may include, for example, a building and a parking area that is not physically or geographically adjacent to the building.

[0075] A portion of a vehicle transportation network may be estimated from the topological location of the vehicle to travel to the destination. The destination may be a discrete uniquely identifiable geographic location. For example, the vehicle transportation network may include a defined location of the destination, such as a street address, a postal address, a vehicle transportation network address, a GPS address, or a combination thereof. The destination may be associated with one or more portals (such as Figure 3A destination may be associated with one or more docking locations (such as the entry 3500 shown in FIG. 1 ). Figure 3 The docking location 3700 shown in FIG. 3700 may be a designated or undesignated location or area near a destination where a vehicle may stop, dock, or park so that docking operations, such as picking up or dropping off passengers, may be performed.

[0076] Figure 4 FIG. 4 is a diagram of an example of an autonomous vehicle operation management system 4000 according to an embodiment of the present disclosure. The autonomous vehicle operation management system 4000 may be configured to manage an autonomous vehicle (such as a Figure 1 The vehicle 1000 shown, Figure 2 The invention may be implemented in one of the vehicles 2100 / 2110 shown, a semi-autonomous vehicle, or any other vehicle that implements autonomous driving, etc.

[0077] The autonomous vehicle can traverse a vehicle transportation network or a portion thereof, which can include traversing different vehicle operation scenarios. Different vehicle operation scenarios (also referred to herein as scenarios) can include any differently identifiable set of operating conditions that may affect the operation of the autonomous vehicle in the defined spatiotemporal region or operating environment of the autonomous vehicle. Different vehicle operation scenarios can be based on the number or cardinality of roads, road segments, or lanes that the autonomous vehicle can traverse within a defined spatiotemporal distance. Different vehicle operation scenarios can be based on one or more traffic control devices that may affect the operation of the autonomous vehicle in the defined spatiotemporal region or operating environment of the autonomous vehicle. Different vehicle operation scenarios can be based on one or more identifiable rules, regulations, or laws that may affect the operation of the autonomous vehicle in the defined spatiotemporal region or operating environment of the autonomous vehicle. Different vehicle operation scenarios can be based on one or more identifiable external objects that may affect the operation of the autonomous vehicle in the defined spatiotemporal region or operating environment of the autonomous vehicle.

[0078] Examples of different vehicle operation scenarios include: a different vehicle operation scenario where the autonomous vehicle is crossing an intersection; a different vehicle operation scenario where pedestrians are crossing or approaching the expected path of the autonomous vehicle; and a different vehicle operation scenario where the autonomous vehicle is changing lanes. The different vehicle operation scenarios may include merging alone, or the different vehicle operation scenarios where the autonomous vehicle is changing lanes may also include merging.

[0079] For simplicity and clarity, similar vehicle operating scenarios may be described herein with reference to vehicle operating scenario types or classes. The type or class of vehicle operating scenario may refer to a specific mode or set of modes on a scenario. For example, a vehicle operating scenario including pedestrians may be referred to herein as a pedestrian scenario, and a pedestrian scenario refers to a type or class of vehicle operating scenario including pedestrians. As an example, a first pedestrian vehicle operating scenario may include a pedestrian crossing a road at a crosswalk, and a second pedestrian vehicle operating scenario may include a pedestrian crossing a road by jaywalking. Although pedestrian vehicle operating scenarios, intersection vehicle operating scenarios, and lane change vehicle operating scenarios are described herein, any other vehicle operating scenario or vehicle operating scenario type may be used.

[0080] Aspects of the operating environment of the autonomous vehicle can be represented within the respective different vehicle operating scenarios. For example, the relative orientation, trajectory, expected path of external objects can be represented within the respective different vehicle operating scenarios. In another example, the relative geometry of the vehicle transportation network can be represented within the respective different vehicle operating scenarios.

[0081] As an example, a first different vehicle operating scenario may correspond to a pedestrian crossing a road at a crosswalk, and the relative orientation and expected path of the pedestrian may be represented within the first different vehicle operating scenario, such as crossing from left to right or from right to left, etc. A second different vehicle operating scenario may correspond to a pedestrian crossing a road by jaywalking, and the relative orientation and expected path of the pedestrian may be represented within the second different vehicle operating scenario, such as crossing from left to right or from right to left, etc.

[0082] The autonomous vehicle may traverse a plurality of different vehicle operating scenarios within the operating environment, which may be aspects of a composite vehicle operating scenario. For example, a pedestrian may approach the intended path of the autonomous vehicle through an intersection.

[0083] The autonomous vehicle operations management system 4000 can operate or control an autonomous vehicle to traverse different vehicle operating scenarios subject to defined constraints (such as safety constraints, legal constraints, physical constraints, user acceptability constraints, or any other constraints or combinations of constraints that can be defined or derived for the operation of an autonomous vehicle).

[0084] Controlling an autonomous vehicle across different vehicle operating scenarios may include: identifying or detecting different vehicle operating scenarios; identifying candidate vehicle control actions based on the different vehicle operating scenarios; controlling the autonomous vehicle across a portion of a vehicle transportation network based on one or more of the candidate vehicle control actions; or a combination thereof.

[0085] A vehicle control action may indicate a vehicle control operation or maneuver, such as accelerating, decelerating, turning, stopping, pulling over, or any other vehicle operation or combination of vehicle operations that may be performed by an autonomous vehicle in conjunction with traversing a portion of a vehicle transportation network.

[0086] The autonomous vehicle operation management controller 4100 or other units of the autonomous vehicle can control the autonomous vehicle to traverse the vehicle transportation network or a portion thereof based on the vehicle control actions. Examples of vehicle control actions include a "stop" vehicle control action that stops or otherwise controls the autonomous vehicle to become or remain stationary, a "forward" vehicle control action that slowly moves the autonomous vehicle forward a short distance (such as a few inches or a foot, etc.), an "accelerate" vehicle control action that accelerates the autonomous vehicle (e.g., at a defined acceleration rate or within a defined range), a "decelerate" vehicle control action that decelerates the autonomous vehicle (e.g., at a defined deceleration rate or within a defined range), a "maintain" vehicle control action that maintains current operating parameters (e.g., current speed, current path or route, current lane heading, etc.), a "turn" vehicle control action (which may include the angle of the turn), a "go" vehicle control action that starts or resumes a previously identified set of operating parameters, or any other standard vehicle operation.

[0087] The vehicle control action may be a composite vehicle control action, which may include a sequence, combination, or both of vehicle control actions. For example, a "go" or "pull over" vehicle control action may indicate a "stop" vehicle control action, a subsequent "accelerate" vehicle control action associated with a defined acceleration rate, and a subsequent "stop" vehicle control action associated with a defined deceleration rate, such that controlling the autonomous vehicle according to the "go" vehicle control action includes controlling the autonomous vehicle to slowly inch forward a short distance, such as a few inches or a foot.

[0088] Autonomous vehicle operations management system 4000 may include autonomous vehicle operations management controller 4100, obstruction monitor 4200, operating environment monitor 4300, SSOCEM 4400, or a combination thereof. Although described separately, obstruction monitor 4200 may be one or more instances of operating environment monitor 4300.

[0089] The autonomous vehicle operations management controller 4100 may receive, identify, or otherwise access operating environment information representing an operating environment of an autonomous vehicle, such as a current operating environment or an expected operating environment or one or more aspects thereof. The operating environment of an autonomous vehicle may include a set of variously identifiable operating conditions that may affect the operation of the vehicle within a defined spatiotemporal region of the vehicle.

[0090] The operating environment information may include vehicle information for the autonomous vehicle, such as information indicating the geospatial location of the vehicle, information relating the geospatial location to information representing a transportation network for the vehicle, the route of the vehicle, the speed of the vehicle, the acceleration state of the vehicle, passenger information of the vehicle, or any other information related to the vehicle or the operation of the vehicle.

[0091] The operating environment information may include information representing a vehicle transportation network that is proximate to the autonomous vehicle (such as within a defined spatial distance of the vehicle (e.g., 300 meters)), information indicating the geometry of one or more aspects of the vehicle transportation network, information indicating conditions of the vehicle transportation network (such as surface conditions, etc.), or any combination thereof.

[0092] Operating environment information may include information representing external objects within the operating environment of the autonomous vehicle, such as information representing pedestrians, non-human animals, non-motorized transportation devices (such as bicycles or skateboards), motorized transportation devices (such as tele-transport vehicles), or any other external objects or entities that may affect the operation of the vehicle.

[0093] The autonomous vehicle operations management controller 4100 may monitor aspects of the autonomous vehicle's operating environment or its definition. Monitoring the operating environment may include identifying and tracking external objects, identifying different vehicle operating scenarios, or a combination thereof.

[0094] For example, the autonomous vehicle operations management controller 4100 can utilize the operating environment of the autonomous vehicle to identify and track external objects. Identifying and tracking external objects can include: identifying the spatiotemporal position of the corresponding external object (which can be relative to the vehicle), identifying one or more expected paths of the corresponding external object (which can include identifying the speed, trajectory, or both of the external object). Descriptions of positions, expected positions, paths, expected paths, etc. herein may omit explicit indications that the corresponding positions and paths refer to geographic space and time components; however, unless explicitly indicated herein or otherwise clearly clear from the context, the positions, expected positions, paths, expected paths, etc. described herein may include geographic space components, time components, or both.

[0095] Operating environment monitors 4300 may include pedestrian operating environment monitors 4310, intersection operating environment monitors 4320, lane change operating environment monitors 4330, or a combination thereof. Operating environment monitors 4340 are shown using dashed lines to indicate that autonomous vehicle operations management system 4000 may include any number of operating environment monitors 4300.

[0096] One or more different vehicle operating scenarios may be monitored by corresponding operating environment monitors 4300. For example, pedestrian operating environment monitor 4310 may monitor operating environment information corresponding to a plurality of pedestrian vehicle operating scenarios, intersection operating environment monitor 4320 may monitor operating environment information corresponding to a plurality of intersection vehicle operating scenarios, and lane change operating environment monitor 4330 may monitor operating environment information corresponding to a plurality of lane change vehicle operating scenarios.

[0097] The operating environment monitor 4300 may receive or otherwise access operating environment information, such as operating environment information generated or captured by one or more sensors of the autonomous vehicle, vehicle transportation network information, vehicle transportation network geometry information, or a combination thereof, etc. For example, the pedestrian operating environment monitor 4310 may receive or otherwise access information, such as sensor data, that may indicate, correspond to, or may otherwise be associated with one or more pedestrians in the operating environment of the autonomous vehicle.

[0098] The operating environment monitor 4300 may associate the operating environment information, or a portion thereof, with the operating environment or an aspect thereof, such as with an external object (such as a pedestrian, a remote vehicle, or an aspect of the vehicle transportation network geometry).

[0099] The operating environment monitor 4300 may generate or otherwise identify information representing one or more aspects of the operating environment, such as having external objects (such as pedestrians, remote vehicles, or an aspect of the vehicle transportation network geometry), which may include filtering, abstracting, or otherwise processing the operating environment information.

[0100] The operating environment monitor 4300 may output information representing one or more aspects of the operating environment to the autonomous vehicle operations management controller 4100 or output the information for access by the autonomous vehicle operations management controller 4100, such as by storing the information representing one or more aspects of the operating environment in a memory of the autonomous vehicle (such as a memory device) accessible by the autonomous vehicle operations management controller 4100. Figure 1The operating environment monitor 4300 may output information representing one or more aspects of the operating environment to one or more elements of the autonomous vehicle operation management system 4000 (such as the obstruction monitor 4200, etc.).

[0101] The pedestrian operating environment monitor 4310 may correlate, associate, or otherwise process the operating environment information to identify, track, or predict the actions of one or more pedestrians. For example, the pedestrian operating environment monitor 4310 may receive information such as sensor data from one or more sensors, which may correspond to one or more pedestrians. The pedestrian operating environment monitor 4310 may associate the sensor data with the identified one or more pedestrians (which may include identifying the direction of travel, path (such as an expected path, etc.), current or expected speed, current or expected acceleration rate, or a combination thereof of one or more of the corresponding identified pedestrians), and the pedestrian operating environment monitor 4310 may output the identified, associated, or generated pedestrian information to the autonomous vehicle operation management controller 4100 or output the pedestrian information for access by the autonomous vehicle operation management controller 4100.

[0102] The intersection operating environment monitor 4320 may correlate, associate, or otherwise process the operating environment information to identify, track, or predict the actions of one or more remote vehicles in the operating environment of the autonomous vehicle, identify an intersection or an aspect thereof in the operating environment, identify vehicle transportation network geometry, or a combination thereof. For example, the intersection operating environment monitor 4320 may receive information such as sensor data from one or more sensors that may correspond to one or more remote vehicles in the operating environment, an intersection or one or more aspects thereof in the operating environment, a vehicle transportation network geometry, or a combination thereof. The intersection operating environment monitor 4320 can associate sensor data with one or more identified remote vehicles in the operating environment, intersections in the operating environment or one or more aspects thereof, vehicle transportation network geometry, or a combination thereof (which can include identifying one or more current or expected directions of travel, paths (such as expected paths, etc.), current or expected speeds, current or expected acceleration rates, or a combination thereof of the corresponding identified remote vehicles), and the intersection operating environment monitor 4320 can output the identified, associated, or generated intersection information to the autonomous vehicle operation management controller 4100 or output the intersection information for access by the autonomous vehicle operation management controller 4100.

[0103] The lane change operating environment monitor 4330 may correlate, associate, or otherwise process the operating environment information to identify, track, or predict the actions of one or more remote vehicles in the operating environment of the autonomous vehicle (such as information indicating a slow or stationary remote vehicle along the vehicle's expected path, etc.), identify one or more aspects of the operating environment (such as the vehicle transportation network geometry in the operating environment, etc.), or a combination thereof (corresponding to the current or expected lane change operation in geospatially). For example, the lane change operating environment monitor 4330 may receive information such as sensor data from one or more sensors that may correspond to one or more remote vehicles in the operating environment of the autonomous vehicle, one or more aspects of the operating environment, or a combination thereof (corresponding to the current or expected lane change operation in geospatially). The lane change operating environment monitor 4330 may associate sensor data with one or more identified remote vehicles in the operating environment of the autonomous vehicle, one or more aspects of the operating environment, or a combination thereof that corresponds to a current or expected lane change operation in geospatially, which may include identifying a current or expected direction of travel, a path (such as an expected path, etc.), a current or expected speed, a current or expected acceleration rate, or a combination thereof for one or more of the corresponding identified remote vehicles. The lane change operating environment monitor 4330 may output the identified, associated, or generated lane change information to the autonomous vehicle operation management controller 4100 or output the lane change information for access by the autonomous vehicle operation management controller 4100.

[0104] The autonomous vehicle operation management controller 4100 may identify one or more different vehicle operation scenarios based on one or more aspects of the operating environment represented by the operating environment information. The autonomous vehicle operation management controller 4100 may identify different vehicle operation scenarios in response to identifying or based on the operating environment information indicated by one or more operating environment monitors 4300. For example, the operating environment information may include information indicating that a pedestrian is approaching an intersection along the expected path of the autonomous vehicle, and the autonomous vehicle operation management controller 4100 may identify a pedestrian vehicle operation scenario, an intersection vehicle operation scenario, or both.

[0105] The autonomous vehicle operations management controller 4100 may instantiate one or more corresponding instances of the SSOCEM 4400 based on one or more aspects of the operating environment represented by the operating environment information. For example, the autonomous vehicle operations management controller 4100 may instantiate corresponding instances of the SSOCEM 4400 in response to identifying an upcoming scenario. The upcoming scenario may be a different vehicle operation scenario that the autonomous vehicle operations management controller 4100 determines to be likely to encounter if the autonomous vehicle continues in its path. The upcoming scenario may be expected (e.g., may be determined from the route of the autonomous vehicle) or unexpected. An unexpected upcoming scenario may be a scenario that may be detected by the vehicle's sensors and cannot be determined without sensor data.

[0106] Once instantiated, SSOCEM 4400 can receive operating environment information including sensor data to determine and output candidate vehicle control actions (also referred to as candidate actions in this article). Candidate actions are vehicle control actions that are identified by a specific SSOCEM 4400 as possible optimal actions for a vehicle to perform to deal with a specific scenario. For example, a SSOCEM 4400 configured to deal with an intersection (e.g., intersection SSOCEM 4420) can output "travel", i.e., a candidate action that suggests traveling through an intersection. At the same time, a SSOCEM 4400 used to deal with lane changes (e.g., lane change SSOCEM 4430) can output a "left turn" candidate action, which indicates that the vehicle should merge to the left by two degrees. In some implementations, each SSOCEM 4400 outputs a confidence score indicating the degree of confidence in the candidate actions determined by the SSOCEM 4400. For example, a confidence score greater than 0.95 may indicate a very high confidence in the candidate action, while a confidence score less than 0.5 may indicate a relatively low degree of confidence in the candidate action.Further details of the SSOCEM 4400 are described below.

[0107] The autonomous vehicle operation management controller 4100 receives candidate actions and determines a vehicle control action based on the received candidate actions. In some implementations, the autonomous vehicle operation management controller 4100 uses hard-coded logic to determine the vehicle control action. For example, the autonomous vehicle operation management controller 4100 can select the candidate action with the highest confidence score. In other implementations, the autonomous vehicle operation management controller 4100 can select the candidate action that is least likely to cause a collision. In other implementations, the autonomous vehicle operation management controller 4100 can generate a composite action based on two or more non-conflicting candidate actions (for example, compounding "travel" and "turn left two degrees" to obtain a vehicle control action that turns the vehicle left and travels through the intersection). In some implementations, the autonomous vehicle operation management controller 4100 can use a machine learning algorithm to determine a vehicle control action based on two or more different candidate actions.

[0108] For example, identifying a vehicle control action from candidate actions may include: implementing a machine learning component (such as supervised learning for a classification problem, etc.), and training the machine learning component using examples (such as 1,000 examples, etc.) of corresponding vehicle operation scenarios. In another example, identifying a vehicle control action from candidate actions may include implementing a Markov decision process (MDP) or a partially observable Markov decision process (POMDP), which may describe how a corresponding candidate action affects subsequent candidate actions and may include a reward function that outputs a positive or negative reward for a corresponding vehicle control action.

[0109] The autonomous vehicle operations management controller 4100 may de-instantiate an instance of the SSOCEM 4400. For example, the autonomous vehicle operations management controller 4100 may identify different sets of operating conditions as indicative of different vehicle operation scenarios for an autonomous vehicle, instantiate instances of the SSOCEM 4400 for the different vehicle operation scenarios, monitor the operating conditions, then determine that one or more of the operating conditions has expired or has a probability of affecting the operation of the autonomous vehicle below a defined threshold, and the autonomous vehicle operations management controller 4100 may de-instantiate an instance of the SSOCEM 4400.

[0110] The obstruction monitor 4200 may receive operating environment information representative of the operating environment of the vehicle or an aspect thereof. For example, the obstruction monitor 4200 may receive operating environment information from the autonomous vehicle operations management controller 4100, from sensors of the vehicle, from external devices such as remote vehicles or infrastructure devices, or a combination thereof. The obstruction monitor 4200 may receive operating environment information from a memory (such as a memory of the autonomous vehicle (such as a Figure 1The memory 1340 shown) etc.) reads the operating environment information or a portion thereof.

[0111] The blocking monitor 4200 may determine a corresponding availability probability or a corresponding blocking probability for one or more portions of the vehicle transportation network. These portions may include portions of the vehicle transportation network corresponding to the expected path of the autonomous vehicle.

[0112] The availability probability or corresponding blocking probability can indicate the probability or likelihood that the autonomous vehicle will safely (such as without being obstructed by external objects (such as remote vehicles or pedestrians, etc.) traverse a portion of the vehicle transportation network or a spatial location within the vehicle transportation network. For example, a portion of the vehicle transportation network may include obstructions such as stationary objects, etc., and the availability probability of that portion of the vehicle transportation network may be low, such as 0%, etc., which can be represented as a high blocking probability of that portion of the vehicle transportation network, such as 100%, etc. The blocking monitor 4200 can identify the corresponding availability probability of each of the multiple portions of the vehicle transportation network within the operating environment of the autonomous vehicle (such as within 300 meters, etc.).

[0113] The availability probabilities corresponding to various external objects in the operating environment of the autonomous vehicle may be indicated by the obstruction monitor 4200, and the geospatial region may be associated with multiple availability probabilities corresponding to multiple external objects. Aggregate availability probabilities corresponding to various types of external objects in the operating environment of the autonomous vehicle (such as an availability probability for pedestrians and an availability probability for remote vehicles, etc.) may be indicated by the obstruction monitor 4200, and the geospatial region may be associated with multiple availability probabilities corresponding to multiple types of external objects.

[0114] The obstruction monitor 4200 can identify an external object, track an external object, predict the location information, path information, or both of the external object, or a combination thereof. For example, the obstruction monitor 4200 can identify an external object and identify an expected path of the external object based on operating environment information (e.g., the current location of the external object), information indicating the current trajectory and / or speed of the external object, information indicating the classification type of the external object (e.g., a pedestrian or a remote vehicle), vehicle transportation network information (e.g., a crosswalk near the external object), previously identified or tracked information associated with the external object, or any combination thereof. The expected path can indicate a sequence of expected spatial locations, expected temporal locations, and corresponding probabilities.

[0115] The blocking monitor 4200 may communicate the available probabilities or the corresponding blocking probabilities to the autonomous vehicle operation management controller 4100. The autonomous vehicle operation management controller 4100 may communicate the available probabilities or the corresponding blocking probabilities to the corresponding instantiated instance of the scenario-specific operation control evaluation module 4400.

[0116] Despite Figure 4 Not explicitly shown, but the autonomous vehicle operations management system 4000 may include a predictor module that can generate prediction information and send it to the obstruction monitor 4200, and the obstruction monitor 4200 can output available probability information to one or more of the operating environment monitors 4300.

[0117] Each SSOCEM 4400 may model a corresponding different vehicle operation scenario. The autonomous vehicle operation management system 4000 includes any number of SSOCEM 4400, each SSOCEM 4400 models a corresponding different vehicle operation scenario. Modeling different vehicle operation scenarios may include: generating and / or maintaining state information representing various aspects of the vehicle's operating environment corresponding to different vehicle operation scenarios; identifying potential interactions between the modeled aspects of the respective states; and determining candidate actions for solving the model. More simply, the SSOCEM 4400 may include one or more models that are configured to determine one or more vehicle control actions for dealing with a scenario given a set of inputs. Models may include, but are not limited to, partially observable Markov decision process (POMDP) ​​models, Markov decision process (MDP) models, classical programming (CP) models, partially observable stochastic game (POSG) models, decentralized partially observable Markov decision process (Dec-POMDP) ​​models, reinforcement learning (RL) models, artificial neural networks, hard-coded specialized logic, or any other suitable type of model. Examples of different types of models are provided below. Each SSOCEM 4400 includes computer executable instructions for defining the manner in which the model operates and the manner in which the model is utilized.

[0118] SSOCEM 4400 can implement a CP model, which can be a single-agent model that models different vehicle operation scenarios based on defined input states. The defined input states can indicate corresponding non-probabilistic states of elements of the operating environment of an autonomous vehicle for different vehicle operation scenarios. In the CP model, one or more aspects (e.g., geospatial locations) of a modeled element (e.g., an external object) associated with a temporal position can be non-probabilistically different from corresponding aspects associated with another temporal position (e.g., an immediately subsequent temporal position) by a defined or fixed amount. For example, at a first temporal position, a remote vehicle can have a first geospatial location, and at an immediately subsequent second temporal location, the remote vehicle can have a second geospatial location that differs from the first geospatial location by a defined geospatial distance along the expected path of the remote vehicle, such as a defined number of meters, etc.

[0119] SSOCEM 4400 can implement discrete-time stochastic control processing, such as an MDP model, which can be a single-agent model that models different vehicle operation scenarios based on defined input states. Changes in the operating environment of the autonomous vehicle (such as changes in the position of external objects, etc.) can be modeled as probabilistic changes. The MDP model can utilize more processing resources and can model different vehicle operation scenarios more accurately than the CP model.

[0120] The MDP model can use a set of states, a set of actions, a set of state transition probabilities, a reward function, or a combination thereof to model different vehicle operation scenarios. In some embodiments, modeling different vehicle operation scenarios may include using a discount factor that can adjust or discount the output of the reward function applied to subsequent time periods.

[0121] The state set may include the current state of the MDP model, one or more possible subsequent states of the MDP model, or a combination thereof. The state represents an identified condition (which may be an expected condition) of a corresponding defined aspect of the vehicle's operating environment (such as external objects and traffic control devices, etc.) that may probabilistically affect the operation of the vehicle at a discrete time location. For example, a remote vehicle operating in proximity to the vehicle may affect the operation of the vehicle and may be represented in the MDP model. The MDP model may include the following identified or expected information representing the remote vehicle corresponding to the corresponding time location: the geospatial location of the remote vehicle, the path, heading, or both of the remote vehicle, the speed of the remote vehicle, the acceleration rate or deceleration rate of the remote vehicle, or a combination thereof. When instantiated, the current state of the MDP model may correspond to a simultaneous state or condition of the operating environment.

[0122] Although any number or cardinality of states may be used, the number or cardinality of states included in the model may be limited to a defined maximum number of states. For example, the model may include the 300 most likely states for a corresponding scenario.

[0123] The action set may include vehicle control actions that can be used in the MDP model in each state in the state set. A corresponding action set may be defined for each different vehicle operation scenario.

[0124] The state transition probability set may probabilistically represent potential or expected changes to the operating environment of a vehicle, as represented by a state, in response to an action. For example, the state transition probability may indicate the probability that the operating environment corresponds to a corresponding state at a corresponding time position immediately following a current time position corresponding to the current state, in response to the vehicle traversing the vehicle transportation network from a current state according to a corresponding action.

[0125] The state transition probability set may be identified based on the operating environment information. For example, the operating environment information may indicate the type of area (such as urban or rural), time of day, ambient light level, weather conditions, traffic conditions (which may include expected traffic conditions, such as rush hour conditions, event-related traffic congestion, or holiday-related driver behavior conditions), road conditions, jurisdiction conditions (such as country, state, or city conditions), or any other condition or combination of conditions that may affect the operation of the vehicle.

[0126] Examples of state transition probabilities associated with pedestrian vehicle operation scenarios may include: a defined probability of a pedestrian jaywalking (e.g., based on the geospatial distance between the pedestrian and the corresponding road segment); a defined probability of a pedestrian stopping at an intersection; a defined probability of a pedestrian crossing at a crosswalk; a defined probability of a pedestrian giving way to an autonomous vehicle at a crosswalk; or any other probability associated with a pedestrian vehicle operation scenario.

[0127] Examples of state transition probabilities associated with intersection vehicle operation scenarios may include: a defined probability of a remote vehicle arriving at an intersection; a defined probability of a remote vehicle cutting off an autonomous vehicle; a defined probability of a remote vehicle crossing an intersection immediately after and adjacent to a second remote vehicle crossing an intersection, such as without the right of way (piggybacking); a defined probability of a remote vehicle stopping near an intersection prior to crossing the intersection in accordance with a traffic control device, regulation, or other right-of-way indication; a defined probability of a remote vehicle crossing an intersection; a defined probability of a remote vehicle deviating from an expected path approaching an intersection; a defined probability of a remote vehicle deviating from an expected right-of-way priority; and any other probability associated with an intersection vehicle operation scenario.

[0128] Examples of state transition probabilities associated with a lane-changing vehicle operation scenario may include: a defined probability of a remote vehicle changing rate, such as a defined probability of a remote vehicle behind the vehicle increasing its rate or a defined probability of a remote vehicle in front of the vehicle decreasing its rate; a defined probability of a remote vehicle in front of the vehicle changing lanes; a defined probability of a remote vehicle approaching the vehicle changing speed to allow the vehicle to merge into the lane; or any other probability associated with a lane-changing vehicle operation scenario.

[0129] The reward function may determine a corresponding positive or negative (cost) value accumulated for each combination of state and action. The accumulation represents the expected value of the vehicle traversing the vehicle transportation network from the corresponding state to the subsequent state according to the corresponding vehicle control action.

[0130] For example, the POMDP model may include an autonomous vehicle at a first geospatial location and a first time location corresponding to a first state. The model may instruct the vehicle to identify and perform or attempt to perform a vehicle control action to traverse the vehicle transportation network from the first geospatial location to a second geospatial location at a second time location immediately after the first time location. The set of observations corresponding to the second time location may include identified operating environment information corresponding to the second time location, such as geospatial location information of the vehicle, geospatial location information of one or more external objects, availability probabilities, expected path information, etc.

[0131] The conditional observation probability set may include the probability of making corresponding observations based on the operating environment of the autonomous vehicle. For example, the autonomous vehicle may approach an intersection by crossing a first road, while the remote vehicle may approach an intersection by crossing a second road, and the autonomous vehicle may identify and evaluate the operating environment information (such as sensor data, etc.) corresponding to the intersection, which may include the operating environment information corresponding to the remote vehicle. The operating environment information may be inaccurate, incomplete or wrong. In the MDP model, the autonomous vehicle may non-probabilistically identify the remote vehicle, which may include identifying the location, expected path, etc. of the remote vehicle, and the identified information (such as the identified location, etc.) based on the inaccurate operating environment information may be inaccurate or wrong. In the POMDP model, the autonomous vehicle may identify information that probabilistically identifies the remote vehicle (such as probabilistically identifying the location information of the remote vehicle). The conditional observation probability corresponding to the observation or probabilistic identification of the location of the remote vehicle represents the probability that the identified operating environment information accurately represents the location of the remote vehicle.

[0132] The set of conditional observation probabilities may be identified based on operating environment information, such as operating environment information described about a reward function.

[0133] SSOCEM 4400 may implement a Dec-POMDP model, which may be a multi-agent model that models different vehicle operation scenarios. The Dec-POMDP model may be similar to the POMDP model, except that the POMDP model models a vehicle and a suitable subset of external objects (such as one, etc.), and the Dec-POMDP models a collection of autonomous vehicles and external objects.

[0134] SSOCEM 4400 may implement a POSG model, which may be a multi-agent model that models different vehicle operation scenarios. The POSG model may be similar to Dec-POMDP, except that the Dec-POMDP model includes a reward function for the vehicle and the POSG model includes a reward function for the vehicle and a corresponding reward function for each external object.

[0135] The SSOCEM 4400 may implement an RL model that may be a learning model that models different vehicle operation scenarios. The RL model may be similar to an MDP model or a POMDP model, except that defined state transition probabilities, observation probabilities, reward functions, or any combination thereof may be omitted from the RL model. Alternatively, for example, the RL model may be a model-based RL model that generates state transition probabilities, observation probabilities, reward functions, or any combination thereof based on one or more modeled or observed events.

[0136] In the RL model, the model can evaluate one or more events or interactions (which may include simulated events), and can generate or modify a corresponding model or its solution in response to the corresponding event. Simulated events may include, for example, crossing an intersection, crossing a vehicle transportation network near pedestrians, or changing lanes. An example of using an RL model to cross an intersection includes the RL model indicating a candidate action for crossing an intersection. The autonomous vehicle then uses the candidate action as a vehicle control action for a temporal position to cross the intersection. The result of crossing the intersection using the candidate action can be determined to update the RL model based on the result.

[0137] The autonomous vehicle operation management system 4000 may include any number or combination of model types. For example, the pedestrian SSOCEM 4410, the intersection SSOCEM 4420, and the lane change SSOCEM 4430 may implement the POMDP model. In another example, the pedestrian SSOCEM 4410 may implement the MDP model, and the intersection SSOCEM 4420 and the lane change SSOCEM 4430 may implement the POMDP model. In addition, the autonomous vehicle operation management controller 4100 may instantiate any number of instances of the SSOCEM 4400 based on the operating environment information. The module 4440 is shown using a dotted line to indicate that the autonomous vehicle operation management system 4000 may include any number or additional types of SSOCEM 4400.

[0138] One or more of the autonomous vehicle operation management controller 4100, the obstruction monitor 4200, the operating environment monitor 4300, or the SSOCEM 4400 may operate continuously or periodically, such as at a frequency of 10 Hertz (10 Hz), etc. For example, the autonomous vehicle operation management controller 4100 may identify vehicle control actions multiple times per second (such as ten times, etc.). The operating frequencies of the various components of the autonomous vehicle operation management system 4000 may be synchronized or asynchronous, and the operating rate of one or more of the autonomous vehicle operation management controller 4100, the obstruction monitor 4200, the operating environment monitor 4300, or the SSOCEM 4400 may be independent of the operating rates of the other components.

[0139] Figure 5 5 is a diagram of an example of a capability awareness system (CAS) 5000 according to an embodiment of the present disclosure. CAS 5000 includes an autonomous cognitive agent (ACA) 5002. As described below with respect to ACA 5002, any one of the SSOCEMs (such as Figure 4 One of the SSOCEM 4410, 4420, 4430, 4440) can be an ACA. ACA5002 can have multiple autonomy levels. CAS 5000 can be Figure 4 Autonomous Vehicle Operations Management System 4000.

[0140] ACA5002 can operate at multiple autonomous operation levels (i.e., autonomy levels) and plan for / utilize multiple autonomous operation levels (i.e., autonomy levels). When the AV performs a vehicle control action at an autonomy level, the AV (more specifically, the ACA) can receive a feedback signal (or simply feedback) in response to the AV's execution of the vehicle control action. For example, the AV can perform a vehicle control action while being monitored by a human. The vehicle can be remotely monitored by a teleoperator. The vehicle can be monitored by a human within the vehicle. The human can cause a signal indicating feedback to be sent to the AV (or a module, component, circuit, etc. therein (such as ACA5002, etc.)) or received by the AV.

[0141] ACA 5002 may proactively determine which autonomy level to enter. Thus, ACA 5002 may have a gradient of autonomy levels. Thus, ACA 5002 may maintain robustness without requiring the same expense of all-or-none (i.e., binary) autonomy. Thus, ACA 5002 may proactively avoid scenarios where it cannot navigate autonomously, and may proactively issue a request for help before the AV is stuck. Proactively requesting help may reduce wasted operational time and resources.

[0142] As described further below, ACA 5002 can learn from its experience traversing a portion of the vehicle transportation network what level of autonomy to enter when traversing the same or similar portion of the vehicle transportation network. That is, ACA 5002 can learn from its experience to predict the feedback to be received from a human operator (such as a teleoperator or a human in a vehicle, etc.).

[0143] ACA5002 can modify its selection of autonomy level based on new knowledge acquired during traversal of this portion of the vehicle transportation network. The learned predictions can be used to modify how ACA5002 traverses this portion of the vehicle transportation network. For example, a request for human assistance should not be issued when ACA5002 has high confidence in the response that will be received from a human. Therefore, ACA5002 only requests assistance in scenarios where ACA5002 is confident that ACA5002 cannot handle the scenario autonomously. In addition, ACA 5002 can actively avoid scenarios that ACA5002 determines (based on its capability level) as ACA5002 requiring human assistance. Therefore, ACA5002 can support scenarios that it can navigate autonomously. For example, ACA5002 can replan (e.g., determine a new route) to avoid the scenario.

[0144] ACA 5002 uses domain model (DM) 5004, human model (HM) 5006, and autonomy model (AM) 5008 as input. HM 5006 may also be referred to as a feedback model. ACA 5002 updates HM 5006 as shown by arrow 5012 and as further described below. ACA 5002 updates AM 5008 as shown by arrow 5010 and as further described below.

[0145] As described above, DM 5004 may model the environment in which ACA 5002 is operating (ie, the operating environment).

[0146] AM 5008 can model the levels of autonomy at which ACA 5002 can operate, when (e.g., under what conditions or constraints) ACA 5002 can be allowed to operate at each autonomy level, and what the corresponding utility of the autonomy level is. The utility can indicate the expected value of performing a vehicle control action at the corresponding autonomy level. The utility value can be used to express a preference between autonomy levels.

[0147] HM 5006 may describe a feedback model that models the feedback that ACA 5002 may receive from a human (eg, a teleoperator), how costly each type of feedback is, and how likely ACA 5002 is to receive each type of feedback.

[0148] As mentioned, DM 5004 can model the environment (i.e., operating environment) in which ACA5002 is operating. For example, DM 5004 can describe (e.g., include) the transition and / or cost dynamics of the environment for ACA5002. In an example, DM5004 can be modeled as a stochastic shortest path (SSP) problem. As known, SSP is a formal decision model for reasoning in a fully observable random environment, wherein the purpose can be to find the minimum cost path from the start state to the target state. For example, the goal of ACA5002 can be to successfully cross the intersection. Therefore, the start state can be the first time position before the intersection (e.g., 50 meters before the intersection), and the target state can be the second time position after the intersection (e.g., 50 meters after the intersection). When ACA5002 is approaching the intersection, ACA 5002 plans the trajectory including the action set to achieve the goal. As a result of planning, ACA5002 selects the next action to be performed under the autonomy level according to HM 5006 and AM 5008.

[0149] DM 5004 can be formally modeled as a tuple <S,A,T,C,s o,G>. S can be a finite set of states (i.e., a set of operating environment information). A can be a finite set of actions (i.e., a set of vehicle control actions). T (i.e., T:S×A→Δ |S| ) can represent the transition function describing the probability distribution of the subsequent state when taking action a∈A in state s∈S. C (i.e., ) can represent a cost function describing the cost of taking action a∈A in state s∈S. s0∈S can be the initial state. Can be a finite set of target states.

[0150] The solution to the SSP of DM 5004 may be a policy π: S→A. That is, under policy π, an action a (i.e., π(s)) is selected for state s. That is, policy π may indicate that an action π(s)∈A should be taken in state s. Policy π may include a value function V π :S→C, the value function V π : S→C can represent the expected cumulative cost V of following strategy π from state s to the target state G π (s). That is, the value function provides each intermediate state s from the starting state until the target state is reached i , the cost (i.e., value) of . The optimal policy π minimizes the expected cumulative cost.

[0151] Thus, for each state configuration of interest, a policy can be used to determine the action that the AV will take in that state. Descriptive examples of states may be "pedestrian in front of AV" and "AV at stop sign". Such a state may be associated with an action "stop" to avoid hitting a pedestrian. Another descriptive example of a state may be "AV in intersection" and "other vehicle at stop sign". Such a state may be associated with an action "go ahead" so that the AV can complete crossing the intersection.

[0152] Intuitively, DM 5004 may include a descriptor of how the world (i.e., the operating environment) changes in one time step for all combinations of states. DM 5004 may include the concept of good states and bad states. That is, DM 5004 may include a descriptor of what configuration of the world is good. For example, in a scenario of crossing an intersection, a good state is a state in which the AV completes crossing the intersection; and a bad state may correspond to a state in which the AV collides with another vehicle or the AV violates a law. A negative reward may be associated with a bad state, and a positive reward may be associated with a good state. Given such descriptors and rewards, an optimal policy π may be computed based on how the operating environment evolves over time. * , the optimal strategy π * is the set of actions (or equivalently, the path through the operating environment) that maximizes the reward.

[0153] Other fully observable stochastic models (such as MDPs) can be used with minimal changes, but partially observable MDPs (POMDPs) may introduce additional sources of uncertainty (especially with respect to human interactions) that make POMDPs difficult to use. AMs describe the levels of autonomy at which an agent can operate, restrictions on the scenarios in which each level is permitted, the utility of each level, a collection of system subcapabilities, or any combination thereof.

[0154] As mentioned, AM 5010 can model the degree of autonomous operation (i.e., the autonomy level) with which ACA 5002 can operate. The autonomy level can indicate (as described below with respect to the autonomy level set Both the actual different forms or degrees of autonomous operation (as described below) and when each level of autonomy may be allowed under some external constraints (as described below with respect to autonomy profiles κ).

[0155] AM 5008 can be formally modeled as a tuple in represents a finite set of autonomy levels, where each level corresponds to a certain set of constraints on the autonomous operation of ACA 5002. ) is an autonomy profile and describes which autonomy levels are allowed when taking an action a∈A in a state s∈S μ(i.e., ) is the autonomy cost and describes the cost of autonomy for a given agent at level If the next action has been taken, in state s∈S, the level The cost of taking action a∈A. is a collection of sub-capabilities, where each sub-capability is the mapping τ i :S×A→Δ |S| .

[0156] It can be a collection of autonomy levels. Each autonomy level l i May correspond to a set of constraints on the autonomous operation of ACA5002. In the example, the action set can be a partially ordered set (i.e., poset). That is, the set The actions have an order or sequence indicating, for example, increasing levels of autonomy.

[0157] In the example, the collection The autonomy level of the system may include four autonomy levels; namely, a "no autonomy" level (i.e., l0), a "verified autonomy" level (i.e., l1), a "supervised autonomy" level (i.e., l2), and an "unsupervised autonomy" level (i.e., l4). The disclosure herein is not limited to such a set of autonomy levels. That is, other levels of autonomy with different semantics are possible.

[0158] The “no autonomy” level 10 may indicate that the ACA 5002 requires a human to act on behalf of the ACA 5002. The no autonomy level may be summarized as the AV requesting the human to fully control the AV so that the human can disengage the AV from a situation (eg, an obstacle scene).

[0159] "Verified autonomy" level 11 may indicate that the ACA 5002 must ask for and receive explicit approval from a human operator before even attempting a selected (e.g., identified, determined, etc.) action. For example, in a sequence of actions (i.e., a plan) determined by the ACA 5002, the ACA must seek explicit approval for each action before proceeding.

[0160] The "supervised autonomy" level l2 may indicate that the ACA 5002 may perform actions autonomously as long as there is a human supervising (e.g., remotely or otherwise monitoring) the ACA 5002. In the "supervised autonomy" level l2, the human may intervene if an error occurs while the action is being performed autonomously. For example, a sequence of actions may be performed as long as the human is supervising the AV. If a failure is detected before or after performing an action in the action sequence, the ACA may request human (e.g., teleoperator) assistance.

[0161] To clarify the boundary between the "verified autonomy" level l1 and the "supervised autonomy" level l2, an example is now given. In the "supervised autonomy" level l2, monitoring does not need to be remote. For example, the testing process of the AV can be considered "supervised autonomy" because the AV can drive autonomously, but it can only do so if there is a supervisory human in the AV who is ready to override the AV and take control in dangerous scenarios. As a further boundary, the "verified autonomy" level l1 may require ACA5002 to receive explicit permission from a human (who may be in the AV or remote) before performing the desired action. In particular, receiving explicit permission may mean that ACA5002 should stop until it receives permission. On the other hand, in the "supervised autonomy" level l2, there is no such requirement as long as there is a supervisory human. That is, ACA5002 does not need to stop and can continue to perform its desired actions, relying on the supervisory human authority to override in potentially dangerous situations.

[0162] "Unsupervised Autonomy" level 13 may indicate that the ACA 5002 may be in fully autonomous operation without the need for human approval, supervision, or monitoring.

[0163] The autonomy profile κ (i.e. ) can map states s∈S and actions a∈A to a set of autonomy levels A subset of . Represents a set of autonomy levels The autonomy profile κ may specify constraints on the allowed autonomy levels for any scenario (ie, the state of the DM 5004). Given the current state of the environment and the action to be performed next, the autonomy profile κ defines a set of acceptable autonomy levels.

[0164] A constraint may be or may include a hard constraint. For example, a constraint may be a technical, legal, or ethical constraint. To illustrate, a non-limiting example of a legal constraint may be that an autonomous vehicle cannot operate autonomously (i.e., at an unsupervised autonomy level 14) in a school zone. A non-limiting example of a social constraint may be a rule of the road whereby when a traffic signal turns green, oncoming traffic yields to the first vehicle turning left. Thus, the constraint is that the first vehicle turning left must proceed rather than wait for traffic to clear.

[0165] The constraints may be, may include, or may serve as temporary conservative constraints that may be updated over time as the ACA 5002 improves. The autonomy profile κ may constrain the space of all policies (π) such that the ACA 5002 is only allowed to follow policies that never violate the autonomy profile κ.

[0166] The utility μ represents the following: given that the action at time step t was taken at autonomy level l, what is the utility of taking another action at another autonomy level l′ at time step t+1? The action at time step t+1 need not be but can be the same action as the action taken at time step t. In some scenarios, there may be a negative utility associated with swapping autonomy levels. For example, in dynamic scenarios (e.g., complex intersections), where the ACA5002 can swap between "supervised autonomy" and "unsupervised autonomy" at each time step given no utility μ. The constant swapping of autonomy levels may actually be more uncomfortable for humans who have to constantly swap attention than simply remaining in "supervised" mode the entire time.

[0167] As further described below, AM 5008 can evolve. That is, AM 5008 can be trained based on the experience of ACA5002. For illustration, assume that the first AV will be deployed to a first market (e.g., Japan) and the second AV will be deployed to a second market (e.g., France). The first AV and the second AV may initially include the same autonomy model operating in a binary autonomy mode. That is, the AV can be trusted to deal with scenarios that it is a priori programmed to discern and traverse, or the AV requests human (e.g., teleoperator) assistance. Since the first market and the second market can each have different (e.g., social) road rules, the autonomy model of the first AV and the autonomy model of the second AV evolve in different ways based on the feedback each receives from humans. Once the autonomy model learns the scenarios that it is capable of dealing with in its respective market, the ACA no longer needs to request human (e.g., teleoperator) assistance for the learned scenarios (i.e., scenes).

[0168] Without loss of generality, we can assume is a total ordered set, and if the level of autonomy can be changed from one level to another, then can be extended to any graph connecting two levels. The constraints corresponding to each autonomy level can be technical in nature (i.e., internally imposed constraints, such as the need for human supervision in severe weather, which may be known a priori to cause errors), as well as externally imposed in nature (such as ethical or legal constraints). In addition, κ can be defined to reflect not only static constraints, but also temporary constraints that can be updated over time and used to achieve more conservative autonomous behavior while the system is still learning. Each constraint can be associated with a corresponding form of human assistance or participation. For example, in the "supervised autonomy" level, the agent can act fully autonomously subject to the presence of a human who actively monitors the agent's execution and is able to override any actions that are deemed unsafe or undesirable. Therefore, the higher the autonomy level, the lower the cost of human participation, although this is not a requirement of the model.

[0169] The capability of a system may depend on the behavior of a subsystem that informs the master process that the system is planning. For example, a perception system that has a high probability of producing incorrect state updates in some scenarios may result in lower capability in those scenarios. A subcapability may be a mathematical representation of the behavior of a subsystem that informs the master process at the level of abstraction that the planning model is reasoning about. A perception subcapability may be defined as the probability that a perception will fail in a given state. In this case, a perception may be a subsystem that informs the master process by providing state update information. Subcapabilities that are known completely a priori or assumed to be known completely a priori may be included in the domain model as part of the transition function and therefore omitted from the autonomy model. Subcapabilities that are not known a priori may be represented by functions Estimated based on observations and data collected while operating online. Including different sub-capabilities may require additional autonomy levels that provide system procedures or operating modes that enable the system to obtain the feedback needed to learn the true sub-capability, or otherwise provide resources for the agent to perform safely and reliably in scenarios with poor sub-capabilities. For example, to better cope with scenarios with poor perception capabilities, additional autonomy levels may be included that require the agent to slow down its movements in order to provide additional online time for its perception algorithm, thereby providing higher quality output that can reduce the associated likelihood of failure.

[0170] As mentioned above, HM 5006 (ie, feedback model) can model the beliefs of ACA 5002 regarding the interaction of ACA 5002 with a human operator. HM 5006 can be formally represented as a tuple where ∑ represents a finite set of feedback signals that the agent can receive from humans, λ represents a feedback profile, and In the case of the operation under the state s∈S, the level The probability distribution over the feedback signal that the agent will receive when performing action a∈A, ρ represents the human cost function, and represents the ... In the case of the operation under the condition that the agent is in state s∈S with level The cost to humans of performing action a∈A, and represents the human state transition function, and represents the probability distribution over the successor state s′∈S when the human takes control of the system when the agent attempts to perform action a∈A in state s∈S.

[0171] ∑={σ0,…,σ n} may be a set of possible feedback signals that the ACA5002 may receive from a human operator. Figure 6 Describes a non-limiting example of a feedback signal. The feedback profile λ may represent a feedback signal when a given ACA 5002 is in state s∈S and the ACA 5002 is at an autonomy level Under the circumstances of operation at the autonomy level The probability that ACA 5002 receives signal σ∈∑ when performing action a∈A. Therefore, the feedback profile λ can be expressed symbolically as

[0172] "Operated under..." may mean the following: At time step t, "operated under..." may mean the level of autonomy at which the action taken by ACA at time step t-1 was performed. As an example, suppose that at time step t, ACA performed action a with autonomy level l2 (i.e., "supervised autonomy"). Therefore, a human has been involved and is observing the behavior of ACA. If, at time step t+1, ACA performs action a' again with autonomy level l2, the probability that a human will override action a' may be lower than the probability if ACA performs action a with autonomy level l3 (i.e., "unsupervised autonomy"), in which case the human may be more surprised and therefore may be more likely to override the action.

[0173] The human cost function ρ can return to the human the cost of a given ACA 5002 being in state s∈S and the ACA 5002 being at autonomy level Under the circumstances of operation at the autonomy level The positive cost of performing an action a∈A. The human cost function ρ can be expressed symbolically as

[0174] The human state transition function τ may represent the human bringing the ACA 5002 to state s∈S when the ACA 5002 chooses to perform an action a∈A in state s∈S and a human (e.g., a teleoperator) takes control of the AV. ′ ∈S. “Bringing ACA5002 to state s” means that the human operates the AV so that state s is achieved. Human state transition function It can be expressed in symbols as For example, suppose the state is s (e.g., s = "at an intersection"), and the ACA intends to take action a (e.g., a = "turn left"), but the human overrides the ACA and takes control. In this case, the human state transition function represents the probability that a human will bring the ACA to a certain state (e.g., complete the left turn, or go straight instead) given the state the ACA is in (i.e., state s) and the action the agent intends to take (i.e., turn left).

[0175] Note that in practice, the feedback profile λ and the human state transition function is not known a priori. Therefore, ACA 5002 may maintain the feedback profile λ and the human state transition function based on previous data collected by ACA 5002 in the same or similar situations. The updating of HM 5006 is illustrated by arrow 5012. Thus, and as with respect to Figure 6To further describe, after ACA 5002 performs an action (in the action execution phase), system 5000 can record the feedback (if any) received by ACA 5002 from the human operator, and can use the feedback to update the feedback profile λ and the human state transition function At least one of them.

[0176] The system 5000 (ie, capability awareness system (CAS)), and more specifically the ACA 5002, may be considered as a solution to the problem of combining the DM 5004, HM 5006, and AM 5008 in the context of automated planning and decision making.

[0177] While DM 5004 may represent the basic underlying SSP for ACA 5002 to find a solution, ACA 5002 may use AM 5008 to actively generate plans that operate across multiple autonomy levels (e.g., for and using different autonomy levels). This is in contrast to autonomous agents that can adjust plans during plan execution. Actively generated plans may be subject to a set of constraints κ. ACA 5002 may use HM 5006 to predict the likelihood of various feedback signals in advance so that ACA 5002 can avoid scenarios where ACA 5002 is unlikely to operate autonomously.

[0178] System 5000 may combine all three of DM 5004, HM 5006, and AM 5008 into one decision framework. System 5000 (and more specifically ACA 5002) is used to solve the problem of generating a strategy for accomplishing its mission (eg, successfully traversing an intersection).

[0179] The problem can be formally defined as an extended SSP, the details of which are now presented. A capability-aware system (CAS) can be represented as a tuple in:

[0180] - is a set of decomposed states, each of which consists of a domain state s∈S and an autonomy level constitute;

[0181] - is a set of decomposed actions, each of which consists of a domain action a∈A and an autonomy level constitute;

[0182] - Is has and Δ |S| And each τ i : The transformation function of S×A→Δ|S|∈\Tau,

[0183] - Yes (i.e., the expected immediate cost of taking action a∈A in state s∈S), utility and human cost function The cost function of

[0184] - is the initial state, so that for a certain autonomy level as well as

[0185] - is the set of target states.

[0186] CAS Status Represents the current domain state s of the CAS and the level of autonomy l that the CAS performed in its last action. CAS Action Representing a domain action a to be performed at a given autonomy level l can change both the mechanics of how the action is performed and the type of feedback the agent receives from human authority. and It is a transformation and cost function consisting of the transformation and cost dynamics of the basic model.

[0187] A solution for a given CAS could be to combine state and level Mapping to actions and levels The space of strategies that the agent can consider is restricted by the autonomy profile κ in the following way. Let It is a set of decomposed actions The decomposition action in . Given a decomposition state set Decomposition state in If the action l∈κ(s,a), then the decomposition of the combination of states and actions can be allowed That is, if the action is allowed under the autonomy profile κ and status The combination of states and actions is allowed to be decomposed Furthermore, if for each state Allowed status and the combination of actions under strategy π (i.e. ), then policy π is allowed. Let Π denote the set of all policies. Therefore, policy π∈Π. The set of allowed policies given κ can be expressed as Π κ . You may need to The solution to the above extended SSP problem is obtained in

[0188] Since the strategy is restricted to κ , so if the autonomy profile κ is changed, the space of allowed strategies is also changed. This in turn can mean that the optimal strategy π* can intuitively be only as good as the function κ. Thus, there is a tradeoff in deciding the initial constraint (κ) on the allowed autonomy.

[0189] In implementation, a conservative approach that significantly constrains the system can be chosen, such as by setting |κ(s,a)|=1 for each (s,a)∈S×A, thereby reducing the problem complexity of solving the underlying domain model at a deterministic level. However, doing so will result in a globally suboptimal strategy with respect to the autonomy level. There is a risk, and the globally optimal policy may not be reached according to the initial autonomy profile κ.

[0190] In another implementation, a risky approach can be chosen by not constraining the system a priori at all, leaving the decision of choosing the level of autonomy entirely to the system. While an optimal policy must be included (subject to the model of ACA), this approach is naturally slower due to the larger policy space, and inherently less safe since ACA can take actions at unexpected levels, in a sense defeating the entire purpose of the model.

[0191] In yet another implementation, in most domains, the ideal initialization may be somewhere closer to the middle of the extremes described above. In scenarios where the expected cost of failure is relatively low, the autonomy profile κ may be less constrained, and in situations where the cost of failure is high, the autonomy profile κ may be more constrained. For example, in an AV, the autonomy profile κ may initially be more constrained in scenarios involving pedestrians, poor visibility, or cluttered environments (such as large intersections with multiple vehicles); however, driving along a highway is typically low risk and may benefit much less from a constrained autonomy profile.

[0192] A component of the CAS model is the skill of using what the system has learned to adjust its autonomy profile over time to optimize its autonomy by reducing unnecessary reliance on human assistance, regardless of how the autonomy profile was initialized. However, before operating at a new autonomy level, the system may not have knowledge of how humans will interact with it at that level, i.e., the feedback profile at that new level may be initialized to some baseline distribution by default. Therefore, it may be necessary for the system to explore autonomy levels that it has reason to believe may be more cost-effective than its current level, so that the system can generate the data it needs to improve the accuracy and confidence of the feedback profile at those levels.

[0193] However, allowing the system to change its own autonomy profile can lead to serious consequences in the real world if not done carefully. Therefore, embodiments disclosed herein perform gated exploration, where the system is configured to obtain permission from a human before exploring new (i.e., disallowed) autonomy levels. Thus, the system must first ask a human to update the autonomy profile to allow such exploration, thereby gating human permission to explore disallowed levels to prevent the agent from randomly performing dangerous actions.

[0194] Embodiments disclosed herein may use a variation of the ∈-greedy explore-exploit strategy, where ∈ is not fixed, but instead is proportional to the relative expected cost of taking a given action at various autonomy levels. The probability of level l' adjacent to the current level l in is proportional to the softmax of the negative q-values ​​operating at level l' over all levels adjacent to l:

[0195]

[0196] Among them, if l and l′ are in If l is adjacent or l = l′, then adj(l,l″) is 1, otherwise it is 0. To ensure efficient exploration of each allowed level, a potential-based mechanism can be implemented, where for each The potential γ is maintained and updated at each level of exploration step as follows l ∈[0,1]:

[0197]

[0198] Various properties of CAS can be used to demonstrate key results for capability-aware systems. In the example provided below, it can be assumed that there is a single human authority that interacts with the semi-autonomous system in CAS. The single human authority can be represented as Can be composed of tuples represents, where:

[0199] - When providing feedback The set of features used,

[0200] - yes The static distribution of the feedback signal follows, and

[0201] - From the state-action pair to A fixed mapping of the set of autonomy levels that will allow the autonomous agent to operate with non-zero probability.

[0202] represents the human authority’s belief in the agent’s capabilities. By definition, Will never allow The image is not included in any level.

[0203] Feedback consistency is a property of how consistent a human authority is in providing poor feedback given the same query from an agent. In the example, let Is a collection of features used by human authority And make The ground truth feedback function is a deterministic mapping if but Completely consistent. If (in but Unless otherwise stated, human authority can be assumed to be ∈-consistent for the following examples.

[0204] make is the static distribution of feedback signals followed by human authority. Given and each Complete knowledge of CAS The ability (expressed as ) is from A mapping to the optimal (minimum cost) autonomy level. Formally:

[0205]

[0206] in The feedback distribution based on human authority and real sub-capabilities For conditions, in state Take action When the optimal strategy π * The cumulative expected cost under .

[0207] Basically, if the system knows the real human feedback distribution and its own sub-capabilities, then the system is in state The ability to perform action a is the most beneficial (e.g., cost-effective) level of autonomy. When it is an ordered set, it can usually be equal to That is, the highest level of autonomy allowed by humans; however, this is not necessarily the case. In principle, the highest allowed level of autonomy may require more frequent human intervention, which may make it less efficient overall relative to lower autonomy levels. Therefore, the optimal level of autonomy is based on the expected costs across the board.

[0208] This definition of capability relies on and is thus a definition of competence over the entire human-agent system, and explicitly not just the technical capabilities of the underlying agent (i.e. and ) measure. A corollary of this fact is that a CAS is only as capable as a human authority deems it to be; a human authority with a poor understanding of the system's skills may result in a system with less capability than a human authority that knows the system's limitations and capabilities. One reason to model capability in this way may be to avoid reliance on arbitrary thresholding based on evaluation metrics to determine when a system is or is not capable.

[0209] If the expected distribution from the true Any new feedback extracted will not affect λ enough to change the and the optimal autonomy level of a∈A, then CASS can be λ-static. In some implementations, under the standard assumption, can converge to λ-staticity.

[0210] make is CAS. If for each state And for each action a∈A, The expected value of sample information (EVSI) on σ∈∑ is less than ∈ (where any ∈ is greater than 0), then is λ static.

[0211] In the first proposal, let After receiving the After t feedback signals, it is indicated A random variable where each signal is from the real distribution Then, since t→∞, the sequence The distribution converges to To prove the first proposition, since independent and identically distributed Extracting the individual signals, by directly applying the law of large numbers, the sequence will converge to This directly implies the statement.

[0212] In the second proposal, After receiving the After t feedback signals, it is indicated A random variable where each signal is from the real distribution Sampling. Since t→∞, if there is no If lacking, can converge to λ-staticity. To prove the second proposal, let And a∈A. Since and a are arbitrary, and it is assumed that there is no lack, so it is shown that due to t→∞ for Convergence to staticity is sufficient. According to the first proposal, given the assumption, will converge to make is given Under the optimal strategy The q value of Perform action a at level l. Then

[0213]

[0214] because converges in the distribution, Therefore, in the limit, after t steps The probability p t (λ) defines the point mass as is the Dirac increment function centered at . Therefore, we get

[0215]

[0216] Secondly, if under the current optimal strategy, CAS The action taken in a state is performed under the ability of the state-action pair, then CAS It is the best level in this state.

[0217] make is CAS. If the following equation holds, then Yes Status The best of the next level:

[0218]

[0219] If this is true for all states, then is level optimal. Similarly, if this is true for γ∈(0,1) in If the state is established, then It is the best at γ level.

[0220] Under certain conditions, it is possible to ensure that the ability awareness system level optimality. Therefore, the system will be guaranteed to reach a point where it operates at its true capacity in all scenarios. It can be observed that if The sub-capability set in is not empty (i.e., there are sub-capabilities that are assumed to be completely unknown a priori), then it may be necessary to have a mechanism to learn these sub-capabilities online. If there are any sub-capabilities that are not completely known a priori and the system has no means of learning, then the true capabilities of the system cannot be learned online and level optimality cannot be achieved. Therefore, it can be assumed that: given enough data, all sub-capabilities can be learned online In other words, given enough data, each estimator will converge to the true value τ in the distribution i .

[0221] To prove that a capability-aware system will reach level optimality, we can rely on the concept of gated exploration. However, we can also use the following exploit: for any given if Having reached λ staticity, it no longer explores under the exploration strategy, but instead exploits its knowledge by deterministically choosing the optimal autonomy level at that point, i.e., for a given The system will use the level This can be called exploitation under static properties. Finally, it can be observed that in practice it is not feasible to obtain an infinite number of samples as required by the formal theory. However, in practice, one can choose a sufficiently small positive value of If the expected value of the information is lower than The system can then operate as if staticity had been reached.

[0222] make is CAS, and let κ t represents the autonomy profile κ at time t. Given and a∈A, we can say that if there exists arrive At least one path where all levels along the path are ), then is from Attainable.

[0223] In the first theorem, let is a CAS that follows a gated exploration strategy and exploits static properties, where for all is reachable from κ0. Then, if there is no Lack, then due to t→∞, will converge to level optimality. To prove the first theorem, fix It can be shown that, in the limit, According to the second proposal, (all a∈A), converges to λ-staticity, and according to the assumption, each will converge to τi .therefore, The following point can be reached, at which point, in the case of ; that is, at such a time, According to the first proposal, assuming This value is exactly According to the assumption, is from reachable, so given the gated exploration strategy, there is a reachable The non-zero probability of . Therefore, due to is arbitrary, and therefore the evidence is complete.

[0224] Many problems in the open world are too complex to fully specify a priori all features that are relevant during the system's deployment, even with domain expertise. Thus, while the CAS model enables a semi-autonomous system to optimize its autonomy over time, it is limited by features in its fixed model that may appear insufficient to fully model human feedback.

[0225] For example, consider a robot deployed on a campus with the task of delivering packages to various offices in different buildings. Initially, the robot can detect doors and knows that it must seek approval or supervision before opening the door; however, since the nature of doors varies from building to building and from campus to campus, no additional information about the doors is used in its initial model. Supervising human authority can use additional features of the door, such as its size, shape, and whether the door can be pushed open. Since the CAS model does not represent these features, human feedback may appear inconsistent in state in the case of doors because the feedback is effectively normalized across all of these additional features. This, in turn, can lead to low capabilities, poor performance, and unnecessary additional reliance on humans due to missing information. To address this shortcoming, one can implement features such as Figure 6 Methods such as the method shown.

[0226] Figure 6 is a flow chart of an example of a method 6000 for providing a CAS with skills that improve its capabilities over time. At 6002, the method 6000 includes increasing the granularity of its state representation through online model updates. At 6004, the method 6000 includes identifying states that are considered to be non-discriminatory (i.e., human feedback cannot be predicted with high confidence) under the current feedback profile of the system. At 6006, the method 6000 includes determining a feature or set of features that is available to the system but not currently used that best distinguishes human feedback. This can result in a more subtle drawing of the boundaries between regions of the state space with different capability levels.

[0227] Figure 7 yes Figure 6 Figure 2 is a diagram of an illustrative example of a method. The example is an illustration of an embodiment in a navigation task of a robotic system (e.g., a robot) that can exploit two key properties of a CAS model. The first property that can be exploited can be existing information that is available in the form of human feedback in a standard CAS model to identify where new features should be added, thereby not adding additional work to the human. The second property that can be exploited can include properties of human-agent interaction to avoid the need to directly change the transition function or the reward function, thereby only directly modifying the state space. Therefore, the entire process can be performed online and completely autonomously.

[0228] Figure 7 A navigation environment 7000 is included with several corridors and doors. The doors are shown as red (R) doors, blue (B) doors, and green (G) doors, where each color represents a different type of door. The navigation environment 7000 includes several paths corresponding to optimal paths at different granularities of the state space representation. For example, path 7002 (shown in solid lines) may correspond to an optimal path for a robot to traverse using an R door. Path 7004 (shown in large dashed lines) may correspond to an optimal path for a robot to traverse using a B door. Path 7006 (shown in small dashed lines) may correspond to an optimal path for a robot to traverse using a G door. When features are identified and added to the state representation, the system can better learn to utilize human assistance and traverse paths that better match the capabilities of the system.

[0229] The embodiments described herein can determine an indistinguishable state. In the example, let is the capability-aware system. When a robotic system is deployed into the open world, both the exact environment in which the system will operate and the human authority to interact with the system may not be known a priori. Naively including all possible features available to the system from perception or external sources in the planning model may result in intractable planning in a way that does not benefit from many features that do not add useful information and only serve to increase the number of states. Therefore, assuming In its planning model, it has a complete feature space available to it, which can be divided into The activity feature space used and Inactive feature space that has not been used. However, when As additional feedback is received over time, It will learn to exploit some inactive features and thus add these features to its state representation to more efficiently align its features with those used by human authorities.

[0230] The complete feature space available (e.g., from its sensors or other external sources) can be divided into The activity feature space used and Inactive feature space that has not been used. As additional feedback is received over time, The inactive features will be learned to be more efficient in aligning with the features used by human authorities.

[0231] Given The available complete feature space F = {F1, F2, ..., F n}, the activity feature space is represented as And the inactive feature space is represented as If the activity feature space lacks the ability to correctly distinguish the states received from humans The feedback characteristics required, then the state The condition states that for at least one action, there must be no feedback signal that can be predicted with high probability under the system's current feedback profile. Under the assumptions of ∈ consistency and ground truth feedback, scenarios where the agent cannot predict feedback with high probability indicate that features may be missing from its state representation, causing the probability mass to be normalized over the remaining features in its active feature space.

[0232] In the example, the human permissions for is ∈ consistent. If there is at least one action Status is indistinguishable, where for each feedback signal σ∈∑, we have the following formula:

[0233]

[0234] Here, δ is called the discriminative slack, and is used to determine the prediction confidence required for a state to be declared indiscriminate. For example, the lower the discriminative slack is set, the higher the confidence required. The discriminative slack is used to provide a formal trade-off mechanism between increasing the complexity of the base planning model and the completeness of the capability-aware model. The determination of how to set δ can be made via expert knowledge, offline evaluation, or tuned online in a dynamic manner.

[0235] Given the concept of indistinguishable states, the central idea of ​​the method can be defined. A discriminator is any subset of the inactive feature space that helps the agent better distinguish between indistinguishable states and the features from As an example, consider an agent that initially considers only the presence of doors in its active feature space, but has additional features representing the size of doors in its inactive feature space. Without these additional features, the agent may perceive that it has received equal approvals and disapprovals from human authority, resulting in a feedback profile with low probability for any feedback signal about all doors, while The robot is always not allowed to open doors of a certain size to prevent the robot from damaging itself. By including these features representing the size of the door into the active feature space, the agent's new feedback profile is able to predict the correct feedback signal for both small and large doors with high probability.

[0236] Embodiments disclosed herein may perform iterative state space refinement. In an example, the distinguisher may be any subset If you add it to Then for some α∈(0,1), improve the accuracy of λ by at least α. The larger α is set, the stricter the requirement for including new features. Determining α may include setting it to a fixed threshold, or may be based on the value of information or other information theoretic measures.

[0237] Figure 8 8000. The single-step state space refinement algorithm 8000 represents pseudo code for improving the capabilities of CAS through iterative partitioning of the state space by adding new features to the state representation over time. Figure 8 As shown, the single-step state space refinement algorithm first identifies 8002 the current set of indistinguishable states. To avoid marking sparsely sampled state-action pairs as accidentally indistinguishable, the process can be restricted to only consider state-action pairs where: The probability of all labeled instances of this element in (conditional on the assumption that there is a true correct feedback signal returned by a human with probability of at least ∈ for each state-action pair) (called is at least some threshold p ∈ 8004.

[0238] Next, the single-step state space refinement algorithm 8000 samples 8006 the indistinguishable state and identifies 8008 the most likely discriminator for that state using any standard feature selection technique, such as minimum redundancy maximum relevance (mRMR). For each potential discriminator, a portion of the full data set is used to train 8010 a new feedback profile, where the discriminator is temporarily added to the active feature set. The discriminator that results in the best performing feedback profile is selected 8012. In an example, the discriminator that results in the best performing feedback profile may be the highest Matthews coefficient. If validation is successful, the discriminator is added to the active feature set and the system is updated 8014.

[0239] Two assumptions may be made in the design and use of the single-step state-space refinement algorithm 8000. First, it may be assumed that the initial transition function provided in the domain model is Any scenario that allows the agent to act autonomously is correct enough. In some examples, the ability can be improved by iteratively refining the state space. The ability can also be increased by directly updating the transition function and replanning, because the human authority improves its understanding of the agent's capabilities.

[0240] Second, it can be assumed that human authorities have sufficient understanding of the capabilities of the agent to prevent the execution of actions that the agent cannot successfully perform and also provide consistent feedback. This assumption can be made for two reasons. First, there can be different ways to improve the authority's understanding of the system's capabilities so that there is appropriate trust or reliance on the system. These can include pre-deployment training of the system, standardized feedback criteria, and expertise. Second, identifying potential failures and recovering from failures are separate areas of activity research that are orthogonal to the examples described in this article.

[0241] Under these assumptions, there may not be a need to directly update the domain model’s transformations or reward function at any point. It is sufficient for the agent to be able to distinguish between actions it is capable of performing autonomously and those that require human involvement.

[0242] Adding a discriminator will not cause a discriminant state to become indiscriminate. Any given discriminant state will either be affected by the discriminator or it will not be affected by the discriminator. If the state is not affected, then the feedback profile of the state will not change. If the state is affected, then by definition the initial state in question no longer exists. More importantly, it is ensured that each state is ultimately correctly discriminated given a sufficient set of features.

[0243] The following theorem states that if every feature that humans use to determine their feedback is available to the robot, then there must be a point in time when the robot has completely distinguished all states, and beyond that point, no state will become indistinguishable. Let I t is the number of indistinguishable states at time t, and let It means that you have received After t feedback signals A random variable where each signal is from the real distribution Sampling. If is ∈ consistent, δ>0 and no If it is missing, then there exists for all t′>t * And l t′ =0 for some t * >0.

[0244] To prove the above theorem, first observe that due to If exists , then according to the first proposal above, the sequence converges in the distribution, Therefore, there is of a certain * > 0, at which point it is clear that no state will be indistinguishable under δ. Therefore, for the claim to be false, it must be that for every t>0, Choose a sufficiently large t for which there is an indistinguishable state There is a subset It is Since this is true for all t>0 and All of them hold true, so we can achieve t * ,in So complete, or This contradicts the assumption.

[0245] Fig. 9 is an example of a flow chart of a technique 9000 for capability-aware decision making according to an embodiment of the present disclosure. The technique 9000 may be implemented by a capability-aware system (CAS) such as Figure 5 Technology 9000 can be implemented by autonomous cognitive agents such as Figure 5 Some or all of the operations of Technique 9000 may be performed by SSOCEM 4400 or by Figure 4 The other components of the autonomous vehicle operation management system 4000 are implemented.

[0246] Flowchart 9000 illustrates how a CAS or an ACA of a CAS may select actions, receive feedback, and update model(s) as a result of experience(s). Figure 7 Describe flow chart 9000.

[0247] Fig.10 It is used to illustrate Fig. 9 1 is a diagram of a scenario 10000 of autonomous driving using the technology 9000. The scenario 10000 includes an intersection 10002. A vehicle 10004 is approaching the intersection 10000. Stop line 10006 (i.e., a stop sign) is a line where the vehicle 10004 should stop to ensure that it can safely travel along a trajectory 10008 (i.e., a path). The vehicle 10004 may be Figure 1 The vehicle 1000 can be Figure 2 One of the vehicles 2100 / 2110. Vehicle 10004 may include an autonomous vehicle operation management system, such as Figure 4The vehicle 10004 may include a capability awareness system (CAS), such as Figure 5 The system 5000 of FIG. 10004 may be an autonomous vehicle or may be a semi-autonomous vehicle. A human operator (e.g., a teleoperator or an in-vehicle operator) may be responsible for (remotely) monitoring and assisting the vehicle 10004, such as in response to the vehicle 10004 issuing a remote request for assistance from the teleoperator.

[0248] Regarding intersection scenarios (such as Fig.10 The intersection 10002, etc.) related domain models (such as Figure 5 DM5004, etc.) to describe flowchart 9000.

[0249] Fig.10 The domain model of the intersection scene may include abstract (e.g., symbolic) information extracted based on sensor (LiDAR, radar, camera, etc.) information. For example, the domain model may include information such as the location of vehicle 10004 (e.g., "approaching the stop line of the intersection"), other related world objects (e.g., "the vehicle on the left" as vehicle 10010), road configuration (e.g., "east-west road with through traffic", meaning there are no traffic signals, stop signs, yield signs, etc.), other relevant information of the scene (e.g., "the object on the left has not moved for a long time", "the left sensor is blocked", etc.), etc.

[0250] It is understood that when the vehicle 10004 is a certain distance away from the obstruction, the identified obstruction or occluding object (such as vehicle 2010, etc.) can be identified as such. However, when the vehicle 10004 is close to the obstruction or occluding object, it can be determined that the obstruction or occluding object is not such. This may be due to noise in the sensor data. Therefore, in Fig. 9 At 9002, the state associated with the scene is modified and a new plan is calculated.

[0251] More generally, a scene can be described in an abstract state space and / or as a combination of objects detected and associated with the scene. The state can include each object necessary for successful autonomy or at least semi-autonomy. For each of these objects, a state space of the AV (e.g., vehicle 10004) relative to that object (e.g., vehicle 10010) can be maintained. The combination of all state spaces together can form (e.g., constitute, become, etc.) a domain space for which capability modeling is performed. Thus, capabilities are modeled with respect to each entity-to-entity pairing of the AV to another entity associated with the scene. For example, if the scene 10000 includes a relevant pedestrian, capability modeling (i.e., the level of autonomy is determined) is determined separately for the vehicle 10004 to pedestrian pairing.

[0252] The states used in the domain model or used to evolve the system can be specified at different levels of granularity. To illustrate, and as non-limiting examples, the domain model can be for a specific intersection 10002; the domain model can be all similarly configured intersections in any geographic area of ​​interest; the domain model can involve crossing one or more intersections at a certain time of day; the domain model can include additional metadata information, such as the location of the intersection includes proximity to a venue and a concert has just ended; etc.

[0253] Flowchart 9000 is described for action set A. Action sets are domain dependent. For the domain of an intersection, the actions may include the actions "go", "stop", and "pull over". However, the action set may include more, fewer, or a combination of other actions. In other scenarios, the action set may be a different action set. For example, for a passing scenario, the actions may include the actions "follow", "stop", "pass left", and "pass right".

[0254] The action "go" may mean to go along trajectory 10008. The action "stop" means that the vehicle should remain stationary in the next step / action step, or stop if it is moving. The action "pull over" may mean that the vehicle 10004 should pull over slightly from a stop position (such as stop line 10006, etc.). The "pull over" action may be useful when, for example, the sensors of the vehicle 10004 are obstructed in the current position of the vehicle 10004. For example, in scene 10000, the field of view of the left sensor of AV10004 is obstructed by vehicle 10010. Therefore, and as a human driver may do, the "pull over" action causes the vehicle 10004 to move slightly forward into the intersection in an attempt to see outside the vehicle 10010.

[0255] For a set of four autonomy levels To describe flowchart 9000.

[0256] Autonomy level 10 may indicate "no autonomy," which may mean that direct human assistance is required and the ACA of the vehicle 10004 has determined to be incapable of performing the determined action, if any.

[0257] Autonomy level 11 may indicate "verified autonomy," which may mean that the ACA of the vehicle 10004 will ask for and receive human (e.g., teleoperator) approval before performing the selected action. Assistance is required because the ACA of the vehicle 10004 has determined that it is not capable of performing the determined (e.g., selected) action completely autonomously.

[0258] Autonomy level 12 may indicate "supervised autonomy," which requires human presence (eg, remote monitoring) and is available for intervention (eg, overriding the system) in the event of a failure while the selected action is in progress.

[0259] Autonomy level 13 may indicate "unsupervised autonomy," which may mean that the selected action may be performed (ie, carried out) without human intervention, supervision, monitoring, etc. That is, autonomy level 13 indicates that the ACA is fully capable of carrying out the action.

[0260] Flowchart 9000 is described for a feedback signal set Σ including four feedback signals (i.e., set Σ = {no feedback, approval, disapproval, override}). For ease of reference, the "no feedback", "approval", "disapproval" and "override" signals may be represented as and Therefore, the feedback signal set Σ can be As the label suggests, “no feedback” (e.g. ) may mean that the ACA of vehicle 5004 did not receive any explicit feedback from a human; “approval” (e.g., ) may mean human approval of the action selected by the ACA; “disapproval” (e.g., ) may mean that a human does not approve of the action selected by the ACA, which in turn may mean that no action is taken and the ACA should replan another action; and “override” (e.g., ) can mean that a human takes over manual operation of vehicle 10004.

[0261] Additionally, flowchart 9000 assumes that feedback "approval" and "disapproval" can only be received at autonomy level l1 (ie, "verified autonomy"), and feedback "override" can only be received at autonomy level l2 (ie, "supervised autonomy").

[0262] Given and Σ, the state transition function T of the CAS can be specified. Given and It can be defined as follows:

[0263]

[0264] In equation (7), [·] represents an Iverson bracket. Equation (7) can be summarized as follows: when the ACA operates at autonomy level l0 (i.e., "no autonomy"), the ACA can follow the transition dynamics of the human (e.g., teleoperator) who takes control; when the ACA operates at autonomy level l1 (i.e., "verified autonomy"), the probability that the vehicle 10004 reaches state s' is the probability that the ACA is approved to take an action multiplied by the probability that the ACA succeeds in following T plus the probability that the action is not approved and the state remains the same; when the agent operates at autonomy level l2 (i.e., "supervised autonomy"), the probability that the vehicle 5004 reaches state s' is the probability that the ACA succeeds in following T without any human intervention plus the probability that the human overrides the action selected by the ACA and the human brings the vehicle to that state (i.e., ); and when the ACA operates at autonomy level l3 (ie, “unsupervised autonomy”), the ACA follows the transformation dynamics of its own model (eg, domain model).

[0265] According to formula (8), the state and actions The associated costs

[0266]

[0267] Formula (8) shows that in the extended SSP, Take action Cost = equal to the domain cost of taking action a in state s (e.g., time + energy to make a left turn at an intersection) plus the cost of taking action a given that the system is just in state The utility of performing action a at autonomy level l′ in the case of The human cost of performing action a at autonomy level l′ (this cost could be 0, such as if autonomy level l′ does not involve humans, etc.) in the case of seeking teleoperation. is the sum of the different cost function components of CAS. More complex aggregation functions related to C, μ, and ρ can be used, especially in multi-purpose problem settings.

[0268] Having described a model of a capability-aware system or an autonomous cognitive agent therein, the technique 9000 can be summarized as including three phases: an action selection phase, an action execution phase, and a model update phase. In the action selection phase, an action and an autonomy level are selected. In the action execution phase, actions are executed according to the autonomy level, and feedback, such as from a human operator, may be received. In the model update phase, the technique 9000 is updated with feedback models (e.g., Figure 5 More specifically, the technique 9000 updates the feedback profile λ and / or the human body state transition function

[0269] At 9002, technique 9000 can detect a vehicle (such as Fig.10 The state s of the world around the vehicle 10004, etc.). The state s is the domain model (e.g., Figure 5 The status of the DM 5004, etc. can be determined using the vehicle's sensors (such as Figure 1 One or more than one of the sensors 1360, etc.) to detect state s.

[0270] At 9004, the detected state s may be used to select a strategy π. As described above, strategy π may be constrained by an autonomy profile κ, which may include one or more constraints. For example, if the autonomy level selected by the ACA is not in the autonomy profile κ, the ACA may reprogram to select a new action and a new autonomy level allowed by the autonomy profile.

[0271] A corresponding capability probability can be associated with each autonomy level. That is, a capability probability can be associated with taking an action in a particular state.

[0272] For example, a 10% capability probability associated with choosing to perform an action (e.g., "go forward") autonomously (i.e., under "unsupervised autonomy") in a particular state may mean that there is a 10% chance that the action will be performed autonomously in the particular state at the correct level of autonomy. Therefore, if the action is performed autonomously, but the resulting state is a bad state, there may be a probability of 0.1 that a large penalty may be incurred. In contrast, there may be a probability of 0.9 that the capability level will be correctly selected. Therefore, capabilities are fused into the transition of states (the probability of going from s at time T to s' at time T+1); and therefore knowing that capabilities are fused therein, when evaluating rewards over time, the system will autonomously choose whether to perform all calculations of similar risks as Actuarial Sciences. The system essentially performs a huge risk assessment of all of these probabilities over time and determines at what point it should be autonomous and when it should seek help and when it should seek approval.

[0273] The capability probability may be considered (eg, used, etc.) when calculating the policy π. The capability probability may be considered in a state transition from a first state s1 at time T to a second state s2 at time T+1.

[0274] The corresponding cost can be associated with the selected autonomy level. In the example, the cost (e.g., reward) can be in units of time. However, other units of measure can be used. It takes a long time to seek help from humans, for example because the teleoperator needs to first gain situational awareness of the situation, analyze the situation, take control of the AV, and actually control the AV to perform actions. Such an interaction may take, for example, 60 seconds, which can be considered a huge cost. In the case of autonomy level l3 (i.e., "unsupervised autonomy"), no delay is incurred. Therefore, when calculating the strategy π, the cost of selecting different autonomy levels that involve humans in different amounts to different degrees in terms of time or some other unit can be considered (e.g., using, etc.).

[0275] At 9006, technique 9000 can output action a and autonomy level l. Thus, at 9004, a tuple (a, l) can be output. Autonomy level l indicates how the ACA of vehicle 10004 performed action a. Examples of interpreting the tuple (a, l) include "ACA selected the stop action; however, ACA will seek assistance as to whether stopping is the right action" or "ACA selected the forward action and will proceed because ACA is fully capable."

[0276] If the autonomy level is 10 (ie, "no autonomy"), the technique 9000 proceeds to 9014. For example, a human operator may remotely control the vehicle 10004 or may send commands to the vehicle 10004 to perform an action. After the manual operation at 9014 is completed, the technique 9000 proceeds to 9016.

[0277] If the autonomy level is l1 (i.e., “verified autonomy”), technique 9000 proceeds to 9008. In a simple example, the ACA of vehicle 10004 may determine that because the field of view is obstructed by vehicle 10010, the action that should be taken is to “pull over” forward toward the middle of intersection 10002. However, before completing the action, vehicle 10004 may forward the action to a human (e.g., a teleoperator) for approval. At 9014, the teleoperator may send a “disapproved” feedback signal (and the ACA may receive a “disapproved” feedback signal), which causes the autonomy level to switch to “no autonomy”. The ACA may receive an “approved” feedback signal. In the event of an “approved” feedback signal, technique 9000 proceeds to 9010 to perform the selected action.

[0278] If the autonomy level is 12 (i.e., "supervised autonomy"), the technique 9000 proceeds to 9010. At 9010, before attempting the selected action, the ACA of the vehicle 10004 can ensure that the teleoperator is monitoring the vehicle 10004 while the action is being performed. While the ACA is performing the selected action, at 9012, the ACA can complete the action without receiving any feedback from a human. Thus, after the action is completed, the technique 9000 proceeds to 9016. On the other hand, while the action is being performed, at 9014, the ACA can receive an override feedback signal, which causes the autonomy level to transition to a "no autonomy" level.

[0279] If the autonomy level is l3 (ie, “unsupervised autonomy”), the action is performed autonomously by vehicle 5004 and technique 9000 proceeds to 9016.

[0280] At 9016, technique 9000 updates the feedback profile λ and the human state transition function based on the feedback (which may be the received feedback) or the absence of feedback.

[0281] How the feedback profile λ is updated may depend on the classifier being used. However, in general, the feedback profile λ may be updated by augmenting the dataset of received human feedback signals and retraining the classifier (decider) on the new data.

[0282] When the human takes over control, the human state transition function can be updated by observing what new state the ACA ends up in and usually taking the maximum a posteriori estimate In practice, this can be accomplished by simply implementing the ACA in a state where it and intend or attempt to take action But humans eventually take over control by counting the frequencies of each state.

[0283] The model can be updated using model-free reinforcement learning, model-based reinforcement learning, or some other learning technique. In model-free reinforcement learning, the probability value can be adjusted up or down. In model-based reinforcement learning, the probability value can be calculated based on the count of successfully reaching the goal state in a scenario compared to all the times the scenario was encountered.

[0284] In the example of reinforcement learning, an estimate of how much time (e.g., cost) it takes for an ACA to reach its goal state (e.g., through an intersection) is maintained. Such times can be maintained with different levels of autonomy. The probabilities can be adjusted downward in the case of not reaching the goal state (e.g., not crossing an intersection), requesting human assistance, human assistance, or overriding the ACA, etc. The probabilities can be adjusted upward in the case of "unsupervised autonomy" or "no feedback." That is, the probabilities can be constantly adjusted up and down for all different combinations of states.

[0285] In some implementations, technique 9000 may proceed from 9016 to 9018; in other implementations, technique 9000 proceeds back to 9002. That is, block 9018 may be an optional block.

[0286] At 9018, technique 9000 can engage in gated exploration (GE).

[0287] As described above, a fundamental component of CAS is that the system can use what it has learned to adjust the autonomy profile κ over time to optimize its autonomy by reducing unnecessary reliance on humans. However, prior to operating at a new autonomy level, the ACA may not have knowledge of how humans will interact with the ACA at that autonomy level. Therefore, the feedback profile at the new autonomy level may be uniformly random, since no data has been received. As a result, it is necessary for the CAS to 'explore' autonomy levels that the CAS has reason to believe may be more cost-effective than its current level. Therefore, the CAS can generate the data needed to improve the accuracy and confidence of the feedback profile at these levels.

[0288] However, any kind of random or pseudo-random exploration (e.g., random exploration, ∈-greedy exploration) may lead to frequent failures, which may have serious consequences in the real world (e.g., collisions, causing traffic jams, etc.). To this end, a simple extension to traditional exploration methods (referred to as gated exploration in this article) can be used.

[0289] In gated exploration, the CAS may still follow a random or pseudo-random exploration strategy to attempt actions at autonomy levels not permitted by the autonomy profile κ. When this occurs, instead of simply performing the action, the ACA may first request human permission for the ACA to change the autonomy profile κ so that the autonomy level the ACA wants to try is now permitted.

[0290] In the reinforcement learning literature, in the domain-unknown situation, the agent must compromise between exploiting the information it has and simply taking the action that performed best in the past, and exploring new actions and new states (or actions and states that were only suboptimal in the past) that may turn out to be better.

[0291] This paper exploits this idea. In the model described in this paper, it is assumed that the optimal autonomy level is not known a priori, otherwise the ACA would simply be designed to proceed at that level. Therefore, the ACA must learn over time what the optimal level is. However, since the ACA is also operating in the real world while learning, the ACA must make the same trade-offs. That is, given the history of human feedback signals that the ACA has received, the ACA must either operate at the current optimal autonomy level, or the ACA tries a new, possibly higher autonomy level, for which the ACA has no feedback signal or has a limited feedback signal. This is the 'exploration' part of gated exploration (GE).

[0292] However, this would defeat the purpose of the model described herein, to allow an ACA to operate at an autonomy level that the ACA is not allowed to operate at simply to try and see if that works out better. Therefore, when an ACA wishes to change the highest autonomy level it can operate at, the ACA must first ask a human to change its allowed autonomy level. This is the 'gating' part of Gated Exploration (GE).

[0293] It is worth noting that while the ACA is generally expected to be exploring higher autonomy levels, this is not necessary given the assumption of a conservative initial model, and the agent can actually explore downwards. That is, the ACA can ask the human to actually lower the maximum autonomy level at which it can act in certain scenarios. Doing so will force the ACA to act at an autonomy level that the ACA would not normally act at in other situations, allowing the ACA to acquire more data at that level and improve the quality of its model.

[0294] From 9018, technology 9000 enters back to 9002 to repeat the above operations, whereby technology 9000 detects the current state of the world, selects the actions to be performed and the level of autonomy, etc.

[0295] To test the capability awareness system, the CAS model can be implemented in two simulated autonomous vehicle domains at different levels of abstraction. The first domain can be a high-level navigation problem, where the autonomous vehicle must plan and execute the optimal route to take between two locations, given its knowledge of different intersections and streets and its own ability to perform different maneuvers that it could have performed in the previous domain (i.e., passing an obstacle that is blocking its lane).

[0296] In the autonomous vehicle navigation domain, the autonomous vehicle operates in a known map represented by a directed graph G = (V, E), where each vertex v∈V represents an intersection and each edge e∈E represents a road. The task of the autonomous vehicle is simply to navigate the map from the start node to the goal node. Each vertex state v∈S is represented by the tuple<ID,p,o,v,θ> , where ID is the vertex ID, p is a Boolean value indicating the presence of a pedestrian, o is a Boolean value indicating the presence of an occluder, v is the number of other vehicles (if any), and θ is the heading of the vehicle. Each edge state e∈S is represented by the tuple<u,v,l,θ,o> , where u and v are the IDs of the start and end states of the edge, respectively, l is the number of lanes along the edge, θ is the direction of travel, and o is a Boolean value indicating the presence of an obstacle blocking the AV's lane. In addition, each edge is associated with a length and speed of travel. Model parameters such as the probability of encountering a pedestrian at a vertex or an obstacle on an edge are given as part of the model input.

[0297] In vertex states, the agent can Go Straight, Turn Right, Turn Left, U-Turn, or Wait. All maneuvers are assumed to succeed deterministically. In edge states, the agent can Continue, Overtake an obstacle, or Wait. Assume that Overtake succeeds with probability [0.2, 0.5, 0.8] depending on the number of lanes, and Continue fails deterministically when an obstacle is present, otherwise transitions the agent to the end vertex of the edge with probability p∝speed(e) / length(e). Each action has unit cost in the domain model.

[0298] The autonomy profile κ is initialized to L in edge states with no obstacles and in node states with no pedestrians, occluders, other vehicles, or when the action Wait is taken. In all other cases, κ is initialized to {l0,l1}. The feedback profile λ is initialized to be uniformly random over the possible feedback signals. There is a cost of 8.0 for humans for operating at l0, as humans need to manually control the vehicle, a cost of 3.0 for operating at l1, a cost of 1.0 at level l2, and no additional cost for humans when operating at l3. The system incurs a cost of 1.0 when receiving a negative response at l1, and a cost of 3.0 when receiving an override at l2.

[0299] In the autonomous vehicle obstacle negotiating domain, an autonomous vehicle must pass an obstacle that blocks its lane on a single-lane road and, importantly, must move into the oncoming traffic lane in order to do so. A state s∈S is represented by the tuple<p,o,t,d,w> , where p is the position of the vehicle (0 to 4), o is the position of the nearest oncoming vehicle (1 to 3, 0 if none, -1 if unknown), t indicates the presence of a trailing vehicle, d indicates whether the obstacle is dynamic (e.g., a slow-moving tractor) or static (i.e., gravel or a stalled car), and w indicates whether the nearest oncoming vehicle has stopped.

[0300] The autonomous vehicle can take the following actions: Wait, Edge, and Go. Edge advances the AV's position by 1 with probability 0.5, unless at position 0 (in which case Edge advances the AV's position deterministically to position 1 for the pull-over position). Go deterministically advances the AV's position by 1 in all positions except position 0, in which case Go advances the AV to position 2 (i.e., skipping the pull-over position). All actions have unit cost, unless the AV and the oncoming vehicle share a position or intersect positions, in which case a high-cost collision occurs.

[0301] The autonomy profile κ is initialized to {l2} in all cases; i.e., in such a safety-critical domain, it is expected that initially the human is always aware and ready to override the system. The feedback profile λ is initialized to uniform random as described above. The human incurs a large cost of 10.0 when the CAS operates at l0 but is assumed to successfully complete the maneuver (i.e., the human does not hand back control in the process of passing the obstacle), a cost of 1.0 when supervising at l2, and no cost at l3. When overridden by a human, the system receives a large penalty of 12.0.

[0302] To evaluate the iterative state space refinement approach, a simulation domain involving a mobile robot tasked with delivering packages to different rooms in various buildings across a small campus can be implemented. To achieve its goal, the robot must contend with two major obstacles (doors and crosswalks), which can lead to costly failures if the agent does not cope with these obstacles correctly. In this case, CAS may be missing certain features from its initial domain model that humans use when determining feedback.

[0303] In the delivery robot domain, the robot operates in a known map and is tasked with delivering a package from one office to another in a campus environment. The robot must safely navigate the environment, which includes closed doors and crosswalks across major roads. A state s∈S is represented by the tuple<x,y,θ,o> , where: x, y, and θ are the robot's pose, and o represents the presence of a door, traffic conditions (if at a crosswalk), or nothing if there are no obstacles at the robot's current position. Additional information is available from sensor information for individual obstacles but is not used a priori in the domain model. For doors, this information includes door color, height, width, and opening type (i.e., pull or push). For crosswalks, this information includes whether there are obstacles blocking visibility and whether the street is one-way or two-way. Additionally, the time of day is always known to the robot.

[0304] The robot can perform the following actions: Move, Open, Wait, Cross, and Drop-Off. Move causes the agent to move forward in the direction of its heading. Open does nothing if the robot is not at a door, but otherwise, Open deterministically opens the door if the robot is capable, otherwise causes the robot to incur a penalty (Note: the initial model of the robot does not differ and assumes that all doors can be opened). Wait does nothing unless the robot is at a crosswalk, in which case there is a chance that traffic conditions have changed. Cross does nothing unless the robot is at a crosswalk, in which case the robot deterministically crosses if there is no traffic (empty), successfully crosses with a 50% probability if there is light traffic (otherwise stays in the same position), and has a 10% chance of crossing, a 10% chance of colliding and causing a dead end, and an 80% chance of staying in the same position if there is heavy traffic.

[0305] When the robot is operating before dropping off its package, a unit negative reward is incurred at each time step. If the robot attempts to open certain kinds of doors that it is not capable of opening, it may damage itself and incur a small penalty (-10). If the robot collides with a vehicle while crossing a street, it incurs a very large penalty (-100).

[0306] The autonomy profile κ is initialized to L in the absence of obstacles or when the action Wait is in progress, and to {l0,l1} otherwise. The feedback profile λ is initialized to uniform random as described above. There is a cost of 10.0 for operating at level l0, a cost of 2.0 for operating at level l1, a cost of 1.0 for operating at level l2, and no additional cost at level l3. No additional cost is incurred in this domain for the feedback received.

[0307] Fig.11 is a flow chart of an example of a technique 11000 for autonomous driving of an autonomous vehicle (AV) according to an embodiment of the present disclosure. Fig.11 The technology 11000 can be Figure 5 The technology 11000 can be implemented in an AV, which can be Figure 1 The vehicle 1000 shown, Figure 2 One of the vehicles 2100 / 2110 shown, a semi-autonomous vehicle, or any other vehicle that enables autonomous driving.

[0308] At 11110, the technique 11000 detects an environmental state based on the sensor data. In an example, the environmental state may be a decomposed state as described above with respect to the extended SSP problem. In an example, the state may be a state s in a state set S. Fig. 9 The detection status is as described in 9002.

[0309] At 11120, technique 11000 selects an action based on the state of the environment. Fig. 9 Select the action as described in 9006.

[0310] At 11130, technique 11000 identifies the current set of indistinguishable states. To avoid labeling sparsely sampled state-action pairs as indistinguishable by chance, the process can be restricted to only consider state-action pairs where all labeled instances of that element in the existing dataset D have been observed (conditional on the assumption that there is a true correct feedback signal returned by a human with a probability of at least ∈ for each state-action pair) with a probability of at least some threshold p ∈ .

[0311] At 11140, technique 11000 samples an indistinguishable state from the set of indistinguishable states and uses a feature selection technique (such as mRMR, etc.) to identify one or more possible discriminators for the state.

[0312] At 11150, for each potential differentiator, a new feedback profile may be trained using a portion of the full data set with the differentiator temporarily added to the active feature set. If validation is successful, the differentiator is added to the active feature set and the system is updated.

[0313] At 11160, the technique 11000 determines the autonomy level associated with the environmental state and the action. Fig. 9 As described in 9004, the autonomy level can be selected based on at least an autonomy model and a feedback model. The autonomy model can be as described in relation to Figure 5 Thus, in an example, the autonomy model may include a utility model and an autonomy profile. The utility model may describe the utility of performing a first action at a first autonomy level for a first environmental state given that the AV has operated at a second autonomy level. The autonomy profile may map corresponding environmental states to corresponding actions and may specify constraints on the allowed autonomy levels for a particular environmental state. The feedback model may be as described with respect to Figure 5 as described in HM 5006.

[0314] In an example, the autonomy level may be selected from a set of autonomy levels including a first autonomy level indicating "no autonomy", a second autonomy level indicating "verified autonomy", a third autonomy level indicating "supervised autonomy", and a fourth autonomy level indicating "unsupervised autonomy".

[0315] At 11170, technology 11000 acts according to the autonomy level.

[0316] In the example of 11170, the autonomy level can be a second autonomy level indicating "verified autonomy," and acting according to the autonomy level can include receiving an approval feedback signal or a disapproval feedback signal for the action. The feedback signal can be received from a human. In an example, the human can be a human in a vehicle. In an example, the human can be a teleoperator. In an example, acting according to the autonomy level can include querying for an approval feedback signal before receiving the approval feedback signal.

[0317] In the example of 11170, the autonomy level can be a third autonomy level indicating "supervised autonomy," and acting according to the autonomy level can include determining that the AV is being monitored by a human before acting. For example, the technology 11000 can receive a signal from a human indicating that the human is monitoring the AV.

[0318] In the example of 11170, and as with respect to Fig. 9 As described, an approval feedback signal may be received, and taking action according to the autonomy level may include: determining that the AV is being monitored; taking action in response to determining that the AV is being monitored; receiving an override signal; and in response to receiving the override signal, ceasing the action and switching to a manual operation mode of the AV.

[0319] In the example of 11170, performing an action according to the autonomy level may include determining whether to request approval of the action from a human before performing the action based on the autonomy level.

[0320] In an example, the autonomy level may be "no autonomy" so that Fig. 9 As described in 9014, the AV is not allowed to perform autonomous actions, and taking actions according to the autonomy level may include enabling the AV to be manually controlled by a human.

[0321] In an example, technique 11000 may include updating at least one of an autonomy profile, a feedback profile, and a human transition function in response to performing an action. Updating each of the autonomy profile, the feedback profile, and the human transition function of the AV may be based on feedback signals received by the AV itself. However, the updates may be based on feedback signals received by many AVs (such as AVs in a queue, etc.). For example, all signals sent to the AVs may be aggregated at some central location, the (one or more) models updated, and the updated models redistributed to the AVs. However, other ways of updating the models based on feedback signals received by (i.e., sent to) multiple AVs are possible.

[0322] As described above, aspects of the disclosed implementations may include a system including a memory (such as Figure 1 1340, etc.) and a processor (such as Figure 1 The memory may include instructions that may be executed by the processor to compute a policy for solving a task by solving an extended stochastic shortest path (SSP) problem as described above with respect to the extended SSP problem. The policy may map environmental states and autonomy levels to actions and autonomy levels. Computing the policy may include generating a plan that operates across multiple autonomy levels.

[0323] In an example, generating a plan for operating across multiple autonomy levels may include generating a plan subject to constraints on allowed autonomy levels in corresponding states, as described above with respect to autonomy profile κ.

[0324] In an example, the instructions may include instructions for updating a feedback profile. The feedback profile may represent a first probability that the system receives a first signal when performing a first action at a second autonomy level, given that the system is in a first state and the system has operated at a first autonomy level.

[0325] In an example, the instructions may include instructions for updating a human state transition function. The human state transition function may represent a second probability that the human transitions to a second state of the environment model given the system selects to perform a second action in the first state and the human assumes manual control.

[0326] In an example, the instructions may include instructions to update an autonomy profile. The autonomy profile may define a set of acceptable autonomy levels given a current state and an action to be taken next.

[0327] As used herein, the term "instruction" may include instructions or expressions for performing any method disclosed herein or one or more than one arbitrary part thereof, and may be implemented in hardware, software, or any combination thereof. For example, an instruction may be implemented as information stored in a memory (such as a computer program, etc.), which may be executed by a processor to perform any of the various methods, algorithms, aspects, or combinations thereof as described herein. An instruction or a portion thereof may be implemented as a dedicated processor or circuit, which may include dedicated hardware for performing any of the methods, algorithms, aspects, or combinations thereof as described herein. In some implementations, a portion of an instruction may be distributed across multiple processors on a single device or multiple devices, which may communicate directly or across a network (such as a local area network, a wide area network, the Internet, or a combination thereof, etc.).

[0328] As used herein, the terms "example," "embodiment," "implementation," "aspect," "feature," or "element" indicate use as an example, instance, or illustration. Unless explicitly indicated otherwise, any example, embodiment, implementation, aspect, feature, or element is independent of each other example, embodiment, implementation, aspect, feature, or element and can be used in combination with any other example, embodiment, implementation, aspect, feature, or element.

[0329] As used herein, the terms "determine" and "identify" or any variations thereof include selecting, ascertaining, calculating, searching, receiving, determining, establishing, obtaining, or otherwise identifying or determining in any manner using one or more of the devices shown and described herein.

[0330] As used herein, unless otherwise specified or clear from context, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." Additionally, the articles "a" and "an" as used in this application and the appended claims should generally be understood to mean "one or more than one," unless otherwise specified or clear from context to direct to the singular.

[0331] In addition, for simplicity of explanation, although the accompanying drawings and description herein may include the sequence or series of steps or stages, the elements of the method disclosed herein may occur in various orders or in parallel. Additionally, the elements of the method disclosed herein may occur with other elements that are not clearly presented and described herein. In addition, all elements of the method described herein may not be required to realize the method according to the present invention. Although aspects, features and elements are described in this article with specific combinations, each aspect, feature or element may be used independently, or may be used in various combinations with other aspects, features and elements or in various combinations without other aspects, features and elements.

[0332] In order to make it easy to understand the present invention, the above aspects, examples and implementations are described, but the present invention is not restrictive. On the contrary, the present invention covers various modifications and equivalent arrangements included within the scope of the appended claims, and the scope should be given the broadest interpretation to cover all such modifications and equivalent structures permitted by law.

Claims

1. A method for autonomous driving of an autonomous vehicle (AV), comprising: Detect environmental conditions based on sensor data; selecting an action based on the state of the environment; Identify the current set of indistinguishable states; identifying a distinguisher from the set of current indistinguishable states; Training a feedback model for the discriminator; determining an autonomy level associated with the environmental state and the action, wherein the autonomy level is selected based on at least an autonomy model and the feedback model; and The action is performed according to the autonomy level.

2. The method according to claim 1, wherein: The autonomy level is selected from a set including: a first autonomy level indicating "no autonomy", a second autonomy level indicating "verified autonomy", a third autonomy level indicating "supervised autonomy", and a fourth autonomy level indicating "unsupervised autonomy".

3. The method according to claim 2, in, The autonomy level is the second autonomy level indicating "verified autonomy", and wherein taking the action in accordance with the autonomy level comprises: For the action, an approval feedback signal or a disapproval feedback signal is received.

4. The method according to claim 3, wherein: Performing the actions according to the autonomy level includes: The approval feedback signal is interrogated prior to receiving the approval feedback signal.

5. The method according to claim 3, in, said autonomy level is said third autonomy level indicating "supervised autonomy", and Wherein, performing the action according to the autonomy level includes: Before performing the action, it is determined that the AV is being monitored by a human.

6. The method according to claim 3, in, the approval feedback signal, and Wherein, performing the action according to the autonomy level includes: Determining that the AV is being monitored, In response to determining that the AV is being monitored, performing the action; receiving an override signal; and In response to receiving the override signal, the action is stopped and a switch is made to a manual operation mode of the AV.

7. The method according to claim 1, wherein: Performing the actions according to the autonomy level includes: A determination is made based on the autonomy level whether to request approval of the action from a human before performing the action.

8. The method according to claim 1, in, The autonomy level is "no autonomy" such that the AV is not allowed to perform autonomous actions, and Wherein, performing the action according to the autonomy level includes: Enables the AV to be manually controlled by a human.

9. The method according to claim 1, in, The autonomy model includes a utility model and an autonomy profile, wherein the utility model describes the utility of performing a first action at a first autonomy level for a first environmental state given that the AV has operated at a second autonomy level, and Therein, the autonomy profile maps corresponding environmental states to corresponding actions and specifies constraints on the allowed autonomy level for specific environmental states.

10. The method according to claim 1, further comprising: At least one of an autonomy profile, a feedback profile, and a human transition function is updated in response to performing the action.

11. A system for autonomous driving, comprising: Memory; as well as a processor configured to execute instructions stored in the memory to: The strategy used to solve the task is calculated by solving the extended random shortest path problem, namely the extended SSP problem. identifying a distinguisher from the current set of indistinguishable states; and Training a feedback model for the discriminator; wherein the policy maps environmental states and autonomy levels to actions and autonomy levels, and Wherein, calculating the strategy comprises: Generate plans that operate across multiple autonomy levels.

12. The system according to claim 11, wherein: Generating a plan for operating across the multiple autonomy levels includes: A plan is generated subject to constraints on the allowed autonomy level in the corresponding states.

13. The system according to claim 12, wherein: Constraints map states and actions to subsets of autonomy levels.

14. The system according to claim 11, wherein: The instructions also include instructions for performing the following operations: The feedback model is updated, the feedback model representing a first probability that the system receives a first signal when performing a first action at a second autonomy level, given that the system is in a first state and the system has operated at a first autonomy level.

15. The system according to claim 11, wherein: The instructions also include instructions for performing the following operations: A human state transition function is updated, the human state transition function representing a second probability of the human transitioning to a second state of the environment model given the system selecting to perform a second action in the first state and the human taking manual control.

16. The system according to claim 11, wherein: The instructions also include instructions for performing the following operations: An autonomy profile is updated, wherein the autonomy profile defines a set of acceptable autonomy levels given a current state and an action to be taken next.

17. A method for autonomous driving, comprising: The strategy used to solve the task is calculated by solving the extended random shortest path problem, namely the extended SSP problem. Identify a distinguisher from the current set of undistinguished states; as well as Training a feedback model for the discriminator; wherein the policy maps environmental states and autonomy levels to actions and autonomy levels, and Wherein, calculating the strategy comprises: Generate plans that operate across multiple autonomy levels.

18. The method according to claim 17, wherein: Generating a plan for operating across the multiple autonomy levels includes: A plan is generated subject to constraints on the allowed autonomy level in the corresponding states.

19. The method according to claim 18, wherein: Constraints map states and actions to subsets of autonomy levels.

20. The method of claim 17, further comprising: updating the feedback model, the feedback model representing a first probability of receiving a first signal when performing a first action at a second autonomy level, given that the agent is in a first state and the agent has operated at a first autonomy level; updating a human state transition function representing a second probability that the human will transition to a second state of the environment model given that the agent chooses to perform a second action in the third state and the human assumes manual control; and An autonomy profile is updated, wherein the autonomy profile defines a set of acceptable autonomy levels given a current state and an action to be taken next.

Images