Vehicle decision making using sequential information probes

Abstract

Vehicle decision making is analyzed and can be used to modify the decision making process. For a feature subset including a vehicle operating scenario, a first value is generated that quantifies behavior of an artificial intelligence (AI) agent when the AI agent performs a sequence of actions within a first world model based on a complete set of observations for the feature subset. The first world model is a copy of a second world model used for sequential decision making. A second value is generated that quantifies behavior of the AI agent when the AI agent performs the sequence of actions in the second world model based on an incomplete set of observations for the feature subset. A difference between the first value and the second value determines an impact of the individual feature on the AI agent within the second world model. The decision making process of the AI agent can be updated.

Description

Technical Field

[0001] This disclosure generally relates to the operation and management of autonomous vehicles and autonomous driving, and more particularly to decision-making for autonomous vehicles using sequential information detection. Background Technology

[0002] Vehicles, such as autonomous vehicles, may traverse a portion of a vehicle transportation network (e.g., a road). Traversing a portion of the vehicle transportation network may include generating or capturing data, such as data representing the operating environment of the vehicle or a portion thereof, using sensors on the vehicle. Traversing a portion of the vehicle transportation network may include actions to perform autonomous driving in response to the captured data. These actions may be selected using artificial intelligence (e.g., a trained machine learning model) or other decision-making models. Summary of the Invention

[0003] This paper discloses aspects, features, elements, implementations, and embodiments of a framework for decision-making models (i.e., AI agents) for autonomous vehicles (AVs). This framework facilitates understanding how insufficient information affects the behavior of the decision-making model (e.g., information-driven versus goal-driven behavior). In some examples, the model may be a partially observable Markov decision process (POMDP) ​​model. The understanding derived using this framework can improve the decision-making model and / or decision-making process of AVs.

[0004] A first aspect is a method comprising: determining a first world model, wherein the first world model is a copy of a second world model for sequential decision-making using incomplete state information; for various subsets of features including a vehicle operation scenario within a vehicle transportation system: generating a first value for quantifying the behavior of an artificial intelligence (AI) agent when the AI ​​agent uses the first world model to perform a sequence of actions in response to the vehicle operation scenario, wherein the AI ​​agent includes a complete set of observations for the feature subsets within the first world model; generating a second value for quantifying the behavior of the AI ​​agent when the AI ​​agent uses the second world model to perform a sequence of actions in response to the vehicle operation scenario, wherein the AI ​​agent includes an incomplete set of observations for the feature subsets within the second world model; and determining a difference between the first value and the second value; using the difference to calculate the influence of individual features on the AI ​​agent within the second world model; and selectively updating the decision-making process of the AI ​​agent based on the influence of the individual features.

[0005] The second aspect is an apparatus. The apparatus includes a processor configured to execute instructions stored in a memory to: determine a first world model, wherein the first world model is a copy of a second world model for sequential decision-making using incomplete state information; for various subsets of features including a vehicle operation scenario within a vehicle transportation system: generate a first value for quantifying the behavior of an artificial intelligence (AI) agent when the AI ​​agent uses the first world model to perform a sequence of actions in response to the vehicle operation scenario, wherein the AI ​​agent includes a complete set of observations for the feature subset within the first world model, wherein AI is artificial intelligence; generate a second value for quantifying the behavior of the AI ​​agent when the AI ​​agent uses the second world model to perform a sequence of actions in response to the vehicle operation scenario, wherein the AI ​​agent includes an incomplete set of observations for the feature subset within the second world model; and determine a difference between the first value and the second value; use the difference to calculate the influence of individual features on the AI ​​agent within the second world model; and selectively update the decision-making process of the AI ​​agent based on the influence of the individual features.

[0006] The third aspect is a non-transitory computer-readable medium storing instructions executable to cause one or more processors to operate, the operations including: determining a first world model, wherein the first world model is a copy of a second world model for sequential decision-making using incomplete state information; for various subsets of features including a vehicle operation scenario within a vehicle transportation system: generating a first value for quantifying the behavior of an artificial intelligence (AI) agent when the AI ​​agent uses the first world model to perform a sequence of actions in response to the vehicle operation scenario, wherein the AI ​​agent includes a complete set of observations for the feature subset within the first world model, wherein AI is artificial intelligence; generating a second value for quantifying the behavior of the AI ​​agent when the AI ​​agent uses the second world model to perform a sequence of actions in response to the vehicle operation scenario, wherein the AI ​​agent includes an incomplete set of observations for the feature subset within the second world model; and determining a difference between the first value and the second value; using the difference to calculate the influence of individual features on the AI ​​agent within the second world model; and selectively updating the decision-making process of the AI ​​agent based on the influence of the individual features.

[0007] These and other aspects, features, elements, implementations, and variations of the methods, apparatuses, processes, and algorithms disclosed herein are described in further detail below. Attached Figure Description

[0008] The various aspects of the methods and apparatus disclosed herein will become more apparent by referring to the examples provided in the following description and accompanying drawings, in which otherwise the same reference numerals refer to the same elements.

[0009] Figure 1 This is a diagram illustrating an example of a vehicle that can realize a portion of the aspects, features, and elements disclosed herein.

[0010] Figure 2 The diagram is an example of a vehicle transportation and communication system that can realize the aspects, features and elements disclosed herein.

[0011] Figure 3 This is a diagram illustrating an example of an autonomous vehicle operation management system according to embodiments of the present disclosure.

[0012] Figure 4 This is a diagram of a vehicle decision-making system based on the teachings in this article, including a data determination interface.

[0013] Figure 5 This is a flowchart of a method for vehicle decision-making using sequential information detection, based on the teachings in this paper.

[0014] Figure 6 This is a diagram illustrating an example of a vehicle operation scenario where a decision-making model generates solutions.

[0015] Figure 7A and Figure 7B This is a diagram illustrating the visualized Value of Information (VoI) within an autonomous vehicle. Detailed Implementation

[0016] Vehicles such as autonomous vehicles (AVs) or semi-autonomous vehicles may traverse a portion of a vehicle transport network. A vehicle may include one or more sensors, and traversing the vehicle transport network may include the sensors generating or capturing sensor data, such as sensor data corresponding to the vehicle's operating environment or a portion thereof. For example, sensor data may include information corresponding to one or more external objects (such as pedestrians, remote vehicles, other objects within the vehicle's operating environment, the geometry of the vehicle transport network, or combinations thereof). As used herein, AVs include semi-autonomous vehicles or any other vehicle capable of operating in response to remote commands as discussed below.

[0017] During autonomous driving, and at different time steps (e.g., at each time step), a component of the AV (e.g., a decision-making module or model, such as an inference module or reasoning module) can determine the appropriate action for controlling the AV in response to sensor information. Thus, at a higher level, the components of the AV use inputs (e.g., sensor data) and produce outputs (e.g., actions to control the AV), where the output can be actions used to control the AV.

[0018] The component can be a single component (e.g., a module, model, circuit system, etc.), multiple cooperating components, or a command arbitration module (e.g., an actuator or autonomous vehicle operation management controller) that receives input (e.g., candidate actions) from multiple components and selects one of the candidate actions as the selected action for controlling the AV.

[0019] Decision-making in such situations can be highly opaque due to various reasons (including, but not limited to, the number of parameters used in the process and their varying impacts on the solution). This disclosure manipulates the inputs to the decision-making model to further process the model. Details are described below, beginning with a description of an AV that can be used with the invention.

[0020] Figure 1 This is a diagram illustrating an example of a vehicle that can implement the aspects, features, and elements disclosed herein. As shown, vehicle 100 includes a chassis 110, a powertrain 120, a controller 130, and wheels 140. Although for simplicity, vehicle 100 is shown as including four wheels 140, one or more other propulsion devices (such as pushers or tracks) may be used. Figure 1 In this configuration, the lines that interconnect components such as powertrain 120, controller 130, and wheels 140 indicate information such as data or control signals, power such as electricity or torque, or both information and power that can communicate between the components. For example, controller 130 may receive power from powertrain 120 and may communicate with powertrain 120, wheels 140, or both to control vehicle 100, which may include accelerating, decelerating, steering, or otherwise controlling vehicle 100.

[0021] As shown, powertrain 120 includes a power source 121, a transmission 122, a steering unit 123, and an actuator 124. Other elements or combinations of elements of the powertrain may be included, such as suspension, drive shafts, wheel axles, or exhaust systems. Although shown separately, wheels 140 may be included in powertrain 120.

[0022] Power source 121 may include an engine, a battery, or a combination thereof. Power source 121 may be any device or combination of devices operable to provide energy (such as electrical energy, thermal energy, or kinetic energy). For example, power source 121 may include an engine (such as an internal combustion engine, an electric motor, or a combination of an internal combustion engine and an electric motor) and may be operable to provide kinetic energy as a prime mover to one or more of the wheels 140. Power source 121 may include a potential energy unit, such as one or more dry cell batteries (such as nickel-cadmium (NiCd) batteries, nickel-zinc (NiZn) batteries, nickel-metal hydride (NiMH) batteries, lithium-ion (Li-ion) batteries, etc.), solar cells, fuel cells, or any other device capable of providing energy.

[0023] The transmission 122 can receive energy (such as kinetic energy) from the power source 121 and can transmit that energy to the wheels 140 to provide prime mover. The transmission 122 can be controlled by a controller 130, an actuator 124, or both. The steering unit 123 can be controlled by the controller 130, the actuator 124, or both, and can control the wheels 140 to steer the vehicle. The actuator 124 can receive signals from the controller 130 and can actuate or control the power source 121, the transmission 122, the steering unit 123, or any combination thereof to operate the vehicle 100.

[0024] As shown, controller 130 may include positioning unit 131, electronic communication unit 132, processor 133, memory 134, user interface 135, sensor 136, electronic communication interface 137, or any combination thereof. Although shown as a single unit, any one or more elements of controller 130 may be integrated into any number of individual physical units. For example, user interface 135 and processor 133 may be integrated into a first physical unit, and memory 134 may be integrated into a second physical unit. Although in Figure 1 Although not shown, controller 130 may include a power source such as a battery. Although shown as separate elements, positioning unit 131, electronic communication unit 132, processor 133, memory 134, user interface 135, sensor 136, electronic communication interface 137, or any combination thereof may be integrated into one or more electronic units, circuits, or chips.

[0025] Processor 133 may include any means or combination of means capable of manipulating or processing existing or subsequently developed signals or other information, including optical processors, quantum processors, molecular processors, or combinations thereof. For example, processor 133 may include one or more dedicated processors, one or more digital signal processors, one or more microprocessors, one or more controllers, one or more microcontrollers, one or more integrated circuits, one or more application-specific integrated circuits, one or more field-programmable gate arrays, one or more programmable logic arrays, one or more programmable logic controllers, one or more state machines, or any combination thereof. Processor 133 may be operatively coupled to positioning unit 131, memory 134, electronic communication interface 137, electronic communication unit 132, user interface 135, sensor 136, powertrain 120, and any combination thereof. For example, the processor may be operatively coupled to memory 134 via communication bus 138.

[0026] Memory 134 may include any tangible, non-transitory computer-usable or computer-readable medium capable of, for example, containing, storing, communicating, or transmitting machine-readable instructions or any information associated therewith, for use by or in connection with processor 133. Memory 134 may, for example, be one or more solid-state drives, one or more memory cards, one or more removable media, one or more read-only memories, one or more random access memories, one or more disks (including hard disks, floppy disks, optical disks, magnetic cards, or optical cards), or any type of non-transitory medium suitable for storing electronic information, or any combination thereof.

[0027] Communication interface 137 can be a wireless antenna, a wired communication port, an optical communication port, or any other wired or wireless unit capable of interfacing with wired or wireless electronic communication medium 150, as shown. Although Figure 1 The diagram shows communication interface 137 communicating via a single communication link, but the communication interface can be configured to communicate via multiple communication links. Although Figure 1 A single communication interface 137 is shown, but the vehicle may include any number of communication interfaces.

[0028] Communication unit 132 can be configured to transmit or receive signals via wired or wireless electronic communication medium 150 (such as via communication interface 137). Although in Figure 1 Although not explicitly shown, communication unit 132 can be configured to transmit, receive, or both via any wired or wireless communication medium (such as radio frequency (RF), ultraviolet (UV), visible light, optical fiber, wired lines, or combinations thereof). Figure 1 A single communication unit 132 and a single communication interface 137 are shown, but any number of communication units and any number of communication interfaces can be used. In some embodiments, the communication unit 132 may include a dedicated short-range communication (DSRC) unit, an on-board unit (OBU), or a combination thereof.

[0029] Positioning unit 131 can determine geographic positioning information, such as the longitude, latitude, altitude, direction of travel, or speed of the vehicle 100. For example, the positioning unit may include a Global Positioning System (GPS) unit, such as a National Marine Electronics Association (NMEA) unit with Wide Area Augmentation System (WAAS) enabled, a radio triangulation unit, or a combination thereof. Positioning unit 131 can be used to obtain information, for example, indicating the current heading of the vehicle 100, the current position of the vehicle 100 in two or three dimensions, the current angular orientation of the vehicle 100, or a combination thereof.

[0030] User interface 135 may include any unit capable of interfacing with a person, such as a virtual or physical keypad, touchpad, display, touch display, head-up display, virtual display, augmented reality display, haptic display, feature tracking device (such as an eye-tracking device), speaker, microphone, camera, sensor, printer, or any combination thereof. User interface 135 may be operatively coupled to processor 133 as shown, or operatively coupled to any other element of controller 130. Although shown as a single unit, user interface 135 may include one or more physical units. For example, user interface 135 may include an audio interface for audio communication with a person and a touch display for vision- and touch-based communication with a person. User interface 135 may include multiple displays, such as multiple physically separate units, multiple defined portions within a single physical unit, or combinations thereof.

[0031] Sensor 136 may include one or more sensors (such as a sensor array) operable to provide information that can be used to control the vehicle. Sensor 136 may provide information relating to the current operating characteristics of the vehicle 100. Sensor 136 may, for example, include a rate sensor, an acceleration sensor, a steering angle sensor, a traction-related sensor, a braking-related sensor, a steering wheel position sensor, an eye-tracking sensor, a work position sensor, or any sensor or combination of sensors operable to report information relating to some aspect of the current dynamic state of the vehicle 100.

[0032] Sensor 136 may include one or more sensors operable to obtain information relating to the physical environment surrounding vehicle 100. For example, one or more sensors may detect road geometry and features (such as lane lines) and obstacles (such as stationary obstacles, vehicles, and pedestrians). Sensor 136 may be or may include one or more cameras, laser sensing systems, infrared sensing systems, acoustic sensing systems, or any other suitable type of vehicle-mounted environmental sensing device, or combination thereof, known now or subsequently developed. In some embodiments, sensor 136 and positioning unit 131 may be a combined unit.

[0033] Although not shown separately, vehicle 100 may include a trajectory controller. For example, controller 130 may include a trajectory controller. The trajectory controller is operable to obtain information describing the current state of vehicle 100 and a planned route for vehicle 100, and to determine and optimize the trajectory of vehicle 100 based on that information. In some embodiments, the trajectory controller may output a signal operable to control vehicle 100 such that vehicle 100 follows the trajectory determined by the trajectory controller. For example, the output of the trajectory controller may be an optimized trajectory that may be supplied to powertrain 120, wheels 140, or both. In some embodiments, the optimized trajectory may be a control input such as a set of steering angles, where each steering angle corresponds to a point in time or position. In some embodiments, the optimized trajectory may be one or more paths, lines, curves, or combinations thereof.

[0034] One or more of the wheels 140 may be: a steering wheel that can pivot to a steering angle under the control of the steering unit 123; a drive wheel that can be twisted under the control of the transmission 122 to propel the vehicle 100; or a steering drive wheel that can steer and propel the vehicle 100.

[0035] The vehicle may include those not in Figure 1 The unit or component explicitly shown herein, such as housing, Bluetooth® module, FM radio unit, Near Field Communication (NFC) module, Liquid Crystal Display (LCD) unit, Organic Light Emitting Diode (OLED) display unit, speaker, or any combination thereof.

[0036] Vehicle 100 can be an autonomous vehicle that can be autonomously controlled to traverse a part of a vehicle transport network without direct human intervention. Although in Figure 1Not shown separately, however, autonomous vehicles may include an autonomous vehicle control unit capable of route planning, navigation, and control. The autonomous vehicle control unit may be integrated with another unit of the vehicle. For example, controller 130 may include an autonomous vehicle control unit. The teachings herein also apply to semi-autonomous vehicles.

[0037] The autonomous vehicle control unit can control or operate vehicle 100 to traverse a portion of the vehicle transportation network based on current vehicle operating parameters. The autonomous vehicle control unit can control or operate vehicle 100 to perform defined operations or maneuvers, such as parking the vehicle. The autonomous vehicle control unit can generate a travel route from a starting point (such as the current position of vehicle 100) to a destination based on vehicle information, environmental information, vehicle transportation network data representing the transportation network, or a combination thereof, and can control or operate vehicle 100 to traverse the transportation network according to this route. For example, the autonomous vehicle control unit can output the travel route to a trajectory controller, and the trajectory controller can operate vehicle 100 to travel from the starting point to the destination using the generated route.

[0038] Figure 2 This is a diagram illustrating an example of a vehicle transportation and communication system that can implement the aspects, features, and elements disclosed herein. The vehicle transportation and communication system 200 may include one or more vehicles 210 / 211, such as... Figure 1 The vehicle 100 shown can travel via one or more parts of one or more vehicle transport networks 220, and can communicate via one or more electronic communication networks 230. Although in Figure 2 It is not explicitly stated, but vehicles may traverse areas not explicitly or fully included in the vehicle transport network (such as off-road areas).

[0039] The electronic communication network 230 may be, for example, a multiple access system, and may provide communication between the vehicle 210 / 211 and one or more communication devices 240, such as voice communication, data communication, video communication, message transmission communication, or combinations thereof. For example, the vehicle 210 / 211 may receive information such as information representing the vehicle transportation network 220 from the communication device 240 via the network 230.

[0040] In some embodiments, vehicle 210 / 211 may communicate via a wired communication link (not shown), wireless communication links 231 / 232 / 237, or any combination of wired or wireless communication links. For example, as shown, vehicle 210 / 211 may communicate via terrestrial wireless communication link 231, via non-terrestrial wireless communication link 232, or via a combination thereof. Terrestrial wireless communication link 231 may include an Ethernet link, a serial link, a Bluetooth link, an infrared (IR) link, a UV link, or any link capable of providing electronic communication.

[0041] Vehicles 210 / 211 can communicate with other vehicles 210 / 2110. For example, a primary vehicle or host vehicle (HV) 210 can receive one or more inter-vehicle messages, such as basic safety messages (BSMs), from a remote or target vehicle (RV) 211 via a direct communication link 237 or via a network 230. For example, a remote vehicle 211 can broadcast messages to the primary vehicle within a defined broadcast range (e.g., 300 meters). In some embodiments, the primary vehicle 210 can receive messages via a third party (e.g., a signal repeater (not shown) or another remote vehicle (not shown)). Vehicles 210 / 211 can periodically transmit one or more inter-vehicle messages based on, for example, defined intervals (e.g., 100 milliseconds).

[0042] Automated vehicle-to-vehicle (AVV) messages can include vehicle identification information, geospatial status information (such as longitude, latitude, or altitude), geospatial location accuracy information, kinematic status information (such as vehicle acceleration, yaw rate, speed, heading, braking system status, throttle position, steering wheel angle, or route information), or vehicle operational status information (such as vehicle size, headlight status, turn signal information, wiper status, transmission information, or any other information or combination of information related to the status of the transmitting vehicle). For example, transmission status information can indicate whether the transmission of the transmitting vehicle is in neutral, parked, forward, or reverse.

[0043] Vehicle 210 can communicate with communication network 230 via access point 233. Access point 233, which may include a computing device, can be configured to communicate with vehicle 210, communication network 230, one or more communication devices 240, or a combination thereof via wired or wireless communication links 231 / 234. For example, access point 233 can be a base station, base transceiver station (BTS), node B, enhanced node B (eNode-B), home node B (HNode-B), wireless router, wired router, hub, repeater, switch, or any similar wired or wireless device. Although in Figure 2 The interface is shown as a single unit, but the access point can include any number of interconnecting elements.

[0044] The vehicle 210 can communicate with the communication network 230 via satellite 235 or other non-terrestrial communication devices. The satellite 235, which may include a computing device, can be configured to communicate with the vehicle 210, the communication network 230, and one or more communication devices 240, or a combination thereof, via one or more communication links 232 / 236. Although in Figure 2 The satellite is shown as a single unit, but it can include any number of interconnecting elements.

[0045] Electronic communication network 230 can be any type of network configured to provide voice, data, or any other type of electronic communication. For example, electronic communication network 230 may include a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a mobile or cellular telephone network, the Internet, or any other electronic communication system. Electronic communication network 230 may use communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Internet Protocol (IP), Real-Time Transfer Protocol (RTP), Hypertext Transfer Protocol (HTTP), or combinations thereof. Although in Figure 2 While shown as a single unit, an electronic communication network can include any number of interconnecting elements.

[0046] Vehicle 210 can identify a portion of or the condition of the vehicle transportation network 220. For example, vehicle 210 may include one or more on-vehicle sensors (such as... Figure 1 The sensor 136 shown may include a rate sensor, wheel speed sensor, camera, gyroscope, optical sensor, laser sensor, radar sensor, acoustic sensor, or any other sensor or device or combination thereof capable of determining or identifying a portion of or the condition of the vehicle transportation network 220. Sensor data may include lane line data, remote vehicle location data, or both.

[0047] Vehicle 210 may use information communicated via network 230 (such as information representing vehicle transport network 220, information identified by sensors on one or more vehicles, or a combination thereof) to traverse one or more portions of one or more vehicle transport network 220.

[0048] Although for the sake of simplicity, Figure 2 Two vehicles 210 and 211, a vehicle transportation network 220, an electronic communication network 230, and a communication device 240 are shown, but any number of vehicles, networks, or computing devices can be used. The vehicle transportation and communication system 200 may include... Figure 2 Devices, units, or elements not shown. Although vehicle 210 is shown as a single unit, the vehicle may include any number of interconnecting elements.

[0049] Although vehicle 210 is shown communicating with communication device 240 via network 230, vehicle 210 may communicate with communication device 240 via any number of direct or indirect communication links. For example, vehicle 210 may communicate with communication device 240 via a direct communication link such as a Bluetooth communication link.

[0050] In some embodiments, vehicle 210 / 211 may be associated with entity 250 / 260 (such as the driver, operator, or owner of the vehicle). In some embodiments, entity 250 / 260 associated with vehicle 210 / 211 may be associated with one or more personal electronic devices 252 / 254 / 262 / 264 (such as smartphone 252 / 262 or computer 254 / 264). In some embodiments, personal electronic devices 252 / 254 / 262 / 264 may communicate with the corresponding vehicle 210 / 211 via direct or indirect communication links. Although in Figure 2 One entity 250 / 260 is shown as associated with the corresponding vehicle 210 / 211, but any number of vehicles can be associated with an entity and any number of entities can be associated with a vehicle.

[0051] Vehicle transport network 220 only shows navigable areas (e.g., roads), but it may also include one or more non-navigable areas (such as buildings), one or more partially navigable areas (such as parking areas or pedestrian walkways), or combinations thereof. Vehicle transport network 220 may also include one or more grade-separated interchanges between one or more navigable or partially navigable areas. A portion of vehicle transport network 220 (such as a road) may include one or more lanes and may be associated with one or more directions of travel.

[0052] A transportation network, or a portion thereof, can be represented as transportation network data. For example, transportation network data can be represented as a hierarchical structure of elements (such as markup language elements) that can be stored in a database or file. For simplicity, the accompanying figures depicting the transportation network data representing parts of the network as diagrams or maps; however, transportation network data can be represented in any computer-usable format capable of representing the transportation network or a portion thereof. Transportation network data may include transportation network control information, such as direction of travel, speed limits, toll information, gradient information (such as inclination or angle information), surface material information, aesthetic information, defined hazard information, or combinations thereof.

[0053] A portion or combination of portions of the transportation network 220 can be identified as points of interest or destinations. For example, transportation network data can identify buildings as points of interest or destinations. Discrete, uniquely identifiable geolocations can be used to identify points of interest or destinations. For example, transportation network 220 may include defined locations for destinations, such as street addresses, postal addresses, transportation network addresses, GPS addresses, or combinations thereof.

[0054] Figure 3 This is a diagram illustrating an example of an autonomous vehicle operation management system 300 according to embodiments of the present disclosure. The autonomous vehicle operation management system 300 can operate on autonomous vehicles (such as...) Figure 1 The vehicle 100 shown Figure 2 This can be implemented in one of the vehicles 210 / 211 shown, a semi-autonomous vehicle, or any other vehicle that achieves autonomous driving.

[0055] Autonomous vehicles can traverse a vehicle transportation network or a portion thereof, which may include traversing different vehicle operating scenarios. Different vehicle operating scenarios can include any clearly identifiable set of operating conditions that can affect the operation of the autonomous vehicle within its defined spatiotemporal region or operating environment. For example, different vehicle operating scenarios can be based on the number or base of roads, road segments, or lanes that the autonomous vehicle can traverse within a defined spatiotemporal distance. In another example, different vehicle operating scenarios can be based on one or more traffic control devices that can affect the operation of the autonomous vehicle within its defined spatiotemporal region or operating environment. In another example, different vehicle operating scenarios can be based on one or more identifiable rules, regulations, or laws that can affect the operation of the autonomous vehicle within its defined spatiotemporal region or operating environment. In yet another example, different vehicle operating scenarios can be based on one or more identifiable external objects that can affect the operation of the autonomous vehicle within its defined spatiotemporal region or operating environment.

[0056] For simplicity and clarity, this article may refer to vehicle operation scenario types or classes to describe similar vehicle operation scenarios. A vehicle operation scenario type or class can refer to a scenario definition pattern or set of definition patterns. For example, an intersection scenario may include an autonomous vehicle crossing an intersection; a pedestrian scenario may include an autonomous vehicle crossing a portion of a vehicle transportation network that includes one or more pedestrians or is within a defined proximity of one or more pedestrians, such as when a pedestrian is crossing or approaching the autonomous vehicle's intended path; a lane-changing scenario may include an autonomous vehicle crossing a portion of a vehicle transportation network by changing lanes; a merging scenario may include an autonomous vehicle crossing a portion of a vehicle transportation network by merging from a first lane to a merging lane; and a pass-obstruction scenario may include an autonomous vehicle crossing a portion of a vehicle transportation network by passing an obstacle or obstruction. Although this article describes pedestrian vehicle operation scenarios, intersection vehicle operation scenarios, lane-changing vehicle operation scenarios, merging vehicle operation scenarios, and obstacle-crossing vehicle operation scenarios, any other vehicle operation scenario or vehicle operation scenario type may be used.

[0057] like Figure 3 As shown, the autonomous vehicle operation management system 300 includes an autonomous vehicle operation management controller (AVOMC) 310, an operation environment monitor 320, and an operation control evaluation module (also known as a model) 330.

[0058] AVOMC 310 can receive, identify, or otherwise access operational environment data representing the operational environment of an autonomous vehicle, such as the current operational environment, the expected operational environment, or one or more aspects thereof. The operational environment of an autonomous vehicle may include a set of clearly identifiable operational conditions that can affect the operation of the autonomous vehicle within a defined spatiotemporal region of the autonomous vehicle, within a defined spatiotemporal region of an identified route of the autonomous vehicle, or a combination thereof. For example, operational conditions that can affect the operation of the autonomous vehicle can be identified based on sensor data, vehicle transportation network data, route data, or any other data or combination of data representing the definition or determination of the operational environment of the vehicle.

[0059] Operating environment data may include vehicle information for autonomous vehicles, such as information indicating the geospatial location of the autonomous vehicle, information relating the geospatial location of the autonomous vehicle to information representing a vehicle transportation network, the autonomous vehicle's route, speed, acceleration status, occupant information, or any other information related to the autonomous vehicle or its operation. Operating environment data may include information representing a vehicle transportation network near the autonomous vehicle, an identified route of the autonomous vehicle, or both. For example, this may include information within a defined spatial distance (such as 300 meters) along a portion of an identified route in a vehicle transportation network, information indicating the geometry of one or more aspects of the vehicle transportation network, information indicating the condition of the vehicle transportation network (such as surface condition), or any combination thereof.

[0060] Operating environment data may include information representing external objects within the operating environment of the autonomous vehicle, such as information representing pedestrians, non-human animals, non-motorized transportation devices (such as bicycles or skateboards), motorized transportation devices (such as remote vehicles), or any other external objects or entities that may affect the operation of the autonomous vehicle.

[0061] Various aspects of the operating environment of an autonomous vehicle can be represented within different vehicle operating scenarios. For example, the relative orientation, trajectory, and expected path of an external object can be represented within different vehicle operating scenarios. In another example, the relative geometry of the vehicle's transportation network can be represented within different vehicle operating scenarios.

[0062] As an example, the first different vehicle operation scenario can correspond to a pedestrian crossing the road at a crosswalk, and the relative orientation and expected path of the pedestrian can be represented within the first different vehicle operation scenario, such as crossing from left to right or from right to left. The second different vehicle operation scenario can correspond to a pedestrian crossing the road by jaywalking, and the relative orientation and expected path of the pedestrian can be represented within the second different vehicle operation scenario, such as crossing from left to right or from right to left.

[0063] Autonomous vehicles can traverse multiple different vehicle operation scenarios within an operating environment, which can be various aspects of a composite vehicle operation scenario. For example, a pedestrian can approach an autonomous vehicle along a planned path through an intersection.

[0064] The Autonomous Vehicle Operation Management System 300 can operate or control autonomous vehicles to traverse different vehicle operation scenarios subject to defined constraints (such as safety constraints, legal constraints, physical constraints, user acceptability constraints, or any other constraints or combinations thereof that can be defined or derived for the operation of autonomous vehicles).

[0065] AVOMC 310 can monitor the operating environment of an autonomous vehicle or aspects thereof. Monitoring the operating environment of an autonomous vehicle can include identifying and tracking external objects, identifying different vehicle operating scenarios, or combinations thereof. For example, AVOMC 310 can identify and track external objects within the operating environment of an autonomous vehicle. Identifying and tracking external objects can include identifying the spatiotemporal location of each external object (which may be relative to the autonomous vehicle), and identifying one or more expected paths for each external object (which may include identifying the rate, trajectory, or both of the external object). For simplicity and clarity, the descriptions of location, expected location, path, and expected path herein may omit explicit indications of geospatial and temporal components; however, unless explicitly indicated herein or otherwise clearly understood from the context, location, expected location, path, and expected path as described herein may include geospatial components, temporal components, or both. Monitoring the operating environment of an autonomous vehicle can include using operating environment data received from the operating environment monitor 320.

[0066] The operating environment monitor 320 may include scene-independent monitors, scene-specific monitors, or combinations thereof. As discussed in further detail below, a scene-independent monitor (such as a congestion monitor 321) may monitor the operating environment of the autonomous vehicle, generate operating environment data representing various aspects of the autonomous vehicle's operating environment, and output the operating environment data to one or more scene-specific monitors, AVOMC 310, or combinations thereof. Scene-specific monitors (such as pedestrian monitors 322, intersection monitors 323, lane change monitors 324, merging monitors 325, or obstacle ahead monitors 326, etc.) may monitor the operating environment of the autonomous vehicle, generate operating environment data representing scene-specific aspects of the autonomous vehicle's operating environment, and output the operating environment data to one or more operation control evaluation models 330, AVOMC 310, or combinations thereof.

[0067] For example, pedestrian detector 322 can be an operating environment detector for detecting pedestrians, intersection detector 323 can be an operating environment detector for detecting intersections, lane change detector 324 can be an operating environment detector for detecting lane changes, merging detector 325 can be an operating environment detector for detecting merging, and obstacle ahead detector 326 can be an operating environment detector for detecting obstacles ahead. Operating environment detector 327 is shown with dashed lines to indicate that the autonomous vehicle operation management system 300 may include any number of operating environment detectors 320.

[0068] The operating environment monitor 320 can receive or otherwise access operating environment data, such as operating environment data generated or captured by one or more sensors of the autonomous vehicle, vehicle transportation network data, vehicle transportation network geometry data, route data, or combinations thereof. For example, the pedestrian monitor 322 can receive or otherwise access information such as sensor data that may indicate, correspond to, or otherwise associate with one or more pedestrians in the operating environment of the autonomous vehicle. The operating environment monitor 320 can associate operating environment data or a portion thereof with the operating environment or aspects thereof, such as with external objects (e.g., pedestrians, remote vehicles, or aspects of vehicle transportation network geometry).

[0069] The operating environment monitor 320 can generate or otherwise identify information representing one or more aspects of the operating environment, such as using external objects (e.g., pedestrians, remote vehicles, or aspects of the geometry of a vehicle transportation network), which may include filtering, abstracting, or otherwise processing the operating environment data. The operating environment monitor 320 can output information representing one or more aspects of the operating environment to or make accessible to the AVOMC 310, such as by storing the information representing one or more aspects of the operating environment in the memory of the autonomous vehicle accessible to the AVOMC 310 (e.g., […]). Figure 1 Information representing one or more aspects of the operating environment, or a combination thereof, is sent from the memory 134 (as shown) to the AVOMC 310. The operating environment monitor 320 can output operating environment data to one or more elements (such as the AVOMC 310, etc.) of the autonomous vehicle operation management system 300. Although in Figure 3 Not shown in the diagram, but scene-specific operating environment monitors 322, 323, 324, 325, and 326 can output operating environment data to scene-independent operating environment monitors (such as congestion monitor 321).

[0070] Pedestrian monitor 322 can correlate, associate, or otherwise process operating environment data to identify, track, or predict the movement of one or more pedestrians. For example, pedestrian monitor 322 can receive information such as sensor data from one or more sensors (which may correspond to one or more pedestrians). Pedestrian monitor 322 can associate the sensor data with one or more identified pedestrians. This may include identifying, for one or more of the identified pedestrians, the direction of travel, path (such as expected path), current or expected speed, current or expected acceleration rate, or a combination thereof. Pedestrian monitor 322 can output the identified, associated, or generated pedestrian information to AVOMC 310 or make it accessible to AVOMC 310.

[0071] The intersection monitor 323 can correlate, associate, or otherwise process operating environment data to identify, track, or predict the actions of one or more remote vehicles in the operating environment of an autonomous vehicle, identify intersections or aspects thereof in the operating environment of an autonomous vehicle, identify the geometry of a vehicle transport network, or combinations thereof. For example, intersection monitor 323 can receive information such as sensor data from one or more sensors (which may correspond to one or more remote vehicles in the autonomous vehicle's operating environment, intersections in the autonomous vehicle's operating environment, or one or more aspects thereof, vehicle transport network geometry, or combinations thereof). Intersection monitor 323 can associate the sensor data with one or more identified remote vehicles in the autonomous vehicle's operating environment, intersections in the autonomous vehicle's operating environment, or one or more aspects thereof, vehicle transport network geometry, or combinations thereof. This may include identifying the current or expected direction of travel, path (such as expected path), current or expected speed, current or expected acceleration rate, or combinations thereof for one or more of the identified remote vehicles. Intersection monitor 323 can output the identified, associated, or generated intersection information to or make it accessible to AVOMC 310.

[0072] The lane change monitor 324 can correlate, associate, or otherwise process operating environment data to identify, track, or predict the actions of one or more remote vehicles in the operating environment of the autonomous vehicle (such as information indicating slow or stationary remote vehicles along the expected path of the autonomous vehicle), identify one or more aspects of the operating environment of the autonomous vehicle (such as the vehicle transport network geometry in the operating environment of the autonomous vehicle) or a combination thereof, which geospatially correspond to lane change operations. For example, the lane change monitor 324 can receive information such as sensor data from one or more sensors (which may correspond to one or more remote vehicles, one or more aspects of the autonomous vehicle's operating environment, or a combination thereof (the one or more aspects geospatially corresponding to lane change operations)). The lane change monitor 324 can associate the sensor data with one or more identified remote vehicles, one or more aspects of the autonomous vehicle's operating environment, or a combination thereof (the one or more aspects geospatially corresponding to lane change operations) within the autonomous vehicle's operating environment. This may include identifying the current or expected direction of travel, path (such as expected path), current or expected speed, current or expected acceleration rate, or a combination thereof, of one or more of the identified remote vehicles. The lane change monitor 324 can output the identified, associated, or generated lane change information to or make it accessible to the AVOMC 310.

[0073] The merging monitor 325 can correlate, associate, or otherwise process operating environment information to identify, track, or predict the actions of one or more remote vehicles in the operating environment of an autonomous vehicle, identify one or more aspects of the operating environment of the autonomous vehicle (such as the vehicle transport network geometry in the operating environment of the autonomous vehicle) or a combination thereof, which geospatially correspond to merging operations. For example, the merging monitor 325 can receive information such as sensor data from one or more sensors (which may correspond to one or more remote vehicles, one or more aspects of the autonomous vehicle's operating environment, or a combination thereof (the one or more aspects geospatially corresponding to merging operations)). The merging monitor 325 can associate the sensor data with one or more identified remote vehicles, one or more aspects of the autonomous vehicle's operating environment, or a combination thereof (the one or more aspects geospatially corresponding to merging operations) within the autonomous vehicle's operating environment. This may include identifying the current or expected direction of travel, path (such as expected path), current or expected speed, current or expected acceleration rate, or a combination thereof, of one or more of the identified remote vehicles. The merging monitor 325 can output the identified, associated, or generated merging information to or make it accessible to the AVOMC 310.

[0074] The obstacle forward monitor 326 can correlate, associate, or otherwise process operating environment information to identify one or more aspects of the autonomous vehicle's operating environment that geospatially correspond to obstacle forward overtaking operations. For example, the obstacle forward monitor 326 can identify the geometry of the vehicle transport network within the autonomous vehicle's operating environment. The obstacle forward monitor 326 can identify one or more obstacles or hindrances in the autonomous vehicle's operating environment, such as slow or stationary long-range vehicles along the autonomous vehicle's intended path or its identified route; and the obstacle forward monitor 326 can identify, track, or predict the movement of one or more long-range vehicles in the autonomous vehicle's operating environment. The Forward Obstacle Detector 326 can receive information such as sensor data from one or more sensors, which may correspond to one or more remote vehicles, one or more aspects of the autonomous vehicle's operating environment, or a combination thereof, in geographic space corresponding to forward obstacle avoidance operations. The Forward Obstacle Detector 326 can associate the sensor data with one or more identified remote vehicles, one or more aspects of the autonomous vehicle's operating environment, or a combination thereof (where one or more aspects geographically correspond to forward obstacle avoidance operations). This may include identifying the current or expected direction of travel, path (such as expected path), current or expected speed, current or expected acceleration rate, or a combination thereof, of one or more of the identified remote vehicles. The Forward Obstacle Detector 326 can output identified, associated, or generated forward obstacle information to or make it accessible to the AVOMC 310.

[0075] Although shown as an operating environment monitor 320, the congestion monitor 321 can be a separate monitoring device. The congestion monitor 321 can receive operating environment data representing the operating environment or aspects thereof of the autonomous vehicle. For example, the congestion monitor 321 can receive operating environment information from the AVOMC 310, from the vehicle's sensors, from external devices (such as remote vehicle or infrastructure devices), or a combination thereof. The congestion monitor 321 can also receive operating environment information from memory (such as the autonomous vehicle's memory, etc.). Figure 1 The memory 134 shown reads operating environment information or a portion thereof.

[0076] The congestion monitor 321 can use this input to determine a corresponding probability of availability (POA) or a corresponding probability of congestion for one or more portions of the vehicle transportation network (such as the portion of the vehicle transportation network near the autonomous vehicle), which may include the portion of the vehicle transportation network corresponding to the expected path of the autonomous vehicle (such as the expected path identified based on the current route of the autonomous vehicle). The probability of availability or the corresponding probability of congestion can indicate the probability or likelihood that the autonomous vehicle can safely (e.g., without obstruction from external objects such as remote vehicles or pedestrians) pass through a portion of the vehicle transportation network or a spatial location within the vehicle transportation network. For example, a portion of the vehicle transportation network may include obstructions (such as stationary objects), and the probability of availability for that portion of the vehicle transportation network may be low (e.g., 0%), which can be represented as a high probability of congestion (e.g., 100%) for that portion of the vehicle transportation network. The congestion monitor 321 can identify the corresponding availability probability for each part of the vehicle transportation network within the operating environment of the autonomous vehicle (e.g., within 300 meters). The congestion monitor 321 can continuously or periodically determine or update the availability probability. The congestion monitor 321 can communicate the availability probability or the corresponding congestion probability to the AVOMC 310.

[0077] Availability probabilities can be indicated by congestion monitors 321 corresponding to various external objects in the operating environment of the autonomous vehicle, and geospatial regions can be associated with multiple availability probabilities corresponding to multiple external objects. Aggregated availability probabilities can be indicated by congestion monitors 321 corresponding to various types of external objects in the operating environment of the autonomous vehicle, such as the availability probability of pedestrians and the availability probability of remote vehicles, and geospatial regions can be associated with multiple availability probabilities corresponding to multiple external object types.

[0078] Congestion monitor 321 can identify external objects, track external objects, and predict the location information, path information, or both, or a combination thereof of the external objects. For example, congestion monitor 321 can identify external objects and their expected paths based on operating environment information (e.g., the current location of the external object), information indicating the current trajectory and / or rate of the external object, information indicating the classification type of the external object (e.g., pedestrian or long-distance vehicle), vehicle transportation network information (e.g., pedestrian crossings near the external object), previously identified or tracked information associated with the external object, or any combination thereof. The expected path can indicate a series of expected spatial locations, expected temporal locations, and corresponding probabilities.

[0079] The congestion monitor 321 can communicate the availability probability or the corresponding congestion probability to the AVOMC 310. The AVOMC 310 can then communicate the availability probability or the corresponding congestion probability to each instantiated instance of the operation control evaluation model 330.

[0080] AVOMC 310 can identify one or more different vehicle operation scenarios based on one or more aspects of the operating environment represented by operating environment data. For example, AVOMC 310 can identify different vehicle operation scenarios in response to identifying operating environment data indicated by one or more operating environment monitors 320, or based on that operating environment data. The different vehicle operation scenarios can be identified based on route data, sensor data, or a combination thereof. For example, AVOMC 310 can identify one or more different vehicle operation scenarios corresponding to an identified route, such as based on map data corresponding to the identified route, in response to identifying a route. Multiple different vehicle operation scenarios can be identified based on one or more aspects of the operating environment represented by operating environment data. For example, operating environment data may include information indicating that a pedestrian is approaching an intersection along a planned path of the autonomous vehicle, and AVOMC 310 can identify a pedestrian vehicle operation scenario, an intersection vehicle operation scenario, or both.

[0081] AVOMC 310 can instantiate instances of one or more operation control evaluation models in the operation control evaluation model 330 based on one or more aspects of the operation environment represented by operation environment data (such as the identification of upcoming scenarios). The upcoming scenario can be different vehicle operation scenarios that AVOMC 310 determines the autonomous vehicle might encounter if it continues on its path. The upcoming scenario can be anticipated (e.g., determined from the autonomous vehicle's route) or unexpected. An unexpected upcoming scenario can be a scenario that can be detected by the vehicle's sensors but cannot be determined without sensor data.

[0082] The operation control evaluation model 330 may include scenario-specific operation control evaluation models (SSOCEMs), such as pedestrian SSOCEM 331, intersection SSOCEM 332, lane change SSOCEM 333, merging SSOCEM 334, obstacle overtaking SSOCEM 335, or combinations thereof. SSOCEM 336 is shown using dashed lines to indicate that the autonomous vehicle operation management system 300 may include any number of SSOCEMs 330. For example, AVOMC 310 may instantiate instances of SSOCEM 330 in response to identifying different vehicle operation scenarios. AVOMC 310 may instantiate multiple instances of one or more SSOCEMs 330 based on one or more aspects of the operation environment represented by operation environment data. For example, the operation environment data may indicate two pedestrians in the autonomous vehicle's operation environment, and AVOMC 310 may instantiate a corresponding instance of pedestrian SSOCEM 331 for each pedestrian.

[0083] AVOMC 310 can send operating environment data, or one or more aspects thereof, to another unit of the autonomous vehicle, such as congestion monitor 321 or one or more instances of SSOCEM 330. For example, AVOMC 310 can communicate availability probabilities or corresponding congestion probabilities received from congestion monitor 321 to the individual instantiated instances of SSOCEM 330. AVOMC 310 can be stored in, for example, the memory of the autonomous vehicle (such as...) Figure 1 The memory 134 shown (etc.) is a medium of storage operation environment data or one or more aspects thereof.

[0084] Despite Figure 3 Although not explicitly shown, the autonomous vehicle operation management system 300 may include a predictor module capable of generating predictive information and sending it to a congestion monitor 321, and the congestion monitor 321 may output availability probability information to one or more of other operating environment monitors 320.

[0085] Once instantiated, SSOCEM 330 can receive operational environment information, including sensor data, to determine and output candidate vehicle control actions (also referred to herein as candidate actions). Candidate actions are vehicle control actions identified by a specific SSOCEM 330 as potentially optimal actions for the vehicle to perform in a particular scenario. For example, an SSOCEM 330 configured to handle intersections (e.g., Intersection SSOCEM 332) can output "proceed," a candidate action suggesting continued passage through the intersection. Similarly, an SSOCEM 330 configured to handle lane changes (e.g., Lane Change SSOCEM 333) can output a "left turn" candidate action, indicating that the vehicle should merge left two degrees. In some implementations, each SSOCEM 330 outputs a confidence score indicating the degree of confidence among the candidate actions determined by the SSOCEM 330. For example, a confidence score greater than 0.95 can indicate a very high level of confidence in a candidate action, while a confidence score less than 0.5 can indicate a relatively low level of confidence in a candidate action. Further details of SSOCEM 330 are described below.

[0086] The AVOMC 310 can receive one or more candidate actions from various instances of the SSOCEM 330. The AVOMC 310 can identify vehicle control actions from the candidate vehicle control actions and can control the vehicle, or it can provide the identified vehicle control actions to another vehicle control unit to traverse the vehicle transport network according to the vehicle control actions.

[0087] Vehicle control actions can instruct vehicle control operations or maneuvers, such as acceleration, deceleration, turning, stopping, or any other vehicle operation or combination of vehicle operations that can be performed by an autonomous vehicle in conjunction with traversing a portion of a vehicle transport network. For example, a "forward" vehicle control action could include slowly moving forward a small distance, such as a few inches or a foot; an "accelerate" vehicle control action could include accelerating at a defined rate, or accelerating at a rate within a defined range; a "decelerate" vehicle control action could include decelerating at a defined rate, or decelerating at a rate within a defined range; a "maintain" vehicle control action could include maintaining current operating parameters, such as maintaining the current speed, current path or route, or current lane orientation; and a "continue" vehicle control action could include starting or resuming a previously identified set of operating parameters. While some vehicle control actions are described herein, others can be used.

[0088] Vehicle control actions may include one or more performance metrics. For example, a "stop" vehicle control action may include deceleration rate as a performance metric. In another example, a "continue" vehicle control action may explicitly indicate route or path information, speed information, acceleration rate, or a combination thereof as a performance metric, or may explicitly or implicitly indicate that the currently or previously identified path, speed, acceleration rate, or a combination thereof can be maintained.

[0089] Vehicle control actions can be composite vehicle control actions, which may include sequences, combinations, or both of vehicle control actions. For example, a "forward" vehicle control action may instruct a "stop" vehicle control action, a subsequent "accelerate" vehicle control action associated with a defined acceleration rate, and a subsequent "stop" vehicle control action associated with a defined deceleration rate, such that controlling the autonomous vehicle based on the "forward" vehicle control action includes controlling the autonomous vehicle to move slowly forward a small distance, such as a few inches or a foot.

[0090] In some implementations, AVOMC 310 utilizes hard-coded logic to determine vehicle control actions from candidate actions. For example, AVOMC 310 can select the candidate action with the highest confidence score. In other implementations, AVOMC 310 can select the candidate action least likely to cause a collision. In still other implementations, AVOMC 310 can generate composite actions based on two or more non-conflicting candidate actions (e.g., combining "continue" and "turn left two degrees" to produce a vehicle control action that causes the vehicle to turn left and continue through the intersection). In some implementations, AVOMC 310 can utilize machine learning algorithms to determine vehicle control actions based on two or more distinct candidate actions.

[0091] For example, identifying vehicle control actions from candidate actions can include implementing machine learning components, such as supervised learning for classification problems, and training the machine learning components using examples of the corresponding vehicle operation scenario (such as 1000 examples). In another example, identifying vehicle control actions from candidate actions can include implementing a Markov Decision Process (MDP) or a POMDP, which can describe how each candidate action affects subsequent candidate actions, and can include a reward function that outputs a positive or negative reward for each vehicle control action.

[0092] AVOMC 310 can cancel the instantiation of an instance of SSOCEM 330. For example, AVOMC 310 can identify different sets of operational conditions as different vehicle operation scenarios indicating autonomous vehicles, instantiate an instance of SSOCEM 330 for each different vehicle operation scenario, monitor the operational conditions, and then determine whether one or more of these operational conditions have expired or have a probability of affecting the operation of the autonomous vehicle below a defined threshold. AVOMC 310 can then cancel the instantiation of the instance of SSOCEM 330.

[0093] As briefly mentioned above, SSOCEM 330 can model different vehicle operation scenarios. The Autonomous Vehicle Operation Management System 300 includes any number of SSOCEM 330s, each modeling different vehicle operation scenarios. Modeling different vehicle operation scenarios may include: generating and / or maintaining state information representing aspects of the vehicle's operating environment corresponding to different vehicle operation scenarios; identifying potential interactions between the modeled aspects regarding corresponding states; and determining candidate actions to solve the model. More simply, SSOCEM 330 may include one or more models configured to determine one or more vehicle control actions for handling a scenario given a set of inputs. Models may include, but are not limited to, POMDP models, MDP models, classical programming (CP) models, partially observable stochastic game (POSG) models, distributed partially observable Markov decision process (Dec-POMDP) ​​models, reinforcement learning (RL) models, artificial neural networks, hard-coded expert logic, or any other suitable type of model. Examples of different model types are provided below. Each SSOCEM330 includes computer-executable instructions that define how the model is operated and how it is utilized.

[0094] SSOCEM 330 can implement a CP model, which can be a single-agent model that models different vehicle operation scenarios based on defined input states. The defined input states can indicate various non-probabilistic states of elements of the autonomous vehicle's operating environment for different vehicle operation scenarios. In the CP model, one or more aspects (e.g., geospatial location) of a modeling element (e.g., an external object) associated with a time location can non-probabilistically differ (by a defined or fixed amount) from the corresponding aspect associated with another time location (e.g., a immediately following time location). For example, at a first time location, a remote vehicle may have a first geospatial location, and at a immediately following second time location, the remote vehicle may have a second geospatial location that differs from the first geospatial location along the remote vehicle's expected path by a defined geospatial distance, such as a defined number of meters.

[0095] SSOCEM 330 can implement discrete-time stochastic control processes such as the MDP model, which can be a single-agent model that models different vehicle operation scenarios based on defined input states. Changes in the autonomous vehicle's operating environment (such as changes in the position of external objects) can be modeled as probabilistic variations. The MDP model can utilize more processing resources and can model different vehicle operation scenarios more accurately than the CP model.

[0096] MDP models can use sets of states, sets of actions, sets of state transition probabilities, reward functions, or combinations thereof to model different vehicle operation scenarios. In some embodiments, modeling different vehicle operation scenarios may include using a discount factor that adjusts or discounts the output of the reward function applied to subsequent time periods.

[0097] The state set can include the current state of the MDP model, one or more possible subsequent states of the MDP model, or a combination thereof. A state represents the identified (or anticipated) conditions of various defined aspects of the vehicle's operating environment (such as external objects and traffic control devices), which can probabilistically affect the vehicle's operation at discrete-time locations. For example, a remote vehicle operating near the vehicle can influence the vehicle's operation and can be represented in the MDP model. The MDP model can include identified or anticipated information representing the remote vehicle, such as its geospatial location, its path, heading, or both, its speed, its acceleration or deceleration rate, or a combination thereof, corresponding to the corresponding time location. At instantiation, the current state of the MDP model can correspond to the concurrent state or condition of the operating environment.

[0098] While any number or cardinality of states can be used, the number or cardinality of states included in the model can be limited to the maximum number of states defined. For example, the model can include 300 most likely states for the corresponding scenario.

[0099] The action set can include the vehicle control actions that can be used at each state in the state set in the MDP model. Different action sets can be defined for different vehicle operation scenarios.

[0100] A set of state transition probabilities can probabilistically represent potential or anticipated changes in the operating environment of a vehicle in response to an action (as represented by a state). For example, a state transition probability can indicate the probability that, in response to a vehicle traversing the vehicle transport network from its current state according to a corresponding action, the operating environment corresponds to a state at a corresponding time position immediately following the current time position corresponding to the current state.

[0101] A set of state transition probabilities can be identified based on operating environment information. For example, operating environment information can indicate region type (such as urban or rural), time of day, ambient light level, weather conditions, traffic conditions (which may include expected traffic conditions, such as peak hour conditions, event-related traffic congestion, or holiday-related driver behavior), road conditions, jurisdiction conditions (such as national, state, or municipal conditions), or any other conditions or combinations of conditions that may affect the operation of the vehicle.

[0102] Examples of state transition probabilities associated with pedestrian vehicle operation scenarios may include the defined probability of a pedestrian jaywalking (e.g., based on the geospatial distance between the pedestrian and the corresponding road segment); the defined probability of a pedestrian stopping at an intersection; the defined probability of a pedestrian crossing at a crosswalk; the defined probability of a pedestrian yielding to an autonomous vehicle at a crosswalk; and any other probabilities associated with pedestrian vehicle operation scenarios.

[0103] Examples of state transition probabilities associated with intersection vehicle operation scenarios may include: the defined probability of a remote vehicle arriving at the intersection; the defined probability of a remote vehicle forcibly cutting in front of an autonomous vehicle; the defined probability of a remote vehicle immediately following and then crossing the intersection, such as in the absence of right-of-way (piggybacking); the defined probability of a remote vehicle stopping near the intersection before crossing it, based on traffic control, regulations, or other instructions regarding right-of-way; the defined probability of a remote vehicle crossing the intersection; the defined probability of a remote vehicle deviating from its expected path near the intersection; the defined probability of a remote vehicle deviating from its expected right-of-way priority; and any other probabilities associated with intersection vehicle operation scenarios.

[0104] Examples of state transition probabilities associated with lane-changing vehicle operation scenarios may include: defined probabilities of a remote vehicle changing speed, such as defined probabilities of a remote vehicle behind a vehicle increasing speed or a remote vehicle in front of a vehicle decreasing speed; defined probabilities of a remote vehicle in front of a vehicle changing lanes; defined probabilities of a remote vehicle approaching a vehicle changing rate to allow the vehicle to merge into the lane; or any other probabilities associated with lane-changing vehicle operation scenarios.

[0105] The reward function determines the corresponding positive or negative (cost) value accumulated for each combination of states and actions. This accumulation represents the expected value of a vehicle traversing the vehicle transportation network from the corresponding state based on the corresponding vehicle control action to the subsequent state.

[0106] For example, a POMDP model could include an autonomous vehicle at a first geospatial location and a first time location corresponding to a first state. This model could instruct the vehicle to identify and perform or attempt vehicle control actions to traverse the vehicle transport network from the first geospatial location to a second geospatial location immediately following the first time location. The set of observations corresponding to the second time location could include identified operational environment information corresponding to the second time location, such as the geospatial location information of the vehicle, the geospatial location information of one or more external objects, availability probabilities, or expected path information.

[0107] The conditional observation probability set can include the probability of each observation based on the operating environment of the autonomous vehicle. For example, an autonomous vehicle can approach an intersection by crossing a first road, while a remote vehicle can approach the intersection by crossing a second road. The autonomous vehicle can identify and evaluate operating environment information (such as sensor data) corresponding to the intersection, which can include operating environment information corresponding to the remote vehicle. This operating environment information may be inaccurate, incomplete, or erroneous. In the MDP model, the autonomous vehicle can non-probabilistically identify the remote vehicle, which may include identifying its location or expected path, and the identified information (such as the identified location) based on inaccurate operating environment information may be inaccurate or erroneous. In the POMDP model, the autonomous vehicle can identify information used for probabilistically identifying the remote vehicle (such as location information for probabilistically identifying the remote vehicle). The conditional observation probability corresponding to the observation or probabilistic identification of the remote vehicle's location represents the probability that the identified operating environment information accurately represents the location of the remote vehicle.

[0108] SSOCEM 330 can implement the Dec-POMDP model, which can be a multi-agent model that models different vehicle operation scenarios. The Dec-POMDP model can be similar to the POMDP model, except that the POMDP model models the vehicle and an appropriate subset of external objects (such as a subset), and the Dec-POMDP model models the autonomous vehicle and the set of external objects.

[0109] SSOCEM 330 can implement the POSG model, which can be a multi-agent model that models different vehicle operation scenarios. Except that the Dec-POMDP model includes the vehicle's reward function, and the POSG model includes the vehicle's reward function and the corresponding reward functions of each external object, the POSG model can be similar to Dec-POMDP.

[0110] SSOCEM 330 can implement RL models, which can be learning models that model different operational scenarios of vehicles. Except that defined state transition probabilities, observation probabilities, reward functions, or any combination thereof can be omitted from the model, RL models can resemble MDP or POMDP models. Conversely, for example, RL models can be model-based RL models that generate state transition probabilities, observation probabilities, reward functions, or any combination thereof based on one or more modeled or observed events.

[0111] In an RL model, the model can evaluate one or more events or interactions (which may include simulated events) and can generate or modify the corresponding model or its solution in response to those events. Simulated events may include, for example, crossing an intersection, crossing a vehicle network near pedestrians, or changing lanes. An example of using an RL model to cross an intersection involves the RL model indicating candidate actions for crossing the intersection. The autonomous vehicle then uses the candidate actions as vehicle control actions at the time location to cross the intersection. The outcome of crossing the intersection using the candidate actions can be determined, and the RL model can be updated based on that outcome.

[0112] The Autonomous Vehicle Operation Management System 300 may include any number or combination of models. For example, pedestrian SSOCEM 331, intersection SSOCEM 332, lane change SSOCEM 333, merging SSOCEM 334, and obstacle overtaking SSOCEM 335 may be POMDP models. In another example, pedestrian SSOCEM 331 may be an MDP model, and intersection SSOCEM 332 may be a POMDP model. AVOMC 310 may instantiate any number of instances of SSOCEM 330 based on operational environment data. Module 336 is shown with dashed lines to indicate that the Autonomous Vehicle Operation Management System 300 may include any number or additional types of SSOCEM 330.

[0113] One or more of the AVOMC 310, the operating environment monitor 320, and the SSOCEM 330 can operate continuously or periodically (e.g., at a frequency of 10 Hz). For example, the AVOMC 310 can identify vehicle control actions multiple times per second (e.g., ten times). The operating frequencies of the components of the autonomous vehicle operation management system 300 can be synchronous or asynchronous, and the operating rate of one or more of the AVOMC 310, the operating environment monitor 320, and the SSOCEM 330 can be independent of the operating rates of the other components.

[0114] As is clear from the above description, these models are complex, and their results are difficult to evaluate. The teachings in this paper provide a representation of the strategies (or tactics) employed by the vehicle, which can be used to determine which factors justify the decisions the AV will make. This determination can be used to modify decision-making strategies to address challenging vehicle operation scenarios.

[0115] Figure 4 This is a diagram of a vehicle decision-making system's data pipeline 400, including a data-defined interface 410, based on the teachings herein. As previously mentioned, the interface lies between perception and decision-making. Figure 4 As shown, the data determination interface 410 is located between the perception system 402 and the decision-making component 412. The perception system 402 may include map data 404, an object information module 406, and a world model 408. In some implementations, the perception system 402 receives map data 404 as input, such that the map is not part of the perception system 402.

[0116] Map data 404 includes any map data representing the operating environment (including the vehicle transportation network) related to the AV. Map data 404 may include HD map data, SD map data, or some combination of HD map data and SD map data. For example, some areas of the operating environment may be represented by HD map data, while other areas may be represented by SD map data. In some implementations discussed in further detail below, map data 404 may not be available for some AVs and / or may not be available for at least a portion of the vehicle transportation network.

[0117] The object information module 406 can receive raw sensing data from the AV's sensors (such as sensor 136). Sensor 136 may include a camera (e.g., an image camera), LiDAR, GPS sensor or unit, or any other sensor or combination of sensors that captures, identifies, or otherwise detects the operating environment surrounding the AV. The object information module 406 can be referenced above. Figure 2 The described wired and wireless signal links receive data from other sources, such as fixed infrastructure cameras, other vehicles within a vehicle transport system, remote vehicle support systems, etc. Object information module 406 can perform object association. For example, object association can include identifying objects from received signals. Object association can associate location information within individual signals with corresponding road objects (e.g., vehicles, pedestrians, or non-motorized vehicles) within the vehicle transport network. Object information module 406 can generate or maintain the state of at least some of the identified objects, such as speed (when the object is dynamic rather than static), pose, geometry (e.g., width, height, and depth), classification (e.g., bicycle, large truck, pedestrian, road sign, etc.), lane position, or some combination thereof.

[0118] World model 408 can output object information, including individually tracked objects and their corresponding trajectories, for use in decision-making within the AV. World model 408 can output location information, such as the position of an object relative to roads and / or lanes in a vehicle transportation network. World model 408 can receive sensed objects from object information module 406 over time. Sensed objects can be fused where appropriate using data such as position and available heading and speed information. That is, data associated with each object can be compared to determine whether objects identified by individual sources (e.g., individual signals from input to object information module 406) can be the same object. Any technique used to compare the data of the sensed objects can be used. The more similar the data, the more likely the two objects are the same. Data of objects identified as the same object are fused to generate an object that includes the tracked object at its location over time (e.g., a fused trajectory).

[0119] In some implementations, some or all components of the perception system 402 may correspond to one or more components of the autonomous vehicle operation management system 300. In the example, object information module 406, world model 408, or both correspond to the operating environment monitor 320. Map data 404 may be part of the operating environment monitor 320, or more likely may be otherwise incorporated elsewhere, such as stored in the memory 134 of the vehicle 100 and / or received remotely from the communication unit 132. Although the perception system 402 is shown as a single component of the data pipeline 400, at least some components of the perception system 402 may be replicated (e.g., because multiple scenes are indicated by detected objects). For example, a single world model 408 may be used for all operating environment monitors 320, while a corresponding object information module 406 (e.g., each associated with an object within a scene) may be used for each operating environment monitor 320. Other variations are possible.

[0120] Decision-making component 412 recommends actions (e.g., candidate vehicle control actions) for the AV, such as go, yield / edge, or stop. This action can be performed automatically by the AV. For example, the action can be performed by controlling one or more of the AV's braking, acceleration (e.g., accelerator pedal), steering (e.g., steering wheel), etc., through the AV's processor (e.g., processor 133). Decision-making component 412 may correspond to a component of the Autonomous Vehicle Operation Management System 300. In the example, decision-making component 412 corresponds to the model of SSOCEM 330. Therefore, more than one decision-making component 412 can be used in the implementation of the teachings herein. Outputs from the various decision-making components 412 (e.g., candidate actions) can be used to select the AV's control action, such as using the previously described AVOMC 310.

[0121] In vehicle operation environments where multiple decision-making components 412 are required (e.g., because multiple scenarios are indicated by detected objects), a corresponding data determination interface 410 can be associated with each decision-making component 412. This is because, as described in more detail below, the data determination interface 410 can be designed to use different data sources from the AV to determine, generate, or otherwise produce the output required by a particular decision-making component 412. In some implementations of the AV incorporating an autonomous vehicle operation management system, the autonomous vehicle operation management system 300 can be modified to include a data determination interface 410 between various operation environment monitors 320 and SSOCEM 330 (such as between an intersection monitor 323 and an intersection model (such as an intersection SSOCEM 332, etc.)).

[0122] Figure 5 This is a flowchart of a method 500 for making vehicle decisions using sequential information detection, based on the teachings herein. The method can be performed by any combination of a computer, processor, controller, or hardware, whether or not software is used. The method can be performed by an AV (such as by processor 133), or it can be performed remotely, such as by a processor or other hardware and optionally software from a remote support facility incorporating communication device 240.

[0123] At operation 502, the first-world model is determined. The first-world model (also known as the mirror world model) can be a copy of the modified second-world model (also known as the original world model) as described below. The second-world model can be used for sequential decision making and can have incomplete state information. Both the first-world and second-world models can be components of AVOMC 310. The first-world and second-world models can provide information for use in SSOCEM (i.e., Figure 3 The input used within SSOCEM 330. The second-world model can be... Figure 4 World Model 408.

[0124] The second-world model can be a representation of different vehicle operation scenarios based on one or more aspects (i.e., features) of an operating environment (e.g., a vehicle transportation system) represented by operating environment data.

[0125] Figure 6 These are diagrams illustrating examples of vehicle operation scenarios 610, 620, and 630 where the decision-making model generates solutions (also referred to as decisions in this paper). These examples are used to explain how the teachings in this paper will be applied to the model.

[0126] Vehicle operation scenario 610 illustrates intersection 602. Vehicle 604 is approaching intersection 602. The objective of vehicle 604 is to safely cross intersection 602 by making a right turn. Vehicle 604 can be... Figure 1 The vehicle 100. The vehicle 604 can be Figure 2 One of the vehicle vehicles 210 / 211. Vehicle vehicle 604 may include an autonomous vehicle operation and management system, such as... Figure 3 The autonomous vehicle operation and management system 300, etc. Thus, the vehicle 604 can be an autonomous vehicle or a semi-autonomous vehicle.

[0127] Vehicle operation scenario 610 also includes a vehicle 606 approaching intersection 602 from the left side of vehicle 604. Vehicle 606 has the right-of-way and does not have a stop sign. Therefore, vehicle 606 can continue through the intersection without first stopping. However, vehicle 606 may begin to slow down to stop or turn before crossing the path of vehicle 604. Therefore, intersection 602 may be a T-junction relative to vehicle 604. Furthermore, if vehicle 606 does not stop or turn before crossing the path of vehicle 604, and vehicle 604 continues to make a right turn before vehicle 606 has passed, vehicle 606 may collide with vehicle 604. Thus, vehicle 604 can wait until vehicle 606 has passed before making a right turn to prevent a collision and safely cross intersection 602.

[0128] Vehicle operation scenario 620 illustrates intersection 602. Vehicle 604 is approaching intersection 602. The objective of vehicle 604 is to safely cross intersection 602 by making a right turn.

[0129] Vehicle operation scenario 620 also includes a vehicle 608 approaching intersection 602 from the left side of vehicle 604. Vehicle 608 has the right-of-way and does not have a stop sign. Therefore, vehicle 608 can continue through the intersection without first stopping. However, vehicle 608 may begin to slow down and stop before crossing the path of vehicle 604. Therefore, intersection 602 may be a T-junction relative to vehicle 604. In this scenario, since the path of vehicle 608 will not intersect with that of vehicle 604, vehicle 604 can make a right turn regardless of whether vehicle 608 stops or continues through the intersection.

[0130] Vehicle operation scenario 630 illustrates intersection 602. Vehicle 604 is approaching intersection 602. The objective of vehicle 604 is to safely cross intersection 602 by making a right turn.

[0131] Vehicle operation scenario 630 also includes a vehicle 610 approaching intersection 602 from the left side of vehicle 604. Vehicle 610 has the right-of-way and does not have a stop sign. Therefore, vehicle 610 can continue through the intersection without first stopping. However, vehicle 604 cannot determine the lane in which vehicle 610 is traveling. Additionally, vehicle 610 may begin to slow down to stop or turn before crossing vehicle 604's path. Therefore, intersection 602 may present a T-junction relative to vehicle 604. In this scenario, vehicle 604 may safely complete the right turn or may not be able to do so safely, depending on the lane in which vehicle 610 is traveling. Thus, vehicle 604 may wait until the lane in which vehicle 610 is traveling can be determined before completing the right turn.

[0132] Refer again Figure 5 The first-world model determined at operation 502 is a modified copy of the second-world model. The second-world model can represent the vehicle's operational scenario, thus including incomplete state information. For example, operational environment data may be incomplete, therefore associated state information is unavailable. (See above reference...) Figure 3 The explained scenario describes a vehicle operation scenario formed by features, the state of which can be used for decision-making. For example, in... Figure 6 In the vehicle operation scenario 630, the lane in which the vehicle 610 is traveling has not yet been determined. Therefore, the vehicle 610 is observed, but its status information is incomplete.

[0133] At operation 502, the first-world model is modified to include perfect information (also known as complete state information) by providing states for different subsets of features of the second-world model, which has incomplete state information. For example, in vehicle operation scenario 630, the state information used by the first-world model determined at operation 502 could be whether the vehicle 610 is in the left or right lane—state information that is not available in the second-world model.

[0134] At operation 504, method 500 generates a first value for quantifying the behavior of the first artificial intelligence (AI) agent. That is, the first AI agent performs a sequence of actions in response to a given vehicle operation scenario, based on a model. In the example, the first AI agent can be represented by a POMDP agent. POMDP is a tuple represented by equation (1). (1)

[0135] In equation (1), It is a finite set of states Each state is represented by a feature set F = Composition. A is a finite set of actions. O is a finite set of observations. T is the state transition function represented by equation (2). (2)

[0136] Indicates that in a given action In the case of state Move to state The probability of. It is the observation function represented by equation (3). (3)

[0137] Taking action and with state The probability of receiving observation o after the event ends. R is the reward function represented by equation (4). (4)

[0138] here, Discount factor The goal of the POMDP agent is to maximize the expected cumulative discount reward, as expressed by Equation (5). (5)

[0139] here, It is the state of the agent at time t, and Let be the action taken by the agent at time t. The solution is the optimal policy that maximizes the expected reward. The policy is a mapping from the agent's state to the prescribed action. However, the agent's state is only partially observable—thus relying on the concept of belief. yes The probability distribution on the POMDP strategy. Belief Distribution mapped to actions .Strategy It can be used to quantify beliefs Start executing the strategy The expected total reward can be expressed by equation (6). (6)

[0140] here, It is used when taking action Subsequent observations transformed into beliefs The probability. Alternatively, if the optimal strategy is used. Then the optimal value is obtained. Furthermore, the action value function (i.e., the Q function) for a given strategy can be represented by formula (7). (7)

[0141] To generate (derive, determine, compute) the first value at operation 504, a first-world model is used. As mentioned above, a first-world model comprises a complete set of observations of a subset of features. For example, a first-world model could be an operational scenario for different vehicles (such as...) Figure 6 The representation of the vehicle operation scenario 630, etc. In this example, the uncertainties depicted in the first-world model can be removed.

[0142] In other words, the first-world model and the second-world model are equivalent in every respect, except that the first AI agent can perfectly observe a subset of features in the first-world model. The subset of features can be represented by Equation (8). (8)

[0143] here, yes The power function. This allows the first AI agent to perfectly observe the feature subset, enabling the isolation of the influence of individual features within the action sequence of the vehicle operation scenario. For example, when the first AI agent performs an action sequence to navigate in vehicle operation scenario 630, the first-world model can represent vehicle 610 as existing in the left lane. The first value can be obtained using equation (6) (execution strategy). The expected total reward can be used to calculate the first value, or the first value can be calculated using equation (7) (Q function).

[0144] After the first AI agent has performed an action sequence within the first world model, the belief state of the second AI agent is updated in the second world model. One of three different methods (i.e., a probe strategy) can be used to update the belief state of the second AI agent based on the first AI agent in the first world model. Using the first method (KS), the first AI agent stays exactly K (i.e., a predefined number) steps in the first world model before updating the belief state of the second AI agent in the second world model. Using the second method (GE), the first AI agent stays K (i.e., a predefined number) steps in the first world model with an expectation, where K ~ Geometric ( However, after each step, the first AI agent can update the second AI agent in the second world with probability λ. Using the third method (MY), the second AI agent lacks awareness of the first-world model. This lack of awareness hinders the second AI agent from adjusting the long-term strategy used during the execution of K steps, leading to short-sighted use of given information.

[0145] At operation 508, method 500 generates (derives, determines, computes) a second value for quantifying the behavior of the second artificial intelligence (AI) agent. That is, the second AI agent uses a second-world model to perform a sequence of actions in response to a given vehicle operation scenario. The second AI agent can be represented by a POMDP agent as described above. The second-world model comprises an incomplete set of observations of a subset of features. For example, the second-world model could be different vehicle operation scenarios (such as...) Figure 6 The representation of the vehicle operation scenario (630, etc.). Thus, the second-world model does not include complete state information.

[0146] At operation 510, method 500 determines the difference between the first value and the second value. That is, when the first AI agent can perfectly observe the feature subset, method 500 compares the values ​​of the information derived from the feature subset. For example, when the AV approaches an intersection with the goal of making a right turn to cross it, it can present the AI ​​agent with information such as... Figure 6 The vehicle operation scenario 630 is described below. If the AI ​​agent is unsure of the lane where the vehicle 610 traveling toward the AV (vehicle 604) is located, the AI ​​agent may make a decision to stop and wait to gather more information. The AI ​​agent may wait until the vehicle has completely passed the AV, or the AI ​​agent may wait long enough to determine the vehicle's lane before making a safe decision to complete the right turn. The decision to stop and wait for the vehicle to pass or the decision to stop and wait to determine the vehicle's lane can be quantified by equation (6) or equation (7) or a combination thereof.

[0147] Subsequently, method 500 determines whether there exists any remaining subset of features of the vehicle operation scenario to be applied to the instantiation of the first world model. For example, the initial subset of features used to determine the first and second values ​​may include known state information about the vehicle operation scenario 630 plus state information that the vehicle 610 is traveling in the left lane, while another subset of features may include known state information about the vehicle operation scenario 630 plus state information that the vehicle 610 is traveling in the right lane. This example uses a subset of features that changes the state of one feature. However, multiple states of multiple features can be used sequentially to determine the first world model at operation 502, generate the first value at operation 504, generate the second value at operation 506, and determine the difference at operation 508.

[0148] Once no feature subset remains in response to the query at operation 510, method 500 proceeds to operation 512.

[0149] At operation 512, method 500 uses difference to calculate the impact of individual features within the second-world model. The impact of individual features can be the information value (VoI) or information effect (IoI) of each individual feature. VoI quantifies how imperfect information related to a subset of information affects the performance of the second AI agent. More specifically, VoI quantifies the expected utility lost by the second AI agent due to the lack of information related to the feature subset for the next K time steps. The feature subset can be represented by equation (9). (9)

[0150] If an individual feature has a large VoI, an AI agent using a utility-maximizing strategy is more likely to seek information related to that feature. Alternatively, if an individual feature has a small VoI, an AI agent using a utility-maximizing strategy is less likely to seek that information. Thus, VoI can be used as an indicator of an AI agent's tendency to seek information about an individual feature in the near future. If, for the next K steps and the probing strategy used, information about the individual feature is given to the AI ​​agent... The perfect information allows us to understand how AI agents can derive their current beliefs. The expected utility achieved is represented by VoI.

[0151] For the KS detection strategy, VoI can be represented by equation (10). (10)

[0152] here, Is using The updated beliefs gained .also, It can also be represented as ,and It can be represented by equation (11). (11)

[0153] For the GS detection strategy, VoI can be represented by equation (12). (12)

[0154] here, Indicates a first-world model, and This indicates a second-world model. Based on this, the value of the information sequence can be defined as a value function and information detection. and The difference between them can be expressed by equation (13). (13)

[0155] The MY detection strategy can be expressed using equation (14) according to the KS detection strategy. (14)

[0156] Here, expectation refers to the value of a belief. Execution by The resulting strategy can achieve the following The distribution of beliefs.

[0157] After calculating the VoI for each individual feature, the marginal value is obtained. The marginal value can be obtained using the Shapely Value Framework, as expressed by Equation (15). (15)

[0158] In addition, VoI exhibits several important properties: (1) if but (2) If but (3) Efficiency of marginal VoI And (4) the relationship between detection strategies (i.e., transportation methods). .

[0159] While VoI quantifies the imperfection with which information influences the behavior of an AI agent in the near future, it does not provide a direct description of that behavior. However, Information Influence (IoI) can be used to quantify the likelihood of observed behavior when probed with sequential information. IoI calculates the ratio of the negative log-likelihood (NLL) of observed behavior given distinct subsets of features with perfect observability. (i.e., behavior...) ) can be represented by equation (16). (16)

[0160] The NLL behavior can be represented by equation (17). (17)

[0161] Furthermore, if an entropy regularization strategy is used... Then the action The probability distribution on can be represented by equation (18). (18)

[0162] Alternatively, if a deterministic strategy is used Then, as shown in equation (19), the action can be used. The Laplace smoothed probability distribution on. (19)

[0163] Sequential information detection strategy and The following behavior The effect of the ratio of NLL can be expressed by equation (20). (20)

[0164] here, and , Is it caused The strategy, and It has an indicator State Time .variable Indicates the remaining steps, or in the case of detecting strategic GE, Indicate whether the agent is in a first-world model or a second-world model. Additionally, when using the KS probe strategy, the T term eliminates the O term, thus allowing the use of the simplified formula (21) to represent IoI. (twenty one)

[0165] When using the GE detection strategy, the IoI can be represented by an equation similar to (21). However, the GE detection strategy contains only what depends on... and The additional term can be ignored. For the MY detection strategy, IoI can be represented by equation (22). (twenty two)

[0166] IoT also exhibits several important attributes: (1) If but (2) If but and (3) the efficiency of marginal IoI. .

[0167] VoI and IoI can be calculated for both discrete and continuous state spaces. For the discrete space, the main components of VoI are... It can be computed by generating a combination of first-world and second-world models (e.g., POMDP). For example, the combined POMDP model can be represented by equation (23). (twenty three)

[0168] Regarding the KS detection strategy It can be defined as follows: It has The enhanced state space, in which It also indicates the number of steps required to achieve perfect observability. It is by... The true value of the feature in The finite set of observations obtained by addition. For example, instead of in Receiving Intelligent agents in Receive Otherwise receive . It is the enhanced state transition function represented by equation (24). (twenty four)

[0169] It is the enhanced observation function, which can be represented by equation (25). (25)

[0170] Regarding GE's detection strategy It can be defined as follows: It has The enhanced state space, in which Indicates whether the AI ​​agent is in the first or second world. Similar to the KS detection strategy. It is by... The finite set of observations is obtained by adding the true values ​​of the features in the dataset. It is the enhanced state transition function represented by equation (26). (26)

[0171] It is the enhanced observation function, which can be represented by equation (27). (27)

[0172] GE's combined detection strategy POMDP has The fewer states and The number of observations is reduced by a factor of 1. Furthermore, for the MY detection strategy, only the value function of POMDP relative to the first-world model is calculated. After solving the POMDP model using the α vector, the strategy can be calculated using equation (28). . (28)

[0173] here, It is in the AI ​​agent that has beliefs and The actual state at that time is represented by equation (29). (29)

[0174] Using the above formula, VoI and IoI can be calculated directly; however, if the real state is not available, formula (30) can be used to calculate them. . (30)

[0175] Similarly, IoI can be calculated using equation (31) without accessing the actual state in the observed behavior. (31)

[0176] here, From The missing state sequence, and It is being sampled The distribution. Natural selection of the distribution is... The choice of distribution is influenced by the observations in the data; however, the choice of distribution may not be readily applicable to continuous state spaces, and different distributions such as uniform distributions can be considered.

[0177] For continuous state spaces and Explicit representations of the belief state may be unavailable. Furthermore, large or infinite state spaces may prevent the application of alpha vector policies. Instead, it is common practice to train AI agents to simulate the environment using deep reinforcement learning of the belief state. Additionally, if explicit belief updates are unavailable, the AI ​​agent can jointly learn the belief update function and the policy.

[0178] To calculate VoI and IoI, different feature sets can be used. The corresponding set of policy values. A meta-deep Q-learning algorithm can be specified, which jointly learns each feature set. The Q-value function. The meta-deep Q-learning algorithm can be represented by algorithm (1). 1. 2. 3. 4. while condition not met do 5. 6. 7. 8. 9. 10. end while 11. return

[0179] Algorithm (1) Obtain the input Q-function and belief update function of POMDP relative to the second-world model Algorithm (1) initializes the meta-Q function. Metabelief function The replay buffer is used as the starting point. There are three key differences in the training process compared to the standard implementation of deep Q-learning. First, the initial history and state distribution are defined by the original policy (line 5 of Algorithm (1)). This is done to account for calculating VoI and IoI from different histories under the original policy. Second, the Collecton_Transitions function takes the initial state, history, and a set of features to be fed perfect information as input and generates transitions for training. Notably, when using the GE probe strategy, the simulation can stop at each step with probability λ, providing information about... Finally, the third key difference is that the Q-function is trained using the standard deep Q-learning loss function expressed by equation (32). (32)

[0180] here, It's about batch size. Importantly, three regularizations were introduced to... Forced simulation This is desirable for several reasons (including accelerating training, maintaining fidelity to the original policy, and using the MY probe strategy). The first regularization can be used in... The difference between Q estimates is kept low and can be expressed by equation (33). (33)

[0181] The second regularization can be used to ensure that in The time is similar and can be represented by equation (34). (34)

[0182] The third regularization can be used to ensure that computations using the MY probing strategy are performed correctly. At that time, the two beliefs were expressed in the original strategy This results in a similar Q value. The third regularization can be represented by equation (35). (35)

[0183] In common, all loss functions can be represented by equation (36). (36)

[0184] Only Regularization should be applied at that time. Once... and If it is known, then equation (37) can be used to calculate. . (37)

[0185] here, It is a generator model that captures the distribution of a given history of states. Furthermore, if the simulator does not have the ability to generate the distribution of a given history of states, the simulator can learn that distribution from the data. It is the current history, where the last observation is replaced with observations from the information being probed. IoI can be calculated using a method similar to VoI.

[0186] Using the disclosed method, explanations can be generated for approximately 20 POMDPs at a rate of 10 explanations per second per POMDP.

[0187] Once the impact of individual characteristics is calculated (determined, derived, inferred) at operation 512, method 500 can use this information at operation 514 to update the decision-making process of the second AI agent. The updated decision-making process of the second AI agent can be used to control the vehicle using the vehicle's control system. The vehicle's control system can be... Figure 1The controller 130. That is, the second AI agent can change how it prioritizes which information is important for achieving its goal. For example, an AV can approach an intersection, such as... Figure 6 For example, at intersection 602, the AV (Autonomous Vehicle) can have the goal of crossing the intersection by making a right turn. The second AI agent can make a decision to stop and wait, or it can make a decision to turn immediately based on the vehicle's operating scenario. Knowing which features have higher VoI or higher IoI, the second AI agent can prioritize these features and seek that information the next time it encounters the same vehicle operating scenario.

[0188] Figure 7A and Figure 7B This is a diagram illustrating the value of information (VoI) within an autonomous vehicle. Diagram 700 depicts intersection 702, stop line 704, vehicle 706, and traffic signal 708. Vehicle 706 can be... Figure 1 Vehicle 100. Vehicle 706 can be Figure 2 One of the vehicle vehicles 210 / 211. Vehicle vehicle 706 may include an autonomous vehicle operation and management system, such as... Figure 3 The autonomous vehicle operation management system 300, etc. Stop line 704 represents the minimum safe distance at which a vehicle can stop before entering intersection 702. When vehicle 706 approaches intersection 702, the state of traffic signal 708 is known, making there no uncertainty about the state of traffic signal 708. Because there is no uncertainty about the state of traffic signal, method 500 does not use traffic signal 708 as a feature modified to generate a first-world model and calculate VoI and IoI in the vehicle operation scenario.

[0189] Figure 720 depicts the same intersection 702, stop line 704, vehicle 706, and traffic signal 708. Additionally, Figure 720 depicts traffic light 710 and VoI indicator 712, and external vehicle 714 and VoI indicator 716. After the vehicle stops at traffic signal 708, traffic signal 708 changes from red to green. While the traffic signal changes from red to green, as indicated by traffic signal 710, the traffic signal state is unknown. The objective of vehicle 706 is to safely cross the intersection.

[0190] Knowing the traffic status is an important signal for achieving this goal. Therefore, the VoI (Voice of Interest) of the status of traffic signals 708 and 710 can be high. This is indicated in Figure 720 by the visibility (i.e., opacity) of the VoI indicator 712 relative to the VoI value. That is, the VoI indicator is more prominently displayed when the VoI of features (e.g., traffic signals 708, 710) is high. Additionally, vehicle 706 is unaware of the status of external vehicle 714; however, the VoI of the external vehicle is low relative to the target of vehicle 706. Therefore, the VoI indicator 716 is less visible (i.e., less opaque) relative to the VoI value.

[0191] As used herein, the term "instructions" can include indications or expressions for performing any of the methods disclosed herein or any one or more of them, and can be implemented in hardware, software, or any combination thereof. For example, instructions can be implemented as information stored in memory (such as a computer program), and instructions can be executed by a processor to perform any of the various methods, algorithms, aspects, or combinations thereof described herein. Instructions, or a portion thereof, can be implemented as a dedicated processor or circuit, which can include dedicated hardware for performing any of the methods, algorithms, aspects, or combinations thereof described herein. In some implementations, a portion of the instructions can be distributed across multiple processors on a single device, or across multiple devices that can communicate directly or across networks (such as local area networks, wide area networks, the Internet, or combinations thereof).

[0192] As used herein, the terms “example,” “implementation,” “aspect,” “feature,” or “element” indicate that they are used as examples, instances, or illustrations. Unless expressly indicated, any example, embodiment, implementation, aspect, feature, or element is independent of various other examples, embodiments, implementations, aspects, features, or elements, and may be used in combination with any other example, embodiment, implementation, aspect, feature, or element.

[0193] As used herein, the terms “determine” and “identify” or any variation thereof include using one or more of the devices shown and described herein to select, identify, calculate, locate, receive, determine, establish, obtain or otherwise identify or determine in any way.

[0194] As used herein, unless otherwise stated or clearly indicated from the context, the term "or" is intended to mean inclusive "or" rather than exclusive "or". Additionally, unless otherwise specified or clearly indicated from the context, the articles "a" and "an" as used in this application and the appended claims should generally be understood to mean "one or more".

[0195] Furthermore, for the sake of simplicity, although the accompanying drawings and descriptions may include a sequence or series of steps or stages, the elements of the methods disclosed herein may occur in various orders or in parallel. Additionally, the elements of the methods disclosed herein may occur together with other elements not explicitly presented and described herein. Moreover, it is not necessary to require all elements of the methods described herein to implement the method according to this disclosure. Although aspects, features, and elements are described herein in specific combinations, each aspect, feature, or element may be used independently, or in various combinations with other aspects, features, and elements, or in various combinations without other aspects, features, and elements.

[0196] The foregoing aspects, examples, and implementations have been described to allow for easy understanding that this disclosure is not restrictive. In contrast, this disclosure covers various modifications and equivalent arrangements included within the scope of the appended claims, which should be interpreted in the broadest possible sense to encompass all such modifications and equivalent structures permitted by law.

Claims

1. A method comprising: Determine a first-world model, wherein the first-world model is a copy of a second-world model used for sequential decision-making using incomplete state information; For various feature subsets including vehicle operation scenarios within a vehicle transportation system: generate The first value is used to quantify the behavior of the first AI agent when it uses the first world model to perform an action sequence in response to the vehicle operation scenario. The first AI agent includes a complete set of observations for the feature subset within the first world model, and AI refers to artificial intelligence. A second value is generated, which is used to quantify the behavior of the second AI agent when the second AI agent uses the second world model to perform an action sequence in response to the vehicle operation scenario, wherein the second AI agent includes an incomplete set of observations for the feature subset within the second world model, and Determine the difference between the first value and the second value; The difference is used to calculate the impact of individual characteristics on the second AI agent within the second world model; and The decision-making process of the second AI agent is selectively updated based on the influence of the individual characteristics.

2. The method according to claim 1, wherein, The sequence of actions performed by the first AI agent includes a predefined number of actions.

3. The method according to claim 1, wherein, Generating the second value includes: Introduce the second AI agent into the second world model; and This enables the second AI agent to perform the action sequence within the second world model.

4. The method according to claim 1, wherein, Generating the first value includes: Introduce the first AI agent into the first world model; This enables the first AI agent to perform the action sequence within the first world model; and Use the first AI agent to update the belief state of the second AI agent.

5. The method according to claim 4, wherein, Generating the first value also includes: Provide the first AI agent with information that the first world model is a copy of the second world model.

6. The method according to claim 4, wherein, Updating the belief state of the second AI agent using the first AI agent can be done after each action in the action sequence.

7. The method according to claim 1, wherein, The second AI agent is used by the vehicle's control system to control the vehicle.

8. An apparatus comprising: Memory; and A processor, configured to execute instructions stored in the memory to: Determine a first-world model, wherein the first-world model is a copy of a second-world model used for sequential decision-making using incomplete state information; For various feature subsets including vehicle operation scenarios within a vehicle transportation system: A first value is generated, which is used to quantify the behavior of the first AI agent when it uses the first world model to perform an action sequence in response to the vehicle operation scenario. The first AI agent includes a complete set of observations for the feature subset within the first world model, and AI refers to artificial intelligence. A second value is generated, which is used to quantify the behavior of the second AI agent when the second AI agent uses the second world model to perform an action sequence in response to the vehicle operation scenario, wherein the second AI agent includes an incomplete set of observations for the feature subset within the second world model, and Determine the difference between the first value and the second value; The difference is used to calculate the impact of individual characteristics on the second AI agent within the second world model; and The decision-making process of the second AI agent is selectively updated based on the influence of the individual characteristics.

9. The device according to claim 8, wherein, The sequence of actions performed by the first AI agent includes a predefined number of actions.

10. The device according to claim 8, wherein, Generating the second value includes executing instructions to: Introduce the second AI agent into the second world model; and This enables the second AI agent to perform the action sequence within the second world model.

11. The device according to claim 8, wherein, Generating the first value includes executing instructions to: Introduce the first AI agent into the first world model; This enables the first AI agent to perform the action sequence within the first world model; and Use the first AI agent to update the belief state of the second AI agent.

12. The device according to claim 11, wherein, Generating the first value includes executing instructions to: Provide the first AI agent with information that the first world model is a copy of the second world model.

13. The device according to claim 11, wherein, Updating the belief state of the second AI agent using the first AI agent can be done after each action in the action sequence.

14. The device according to claim 8, wherein, The second AI agent is used by the vehicle's control system to control the vehicle.

15. A non-transitory computer-readable medium storing instructions operable to cause one or more processors to perform operations, the operations including: Determine a first-world model, wherein the first-world model is a copy of a second-world model used for sequential decision-making using incomplete state information; For various feature subsets including vehicle operation scenarios within a vehicle transportation system: generate The first value is used to quantify the behavior of the first AI agent when it uses the first world model to perform an action sequence in response to the vehicle operation scenario. The first AI agent includes a complete set of observations for the feature subset within the first world model, and AI refers to artificial intelligence. A second value is generated, which is used to quantify the behavior of the second AI agent when the second AI agent uses the second world model to perform an action sequence in response to the vehicle operation scenario, wherein the second AI agent includes an incomplete set of observations for the feature subset within the second world model, and Determine the difference between the first value and the second value; The difference is used to calculate the impact of individual characteristics on the second AI agent within the second world model; and The decision-making process of the second AI agent is selectively updated based on the influence of the individual characteristics.

16. The non-transitory computer-readable medium according to claim 15, wherein, The sequence of actions performed by the first AI agent includes a predefined number of actions.

17. The non-transitory computer-readable medium according to claim 15, wherein, Generating the second value includes: Introduce the second AI agent into the second world model; and This enables the second AI agent to perform the action sequence within the second world model.

18. The non-transitory computer-readable medium according to claim 15, wherein, Generating the first value includes: Introduce the first AI agent into the first world model; This enables the first AI agent to perform the action sequence within the first world model; and Use the first AI agent to update the belief state of the second AI agent.

19. The non-transitory computer-readable medium according to claim 18, wherein, Generating the first value also includes: Provide the first AI agent with information that the first world model is a copy of the second world model.

20. The non-transitory computer-readable medium according to claim 18, wherein, Updating the belief state of the second AI agent using the first AI agent can be done after each action in the action sequence.

Images