Systems and methods for utilizing recursive inference graphs in multi-agent reinforcement learning

Abstract

Systems and methods for utilizing recursive inference graphs in multi-agent reinforcement learning, the systems and methods comprising receiving data associated with a self-agent and a target agent traveling within a multi-agent environment; and analyzing the data associated with the self-agent and the target agent utilizing a multi-agent-centric actor critic framework. The systems and methods further comprise performing k-level recursive inference based on the multi-agent actor critic framework to compute higher-level recursive actions for the self-agent and the target agent. The systems and methods further comprise controlling at least one of the self-agent and the target agent to operate within the multi-agent environment based on at least one of an agent action policy associated with the self-agent and an agent action policy associated with the target agent.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application Serial No. 63 / 139,690, filed January 20, 2021, which is expressly incorporated herein by reference. Background Technology

[0003] Many real-world scenarios involve interactions between multiple agents with limited information exchange. Multi-robot navigation and autonomous driving applications (such as highway merging, four-way parking, and lane changing) are examples of situations requiring interaction between multiple mobile agents. For instance, two mobile agents might be attempting to maneuver across each other's paths. Modeling the interactions of various agents can be challenging due to the need for continuous learning. In scenarios where complex interactions may occur between numerous agents, a sufficient machine-based understanding of multi-agent reasoning has not yet been successfully achieved to correctly model such interactions. Summary of the Invention

[0004] According to one aspect, a computer-implemented method for utilizing recursive inference graphs in multi-agent reinforcement learning includes: receiving data associated with an ego agent and a target agent navigating within a multi-agent environment. The computer-implemented method further includes: analyzing the data associated with the ego agent and the target agent using a multi-agent central participant-commentator framework. The computer-implemented method further includes: performing k-level recursive inference based on the multi-agent participant-commentator framework to compute higher-level recursive actions of the ego agent and the target agent. The output of the k-level recursive inference is used to learn agent action policies associated with the ego agent and agent action policies associated with the target agent. The computer-implemented method further includes: controlling at least one of the ego agent and the target agent to operate within the multi-agent environment based on at least one of the agent action policies associated with the ego agent and the agent action policies associated with the target agent.

[0005] According to another aspect, a system for utilizing recursive inference graphs in multi-agent reinforcement learning includes: a memory storing instructions that, when executed by a processor, cause the processor to: receive data associated with an ego agent and a target agent navigating within a multi-agent environment. The instructions further cause the processor to analyze the data associated with the ego agent and the target agent using a multi-agent central participant commentator framework. The instructions further cause the processor to perform k-level recursive inference based on the multi-agent participant commentator framework to compute higher-level recursive actions of the ego agent and the target agent. The output of the k-level recursive inference is used to learn agent action policies associated with the ego agent and agent action policies associated with the target agent. The instructions further cause the processor to control at least one of the ego agent and the target agent to operate within the multi-agent environment based on at least one of the agent action policies associated with the ego agent and the target agent.

[0006] According to another aspect, a non-transitory computer-readable storage medium stores instructions that, when executed by a computer including a processor, perform a method comprising: receiving data associated with a self-subject and a target subject traveling within a multi-subject environment. The method further comprises: analyzing the data associated with the self-subject and the target subject using a multi-subject central participant-commentator framework. The method further comprises: performing k-level recursive reasoning based on the multi-subject participant-commentator framework to compute higher-level recursive actions of the self-subject and the target subject. The output of the k-level recursive reasoning is used to learn subject action policies associated with the self-subject and subject action policies associated with the target subject. The method further comprises: controlling at least one of the self-subject and the target subject to operate within the multi-subject environment based on at least one of the subject action policies associated with the self-subject and the subject action policies associated with the target subject. Brief description of the attached diagram

[0007] The appended claims set forth novel features considered characteristic of this disclosure. In the following description, the same parts are designated by the same numerals throughout the specification and drawings. The drawings are not necessarily drawn to scale, and some figures may be shown in enlarged or generalized form for clarity and brevity. However, the disclosure itself, as well as preferred modes of use, further objects, and advancements thereof, will be best understood by referring to the following detailed description of illustrative embodiments read in conjunction with the accompanying drawings, in which:

[0008] Figure 1 This is a schematic diagram of an exemplary system utilizing recursive reasoning graphs in multi-agent reinforcement learning according to an exemplary embodiment of the present disclosure;

[0009] Figure 2 This is an illustrative example of a multi-agent environment based on exemplary embodiments of this disclosure;

[0010] Figure 3 This is a process flowchart of a method for receiving data associated with a multi-agent environment and processing a model simulating a multi-agent environment according to an exemplary embodiment of this disclosure;

[0011] Figure 4 This is an illustrative example of a simulated multi-agent environment model that virtually represents a multi-agent environment according to an exemplary embodiment of the present disclosure;

[0012] Figure 5 This is a process flowchart of a method for learning agent action strategies to be executed to control the operation of a self-agent and / or target agent in a multi-agent environment, according to an exemplary embodiment of the present disclosure;

[0013] Figure 6 This is an illustrative example of a recursive reasoning graph in a three-player random game according to an exemplary embodiment of this disclosure; and

[0014] Figure 7 This is a flowchart illustrating a method for utilizing recursive reasoning graphs in multi-agent reinforcement learning according to an exemplary embodiment of this disclosure. Detailed Implementation

[0015] The following includes definitions of the selected terms used herein. The definitions include various examples and / or forms of components that fall within the scope of the terminology and can be used for implementation. These examples are not intended to be limiting.

[0016] As used herein, a "bus" refers to an interconnect architecture that is operatively connected to other computer components within or between computers. A bus can transfer data between computer components. A bus can be a memory bus, memory controller, peripheral bus, external bus, crossbar switch, and / or local bus, etc. A bus can also be a vehicle bus, which uses protocols such as System Transmission to Media (MOST), Controller Area Network (CAN), and Local Interconnect Network (LIN) to interconnect components within a vehicle.

[0017] As used herein, “computer communication” refers to communication between two or more computing devices (e.g., computers, personal digital assistants, cellular phones, network devices) and can be, for example, network transmission, file transfer, app transfer, email, Hypertext Transfer Protocol (HTTP) transmission, etc. Computer communication can occur across, for example, wireless systems (e.g., IEEE 802.11), Ethernet systems (e.g., IEEE 802.3), Token Ring systems (e.g., IEEE 802.5), local area networks (LANs), wide area networks (WANs), point-to-point systems, circuit-switched systems, packet-switched systems, etc.

[0018] As used herein, "disk" can be, for example, a disk drive, a solid-state drive, a floppy disk drive, a magnetic tape drive, a Zip drive, a flash memory card, and / or a memory stick. Furthermore, a disk can be a CD-ROM (compressed disc ROM), a CD-R drive, a CD-RW drive, and / or a digital video ROM drive (DVD-ROM). A disk can store an operating system that controls or allocates resources of a computing device.

[0019] As used herein, "memory" can include volatile memory and / or non-volatile memory. Non-volatile memory can include, for example, ROM (Read-Only Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable PROM), and EEPROM (Electrically Erasable PROM). Volatile memory can include, for example, RAM (Random Access Memory), Synchronous RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), and Direct RAM Bus RAM (DRRAM). Memory can store the operating system that controls or allocates the resources of a computing device.

[0020] As used herein, a "module" includes, but is not limited to, a non-transitory computer-readable medium storing instructions, instructions that execute on a machine, hardware, firmware, software that executes on a machine, and / or combinations thereof, to perform a function or action, and / or cause another module, method, and / or system to perform a function or action. A module may also include logic, software-controlled microprocessors, discrete logic circuits, analog circuits, digital circuits, programmable logic devices, memory devices containing instructions for execution, logic gates, combinations of gates, and / or other circuit components. Multiple modules may be combined into one module, and individual modules may be distributed across multiple modules.

[0021] An "operable connection" or an entity "operably connected" is a connection in which signals can be sent and / or received, and physical and / or logical communications can be performed. An operable connection may include a wireless interface, a physical interface, a data interface, and / or an electrical interface.

[0022] As used herein, a "processor" processes signals and performs general computational and arithmetic functions. Signals processed by a processor may include digital signals, data signals, computer instructions, processor instructions, messages, bits, bit streams, or other means that can be received, transmitted, and / or detected. Typically, a processor can be a variety of different processors, including multiple single-core and multi-core processors and coprocessors, as well as multiple single-core and multi-core processor and coprocessor architectures. A processor may include various modules that perform a variety of functions.

[0023] As used herein, “vehicle” means any means of movement capable of carrying one or more human occupants and powered by any form of energy. The term “vehicle” includes, but is not limited to: automobiles, trucks, vans, minivans, SUVs, motorcycles, scooters, boats, go-karts, amusement park vehicles, rail transport, private boats, and aircraft. In some cases, a motorized vehicle includes one or more engines. Furthermore, the term “vehicle” can refer to an electric vehicle (EV) capable of carrying one or more human occupants and powered wholly or partially by one or more electric motors powered by batteries. EVs can include battery electric vehicles (BEVs) and plug-in hybrid electric vehicles (PHEVs). The term “vehicle” can also refer to autonomous vehicles and / or self-driving vehicles powered by any form of energy. Autonomous vehicles may or may not carry one or more human occupants. Additionally, the term “vehicle” can include automated or non-automated vehicles with predetermined routes or freely moving vehicles.

[0024] As used herein, "value" and "level" may include, but are not limited to, numeric values ​​or levels or other kinds of values ​​or levels, such as percentages, non-numeric values, discrete states, discrete values, persistent values, etc. The terms "value of X" or "level of X" as used throughout the detailed description and claims refer to any numeric value or other kind of value used to distinguish two or more states of X. For example, in some cases, the value or level of X may be given as a percentage between 0% and 100%. In other cases, the value or level of X may be a value in the range between 1 and 10. In still other cases, the value or level of X may not be a numeric value but may be associated with a given discrete state, such as "not X", "slightly x", "x", "very x", and "extremely x".

[0025] I. System Overview

[0026] Referring now to the accompanying drawings, the illustrations are for the purpose of explaining one or more exemplary embodiments and not for the purpose of limiting them. Figure 1This is a schematic diagram of an exemplary system 100 utilizing recursive reasoning graphs in multi-agent reinforcement learning according to an exemplary embodiment of this disclosure. Components of system 100, as well as components of other systems, hardware architectures, and software architectures discussed herein, may be combined, omitted, or organized into different architectures for various embodiments.

[0027] Typically, system 100 includes a self-subject 102 and one or more target subjects 104. For simplicity, this disclosure will describe embodiments of system 100 with respect to a single self-subject 102 and a single target subject 104. However, it should be understood that system 100 may include more than one self-subject 102 and more than one target subject 104, and the embodiments and processes discussed herein can be used in environments that include one or more self-subjects 102 and one or more target subjects 104.

[0028] like Figure 2 As illustrated in the illustrative example, self-agent 102 and target agent 104 may travel within a multi-agent environment 200. In one or more configurations, self-agent 102 and / or target agent 104 may include, but are not limited to, vehicles, robots, forklifts, bicycles, aircraft, construction cranes, etc., that may travel within one or more types of multi-agent environments. In one embodiment, the multi-agent environment 200 may include, but is not limited to, areas evaluated to provide navigable paths for self-agent 102 and / or target agent 104 traveling on road 202, such as... Figure 2 An illustrative example is shown.

[0029] In additional embodiments, one or more multi-agent environments may include, but are not limited to, additional types of roads, such as narrow streets or tunnels and / or paths that may exist in restricted locations (such as factory workshops, construction sites, or airport taxiways). For simplicity, the exemplary embodiments and examples discussed herein will be described primarily with reference to a multi-agent environment 200 including roads, such as... Figure 2 As illustrated in the illustrative example. However, it should be understood that the multi-agent environment 200 may include additional types of roads discussed above.

[0030] like Figure 2As shown, self-subject 102 and target subject 104 can travel within adjacent lanes of road 202 in the multi-subject environment 200. Self-subject 102 and target subject 104 can travel in corresponding directions and at locations within specific distances from each other. As shown, self-subject 102 can travel on road 202 to reach target 204 (e.g., a waypoint, destination), while target subject 104 can travel on road 202 to reach target 206 (e.g., a waypoint, destination). In some cases, as shown, path 208 of self-subject 102 may potentially intersect path 210 of target subject 104, because subjects 102 and 104 are each attempting to reach their respective targets 204 and 206.

[0031] refer to Figure 1 and Figure 2 In an exemplary embodiment, system 100 may include a multi-agent recursive reasoning reinforcement learning application (multi-agent application) 106, which may be configured to perform multi-agent reinforcement learning to simultaneously learn policies of multiple agents interacting with each other. As discussed below, one or more policies may be executed to autonomously control ego agent 102 and / or target agent 104 to reach their respective targets 204, 206 while considering each other. As discussed below, multi-agent application 106 may be configured to utilize neural network 108 and may execute instructions to facilitate manipulation relative to agents 102, 104 within a multi-agent environment 200 using a recursive reasoning model in a centrally trained distributed execution framework.

[0032] Multi-agent application 106 can execute a recursive reasoning graph as a graph structure, which can be utilized in the simulation of multi-agent environment 200 to mimic recursive reasoning procedures under a centralized training and distributed execution framework for multi-agent reinforcement learning. Multi-agent application 106 can thus be configured to utilize a multi-agent central participant commentator model (…). Figure 6 (as shown in the figure) to learn one or more subject action strategies, which describe the actions of the self subject 102 and the target subject 104 traveling within the multi-subject environment 200.

[0033] Self-agent 102 and target-agent 104 are evaluated as central participants and treated as nodes to construct a recursive reasoning graph for efficiently computing higher-level recursive actions of the interacting agents. As central participants, self-agent 102 and target-agent 104 are analyzed as self-interested agents attempting to reach their respective goals 204 and 206 in the most efficient manner. The multi-agent central participant commentator model includes one or more iterations of Markov games, in which one or more commentators evaluate one or more actions (output of the participant model) taken by the simulated self-agent and simulated target-agent to determine one or more rewards and one or more states associated with a goal-specific reward function.

[0034] The recursive reasoning graph structure enables the multi-agent application 106 to model the relationship between self-agent 102 and target agent 104, and to explicitly treat their responses as central participants. The recursive actions of each agent 102, 104 are efficiently sampled and shared through message passing in the graph. Accordingly, the recursive reasoning graph executed by the multi-agent application 106 explicitly models the recursive reasoning processes of agents 102, 104 in general and game-theoretic contexts. As discussed below, the multi-agent application 106 can be configured to augment existing centralized training distributed execution algorithms with centralized participants and graph-like message passing to simulated environments (such as...). Figure 4 (As shown) to efficiently train the learning subjects 102 and 104 representing the multi-subject environment 200.

[0035] In other words, the multi-agent application 106 utilizes a multi-agent central participant reviewer model to perform centralized training, followed by distributed execution. The multi-agent application 106 accordingly evaluates the position, location, and path of all agents 102 and 104. As discussed, the multi-agent application 106 can evaluate the actions of self-agent 102 from the perspective of target agent 104 and can evaluate the actions of target agent 104 from the perspective of self-agent 102. Accordingly, the Markov game simulation performed by the multi-agent application 106 can consider the actions of both self-agent 102 and target agent 104 as central participants.

[0036] In one or more embodiments, the multi-agent application 106 can be configured to learn multiple interaction strategies for multiple agents, including but not limited to the self-agent 102 and the target agent 104 navigating within the multi-agent environment 200. Specifically, the graph output is used to thereby learn corresponding agent action strategies, which can be used within the multi-agent environment 200 to enable the self-agent 102 and / or the target agent 104 to autonomously operate while considering each other to reach their respective targets 204, 206.

[0037] This feature allows the multi-agent application 106 to simultaneously acquire knowledge about the actions of all agents 102 and 104, rather than optimizing different agent actions separately. Additionally, this feature allows the multi-agent application 106 to learn multiple agent action strategies associated with each of the self-agent 102 and the target agent 104. As discussed below, the multi-agent application 106 can incorporate actions from the self-agent 102 to learn agent action strategies associated with the target agent 104. Furthermore, the multi-agent application 106 can incorporate actions from the target agent 104 to learn agent action strategies associated with the self-agent 102.

[0038] This can be achieved by using k-level reasoning to incorporate the reasoning of the relative subject into the reasoning of the corresponding subject to reach its respective goal. For example, when learning the subjective action strategy of the ego subject, k-level reasoning allows multiple subject application 106 to assume the reasoning of the target subject 104 relative to following the path 208 to reach its goal 204. Similarly, when learning the subjective action strategy of the target subject, k-level reasoning allows multiple subject application 106 to assume the reasoning of the ego subject 102 relative to following the path 210 to reach its goal 206. k-level reasoning can contain multiple levels (e.g., level zero reasoning, level one reasoning, level two reasoning, k-level reasoning) to provide action determination regarding the actions of the relative subjects in each subject 102, 104 to reach the corresponding goal.

[0039] As discussed below, after learning the subject action strategies associated with self-subject 102 and target subject 104, multi-subject application 106 can be configured to train neural network 108 with the corresponding subject action strategies. As discussed below, multi-subject application 106 can communicate with neural network 108 to receive the corresponding subject action strategies associated with self-subject 102 and / or target subject 104, which are to be executed to control the autonomous operation (e.g., driving) of self-subject 102 and / or target subject 104, thereby maneuvering within multi-subject environment 200 while following a specific path at a specific speed, acceleration rate, steering angle, deceleration rate, etc., to reach the corresponding targets 204, 206 without any conflict between them. Accordingly, the multi-agent application 106 can autonomously control the self-agent 102 and / or target agent 104 within the multi-agent environment 200 and / or similar multi-agent environments (e.g., real-world environments) including similar driving scenarios to navigate safely and efficiently to their respective targets 204, 206 using agent action strategies.

[0040] As discussed below, the multi-agent application 106 can determine a virtual simulation model of the multi-agent environment 200, wherein the self-agent 102 and the target agent 104, along with corresponding targets 204, 206, are virtually represented at each discrete time step. The simulation model can be determined based on image data and / or LiDAR data that may be provided to the multi-agent application 106 by one or more components of the self-agent 102 and / or the target agent 104. For example, in addition to tracks 202b, 202c falling between tracks 202a, 202d in which the self-agent 102 and the target agent 104 are traveling, the simulation model may also include tracks 202a, 202d in which the self-agent 102 and the target agent 104 are traveling. As discussed below, the simulation model includes corresponding observations and corresponding targets, which can be input into a multi-center participant commentator model and used to perform k-level recursive reasoning to learn agent action strategies associated with the self-agent 102 and the target agent 104, respectively.

[0041] Continue to refer to Figure 1 The self-entity 102 and the target entity 104 may include corresponding electronic control units (ECUs) 110a and 110b. ECUs 110a and 110b may execute one or more application, operating system, vehicle system, and subsystem executable instructions. In one or more embodiments, ECUs 110a and 110b may include corresponding microprocessors, one or more application-specific integrated circuits (ASICs), or other similar devices. ECUs 110a and 110b may also include corresponding internal processing memory, interface circuitry, and buses for transmitting data, sending commands, and communicating with multiple components of the self-entity 102 and / or the target entity 104.

[0042] ECUs 110a and 110b may also include corresponding communication devices (not shown) for internally transmitting data to components of the respective bodies 102 and 104 and communicating with externally hosted computing systems (e.g., outside of bodies 102 and 104). Typically, ECUs 110a and 110b communicate with corresponding storage units 114a and 114b to execute one or more applications, operating systems, vehicle systems, and subsystem user interfaces, etc., stored in the respective storage units 114a and 114b.

[0043] In some implementations, storage units 114a and 114b may store separately learned agent action policies associated with ego agent 102 and / or target agent 104, respectively. Accordingly, multi-agent application 106 may access storage units 114a and 114b to store corresponding agent action policies learned by application 106 for corresponding agents 102 and 104 to follow. In some implementations, application 106 may access storage units 114a and 114b to retrieve corresponding agent action policies to autonomously control the operation of ego agent 102 and / or target agent 104 to take into account the presence of each other (e.g., other agents) within multi-agent environment 200.

[0044] In an exemplary embodiment, ECUs 110a and 110b can be configured to operatively control multiple components of the respective subjects 102 and 104. ECUs 110a and 110b can additionally provide one or more commands to one or more control units (not shown) of subjects 102 and 104 (including, but not limited to, respective engine control units, respective brake control units, respective transmission control units, respective steering control units, etc.) to control the autonomous operation of the subject 102 and / or the target subject 104.

[0045] Continue to refer to Figure 1 The corresponding storage units 114a and 114b of the self-subject 102 and the target subject 104 can be configured to store one or more executable files associated with one or more operating systems, applications, associated operating system data, application data, vehicle system and subsystem user interface data, etc., executed by the corresponding ECUs 110a and 110b. In one or more embodiments, the multi-subject application 106 can access the storage units 114a and 114b to store data, such as one or more images, videos, one or more sets of image coordinates, one or more sets of LiDAR coordinates (e.g., LiDAR coordinates associated with the location of an object), one or more sets of location coordinates (e.g., GPS / DGPS coordinates), and / or vehicle dynamic data, respectively associated with the self-subject 102 and the target subject 104.

[0046] ECUs 110a and 110b can be additionally configured to operatively control corresponding camera systems 116a and 116b of the self-subject 102 and the target subject 104. Camera systems 116a and 116b may include one or more cameras positioned at one or more external portions of the respective subjects 102 and 104. The cameras of camera systems 116a and 116b may be positioned to capture the surrounding environment of the respective subjects 102 and 104, said surrounding environment including a predetermined area located around (front / side / rear) the respective subjects 102 and 104 within the multi-subject environment 200.

[0047] In one or more configurations, one or more cameras of the respective camera systems 116a, 116b may be disposed on the external front, rear, and / or side portions of the respective bodies 102, 104, including but not limited to different portions of the bumper, lighting unit, fender / body panel, and / or windshield. One or more cameras may be positioned on a respective planar scanning base (not shown), which allows the one or more cameras to oscillate to capture images of the surrounding environment of the respective bodies 102, 104.

[0048] Relative to the self-subject 102, the multi-subject application 106 can receive image data associated with untrimmed images / videos of the surrounding environment of the self-subject 102 from the camera system 116a, and can perform image logic to analyze the image data and determine observations based on the self-subject images associated with the multi-subject environment 200, one or more target subjects 104 that may be located within the multi-subject environment 200, one or more lines 202a-202d (paths) within the multi-subject environment 200, and / or one or more objects (not shown) that may be located within the multi-subject environment 200.

[0049] Relative to the target subject 104, the multi-subject application 106 can receive image data associated with an untrimmed image / video of the environment surrounding the target subject 104 from the camera system 116b, and can perform image logic to analyze the image data and determine observations based on the target subject image associated with the multi-subject environment 200, self-subjects 102 that may be located within the multi-subject environment 200, one or more lines 202a-202d (paths) within the multi-subject environment 299, and / or one or more objects (not shown) that may be located within the multi-subject environment 200.

[0050] In one or more embodiments, ECUs 110a, 110b may also be operatively connected to corresponding vehicle laser projection systems 118a, 118b, which may include one or more corresponding LiDAR transceivers (not shown). One or more corresponding LiDAR transceivers of the corresponding vehicle laser projection systems 118a, 118b may be disposed on corresponding external front, rear, and / or side portions of the corresponding bodies 102, 104, including but not limited to different portions of the bumper, body panel, fenders, lighting units, and / or windshield.

[0051] One or more corresponding LiDAR transceivers may include one or more planar scanning lasers configured to oscillate and emit one or more laser beams of ultraviolet, visible, or near-infrared light toward the surrounding environment of the corresponding subjects 102, 104. Vehicle laser projection systems 118a, 118b may be configured to receive one or more reflected laser waves based on one or more laser beams emitted by the LiDAR transceivers. One or more reflected laser waves may be reflected from one or more boundaries 212a, 212b of the multi-subject environment 200 (e.g., guardrails) and / or from one or more objects (e.g., other subjects, cones, pedestrians, etc.) that may be located within the multi-subject environment 200.

[0052] In an exemplary embodiment, vehicle laser projection systems 118a, 118b may be configured to output LiDAR data associated with one or more reflected laser waves. Relative to self-subject 102, multi-subject application 106 may receive the LiDAR data transmitted by vehicle laser projection system 118a and may perform LiDAR logic to analyze the LiDAR data and determine self-subject LiDAR-based observations associated with multi-subject environment 200, and more specifically, the lane 202a in which self-subject 102 is traveling, additional lanes 202b-202d included within multi-subject environment 200, one or more target subjects 104 that may be located within multi-subject environment 200, one or more boundaries 212a, 212b of multi-subject environment 200, and / or one or more objects that may be located within multi-subject environment 200.

[0053] Relative to the target subject 104, the multi-subject application 106 can receive LiDAR data transmitted by the vehicle laser projection system 118b and can execute LiDAR logic to analyze the LiDAR data and determine the target subject LiDAR-based observations associated with the multi-subject environment 200, and more specifically, the lane 202d in which the target subject 104 is traveling, additional lanes 202a-202c included within the multi-subject environment 200, the self subject 102 that may be located within the multi-subject environment 200, one or more boundaries 212a, 212b of the multi-subject environment 200 and / or one or more objects that may be located within the multi-subject environment 200.

[0054] In one or more embodiments, the self-subject 102 and the target subject 104 may additionally include corresponding communication units 120a, 120b, which may be operatively controlled by corresponding ECUs 110a, 110b of the respective subjects 102, 104. Communication units 120a, 120b may each be operatively connected to one or more transceivers (not shown) of the respective subjects 102, 104. Communication units 120a, 120b may be configured to communicate via Internet cloud 122 through one or more wireless communication signals, which may include, but are not limited to, wireless communication signals. Signals, Wi-Fi signals, ZigBee signals, Wi-Max signals, etc. In some implementations, the communication unit 120a of the self-subject 102 can be configured to communicate with the communication unit 120b of the target subject 104 via vehicle-to-vehicle (V2V) to exchange information about the position, speed, steering angle, acceleration rate, deceleration rate, etc. of the subjects 102, 104 traveling within the multi-subject environment 200.

[0055] In one implementation, communication units 120a and 120b may be configured to connect to an Internet cloud 122 to send and receive communication signals to and from an externally hosted server infrastructure (external server) 124. The external server 124 may host a neural network 108 and may execute a multi-agent application 106 to utilize processing power to learn one or more corresponding agent action policies and thereby train the neural network 108 with one or more corresponding agent action policies associated with the self-agent 102 and / or the target agent 104.

[0056] Specifically, the neural network 108 can be trained at one or more time steps based on the learning of one or more agent action policies associated with the self-agent 102 and / or target agent 104 moving within the multi-agent environment 200. Training the neural network 108 can allow agents 102, 104 to receive data relating to real-time or similar multi-agent scenarios (e.g., self-agent 102 and target agent 104 positioned relative to each other) that may occur within the multi-agent environment 200, to ensure that policies that can be learned by the self-agent 102 and / or target agent 104 to simultaneously achieve the respective objectives 204, 206 in a non-conflicting manner while considering each other within the multi-agent environment 200.

[0057] In an exemplary embodiment, components of an external server 124, including neural network 108, can be operatively controlled by processor 126. Processor 126 can be configured to operatively control neural network 108 to leverage machine learning / deep learning to provide artificial intelligence capabilities that can be used to construct multi-agent machine learning dataset 112.

[0058] Referring again to external server 124, processor 126 may be operatively connected to memory 130. Memory 130 may store one or more operating systems, applications, associated operating system data, application data, executable data, etc. In one embodiment, processor 126 may be configured to process information derived from one or more agent action policies associated with self-agent 102 and / or target agent 104, learned by multi-agent application 106 at one or more time steps, which may be used to train neural network 108 by updating multi-agent machine learning dataset 112 stored on memory 130.

[0059] In one or more embodiments, the multi-agent machine learning dataset 112 may be configured as a dataset including one or more fields associated with each of the self-agent 102 and the target agent 104, wherein travel path geolocation information associated with one or more corresponding paths and vehicle dynamics data associated with specific speeds, acceleration rates, steering angles, deceleration rates, etc., may be determined to be used by the self-agent 102 and / or the target agent 104 to reach the corresponding targets 204, 206 based on learned agent action strategies associated with the self-agent 102 and / or the target agent 104, respectively.

[0060] In one embodiment, the processor 126 of the external server 124 may be additionally configured to communicate with the communication unit 128. The communication unit 128 may be configured to communicate via the Internet cloud 122 through one or more wireless communication signals, which may include, but are not limited to, these wireless communication signals. Signals, including Wi-Fi signals, ZigBee signals, and Wi-Max signals. In one embodiment, communication unit 128 may be configured to connect to Internet cloud 122 to send and receive communication signals to and from self-subject 102 and / or target subject 104. Specifically, external server 124 may receive image data and LiDAR data that may be transmitted by self-subject 102 and / or target subject 104 based on the use of one or more of camera systems 116a, 116b and vehicle laser projection systems 118a, 118b. As discussed below, such data can be used to determine a simulated multi-subject environment 200 (real world) and for use relative to multi-subject recursive inference reinforcement learning performed by multi-subject application 106.

[0061] II. Applications of Reinforcement Learning in Multi-Agent Recursive Reasoning, Related Methods, and Examples of Illustrative Strategies and Results

[0062] Now, based on exemplary implementation schemes and references Figure 1The components of the multi-subject application 106 are described below. In an exemplary embodiment, the multi-subject application 106 may be stored on memory 130 and executed by processor 126 of external server 124. In another embodiment, the multi-subject application 106 may be stored on storage unit 114a of ego subject 102 and executed by ECU 110a of ego subject 102. In some embodiments, in addition to being stored and executed by external server 124 and / or ego subject 102, application 106 may also be executed by ECU 110b of target subject 104.

[0063] The overall functionality of the multi-agent application 106 will now be discussed. In an exemplary embodiment, the multi-agent application 106 may include a simulation module 132, a policy learning module 134, a neural network training module 136, and an agent control module 138. However, it should be understood that, in addition to modules 132-138, the multi-agent application 106 may include one or more additional modules and / or sub-modules. Methods and examples describing the process steps performed by modules 132-138 of the multi-agent application 106 will now be described in more detail.

[0064] Figure 3 This is a process flowchart of a method 300 for receiving data associated with a multi-agent environment 200 in which the self-agent 102 and the target agent 104 are traveling, and for processing a simulated multi-agent environment model (simulation model) that virtually represents the multi-agent environment 200, according to an exemplary embodiment of this disclosure. (See also...) Figure 1 , Figure 2 and Figure 4 Component description Figure 3 However, it should be understood that Figure 3 The method can be used with other systems / components.

[0065] As discussed above, the simulation model can be determined by application 106 as a virtual representation (e.g., a virtual model) of the multi-agent environment 200 for use within a multi-agent central participant-commentator model. Specifically, the simulation model can be determined by application 106 as a virtual world model of the multi-agent environment 200, which is utilized when performing one or more iterations of a Markov game to learn agent action policies associated with self-agent 102 and / or agent action policies associated with target agent 104.

[0066] In an exemplary embodiment, method 300 may begin at block 302, wherein method 300 may include receiving image data. In one embodiment, simulation module 132 may communicate with camera system 116a of self-subject 102 and / or camera system 116b of target subject 104 to collect uncropped images / videos of the surrounding environment of subjects 102, 104. The uncropped images / videos may include a 360-degree external view of the surrounding environment of subjects 102, 104, said surrounding environment including a multi-subject environment 200.

[0067] refer to Figure 2 As an illustrative example, from the perspective of self-subject 102, such a view may include the observations of self-subject 102, including target subject 104, target 204 of self-subject 102, lines 202a-202d included within the multi-subject environment 200, and boundaries 212a, 212b of the multi-subject environment 200. Alternatively, from the perspective of target subject 104, such a view may include the observations of target subject 104, including self-subject 102, target 206 of target subject 104, lines 202a-202d included within the multi-subject environment 200, and boundaries 212a, 212b of the multi-subject environment 200. In one embodiment, simulation module 132 may package and store image data received from camera system 116a and / or image data received from camera system 116b on memory 130 of external server 124 for further evaluation by simulation module 132.

[0068] Method 300 may proceed to block 304, wherein method 300 may include receiving LiDAR data. In an exemplary embodiment, simulation module 132 may communicate with vehicle laser projection system 118a of self-subject 102 and / or vehicle laser projection system 118b of target subject 104 to collect LiDAR data including LiDAR-based observations from self-subject 102 and / or target subject 104. The LiDAR-based observations may indicate the location, extent, and position of one or more objects from which reflected laser waves are reflected relative to the location / position of the respective subject 102, 104.

[0069] Refer again Figure 2From the perspective of self-subject 102, simulation module 132 can communicate with the vehicle laser projection system 118a of self-subject 102 to collect self-subject LiDAR-based observations, which classify multiple sets of LiDAR coordinates associated with target subject 104, target 204 of self-subject 102, and boundaries 212a, 212b of multi-subject environment 200. Additionally, from the perspective of target subject 104, simulation module 132 can communicate with the vehicle laser projection system 118b of target subject 104 to collect target subject LiDAR-based observations, which classify multiple sets of LiDAR coordinates associated with self-subject 102, target 206 of self-subject 102, and boundaries 212a, 212b of multi-subject environment 200. In one implementation, the simulation module 132 may package and store self-subject-based LiDAR observations received from the vehicle laser projection system 118a and / or target-subject-based LiDAR observations received from the vehicle laser projection system 118b on the memory 130 of the external server 124 for further evaluation by the simulation module 132.

[0070] Method 300 can proceed to block 306, where method 300 may include fusing image data and LiDAR data. In an exemplary embodiment, simulation module 132 may communicate with neural network 108 to provide artificial intelligence capabilities to perform multimodal fusion of image data received from camera system 116a of self-subject 102 and / or camera system 116b of target subject 104 with LiDAR data received from vehicle laser projection system 118a of self-subject 102 and / or vehicle laser projection system 118b of target subject 104. Simulation module 132 may aggregate the image data and LiDAR data into fused environment data associated with the multi-subject environment 200 for further evaluation by simulation module 132.

[0071] As an illustrative example, simulation module 132 can communicate with neural network 108 to provide artificial intelligence capabilities to leverage one or more machine learning / deep learning fusion processes to aggregate image data received from camera system 116a of self-subject 102 and image data received from camera system 116b of target subject 104 into aggregated image data. Accordingly, observations of the multi-subject environment 200 based on self-subject images can be aggregated with observations of the multi-subject environment 200 based on target subject images.

[0072] The simulation module 132 may also utilize the neural network 108 to provide artificial intelligence capabilities to aggregate LiDAR data received from the vehicle laser projection system 118a of the self-subject 102 and the vehicle laser projection system 118a of the target subject 104 into aggregated LiDAR data using one or more machine learning / deep learning fusion processes. Accordingly, self-subject LiDAR-based observations of the multi-subject environment 200 can be aggregated with target-subject LiDAR-based observations of the multi-subject environment 200. The simulation module 132 may additionally employ the neural network 108 to provide artificial intelligence capabilities to aggregate aggregated image data and aggregated LiDAR data into fused environmental data using one or more machine learning / deep learning fusion processes.

[0073] Method 300 can proceed to block 308, where method 300 may include evaluating fused environmental data associated with the multi-agent environment 200 and determining a simulated multi-agent environment model. In an exemplary embodiment, simulation module 132 may communicate with neural network 108 to utilize one or more machine learning / deep learning fusion processes to evaluate the fused environmental data to determine one or more sets of environmental coordinates based on aggregated observations of ego agent 102 and target agent 104.

[0074] One or more sets of environmental coordinates may include the boundaries of the self-subject 102, the target subject 104, the multi-subject environment 200, the corresponding targets 204 and 206 associated with the self-subject 102 and the target subject 104 (source definitions based on image data and / or LiDAR data), and the position coordinates (e.g., x, y grid world coordinates) of the paths in which the self-subject 102 and the target subject 104 can travel within the multi-subject environment 200, for processing the simulation environment.

[0075] One or more sets of environmental coordinates can be used to define a simulation model (e.g., a virtual grid world) that represents a multi-agent environment 200 including self-agent 102 and target agent 104, and can be used to perform one or more iterations of a Markov game to learn single-agent and multi-agent policies associated with self-agent 102 and target agent 104. As discussed below, the simulation model includes a virtual self-agent representing self-agent 102 and a virtual target agent representing target agent 104, as well as virtual markers that can represent corresponding targets 204, 204, lines 202a-202d on the roads of the multi-agent environment 200, and boundaries 212a, 212b of the multi-agent environment 200.

[0076] In an exemplary implementation, after determining the simulation model (at block 308 of method 300), the simulation module 132 can transmit data related to the simulation model to the policy learning module 134. The policy learning module 134 can then use the simulation model to perform one or more iterations of a random game to learn the corresponding agent action policies associated with the self-agent 102 and the target agent 104.

[0077] Figure 4 This includes an illustrative example of a simulation model 400 that virtually represents a multi-agent environment 200 according to an exemplary embodiment of this disclosure. The simulation model 400 can be processed by the simulation module 132 of the multi-agent application 106 based on the execution of method 300, as discussed above. In one embodiment, the simulation model 400 may include a simulated virtual model of a self-agent 102 provided as a virtual self-agent 102a, the virtual self-agent being presented at a corresponding location in a simulation model that replicates the real-world surrounding environment of the self-agent 102 within the multi-agent environment 200. The simulation model 400 may also include a virtual model of a target self 104 provided as a virtual target self 104a, the virtual target self being presented at a corresponding location in the simulation model 400 that replicates the real-world location of the target self 104 within the multi-agent environment 200.

[0078] like Figure 4 As shown, the corresponding targets 204 and 206 of self-subject 102 and target subject 104 can also be represented virtually as corresponding virtual targets 204a and 206a within the simulation model 400. In one or more embodiments, the simulation model 400 can be used to learn one or more subject action strategies associated with self-subject 102 and / or target subject 104, respectively, during one or more executions of a random game relative to the virtual self-subject 102a representing self-subject 102 and the virtual target subject 104a representing target subject 104.

[0079] In some implementations, the simulation model 400 may further include vehicle dynamic data points (not shown) that can be interpreted by the multi-agent application 106. The vehicle dynamic data points may be represented as vectors having real-valued parameters associated with the virtual self-agent 102a and the virtual target agent 104a, respectively. Relative to the virtual self-agent 102a, the real-valued parameters may correspond to the speed of the virtual self-agent 102a, the steering angle of the virtual self-agent 102a, the acceleration rate of the virtual self-agent 102a, the deceleration rate of the virtual self-agent 102a, etc. Similarly, relative to the virtual target agent 104a, the real-valued parameters may correspond to the speed of the virtual target agent 104a, the steering angle of the virtual target agent 104a, the acceleration rate of the virtual target agent 104a, the deceleration rate of the virtual target agent 104a, etc. In one implementation, these real-valued parameters may be adjusted for the self-agent 102 and / or the target agent 104 based on the training of the neural network 108, thereby allowing the self-agent 102 and the target agent 104 to reach their respective targets 204, 206 without any conflict between them.

[0080] Figure 5 This is a process flowchart of a method 500 for learning subject action strategies to be executed to control the operation of ego subject 102 and / or target subject 104 within a multi-subject environment 200, according to an exemplary embodiment of this disclosure. (See also...) Figure 1 , Figure 2 and Figure 4 Component description Figure 5 However, it should be understood that Figure 5 The method can be used with other systems / components. Method 500 may begin at block 502, where method 500 may include receiving data associated with simulation model 400.

[0081] In an exemplary implementation, the simulation module 132 of the multi-agent application 106 can transmit data associated with the simulation model 400 to the policy learning module 134. The policy learning module 134 can evaluate the data and determine observations associated with the multi-agent environment 200 from the perspectives of the ego agent 102 and the target agent 104.

[0082] Specifically, the strategy learning module 134 can evaluate data associated with the simulation model 400 and determine the target 206 of the self-agent 102, the path 202a in which the self-agent 102 is traveling, the additional paths 202b-202d in which the self-agent 102 is traveling, and the boundaries of the multi-agent environment 200, etc. Additionally, the strategy learning module 134 can evaluate data associated with the simulation model 400 and determine the target 204 of the target agent 104, the path 202d in which the target agent 104 is traveling, the additional paths 202a-202c in which the target agent 104 is traveling, and the boundaries of the multi-agent environment 200, etc.

[0083] The strategy learning module 134 can utilize such data to perform Markov games with respect to one or more executions of a virtual self-subject 102a representing self-subject 102 and a virtual target subject 104a representing target subject 104, to learn subject action strategies associated with self-subject 102 and / or target subject 104. Accordingly, simulation model 400 can be used to simulate one or more potential actions that can be performed by virtual self-subject 102a and / or virtual target subject 104a to independently reach their respective virtual targets 204a, 206a regardless of each other. These independent actions can be evaluated using a multi-subject central participant reviewer model and k-level recursive reasoning to learn the corresponding subject action strategies associated with self-subject 102 and target subject 104.

[0084] Method 500 can proceed to block 504, where method 500 may include augmenting each agent 102, 104 with a central agent component to model their conditional responses. In an exemplary embodiment, after evaluating data associated with the simulation model 400 and determining the virtual target 204a of the virtual self-agent 102a, the lanes 202s-202d of the road in which the virtual self-agent 102a and the virtual target agent 104a are traveling, the boundaries of the multi-agent environment 200, etc., the policy learning module 134 may model the relationship between the virtual self-agent 102a and the virtual target agent 104a and take into account their responses to the auxiliary central agent.

[0085] In an exemplary implementation, using simulation model 400, the Markov game can be derived from... The specification is given, where n is the number of subjects 102a and 104a; S is the state space containing the states of all subjects 102a and 104a; A · T represents the action space of subject i (where subject i is virtual self subject 102a when determining the reward for virtual self subject 102a, and virtual target subject 104a when determining the reward for virtual target subject 104a); The transition probability is given by the current state and the actions of all subjects 102a and 104a; r i : s0:S→R represents the reward for subject i; and s0:S→R represents the initial state distribution of all subjects 102a and 104a.

[0086] In one implementation, the learning objective of performing a Markov game is to obtain a set of policies. Where for each subject i, A i :S→A i The state is mapped to its action. However, since the policy learning module 134 can determine the inherent conflict relative to each self-interested agent attempting to reach its corresponding virtual target 204a, 206a in the most efficient way, the policy learning module 134 can utilize the Nash equilibrium in which all agents respond best to each other's current policies. In other words, by utilizing the Nash equilibrium, the virtual self agent 102a and the virtual target agent 104a may not perform better by unilaterally changing their own policies.

[0087] Using the participant-commenter framework, the policy learning module 134 can train the commenter, Qθ(s, a), to estimate the return value of the state-action pair (s, a), with a loss of... in It is a replay buffer for storing exploration experiences, and It is an empirical estimate of the return value. For participants... Training is performed to maximize the return value, with a loss of Additional items such as policy entropy can also be added. To improve training. In one implementation, the policy learning module 134 trains a central commentator for each subject i. To estimate the return values ​​of the state and the joint action; that is Then for each participant Training to minimize loss The strategy learning module 134 uses the equivalent term Q for each subject i. · (s, a, a-).

[0088] Using centralized training and distributed execution, the centralized commentator is defined as: Q i (s, a) i a -i And the dispersed participants are defined as: π i (a i |s). Training can be defined as: Q i (s, a) i a -i )←r+γV(s′); The commentator was thus used to determine how good a particular action was for selecting an optimal set of actions for the virtual self subject 102a and the virtual target subject 104a.

[0089] Refer again Figure 5 Method 500 can proceed to box 506, where method 500 may include using k-level reasoning to process the recursive reasoning graph. In the example, after completing action selection using a multi-agent participant commentator model, the strategy learning module 134 may utilize recursive reasoning as a process of reasoning about other agents during decision-making. For example, the strategy learning module 134 may utilize recursive reasoning as a process of reasoning about a target agent used by virtual self-agent 102a during decision-making. This functionality allows virtual self-agent 102a to consider potential changes in the strategy of virtual target agent 104a, rather than treating virtual target agent 104a as a fixed agent. Similarly, the strategy learning module 134 may utilize recursive reasoning as a process of reasoning about a virtual self-agent used by virtual target agent 104b during decision-making, to allow virtual self-agent 102a to consider potential changes in the strategy of virtual self-agent 102a, rather than treating virtual self-agent 102a as a fixed agent.

[0090] Using k-level reasoning, the policy learning module 134 can perform reasoning at various levels, making the operational decisions of the virtual self subject based on the operational decisions of the virtual target subject, and making the operational decisions of the virtual target subject based on the operational decisions of the virtual self subject. In other words, the policy learning module 134 utilizes a k-level reasoning model to perform recursive reasoning. At level 0, k-level reasoning is used, and all subjects (virtual self subject 102a and virtual target subject 104a) are based on base policy A. (0) The agent selects its corresponding action. The strategy learning module 134 performs k-level reasoning, such that at each level k, each agent selects the optimal strategy by assuming that other agents follow a k-1 level strategy. For example, for levels 1, 2, and k, the virtual self agent 102a can select the optimal strategy by assuming that the virtual target agent 104a follows the virtual self agent's k-1 level strategy.

[0091] In multi-agent RL, the natural level 0 policy is the agent's current policy, i.e., A. i，(0) =A i Given the actions k-1 of other subjects at a given level: a -i，(k-1) The optimal k-th level action for subject i should be

[0092] a i，(k) =arg max Q i (s, a) i a -i，(k-1) (1)

[0093] Q · This is the estimated return value of subject i. This formula applies to general and sum games.

[0094] Approximated by minimizing the following loss

[0095]

[0096] The central participant πci(s,a) was introduced. -i (), which learns i given state s and the actions a of other agents. - The best response.

[0097] In the illustrative example, at level zero, the virtual self-agent 102a can be based on its base policy A (0) The virtual target entity 104a is regarded as an obstacle.

[0098] Level 0: a i，(0) =π i (a i |s)

[0099] The strategy learning module 134 can thus perform first-level reasoning to allow the virtual ego subject 102a to consider the actions of the virtual target subject 104a (if the virtual ego subject 102a merges to reach its virtual target 206a), such that the virtual target subject 104a can brake to allow the virtual ego subject 102a to merge, or the virtual target subject 104a itself can merge simultaneously to reach its virtual target 204a. The strategy learning module 134 can thus perform level k-level reasoning (e.g., level 2 reasoning) to determine the optimal action that the virtual ego subject 102a should take based on the previous level (e.g., level 1) actions of the virtual target subject 104a, since this is based on the fact that both the virtual ego subject 102a and the virtual target subject 104a are central participants and are known to be so. Accordingly, level k:

[0100] Based on message passing among the central participants, k-level reasoning ensures that at each level k after level 0, each agent's policy, when determining the optimal action to take to reach the corresponding virtual targets 204a and 206a of virtual self agent 102a and virtual target agent 104a, takes into account the actions of other agents. In one implementation, the policy learning module 134 may use a recursive reasoning graph R2G: The message passing process in the algorithm is used to calculate a. -i' k. Node set Each subject 102 and 104 has a central participant node, and the edge set ε contains all the edges between interacting subjects 102 and 104. Accordingly, the node set contains the central participant node of the virtual self subject 102a and the central participant node of the virtual target subject 104a, and the edge set contains the edges between the virtual self subject 102a and the virtual target subject 104a.

[0101] Assuming all entities interact with each other, an undirected fully connected graph can be used. The messages in the edges are sampled actions 'a' from the central participant. i' k. The initial level 0 actions are sampled from various policies:

[0102] a i' 0~A i (s)(3) At each level k≥1, we have:

[0103]

[0104] Where AGG is an aggregation function, and It is a node neighborhood function. Since cascading can be used with AGG, therefore... In a fully connected graph, it can be associated with a -i exchange.

[0105] Accordingly, as discussed above, at level k, each central participant node receives a i' Given input message k-1, output its best response a. i' k. Therefore, a complete message traversing the graph ascends one level recursively. Through reparameterization, the entire computation relative to a i' 0 is differentiable. Therefore, it provides values ​​and gradient information for higher-level responses. The computational complexity is linearly related to the number of subjects n and the recursion level k.

[0106] Figure 6 An illustrative example of a recursive reasoning graph in a three-player random game is provided. As shown in the figure, the action a of the central player 602... n k can be evaluated by commentator 604 to output a policy 606 for each central participant 602, which includes the optimal set of actions for the virtual self-subject 102a and the virtual target subject 104a. Accordingly, k-level recursive reasoning uses a message-passing process in the recursive reasoning graph, where the set of nodes in the recursive reasoning graph contains the central participant nodes for each subject, and the set of edges contains the edges between the self-subject and the target subject. Each participant node accepts input messages based on the k-level policy and outputs a response based on the previous level actions of the relative subject.

[0107] In one configuration, each component of the recursive reasoning graph is trained such that, for the central commentator... Using soft Bellman residuals:

[0108]

[0109] The next state value V(s′)Reject is estimated as follows:

[0110]

[0111]

[0112] Among them Reject i It is a trained temperature variable, and This is a delayed update version of the commentator network. Regarding each strategy... Train to use a -i，(k) Compared to The corresponding energy-based distribution represents the one that minimizes the KLdivergence:

[0113]

[0114]

[0115] Central participants Training is performed using the loss given in Equation 2 above. The output of the recursive inference graph will be the subject action policy of each of the virtual self-subject 102a and the virtual target subject 104a. In summary, the subject action policy output based on the recursive inference graph can be expressed as: Policy

[0116] In an exemplary embodiment, the policy learning module 134 may be configured to transmit data associated with the agent action policies learned for virtual self-agent 102a and virtual target agent 104a to the neural network training module 136 of the multi-agent application 106. Upon receiving the data associated with the agent action policies learned by the policy learning module 134, the neural network training module 136 may access the multi-agent machine learning dataset 112 and populate one or more fields associated with each of self-agent 102 and / or target agent 104 with the corresponding agent action policies associated with self-agent 102 and / or target agent 104 for multiple executions based on a recursive inference graph using k-level inference at multiple corresponding time steps. Accordingly, the neural network 108 may be trained at multiple time steps with multiple agent action policies that can be used to autonomously control self-agent 102 and / or target agent 104 to reach their respective targets 204, 206 within the multi-agent environment 200 without any conflict between them.

[0117] Refer again Figure 5 Method 500, after utilizing trained agent action policies based on a recursive reasoning graph to consider higher-level recursive actions on the interacting agents, can proceed to box 508, where method 500 may include analyzing a multi-agent machine learning dataset 112 and implementing agent action policies to manipulate ego agent 102 and / or target agent 104. In an exemplary embodiment, agent control module 138 may access the multi-agent machine learning dataset 112 and may analyze the dataset to retrieve agent action policies associated with ego agent 102 and / or target agent 104 at a specific time step.

[0118] Method 500 can proceed to block 510, wherein method 500 may include autonomously controlling the self-subject 102 and / or the target subject 104 based on the corresponding subject action strategy. In an exemplary embodiment, after retrieving the subject action strategy associated with the self-subject 102 and / or the subject action strategy associated with the target subject 104 at a specific time step, the subject control module 138 may transmit the corresponding data associated with the subject action strategy associated with the self-subject 102 and / or the subject action strategy associated with the target subject 104 to the ECU 110a of the self-subject 102 and / or the ECU 110b of the target subject 104 to autonomously control the corresponding subjects 102, 104 based on the corresponding associated subject action strategy.

[0119] In an exemplary embodiment, the subject control module 138 can analyze the subject action strategy associated with the self-subject 102 and / or the target subject 104 and thereby communicate with the ECU 110a of the self-subject 102 and / or the ECU 110b of the target subject 104 to control the self-subject 102 and / or the target subject 104 to operate autonomously (or semi-autonomously) within the multi-subject environment 200 according to the corresponding multi-subject strategy (e.g., driving). The ECUs 110a, 110b can communicate with one or more of the corresponding system / control unit (not shown) to control the self-subject 102 and / or the target subject 104 to maneuver within the multi-subject environment 200 while following a specific path at a corresponding speed, acceleration rate, steering angle, deceleration rate, etc., to reach the corresponding targets 204, 206 without any conflict.

[0120] As an illustrative example, such as Figure 2As shown, due to the autonomous control of subjects 102 and 104 (based on the execution of separately associated subject action policies based on k-level recursive reasoning), self-subject 102 and target subject 104 must cross paths to reach their respective targets 204 and 206, so subjects 102 and 104 can successfully interact without any conflict. Accordingly, based on the associated subject action policies learned and trained by the multi-subject application 106 into the neural network 108 to be implemented, target subject 104 can be autonomously controlled to decelerate at a specific deceleration rate to allow self-subject 102 to merge towards its target 206. After self-subject 102 passes through the potential overlap point between subjects 102 and 104, target subject 104 can subsequently accelerate to efficiently reach its target 204. It should be understood that alternative types of autonomous control can be followed based on alternative subject action policies based on k-level recursive reasoning.

[0121] Figure 7 This is a flowchart of a method 700 for utilizing recursive reasoning graphs in multi-agent reinforcement learning according to an exemplary embodiment of this disclosure. (Refer to...) Figure 1 Component description Figure 7 However, it should be understood that Figure 7 The method can be used with other systems / components. Method 700 may begin at block 702, wherein method 700 may include receiving data associated with self-subject 102 and target subject 104 traveling within the multi-subject environment 200.

[0122] Method 700 can proceed to box 704, wherein method 700 may include using a multi-subjective participant commentator framework to analyze data associated with self-subject 102 and target subject 104. Method 700 can proceed to box 706, wherein method 700 may include performing k-level recursive inference based on the multi-subjective participant commentator framework to compute higher-level recursive actions for self-subject 102 and target subject 104.

[0123] In one implementation, the output of k-level recursive inference is used to learn subject action policies associated with self-subject 102 and subject action policies associated with target subject 104. Method 700 may proceed to block 708, wherein method 700 may include controlling at least one of self-subject 102 and target subject 104 to operate within the multi-subject environment 200 based on at least one of the subject action policies associated with self-subject 102 and subject action policies associated with target subject 104.

[0124] It will be apparent from the foregoing description that various exemplary embodiments of the present invention can be implemented in hardware. Furthermore, the various exemplary embodiments can be implemented as instructions stored on a non-transitory machine-readable storage medium, such as volatile or non-volatile memory, which can be read and executed by at least one processor to perform the operations described in detail herein. Machine-readable storage media can include any mechanism for storing information in a machine-readable form, such as personal or laptop computers, servers, or other computing devices. Therefore, non-transitory machine-readable storage media do not include transient signals but can include volatile and non-volatile memory, including but not limited to read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash memory devices, and similar storage media.

[0125] Those skilled in the art will understand that any block diagram herein represents a conceptual diagram of an illustrative circuit embodying the principles of the invention. Similarly, it should be understood that any flowchart, diagram, state transition diagram, pseudocode, etc., represents various processes that can be substantially represented in a machine-readable medium and thus executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0126] It should be understood that the various implementations and other features and functions disclosed above, or their alternatives or variations, can be expected to be combined with many other different systems or applications. Furthermore, those skilled in the art can subsequently make various alternatives, modifications, variations, or improvements that are not currently foreseeable or anticipated, and these are also intended to be covered by the appended claims.

Claims

1. A computer-implemented method for utilizing recursive reasoning graphs in multi-agent reinforcement learning, comprising: Receive data associated with a self-subject and a target subject moving within a multi-subject environment, wherein receiving data associated with the multi-subject environment includes receiving image data and LiDAR data from at least one of the self-subject and the target subject; The data associated with the self-subject and the target subject are analyzed using a multi-subject-centered participant-commentator framework. Based on the multi-subject central participant commentator framework, k-level recursive reasoning is performed to compute higher-level recursive actions of the self-subject and the target subject, wherein the output of the k-level recursive reasoning is used to learn subject action strategies associated with the self-subject and subject action strategies associated with the target subject. as well as The vehicle's steering, acceleration, or braking are driven by executing a subject action strategy learned from the k-level recursive inference on an electronic control device of either the ego subject or the target subject, based on at least one of the following: the subject action strategy associated with the ego subject and the subject action strategy associated with the target subject. in, Performing k-level recursive reasoning involves representing the self-subject and the target subject as respective central participant nodes in a recursive reasoning graph, wherein the k-level action of each central participant node is updated through actions of previous levels that include other central participants. The multi-subject central participant commentator framework includes respective central participant components for the self-subject and the target subject, such that each central participant component receives data reflecting actions at previous levels to generate higher-level response actions as part of the k-level recursive reasoning.

2. The computer-implemented method of claim 1, wherein, Multimodal fusion processing is performed to aggregate the image data and the LiDAR data into fused environmental data.

3. The computer-implemented method of claim 2, wherein the image data and the LiDAR data are aggregated to determine a simulated multi-agent environment, and the simulation of a virtual environment is processed in the simulated multi-agent environment to perform at least one iteration of a random game.

4. The computer-implemented method of claim 3, wherein utilizing the multi- agent-centric participant reviewer framework comprises: The self-agent and the target agent are expanded based on the central participant component to model each agent’s conditional response to each other within at least one iteration of the random game.

5. The computer-implemented method of claim 4, wherein utilizing the multi- agent hub participant reviewer framework comprises: The reward is determined for each subject, wherein the transition probability is conditioned on the current state of each subject and the actions of the self subject and the target subject, wherein the Nash equilibrium of the actions of the self subject and the target subject in response to each other's current policies is utilized.

6. The computer-implemented method of claim 1, wherein performing k-level recursive reasoning based on the multi-agent-centric participant reviewer framework comprises: The recursive reasoning graph uses a message passing process, wherein the node set of the recursive reasoning graph contains a central participant node for each subject and the edge set contains edges between the self subject and the target subject, wherein each participant node accepts input messages based on a k-level policy and outputs a response based on the previous level action of the relative subject.

7. The computer-implemented method of claim 1, wherein performing k-level recursive reasoning based on the multi-agent-centric participant reviewer framework comprises: The zero-level policy is implemented as the base policy, wherein the self-agent and the target agent regard each other as obstacles to reaching the corresponding goal based on the zero-level policy.

8. The computer-implemented method of claim 7, wherein performing k-level recursive reasoning based on the multi-agent-centric participant reviewer framework comprises: Implement a k-level policy for each subject, the k-level policy taking into account the actions of the relative subject based on the past actions of the corresponding subject.

9. The computer-implemented method of claim 8, wherein a complete message traversing the recursive reasoning graph ascends one level in the recursion, wherein the subject action policy is output for the self-subject and the subject action policy is output for the target subject based on the corresponding k-level policy output from the recursive reasoning graph.

10. A system for utilizing recursive reasoning graphs in multi-agent reinforcement learning, comprising: The memory stores instructions that, when executed by the processor, cause the processor to: Receive data associated with a self-subject and a target subject moving within a multi-subject environment, wherein receiving data associated with the multi-subject environment includes receiving image data and LiDAR data from at least one of the self-subject and the target subject; The data associated with the self-subject and the target subject are analyzed using a multi-subject-centered participant-commentator framework. Based on the multi-subject central participant commentator framework, k-level recursive reasoning is performed to compute higher-level recursive actions of the self-subject and the target subject, wherein the output of the k-level recursive reasoning is used to learn subject action strategies associated with the self-subject and subject action strategies associated with the target subject. as well as The vehicle's steering, acceleration, or braking are driven by executing a subject action strategy learned from the k-level recursive inference on an electronic control device of either the ego subject or the target subject, based on at least one of the following: the subject action strategy associated with the ego subject and the subject action strategy associated with the target subject. in, Performing k-level recursive reasoning involves representing the self-subject and the target subject as respective central participant nodes in a recursive reasoning graph, wherein the k-level action of each central participant node is updated through actions of previous levels that include other central participants. The multi-subject central participant commentator framework includes respective central participant components for the self-subject and the target subject, such that each central participant component receives data reflecting actions at previous levels to generate higher-level response actions as part of the k-level recursive reasoning.

11. The system of claim 10, wherein, Multimodal fusion processing is performed to aggregate the image data and the LiDAR data into fused environmental data.

12. The system of claim 11, wherein the image data and the LiDAR data are aggregated to determine a simulated multi-agent environment, and the simulation of a virtual environment is processed in the simulated multi-agent environment to perform at least one iteration of a random game.

13. The system of claim 12, wherein utilizing the multi-agent hub participant reviewer framework comprises: The self-agent and the target agent are expanded based on the central participant component to model each agent’s conditional response to each other within at least one iteration of the random game.

14. The system of claim 13, wherein utilizing the multi-agent hub participant reviewer framework comprises: The reward is determined for each subject, wherein the transition probability is conditioned on the current state of each subject and the actions of the self subject and the target subject, wherein the Nash equilibrium of the actions of the self subject and the target subject in response to each other's current policies is utilized.

15. The system of claim 10, wherein performing k-level recursive reasoning based on the multi-agent-centric participant reviewer framework comprises: The recursive reasoning graph uses a message passing process, wherein the node set of the recursive reasoning graph contains a central participant node for each subject and the edge set contains edges between the self subject and the target subject, wherein each participant node accepts input messages based on a k-level policy and outputs a response based on the previous level action of the relative subject.

16. The system of claim 10, wherein performing k-level recursive reasoning based on the multi-agent-centric participant reviewer framework comprises: The zero-level policy is implemented as the base policy, wherein the self-agent and the target agent regard each other as obstacles to reaching the corresponding goal based on the zero-level policy.

17. The system of claim 16, wherein performing k-level recursive reasoning based on the multi-agent-centric participant reviewer framework comprises: Implement a k-level policy for each subject, the k-level policy taking into account the actions of the relative subject based on the past actions of the corresponding subject.

18. The system of claim 17, wherein a complete message traversing the recursive reasoning graph ascends one level in the recursion, wherein the subject action policy is output for the self-subject and the subject action policy is output for the target subject based on the corresponding k-level policy output from the recursive reasoning graph.

19. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer including a processor, perform a method comprising: Receive data associated with a self-subject and a target subject moving within a multi-subject environment, wherein receiving data associated with the multi-subject environment includes receiving image data and LiDAR data from at least one of the self-subject and the target subject; The data associated with the self-subject and the target subject are analyzed using a multi-subject-centered participant-commentator framework. Based on the multi-subject central participant commentator framework, k-level recursive reasoning is performed to compute higher-level recursive actions of the self-subject and the target subject, wherein the output of the k-level recursive reasoning is used to learn subject action strategies associated with the self-subject and subject action strategies associated with the target subject. as well as The vehicle's steering, acceleration, or braking are driven by executing a subject action strategy learned from the k-level recursive inference on an electronic control device of either the ego subject or the target subject, based on at least one of the following: the subject action strategy associated with the ego subject and the subject action strategy associated with the target subject. in, Performing k-level recursive reasoning involves representing the self-subject and the target subject as respective central participant nodes in a recursive reasoning graph, wherein the k-level action of each central participant node is updated through actions of previous levels that include other central participants. The multi-agent center participant commentator framework includes respective center participant components of the ego agent and the target agent such that each center participant component receives data reflecting previous level of action to generate a higher level of response action as part of the k-level recursive reasoning.

20. The non-transitory computer-readable storage medium of claim 19, wherein performing k-level recursive reasoning based on the multi-agent-centric participant reviewer framework comprises: A message passing procedure is used in a recursive reasoning graph, where a node set of the recursive reasoning graph includes a center participant node of each agent and an edge set includes an edge between the ego agent and the target agent, where each participant node accepts an input message based on a k-level policy and outputs a response based on previous level of action of the relative agent.

Images