Decision based on multi-round negotiation

Abstract

According to the implementation of the invention, a scheme for decision making is provided. In this approach, a target agent may receive a set of observation information from a set of associated agents. Further, the target agent may generate a first intermediate decision for the target agent based on a set of observation information and target observation information for the target agent. Additionally, the target agent may also receive a first set of intermediate decisions from a set of associated agents, and may generate a target decision for the target agent based at least on the generated first intermediate decision and the received first set of intermediate decisions. Based on the mode, the embodiment of the invention can support the agent not only to consider the observation information of other associated agents, but also to consider the intermediate decision of other associated agents in the decision making process, so that the execution effect of the decision can be improved in a negotiation mode.

Description

Background Art

[0001] In recent years, with the development of computer technology, reinforcement learning technology has been widely used. In the joint reinforcement learning process involving multiple agents, multiple agents can share information through negotiation, thereby improving the execution effect of decision-making in the environment.

[0002] For example, in a traffic scenario, multiple traffic participants can participate in reinforcement learning-based decision-making as multiple intelligent agents, and improve the execution effect of decisions about traffic behavior in the traffic environment through information sharing, for example, reducing the probability of collision. Summary of the invention

[0003] According to the implementation of the present disclosure, a scheme for decision-making is provided. In the scheme, a target intelligent agent can receive a set of observation information from a group of associated intelligent agents. Further, the target intelligent agent can generate a first intermediate decision for the target intelligent agent based on a set of observation information and target observation information for the target intelligent agent. Additionally, the target intelligent agent can also receive a first group of intermediate decisions from a group of associated intelligent agents, and can generate a target decision for the target intelligent agent based at least on the generated first intermediate decision and the received first group of intermediate decisions. Based on this approach, the embodiments of the present disclosure can support the intelligent agent to consider not only the observation information of other associated intelligent agents in the process of making decisions, but also the intermediate decisions of other associated intelligent agents, thereby improving the execution effect of the decision through negotiation.

[0004] The Summary is provided to introduce the concepts in a simplified form, which will be further described in the Detailed Description below. The Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 A schematic diagram illustrating an example environment according to some implementations of the present disclosure is shown;

[0006] Figure 2 A flowchart illustrating an example method for decision making according to some implementations of the present disclosure is shown;

[0007] Figure 3A and Figure 3B A schematic block diagram of a reinforcement learning system according to some implementations of the present disclosure is shown;

[0008] Figure 4A and Figure 4B shows example tests according to some embodiments of the present disclosure;

[0009] Figure 5 shows an example training process according to some implementations of the present disclosure; and

[0010] Figure 6 A block diagram of an example electronic device is shown in accordance with some implementations of the present disclosure.

[0011] In these drawings, the same or similar reference symbols are used to designate the same or similar elements. DETAILED DESCRIPTION

[0012] The present disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and thus implement the present disclosure, and do not imply any limitation on the scope of the present subject matter.

[0013] As used herein, the term "including" and variations thereof are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "based at least in part on". The terms "an implementation" and "an implementation" are to be interpreted as "at least one implementation". The term "another implementation" is to be interpreted as "at least one other implementation". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0014] As discussed above, in the joint reinforcement learning process involving multiple agents, multiple agents can share information through negotiation. How to improve the execution effect of the agent's decision in the environment during the reinforcement learning process is the focus of this field.

[0015] According to the implementation of the present disclosure, a scheme for decision-making is provided. In the scheme, a target intelligent agent can receive a set of observation information from a group of associated intelligent agents. Further, the target intelligent agent can generate a first intermediate decision for the target intelligent agent based on a set of observation information and target observation information for the target intelligent agent. Additionally, the target intelligent agent can also receive a first group of intermediate decisions from a group of associated intelligent agents, and can generate a target decision for the target intelligent agent based at least on the generated first intermediate decision and the received first group of intermediate decisions. Based on this approach, the embodiments of the present disclosure can support the intelligent agent to consider not only the observation information of other associated intelligent agents in the process of making decisions, but also the intermediate decisions of other associated intelligent agents, thereby improving the execution effect of the decision through negotiation.

[0016] The basic principles and several example implementations of the present disclosure are explained below with reference to the accompanying drawings.

[0017] Example Environment

[0018] Figure 1An example environment 100 is shown according to some implementations of the present disclosure. Figure 1 As shown, the environment 100 may be a reinforcement learning system 120 composed of multiple agents. For example, the reinforcement learning system includes multiple agents 110-1, 110-2, 110-3, ..., 110-N (individually or collectively referred to as agent 110, where N is an integer greater than 1).

[0019] To facilitate subsequent understanding, let’s first briefly introduce reinforcement learning.

[0020] Reinforcement learning, also known as reinforcement learning, evaluation learning or enhanced learning, is one of the machine learning techniques used to describe and solve the problem of how an intelligent agent learns strategies to maximize rewards or achieve specific goals during its interaction with the environment. Reinforcement learning focuses on the interaction between the agent and the environment, and the goal of reinforcement learning is generally to pursue maximum rewards. In other words, reinforcement learning is a learning mechanism that learns how to map from state to behavior in order to maximize the reward obtained. Such an intelligent agent needs to constantly interact and learn in the environment, and continuously optimize the state-behavior correspondence through feedback (rewards) given by the environment.

[0021] like Figure 1 As shown, the agent 110-1 can make its target decision 130 based on the reinforcement learning system 120. As an example scenario, the agent 110 can correspond to multiple traffic participants in the traffic environment, such as vehicles. Accordingly, each traffic participant can make its target decision, such as a decision about traffic behavior, through the reinforcement learning system 120.

[0022] The process of determining the target decision 130 will be described in detail below.

[0023] Example Process

[0024] Figure 2 FIG. 2 is a flowchart of an example process 200 for decision making according to some implementations of the present disclosure. The process 200 may be performed by, for example, Figure 1 The process 200 is implemented by the agent 110 in the example of the target agent. The process 200 is described below using the agent 110-1 as an example of the target agent.

[0025] like Figure 2 As shown, at block 210, the target agent 110-1 receives a set of observation information from a set of associated agents associated with the target agent.

[0026] Traditionally, multiple agents 305 can make decisions through multiple rounds of consultation, the process of which can be expressed as follows:

[0027]

[0028]

[0029]

[0030] For each agent i, the extractor E can be used to process the observation data o i , and store it as a hidden state Furthermore, multiple agents can perform one or more rounds (K rounds) of communication with other agents through the communication function C. Accordingly, the hidden state can be updated as Finally, the updated hidden state can be As input, the decision function D is used to determine the final target decision (or action) a made by the agent. i .

[0031] It can be seen that in the traditional scheme, only the communication between hidden states is involved. However, this approach does not take into account the intention information of other agents, which may affect the decision-making process and further affect the execution effect of the reinforcement learning task.

[0032] The following will refer to Figure 3A and Figure 3B The process 200 is described below. Figure 3A and Figure 3B An example block diagram of an example reinforcement learning system according to some implementations of the present disclosure is shown.

[0033] like Figure 3A As shown, the reinforcement learning system 300A may include a plurality of agents 305-1, 305-i, to 305-n (individually or collectively referred to as agents 305). Figure 3A As shown, agent 305 can utilize communication module 315 to obtain observation information of one or more other associated agents.

[0034] For example, the agent 305 may utilize corresponding extractors 310 - 1 , 310 - i , to 310 - n (individually or collectively referred to as extractors 310 ) to process corresponding observation data, thereby determining its corresponding hidden state as corresponding observation information.

[0035] Figure 3B Further illustrated are example communication modules according to some implementations of the present disclosure. Figure 3B As shown, the communication module 315 may include, for example, three processes: a message encoding process, a communication throttling process, and an information aggregation process, which may be expressed as:

[0036]

[0037]

[0038]

[0039] In the communication process, the encoder 340 can be used to hide the state Encode to get the message Furthermore, considering the communication cost and the fact that not all other agents need to obtain the message of the current agent in the actual process, the current limiting function H can also be used to determine the binary value To indicate whether a message from agent j can be sent to agent i Further, each receiver 345 can aggregate the messages successfully received from other associated agents, and can use the function P to update the hidden state corresponding to the agent to

[0040] In some implementations, the communication throttling process may also be omitted. For example, each agent may always receive observation information (eg, hidden variables) from all other associated agents.

[0041] by Figure 3A As an example, agent 305 - 1 may receive its corresponding observation information, such as hidden states, from one or more other associated agents (eg, agent 305 - i, etc.).

[0042] Further, the agent 305-1 may perform aggregation of the observation information based on the detected observation information and the observation information determined by the extractor 310-1, thereby generating an updated hidden state, also referred to as first aggregate information.

[0043] Continue to refer Figure 2 In box 220, the target agent 110-1 generates a first intermediate decision for the target agent based on a set of observation information and target observation information for the target agent, and the first intermediate decision indicates a first candidate action determined by the target agent.

[0044] Further, Figure 3A As an example, the agent 305-1 can generate an intermediate decision 320-1 based on the first aggregate information (i.e., the updated hidden state) determined by the communication module 315. Such an intermediate decision 320-1 can also be referred to as an intended action of the agent 305-1, which can reflect the intention of the agent 305-1 based on the current input. Taking the traffic scene as an example, the intermediate decision can indicate the traffic behavior intention of the participants based on the current round of communication.

[0045] Similarly, other agents can also make corresponding intermediate decisions based on similar processes, such as intermediate decisions 320-i to intermediate decisions 320-n. Taking intermediate decision 320-i as an example, agent 350-i can obtain observation information from one or more other associated agents (e.g., agent 305-1) and update the hidden state based on the aggregation of the observation information, thereby determining the corresponding intermediate decision 320-i.

[0046] Continue to refer Figure 2 , in block 230 , the target agent 110 - 1 receives a first set of intermediate decisions from a set of associated agents, the first set of intermediate decisions indicating candidate actions determined by the corresponding associated agents.

[0047] Continue with Figure 3A As an example, each agent 305 may provide its hidden state (also referred to as first aggregate information) updated based on the communication module 315 and the corresponding intermediate decision 320 to the communication module 325. Exemplarily, in the communication module 325, the hidden state and / or intermediate decision of each agent 305 may be sent to some associated agents according to the communication flow limiting strategy, or may be sent to all associated agents.

[0048] Taking agent 305 - 1 as an example, agent 305 - 1 can use the communication module 325 to obtain intermediate decisions and corresponding hidden states made by one or more other associated agents.

[0049] Continue to refer Figure 2 In box 240, the target agent 110-1 generates a target decision for the target agent based on at least the generated first intermediate decision and the received first group of intermediate decisions, and the target decision indicates a target action to be performed by the target agent.

[0050] Continue to refer Figure 3A For example, agent 305-1 can utilize communication module 325 to update the hidden state based on the process discussed above to determine the target decision 330-1.

[0051] Specifically, the agent 305-1 may generate second aggregate information, i.e., the hidden state updated by the communication module 325, based at least on the intermediate decision 320-1 and one or more other intermediate decisions received (e.g., the intermediate decision 320-i, etc.). The above process of updating the hidden state may be referred to as a round of negotiation.

[0052] Further, the agent 305-1 may directly generate the target decision 330-1 based on the second aggregate information. That is, the agent 305-1 may directly generate the target decision after two rounds of negotiation.

[0053] Alternatively, the agent 305-1 may also perform more rounds of negotiation. Exemplarily, the agent 305-1 may generate a decision 330-1 based on the second aggregate information. In this case, the decision 330-1 is an intermediate decision, also referred to as a second intermediate decision.

[0054] Additionally, other associated agents (eg, agents 305 - i to 305 - n ) may also generate corresponding intermediate decisions (eg, intermediate decisions 330 - i to 330 - n ) based on a similar process.

[0055] Taking agent 305-i as an example, it can use communication module 325 to update the corresponding hidden state to generate intermediate decision 330-i. In the process of updating the hidden state, agent 305-i can, for example, aggregate intermediate decisions (e.g., intermediate decision 320-1) made by other agents (e.g., agent 305-1) to generate intermediate decision 330-i.

[0056] Further, agent 305 - 1 may perform aggregation of information based on intermediate decision 330 - 1 and one or more intermediate decisions (eg, intermediate decision 330 - i ) received from other associated agents, and update the hidden state accordingly.

[0057] Such a negotiation process may be performed in two or more rounds to generate target decisions corresponding to each agent 305, for example, target decision 335-1, target decision 335-i to target decision 335-n.

[0058] Based on such a process, the implementation of the present disclosure not only considers the observation information of other intelligent agents during reinforcement learning, but also considers the intermediate intentions of other intelligent agents, thereby improving the execution effect of the decisions made by each intelligent agent.

[0059] Specifically, by considering the intermediate intentions of other agents in the process of agent decision-making, the implementation of the present disclosure can strengthen the decision-making consultation between agents, avoid possible decision conflicts among agents, optimize the decision-making process, and thus improve the execution effect of the decision. Taking the traffic scene as an example, through the communication of intermediate intentions, the probability of decision conflicts (e.g., collisions) among the traffic parties can be reduced, thereby improving the safety of traffic travel.

[0060] Furthermore, the present disclosure is designed as follows Figure 4A Two traffic test scenarios 410 and 420 are shown to compare the multi-round negotiation scheme of intermediate intention with other technical schemes, such as the proximal policy optimization (PPO) algorithm, the CommNet algorithm and the target multi-agent communication neural network (TarMAC) algorithm.

[0061] Exemplarily, the test scene 410 includes an intersection consisting of two roads, and its size is 14 grids * 14 grids. The test scene 420 is more complex than the test scene 410, and includes four intersections consisting of two bidirectional roads, and its size is 18 grids * 18 grids.

[0062] In test scenario 410, the actions that each agent (i.e., vehicle) is allowed to perform include: braking, i.e., staying at the original position; advancing, i.e., moving one step along the specified path. In test scenario 420, the actions that each agent (i.e., vehicle) is allowed to perform include: braking, i.e., staying at the original position; advancing, i.e., moving one step along the specified path; accelerating, i.e., moving two steps along the specified path.

[0063] Comparative test results such as Figure 4B As shown in the figure, it can be seen that the multi-round negotiation method based on intermediate intention (NeoNetC and NeoNetT) has a higher success rate and can obtain a better reward value, where the reward value is determined based on the number of collisions with other vehicles:

[0064]

[0065] in, represents the number of collisions between the i-th vehicle and other vehicles, represents the time step after the vehicle is added to the traffic environment, r coll = -10, and r time =-0.01. Therefore, a larger reward value indicates a better performance.

[0066] The training process of the agent will be further described below. Based on the process described above, different from the traditional reinforcement learning process (e.g., Formula 1), the reinforcement learning process based on multi-round negotiation introduced in the present disclosure can be expressed as:

[0067]

[0068]

[0069] Where N represents the negotiation process, K is the number of negotiations, is the intermediate decision of agent i in the k-round negotiation process, is the initialization value. In addition, the k-th round of negotiation process can be expressed as:

[0070]

[0071]

[0072] In the kth round of negotiation, the agent can send hidden states h to other associated agents. k and the intermediate decisions made Furthermore, the agent can communicate through the communication module C and can generate new intermediate decisions based on the content exchanged during the communication process and using the decision module D. After the above process is repeated for multiple rounds, the agent can determine its final target decision In some implementations, the number of negotiations K may be a preset value, may be dynamically adjusted according to actual applications, or may be determined during the training process of the intelligent agent.

[0073] In the process of training the intelligent agent, the embodiments of the present disclosure may utilize an extended Markov decision process MDP. In the extended MDP, each round of negotiation process may be represented as a step in the extended sequential process. Figure 5 As shown, the trajectory τ in the original MDP is t , a t ), ..., s T ) can be converted to:

[0074]

[0075] in, represents the state of the extended MDP at time step t after k rounds of negotiation; represents the intermediate decision of the extended MDP at time step t after k rounds of negotiation, and Accordingly, the reward function of the extended MDP can be defined as:

[0076]

[0077] in Therefore, the above reward function can be expressed as follows: During the training process, the training goal is to maximize the reward corresponding to each intermediate decision made by the agent.

[0078] In some implementations, the agent can also be trained using an advantage function, which can be expressed as:

[0079]

[0080] Where V represents the value function in the original MDP, and Can be initialized to 0, Represents the state transition function in the extended MDP. Thus, the above advantage function can be expressed as: During the training process, the training goal is to maximize the advantage corresponding to each intermediate decision made by the agent.

[0081] Therefore, the overall loss function used in the training process can be expressed as:

[0082]

[0083] Among them, θ can include parameters in the extractor and negotiation process, and KL (Kullback-Leibler) represents the KL divergence operation. The part of the loss function in formula (9) related to the action loss is intended to minimize the action loss corresponding to the intermediate decision during the parameter iteration process, that is, to maximize the advantage function corresponding to each intermediate decision. In addition, the part of the loss function related to the KL loss is intended to maximize the KL divergence, that is, to maximize the KL divergence of the parameters before the update (θ old ) and the updated parameters (θ) enable the policy to make the same intermediate decisions based on the same observations The probability of and π θ As close as possible.

[0084] Example Device

[0085] Figure 6 1 shows a schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure. It should be understood that Figure 6 The device 600 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the implementation described in the present disclosure. Figure 6 As shown, components of device 600 may include, but are not limited to, one or more processors or processing units 610 , memory 620 , storage devices 630 , one or more communication units 640 , one or more input devices 650 , and one or more output devices 660 .

[0086] In some implementations, the device 600 can be implemented as various user terminals or service terminals. The service terminal can be a server, a large computing device, etc. provided by various service providers. The user terminal is such as any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It is also foreseeable that the device 600 can support any type of interface for the user (such as a "wearable" circuit, etc.).

[0087] Processing unit 610 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of device 600. Processing unit 610 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0088] Device 600 generally includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., register, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory) or some combination thereof. Memory 620 can include one or more program products 625, which are configured to perform the functions of various implementations described herein. Storage device 630 can be a removable or non-removable medium, and can include a machine-readable medium, which can be used to store information and / or data and can be accessed in device 600.

[0089] The function of the component of device 600 can be realized with a single computing cluster or multiple computing machines, and these computing machines can communicate by communication connection. Therefore, device 600 can use the logical connection with one or more other servers, personal computers (PC) or another general network node to operate in a networked environment. Device 600 can also communicate with one or more external devices (not shown) by communication unit 640 as needed, and external devices such as databases, other storage devices, servers, display devices, etc. communicate with one or more devices that make users interact with device 600, or communicate with any device (for example, network card, modem, etc.) that makes device 600 communicate with one or more other computing devices. Such communication can be performed via input / output (I / O) interface (not shown).

[0090] The input device 650 may be one or more various input devices, such as a mouse, keyboard, trackball, voice input device, camera, etc. The output device 660 may be one or more output devices, such as a display, speaker, printer, etc.

[0091] Example Implementation

[0092] Some example implementations of the present disclosure are listed below.

[0093] In a first aspect of the present disclosure, a computer-implemented method is provided, comprising: receiving a set of observation information from a group of associated agents associated with a target agent; generating a first intermediate decision for the target agent based on the set of observation information and target observation information for the target agent, the first intermediate decision indicating a first candidate action determined by the target agent; receiving a first group of intermediate decisions from a group of associated agents, the first group of intermediate decisions indicating candidate actions determined by corresponding associated agents; and generating a target decision for the target agent based at least on the generated first intermediate decision and the received first group of intermediate decisions, the target decision indicating a target action to be performed by the target agent.

[0094] In some implementations, generating a first intermediate decision for a target agent includes: generating first aggregate information based on a set of observation information and target observation information for the first agent; and generating a first intermediate decision for the target agent based on the first aggregate information.

[0095] In some implementations, the method further includes sending the first aggregate information and the first intermediate decision to at least one associated agent in a group of associated agents.

[0096] In some implementations, at least one intermediate decision in the first set of intermediate decisions is determined based on target observation information received from the first agent.

[0097] In some implementations, a set of observation information includes observation information sent by at least one associated agent in a set of associated agents to a target agent.

[0098] In some implementations, generating a target decision for a target agent based at least on a generated first intermediate decision and a received set of intermediate decisions includes: generating second aggregate information based at least on the first intermediate decision and the set of intermediate decisions; generating a second intermediate decision based on the second aggregate information; receiving a second set of intermediate decisions from a group of associated agents; and generating a target decision for the target agent based at least on the generated second intermediate decision and the received second set of intermediate decisions.

[0099] In some implementations, at least one intermediate decision in the second set of intermediate decisions is determined based on the first intermediate decision received from the first agent.

[0100] In some implementations, the first agent and at least one of a set of associated agents are trained based on the following process: in the process of training the at least one agent, the at least one agent is trained by maximizing a set of intermediate benefits corresponding to a set of intermediate decisions made by the at least one agent.

[0101] In some implementations, the intermediate benefits are determined based on a reward function or an advantage function associated with the reinforcement learning process.

[0102] In some implementations, the target agent and the set of associated agents correspond to multiple traffic participants in the traffic environment.

[0103] In a second aspect of the present disclosure, an electronic device is provided. The device includes: a processing unit; and a memory, coupled to the processing unit and containing instructions stored thereon, which, when executed by the processing unit, cause the device to perform the following actions: receiving a set of observation information from a group of associated agents associated with a target agent; generating a first intermediate decision for the target agent based on the set of observation information and target observation information for the target agent, the first intermediate decision indicating a first candidate action determined by the target agent; receiving a first group of intermediate decisions from a group of associated agents, the first group of intermediate decisions indicating candidate actions determined by corresponding associated agents; and generating a target decision for the target agent based at least on the generated first intermediate decision and the received first group of intermediate decisions, the target decision indicating a target action to be performed by the target agent.

[0104] In some implementations, generating a first intermediate decision for a target agent includes: generating first aggregate information based on a set of observation information and target observation information for the first agent; and generating a first intermediate decision for the target agent based on the first aggregate information.

[0105] In some implementations, the process further includes sending the first aggregate information and the first intermediate decision to at least one associated agent in a group of associated agents.

[0106] In some implementations, at least one intermediate decision in the first set of intermediate decisions is determined based on target observation information received from the first agent.

[0107] In some implementations, a set of observation information includes observation information sent by at least one associated agent in a set of associated agents to a target agent.

[0108] In some implementations, generating a target decision for a target agent based at least on a generated first intermediate decision and a received set of intermediate decisions includes: generating second aggregate information based at least on the first intermediate decision and the set of intermediate decisions; generating a second intermediate decision based on the second aggregate information; receiving a second set of intermediate decisions from a group of associated agents; and generating a target decision for the target agent based at least on the generated second intermediate decision and the received second set of intermediate decisions.

[0109] In some implementations, at least one intermediate decision in the second set of intermediate decisions is determined based on the first intermediate decision received from the first agent.

[0110] In some implementations, the first agent and at least one of a set of associated agents are trained based on the following process: in the process of training the at least one agent, the at least one agent is trained by maximizing a set of intermediate benefits corresponding to a set of intermediate decisions made by the at least one agent.

[0111] In some implementations, the intermediate benefits are determined based on a reward function or an advantage function associated with the reinforcement learning process.

[0112] In some implementations, the target agent and the set of associated agents correspond to multiple traffic participants in the traffic environment.

[0113] In a third aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored in a non-transitory computer storage medium and includes machine executable instructions, which, when executed by a device, cause the device to perform the following actions: receiving a set of observation information from a group of associated agents associated with a target agent; generating a first intermediate decision for the target agent based on the set of observation information and target observation information for the target agent, the first intermediate decision indicating a first candidate action determined by the target agent; receiving a first group of intermediate decisions from a group of associated agents, the first group of intermediate decisions indicating candidate actions determined by the corresponding associated agents; and generating a target decision for the target agent based at least on the generated first intermediate decision and the received first group of intermediate decisions, the target decision indicating a target action to be performed by the target agent.

[0114] In some implementations, generating a first intermediate decision for a target agent includes: generating first aggregate information based on a set of observation information and target observation information for the first agent; and generating a first intermediate decision for the target agent based on the first aggregate information.

[0115] In some implementations, the process further includes sending the first aggregate information and the first intermediate decision to at least one associated agent in a group of associated agents.

[0116] In some implementations, at least one intermediate decision in the first set of intermediate decisions is determined based on target observation information received from the first agent.

[0117] In some implementations, a set of observation information includes observation information sent by at least one associated agent in a set of associated agents to a target agent.

[0118] In some implementations, generating a target decision for a target agent based at least on a generated first intermediate decision and a received set of intermediate decisions includes: generating second aggregate information based at least on the first intermediate decision and the set of intermediate decisions; generating a second intermediate decision based on the second aggregate information; receiving a second set of intermediate decisions from a group of associated agents; and generating a target decision for the target agent based at least on the generated second intermediate decision and the received second set of intermediate decisions.

[0119] In some implementations, at least one intermediate decision in the second set of intermediate decisions is determined based on the first intermediate decision received from the first agent.

[0120] In some implementations, the first agent and at least one of a set of associated agents are trained based on the following process: in the process of training the at least one agent, the at least one agent is trained by maximizing a set of intermediate benefits corresponding to a set of intermediate decisions made by the at least one agent.

[0121] In some implementations, the intermediate benefits are determined based on a reward function or an advantage function associated with the reinforcement learning process.

[0122] In some implementations, the target agent and the set of associated agents correspond to multiple traffic participants in the traffic environment.

[0123] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0124] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0125] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0126] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate implementation can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0127] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A computer-implemented method comprising: receiving a set of observation information from a set of associated agents associated with a target agent; generating a first intermediate decision for the target agent based on the set of observation information and target observation information for the target agent, the first intermediate decision indicating a first candidate action determined by the target agent; receiving a first set of intermediate decisions from the set of associated agents, the first set of intermediate decisions indicating candidate actions determined by respective associated agents; as well as Based at least on the generated first intermediate decision and the received first set of intermediate decisions, a target decision for the target agent is generated, the target decision indicating a target action to be performed by the target agent.

2. The method according to claim 1, wherein generating a first intermediate decision for the target agent comprises: generating first aggregate information based on the set of observation information and the target observation information for the first agent; as well as Based on the first aggregate information, the first intermediate decision for the target agent is generated.

3. The method according to claim 2, further comprising: The first aggregate information and the first intermediate decision are sent to at least one associated agent in the group of associated agents.

4. The method of claim 1, wherein at least one intermediate decision in the first set of intermediate decisions is determined based on the target observation information received from the first agent.

5. The method according to claim 1, wherein the set of observation information comprises: Observation information sent to the target agent by at least one associated agent in the group of associated agents.

6. The method according to claim 1, wherein generating a target decision for the target agent based on at least the generated first intermediate decision and the received set of intermediate decisions comprises: generating second aggregate information based at least on the first intermediate decision and the set of intermediate decisions; generating a second intermediate decision based on the second aggregated information; receiving a second set of intermediate decisions from the set of associated agents; as well as A target decision for the target agent is generated based at least on the generated second intermediate decision and the received second group of intermediate decisions.

7. The method of claim 6, wherein at least one intermediate decision in the second set of intermediate decisions is determined based on the first intermediate decision received from the first agent.

8. The method of claim 1, wherein the first agent and at least one of the group of associated agents are trained based on the following process: In the process of training the at least one intelligent agent, the at least one intelligent agent is trained by maximizing a set of intermediate benefits corresponding to a set of intermediate decisions made by the at least one intelligent agent.

9. The method of claim 8, wherein the intermediate benefit is determined based on a reward function or an advantage function associated with a reinforcement learning process.

10. The method according to claim 1, wherein the target agent and the group of associated agents correspond to a plurality of traffic participants in a traffic environment.

11. An electronic device, comprising: Processing unit; as well as A memory coupled to the processing unit and containing instructions stored thereon, wherein when the instructions are executed by the processing unit, the electronic device performs the following actions, the actions comprising: receiving a set of observation information from a set of associated agents associated with a target agent; generating a first intermediate decision for the target agent based on the set of observation information and target observation information for the target agent, the first intermediate decision indicating a first candidate action determined by the target agent; receiving a first set of intermediate decisions from the set of associated agents, the first set of intermediate decisions indicating candidate actions determined by respective associated agents; and Based at least on the generated first intermediate decision and the received first set of intermediate decisions, a target decision for the target agent is generated, the target decision indicating a target action to be performed by the target agent.

12. The electronic device of claim 11, wherein generating a first intermediate decision for the target agent comprises: generating first aggregate information based on the set of observation information and the target observation information for the first agent; as well as Based on the first aggregate information, the first intermediate decision for the target agent is generated.

13. The electronic device according to claim 12, wherein the actions further comprise: The first aggregate information and the first intermediate decision are sent to at least one associated agent in the group of associated agents.

14. The electronic device of claim 11, wherein at least one intermediate decision in the first set of intermediate decisions is determined based on the target observation information received from the first agent.

15. The electronic device according to claim 11, wherein the set of observation information comprises: Observation information sent to the target agent by at least one associated agent in the group of associated agents.

16. The electronic device according to claim 11, wherein generating a target decision for the target agent based on at least the generated first intermediate decision and the received set of intermediate decisions comprises: generating second aggregate information based at least on the first intermediate decision and the set of intermediate decisions; generating a second intermediate decision based on the second aggregated information; receiving a second set of intermediate decisions from the set of associated agents; as well as A target decision for the target agent is generated based at least on the generated second intermediate decision and the received second group of intermediate decisions. 17 . The electronic device of claim 16 , wherein at least one intermediate decision in the second set of intermediate decisions is determined based on the first intermediate decision received from the first agent.

18. The electronic device of claim 11, wherein the first agent and at least one of the group of associated agents are trained based on the following process: In the process of training the at least one intelligent agent, the at least one intelligent agent is trained by maximizing a set of intermediate benefits corresponding to a set of intermediate decisions made by the at least one intelligent agent.

19. The electronic device of claim 18, wherein the intermediate benefit is determined based on a reward function or an advantage function associated with a reinforcement learning process.

20. A computer program product tangibly stored in a non-transitory computer storage medium and comprising machine executable instructions which, when executed by a device, cause the device to perform actions according to the following, the actions comprising: receiving a set of observation information from a set of associated agents associated with a target agent; generating a first intermediate decision for the target agent based on the set of observation information and target observation information for the target agent, the first intermediate decision indicating a first candidate action determined by the target agent; receiving a first set of intermediate decisions from the set of associated agents, the first set of intermediate decisions indicating candidate actions determined by respective associated agents; as well as Based at least on the generated first intermediate decision and the received first set of intermediate decisions, a target decision for the target agent is generated, the target decision indicating a target action to be performed by the target agent.

Images