Reinforcement learning and application methods, systems, devices and media for multi-agent scenarios

Abstract

The present invention discloses a reinforcement learning and application method, system, device and medium for multi-agent scenarios, and designs a value function decomposition module based on the self-attention mechanism, which constructs the action values of the agent pairs and then weights them using the connection coefficient, which measures the mutual connection relationship between the agents in the multi-agent system. It solves the problem that the optimal action consistency principle brought about by directly using the self-attention mechanism in the value decomposition method cannot be guaranteed. The value function decomposition module based on the self-attention mechanism can be regarded as a basic module and applied to the existing value decomposition method to improve performance. Empirical results in multiple simulation environments further show that the value function decomposition module based on the self-attention mechanism learns the reasonable internal connection relationship between the agents in the multi-agent system, and the learning performance is significantly improved compared with the existing scheme.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a reinforcement learning and application method, system, device and medium for multi-agent scenarios. Background Art

[0002] In recent years, cooperative multi-agent reinforcement learning has shown broad application prospects in various real-world domains, such as self-driving cars, computer game scenarios, sensor networks, and robotic swarms. In widely adopted cooperative multi-agent reinforcement learning settings, a group of agents typically has limited access to local observations and communication constraints between them. At each time step, all agents take a joint action on the environment, which transitions to a global state at the next time step and assigns a reward to the team. Therefore, agents must learn to coordinate with each other to maximize the team's cumulative reward.

[0003] Recently, a value function decomposition approach based on a centralized training and decentralized execution paradigm has achieved state-of-the-art performance on challenging cooperative multi-agent tasks. More specifically, for each agent, an individual value function network takes its local trajectory and last action as input to model its individual value function. These individual value functions are then integrated into a global value function via a hybrid network. During the training phase, the parameters of the individual value function network and the hybrid network are optimized with the objective of minimizing the temporal difference loss. During the inference phase, agents determine their actions based on their individual value functions, thus achieving decentralized execution.

[0004] Considering the differences between training and execution, although the aforementioned hybrid network is subject to the optimal action consistency principle—that is, it constrains the consistency between the global greedy actions selected according to the global action-value function and the individual greedy actions selected according to their individual action values—existing solutions only capture the contribution of a single agent to the entire team, while ignoring the connections between agents in a multi-agent system. Consequently, current solutions are unable to effectively perform tasks in their respective application scenarios. For example, in self-driving car scenarios, they cannot accurately and effectively control the car, affecting the safety of passengers and other vehicles on the road; in computer game scenarios, they cannot automatically play the game, reducing the player's gaming experience. Summary of the Invention

[0005] The purpose of the present invention is to provide a reinforcement learning and application method, system, device and medium for multi-agent scenarios, which not only meets the principle of optimal action consistency, but also can use the self-attention mechanism to capture the connection relationship between agents, so as to better complete the tasks in the application scenario.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A reinforcement learning and application method for multi-agent scenarios, including:

[0008] Configure multiple agents according to application scenarios, and obtain individual action-value functions generated by each agent based on observation information in the application scenarios, as well as individual action observation history trajectories; wherein the application scenarios include at least: self-driving cars and computer game scenarios;

[0009] Training phase: For the agent pairs formed by the current agent and each agent, the value function decomposition module based on the self-attention mechanism uses the individual action value function and individual action observation history trajectory of the corresponding agent to calculate the action value function of each agent pair. The connection coefficient is calculated by combining the individual action observation history trajectory of the agents in each agent pair. The action value function of the agent pair is fused using the connection coefficient to obtain the fused action value function of the current agent. The fused action value function of all agents is combined through other value function decomposition modules to generate a global action value function. The training target is established by combining the global action value function. The agent, the value function decomposition module based on the self-attention mechanism, and other value function decomposition modules are optimized to obtain the optimized agent.

[0010] Reasoning stage: Each optimized intelligent agent combines the observation information in the application scenario to generate an individual action value function and selects an action to be executed in the corresponding application field.

[0011] A reinforcement learning and application system for multi-agent scenarios, including:

[0012] The agent configuration and data acquisition unit is used to configure multiple agents according to the application scenario, obtain the individual action value function generated by each agent in combination with the observation information in the application scenario, and the individual action observation history trajectory;

[0013] A training unit is used for a training phase, which includes: for the agent pairs formed by the current agent and each agent, respectively, the value function decomposition module based on the self-attention mechanism is used to calculate the action value function of each agent pair using the individual action value function and the individual action observation history trajectory of the corresponding agent, and the connection coefficient is calculated by combining the individual action observation history trajectory of the agents in each agent pair, and the action value function of the agent pair is fused using the connection coefficient to obtain the fused action value function of the current agent; the fused action value function of all agents is combined through other value function decomposition modules to generate a global action value function, and the training target is established by combining the global action value function, and the agent, the value function decomposition module based on the self-attention mechanism and the other value function decomposition modules are optimized to obtain the optimized agent;

[0014] The reasoning unit is used in the reasoning stage. The reasoning stage includes: after training, each optimized intelligent agent combines the observation information in the application scenario to generate an individual action value function and selects an action to be executed in the application field.

[0015] A processing device comprising: one or more processors; a memory for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0017] A readable storage medium stores a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0018] The technical solution provided by the present invention, as can be seen from the above, features a value function decomposition module based on a self-attention mechanism. This module constructs action values for pairs of agents and then weights them using a connection coefficient, which measures the interconnectedness of agents within a multi-agent system. This solves the problem of ensuring optimal action consistency when directly using the self-attention mechanism in value decomposition methods. The self-attention-based value function decomposition module can be considered a basic module and applied to existing value decomposition methods to improve performance. Empirical results in multiple simulation environments further demonstrate that the self-attention-based value function decomposition module learns reasonable internal connections between agents in a multi-agent system, significantly improving learning performance compared to existing solutions. Deployed in specific application scenarios, it can better accomplish the tasks in those scenarios. For example, in autonomous driving scenarios, it can accurately and effectively drive the car, improving the safety of passengers and other vehicles on the road; in computer game scenarios, it can automatically play the game, enhancing the player's gaming experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of a reinforcement learning and application method for a multi-agent scenario provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a reinforcement learning framework based on a value function decomposition module of a self-attention mechanism provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a reinforcement learning system for a multi-agent scenario provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0025] First, the following terms may be used in this article:

[0026] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y includes both “X” or “Y” and “X and Y”.

[0027] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles, etc.) should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0028] The following describes in detail the reinforcement learning and application methods, systems, devices, and media for multi-agent scenarios provided by the present invention. Any information not described in detail in the present embodiments is prior art known to those skilled in the art. For any unspecified conditions in the present embodiments, the conditions generally accepted in the art or recommended by the manufacturer shall prevail.

[0029] Example 1

[0030] The embodiment of the present invention provides a reinforcement learning and application method for a multi-agent scenario, and introduces a new value decomposition method, named the value function decomposition module based on the self-attention mechanism. Compared with the existing multi-layer perceptron structure, the self-attention mechanism captures the intrinsic connection between entities, thereby achieving better results. Although some existing technologies apply the attention mechanism to the value decomposition method, they only capture the contribution of a single agent to the entire team, without considering the connection between the agents in the multi-agent system. In a multi-agent system, it is crucial for the agent to only focus on transferring the most important teammate information and ignore irrelevant parts. For example, a football player with the ball should pay more attention to teammates who can catch the ball rather than his goalkeeper. Figure 1 FIG. 1 is a flowchart of a reinforcement learning and application method for a multi-agent scenario provided by an embodiment of the present invention, which mainly includes the following steps:

[0031] Step 1: Configure multiple agents according to the application scenario, obtain the individual action value function generated by each agent combined with the observation information in the application scenario, and the individual action observation history trajectory.

[0032] Embodiments of this invention can be applied to real-world applications such as self-driving cars, computer game scenarios, sensor networks, and robot swarms. Observation information refers to information observed by an agent within an application scenario. The following uses self-driving cars and computer game scenarios as examples for illustration.

[0033] 1) Autonomous driving car scenario.

[0034] In the autonomous vehicle scenario, each agent controls a vehicle. Observational information includes vehicle information and road environment information. Vehicle information includes speed and location, while road environment information includes the distance between the vehicle and the vehicles ahead and behind. Actions are actions performed to control the vehicle, including controlling the current throttle and throttle level, the current brake and brake level, and the steering wheel angle. The individual action observation history is a collection of observation information and actions over a period of time.

[0035] 2) Computer game scene.

[0036] In a computer game scenario, each agent is configured to control a game unit of its own. Observation information includes information about its own game unit and enemy units within its field of view (e.g., health, position, etc.). Actions are actions to control its own game unit, including controlling the unit to move in a certain direction or selecting an enemy unit to attack.

[0037] In this embodiment of the present invention, the agent's individual action-value function generation network generates the current individual action-value function based on the current observation information and the previous action, thereby selecting the action to execute at the current moment. The collection of historical observation information and actions is the individual action observation history trajectory.

[0038] Step 2, training phase: For the agent pairs formed by the current agent and each agent, the value function decomposition module based on the self-attention mechanism is used to calculate the action value function of each agent pair using the individual action value function and individual action observation history trajectory of the corresponding agent. The connection coefficient is calculated by combining the individual action observation history trajectory of the agents in each agent pair, and the action value function of the agent pair is fused using the connection coefficient to obtain the fused action value function of the current agent; the fused action value function of all agents is combined through other value function decomposition modules to generate a global action value function, and the training target is established by combining the global action value function to optimize the agent, the value function decomposition module based on the self-attention mechanism and other value function decomposition modules to obtain the optimized agent.

[0039] In an embodiment of the present invention, the action-value function and connection coefficient of the agent pair are explicitly constructed based on the historical trajectories of the individual actions of the two agents in the agent pair. The connection coefficient measures the closeness of the connection between two agents in a multi-agent system. For each agent, the fusion value function is the weighted average of the action-value functions of the agent pair, weighted by the connection coefficient between itself and each agent. A regularization term is also introduced for the connection coefficient to induce sparser connections between agents and encourage agents to focus on a few more important teammates. The value function decomposition module based on the self-attention mechanism not only has excellent representation capabilities but also meets the principle of optimal action consistency required by the value function factorization method.

[0040] In an embodiment of the present invention, other value function decomposition modules can be implemented through existing value decomposition methods, and the value function decomposition module based on the self-attention mechanism and other value function decomposition modules together form a hybrid network; that is, the value function decomposition module based on the self-attention mechanism provided by the present invention can be applied to various existing value function decomposition methods.

[0041] In this embodiment of the present invention, during the training phase, the agent, the self-attention-based value function decomposition module, and other value function decomposition modules are all trained by backpropagating the training target gradient. Since the agent cannot be trained directly, the self-attention-based value function decomposition module and other value function decomposition modules can be considered auxiliary modules for training the agent. After training is completed, the optimized agent is retained. The remaining training processes involved can be referred to conventional techniques and are not detailed in this invention.

[0042] Step 3, reasoning phase: Each optimized agent generates an individual action value function based on the observation information in the application scenario and selects an action to be executed in the corresponding application field.

[0043] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the reinforcement learning and application method for a multi-agent scenario provided by an embodiment of the present invention is described in detail below with specific examples.

[0044] 1. Reinforcement learning framework based on the value function decomposition module of self-attention mechanism.

[0045] In complex multi-agent tasks, it's important to prioritize the connections between agents to quickly acquire the most effective information in the system. This requirement aligns with the core concept of the self-attention mechanism. However, directly applying the self-attention mechanism to individual value functions can greatly complicate the gradients between the individual action value function and the joint action value function, making it impossible to guarantee optimal action consistency. To address this, this paper proposes a value function decomposition module based on the self-attention mechanism.

[0046] Figure 2 The reinforcement learning framework of the value function decomposition module based on the self-attention mechanism is presented. Figure 2 The superscript t in the figure represents the number of the time series, the subscript represents the number of the agent, o represents the observation information, and a represents the action; the value function decomposition module based on the self-attention mechanism is based on the individual action value function and individual action observation history As input, the output is the fusion of individual action value functions N is the total number of agents. Specifically, we first model the action value function Q of the agent for (i, j) ij , the fusion of individual action value functions It is based on the connection relationship between the agent pair (i, j). Weighted aggregation. Therefore, this method needs to solve the following two problems under the premise of adhering to the principle of optimal action consistency: 1) How to explicitly represent the action-value function of each agent pair. 2) How to measure the importance of the connection between agents.

[0047] 1. Explicitly represent the action-value function for each agent pair.

[0048] In the embodiment of the present invention, the current agent is denoted as agent i, which forms an agent pair (i, j) with agent j; the agent pair encoder f in the value function decomposition module based on the self-attention mechanism ij Calculate the action value function pair [Q i ,Q j ]’s action-value function Q ij, where i = 1, ..., N, j = 1, ..., N, N is the total number of agents, and the agents are ... ij The weight parameters and bias parameters in are generated by observing the historical trajectory of the individual actions of agents i and j. The agents observe the encoder f ij It can be implemented using a two-layer feedforward neural network. The action value function Q ij The calculation method is expressed as:

[0049]

[0050] Among them, Q i is the individual action value function of agent i, Q j is the individual action value function of agent j, σ represents the activation function, and The agent is the encoder f ij The weight parameter of and The agent is the encoder f ij The bias parameters, and Both are generated by observing the historical trajectories of the individual actions of agents i and j.

[0051] In the embodiment of the present invention, it is assumed that the individual action value function Q i and Q j The connection between them is inferred from the individual action observation history of agents i and j. Therefore, the agent’s encoder f ij The weight parameters and bias parameters in can be generated by the hyperparameter network in the value function decomposition module based on the self-attention mechanism. The input of the hyperparameter network is the hidden state (h i , h j ), the first type of output layer outputs the agent to the encoder f ij The weight parameters and bias parameters in Among them, h i is the hidden state corresponding to the historical trajectory of the individual action observation of agent i, h j is the hidden state corresponding to the historical trajectory of the individual action observation of agent j, such as Figure 2 As shown in Figure 2, the hidden state can be generated by a multilayer perceptron and a gated recurrent unit.

[0052] In this embodiment of the present invention, the parameters of the hyperparameter network are shared by all agents and are expressed as θ p In the training phase, the optimized value function decomposition module based on the self-attention mechanism is actually to optimize the parameters θ of the hyperparameter network. p .

[0053] In addition, in order to satisfy the principle of optimal action consistency, the weight of the agent on the encoder is constrained Is a positive number.

[0054] 2. Measuring the importance of connections between agents.

[0055] In the embodiment of the present invention, the connection coefficient of the agent pair (i, j) is defined as λ ij , which observes the historical trajectory of the two agents through their individual actions (h i , h j ) calculation, the connection coefficient λ ij Used to measure the importance of agent j to agent i, the connection coefficient λ ij The larger it is, the more important the relationship between agent i and agent j is in agent i's decision-making.

[0056] Similar to the agent-to-encoder mentioned above, the connection coefficient λ ij It is also generated by the hyperparameter network, and the input is the hidden state (h i , h j ), output the connection coefficient λ through the second type output layer ij , where the output connection coefficient λ is controlled by the sigmoid activation function in the second type of output layer ij The value range of .

[0057] like Figure 2 As shown, the lower middle part shows the principle of the hyperparameter network, the hidden state of the input (h i , h j ) After being processed by the multi-layer perceptron, the weight parameters and bias parameters are output through the first type of output layer (i.e., the first four rectangular boxes from left to right), and the connection coefficient is output through the second type of output layer (the rightmost rectangular box); Figure 2 The five rectangular boxes corresponding to the two types of output layers shown represent single-layer perceptrons.

[0058] 3. Fusion agent pair action value function.

[0059] In the embodiment of the present invention, the action value functions of the agent pairs are aggregated by combining the connection coefficient to obtain a fused action value function. Under the premise of considering the closeness of the connection, the individual action value functions are effectively aggregated. Specifically, the fused action value function of agent i is named The connection coefficient Weighted agent action value The obtained expression is:

[0060]

[0061] Among them, j = 1,...,N, and finally N fusion action value functions are obtained:

[0062] 4. Generate global action value.

[0063] In the embodiment of the present invention, the fusion action value function can be: Input to other value decomposition modules to generate the global action value Q tot , the other value decomposition modules here can be implemented using existing value decomposition methods. That is, the value function decomposition module based on the self-attention mechanism provided in the embodiment of the present invention can be applied to any value function decomposition method.

[0064] 2. Overall training and reasoning part.

[0065] During the training phase, N fusion action value functions can be obtained through the scheme introduced in the first part of the previous article Then, other value decomposition modules are used to generate the joint action value function Q tot , the value function decomposition module based on the self-attention mechanism and other value decomposition modules are regarded as a hybrid network, and the parameters of other value decomposition modules are recorded as θ m Then all parameters of the hybrid network θ:={θ m ,θ p} can be obtained by minimizing Q tot In addition, a connection coefficient regularization term is proposed to make the agent focus only on the most important teammate information, thus making the connection relationship between agents sparse. Therefore, the overall training goal is:

[0066]

[0067] Among them, λ ij represents the connection coefficient of the agent pair (i, j); L TD Represented by the global action value function Q tot Calculated time difference loss, k is the balance time difference loss and regularization loss ∑ ij λ ij hyperparameters.

[0068] In this embodiment of the present invention, during training, the agent, the self-attention-based value function decomposition module, and other value function decomposition modules are all trained using the overall training objective gradient feedback. Because individual rewards are unavailable, only team rewards are available, the agent is trained using gradient feedback. This gradient feedback implicitly decomposes the global value function into individual value functions. The self-attention-based value function decomposition module introduced in this embodiment of the present invention can better decompose the value function by modeling the connections between agents, thereby assisting in the training of the agents.

[0069] In the inference phase, the optimized agent i selects a greedy action for decentralized execution based on its individual action-value function.

[0070] The above-mentioned solution provided by the embodiment of the present invention designs a value function decomposition module based on the self-attention mechanism, which constructs the action values of the agent pairs and then weights them using the connection coefficient, which measures the mutual connection relationship between the agents in the multi-agent system. It solves the problem that the optimal action consistency principle brought about by directly using the self-attention mechanism in the value decomposition method cannot be guaranteed. The value function decomposition module based on the self-attention mechanism can be regarded as a basic module and applied to the existing value decomposition method to improve performance. Empirical results in multiple simulation environments further show that the value function decomposition module based on the self-attention mechanism learns the reasonable internal connection relationship between the agents in the multi-agent system, and the learning performance is significantly improved compared with the existing scheme.

[0071] Through the description of the above embodiments, those skilled in the art will clearly understand that the above embodiments can be implemented through software or by using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) and includes a number of instructions for causing a computer device (such as a personal computer, a server, or a network device) to execute the methods described in the various embodiments of the present invention.

[0072] Example 2

[0073] The present invention also provides a multi-agent scenario reinforcement learning and application system, which is mainly implemented based on the method provided in the above embodiment, such as Figure 3 As shown, the system mainly includes:

[0074] The agent configuration and data acquisition unit is used to configure multiple agents according to the application scenario, obtain the individual action value function generated by each agent in combination with the observation information in the application scenario, and the individual action observation history trajectory;

[0075] A training unit is used for a training phase, which includes: for the agent pairs formed by the current agent and each agent, respectively, the value function decomposition module based on the self-attention mechanism is used to calculate the action value function of each agent pair using the individual action value function and the individual action observation history trajectory of the corresponding agent, and the connection coefficient is calculated by combining the individual action observation history trajectory of the agents in each agent pair, and the action value function of the agent pair is fused using the connection coefficient to obtain the fused action value function of the current agent; the fused action value function of all agents is combined through other value function decomposition modules to generate a global action value function, and the training target is established by combining the global action value function, and the agent, the value function decomposition module based on the self-attention mechanism and the other value function decomposition modules are optimized to obtain the optimized agent;

[0076] The reasoning unit is used in the reasoning stage. The reasoning stage includes: after training, each optimized intelligent agent combines the observation information in the application scenario to generate an individual action value function and selects an action to be executed in the application field.

[0077] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0078] Example 3

[0079] The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0080] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0081] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example:

[0082] The input device can be a touch screen, image acquisition device, physical button or mouse;

[0083] The output device may be a display terminal;

[0084] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0085] Example 4

[0086] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0087] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0088] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A reinforcement learning and application method for multi-agent scenarios, characterized by: include: Configure multiple agents according to application scenarios, and obtain individual action-value functions generated by each agent based on observation information in the application scenarios, as well as individual action observation history trajectories; wherein the application scenarios include at least: self-driving cars and computer game scenarios; Training phase: For the agent pairs formed by the current agent and each agent, the value function decomposition module based on the self-attention mechanism uses the individual action value function and individual action observation history trajectory of the corresponding agent to calculate the action value function of each agent pair. The connection coefficient is calculated by combining the individual action observation history trajectory of the agents in each agent pair. The action value function of the agent pair is fused using the connection coefficient to obtain the fused action value function of the current agent. The fused action value function of all agents is combined through other value function decomposition modules to generate a global action value function. The training target is established by combining the global action value function. The agent, the value function decomposition module based on the self-attention mechanism, and other value function decomposition modules are optimized to obtain the optimized agent. Inference phase: Each optimized agent generates an individual action-value function based on the observation information in the application scenario and selects an action to be executed in the application domain. The agent pairs formed by the current agent and each agent are respectively based on the value function decomposition module of the self-attention mechanism, and the individual action value function of the corresponding agent and the individual action observation history trajectory are used to calculate the action value function of each agent pair. The method includes: the current agent is denoted as agent i, and it forms an agent pair (i, j) with agent j; the agent pair encoder f in the value function decomposition module based on the self-attention mechanism ij Calculate the action value function Q of the agent for (i, j) ij , where i = 1, ..., N, j = 1, ..., N, N is the total number of agents, and the agents are ... uj The weight parameters and bias parameters in are generated by the hyperparameter network in the value function decomposition module based on the self-attention mechanism through the individual action observation history trajectories of agent i and agent j; The method of calculating the action value function of each agent pair by using the individual action value function of the corresponding agent and the individual action observation history trajectory includes: For an agent pair (i, j), the action-value function Q of the agent pair (i, j) is ij The calculation method is expressed as: Among them, Q i is the individual action value function of agent i, Q j is the individual action value function of agent j, σ represents the activation function, and The agent is the encoder f ij The weight parameters are all positive numbers; and The agent is the encoder f ij The bias parameters, and Both are generated by observing the historical trajectories of the individual actions of agents i and j through the hyperparameter network.

2. The reinforcement learning and application method for a multi-agent scenario according to claim 1, characterized in that: In the autonomous driving car scenario, observation information includes: car information and road environment information, and actions are actions to control the car; in the computer game scenario, observation information includes: information about one's own game units and enemy units within the field of view, and actions are actions to control one's own game units; the individual action observation history trajectory is a collection of observation information and actions in historical time.

3. The reinforcement learning and application method for a multi-agent scenario according to claim 1, characterized in that: The calculation of the connection coefficient by combining the individual action observation history trajectories of the agents in each agent pair and using the connection coefficient to fuse the action value functions of the agent pair to obtain the fused action value function of the current agent includes: Let the current agent be agent i, which forms an agent pair (i, j) with agent j. For the agent pair (i, j), the super-parameter network in the value function decomposition module based on the self-attention mechanism uses the hidden state (h i ,h j ) is calculated as λ ij , the connection coefficient λ ij It is used to measure the importance of agent j to agent i, and use the connection coefficient to fuse the action value function of the agent pair to obtain the fused action value function of the current agent. Expressed as: Where i = 1,...,N, j = 1,...,N, N is the total number of agents; h i is the hidden state corresponding to the historical trajectory of the individual action observation of agent i, h j is the hidden state corresponding to the historical trajectory of the individual action observation of agent j.

4. A reinforcement learning and application method for a multi-agent scenario according to claim 1 or 3, characterized in that: The hyperparameter network includes: a multi-layer perceptron and two types of output layers; The input of the multi-layer perceptron is the hidden state corresponding to the individual action observation history trajectory of the two agents in the agent pair (i, j); the first type of output layer outputs the agent pair encoder f in the value function decomposition module based on the self-attention mechanism ij The weight parameters and bias parameters in the encoder f ij Used to calculate the action value function Q of the agent pair (i, j) ij ; The second type of output layer outputs the connection coefficient λ of the agent pair (i, j) ij ; During the training phase, the optimized value function decomposition module based on the self-attention mechanism refers to optimizing the parameters of the hyperparameter network.

5. The reinforcement learning and application method for a multi-agent scenario according to claim 1, characterized in that: The training objective is expressed as: Among them, θ represents all parameters in the value function decomposition module based on the self-attention mechanism and other value decomposition modules, λ ij represents the connection coefficient of the agent pair (i, j); L TD Represented by the global action value function Q tot Calculated time difference loss, k is the balance time difference loss and regularization loss ∑ ij λ ij hyperparameters.

6. A reinforcement learning and application system for multi-agent scenarios, characterized by: The method according to any one of claims 1 to 5 is implemented, and the system comprises: The agent configuration and data acquisition unit is used to configure multiple agents according to the application scenario, obtain the individual action value function generated by each agent in combination with the observation information in the application scenario, and the individual action observation history trajectory; A training unit is used for a training phase, which includes: for the agent pairs formed by the current agent and each agent, respectively, the value function decomposition module based on the self-attention mechanism is used to calculate the action value function of each agent pair using the individual action value function and the individual action observation history trajectory of the corresponding agent, and the connection coefficient is calculated by combining the individual action observation history trajectory of the agents in each agent pair, and the action value function of the agent pair is fused using the connection coefficient to obtain the fused action value function of the current agent; the fused action value function of all agents is combined through other value function decomposition modules to generate a global action value function, and the training target is established by combining the global action value function, and the agent, the value function decomposition module based on the self-attention mechanism and the other value function decomposition modules are optimized to obtain the optimized agent; The reasoning unit is used in the reasoning stage. The reasoning stage includes: after training, each optimized intelligent agent combines the observation information in the application scenario to generate an individual action value function and selects an action to be executed in the application field.

7. A processing device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images