Multi-agent decision-making method, device, computer equipment and medium for parameterized action space

Abstract

The present invention proposes a multi-agent decision-making method, device, computer equipment and medium for parameterized action space, which sets a task scenario, global state space, action space and reward function for multi-agent collaboration; obtains observation information at the current moment, and the policy network of each agent decides the corresponding behavior action based on the current observation information and the set task scenario, global state space and action space; the value assessment network of each agent generates an individual Q value for evaluating the value of the behavior action decided by each agent; the individual Q value of each agent is nonlinearly compounded through a hybrid network to obtain a global joint Q value for evaluating the effectiveness of the joint action; based on the global joint Q value and the reward function, the hybrid network parameters, the value assessment network parameters of each agent and the policy network parameters of each agent are updated. The present invention can solve the problem that traditional multi-agent reinforcement learning methods can only be applied to discrete action spaces or continuous action spaces.

Description

Technical Field

[0001] The present invention mainly relates to the field of multi-agent intelligent collaborative decision-making technology, in particular to a multi-agent decision-making method, device, computer equipment and medium in a parameterized action space. Background Art

[0002] Intelligent, coordinated penetration of multiple aircraft is key to improving their penetration effectiveness. To enable aircraft to effectively evade interceptor threats and successfully execute their intended missions in highly dynamic adversarial environments, a growing number of experts and scholars are focusing on using reinforcement learning methods to intelligently control the diverse penetration parameters of aircraft.

[0003] Traditional multi-agent reinforcement learning can only be applied to discrete action spaces such as QMIX or continuous action spaces such as MADDPG. In many practical multi-agent decision-making applications, decision-making often requires making decisions in parameterized policy spaces with both discrete and continuous action variables. Extensive research has been conducted on single-agent reinforcement learning algorithms in parameterized action spaces, with proposed methods such as PDDPG, P-DQN, HPPO, and HyAR being suitable for such spaces. However, research on multi-agent reinforcement learning algorithms in parameterized action spaces is relatively limited. Summary of the Invention

[0004] In response to the technical problems existing in the prior art, the present invention proposes a multi-agent decision-making method, device, computer equipment and medium in a parameterized action space.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] In one aspect, the present invention proposes a method for training a multi-agent decision model in a parameterized action space, comprising:

[0007] Obtain the global state information of the multi-agent system at the current moment and the trajectory information of each agent in the multi-agent system, wherein the trajectory information includes the observation information of the agent at the current moment and the action information at the previous moment;

[0008] The trajectory information of each agent is input into the strategy network corresponding to each agent, and the strategy network corresponding to each agent outputs discrete action variables and continuous action variables;

[0009] The trajectory information of each agent at the current moment and the discrete action variables and continuous action variables corresponding to each agent are input into the individual evaluation network corresponding to each agent, and the individual evaluation network corresponding to each agent evaluates the individual Q value of the current individual behavior value of each agent;

[0010] The global state information of the multi-agent system at the current moment and the individual Q values of the individual behavior values of all agents at the current moment are used as the input of the hybrid network. The hybrid network fits the global joint Q value through a multi-layer nonlinear composite method.

[0011] Based on the global joint Q value, the strategy network parameters, individual evaluation network parameters and hybrid network parameters corresponding to each agent are updated until the training of the multi-agent decision model is completed.

[0012] In the multi-agent system of the present invention, observation information refers to the information obtained within the observation range of each agent and the position, speed, and equipment parameter information of the agent itself. Global state information refers to the collection of observation information of all agents from the perspective of the white side.

[0013] Furthermore, the strategy network described in the present invention is a dual-head output MLP multi-layer neural network, which makes decisions based on the observation information of the intelligent agent at the current moment and the action information at the previous moment and outputs the discrete action variables and continuous action variables at the current moment.

[0014] Furthermore, the hybrid network of the present invention compounds the individual Q values of the individual behavior values of all agents twice to obtain a global joint Q value, including:

[0015] The global state information of the multi-agent system at the current moment is input into four different fully connected networks, and the four fully connected networks respectively output the first weight matrix W1, the second weight matrix W2, the first bias and the second bias b2, Respectively represent the m elements in the first bias b1;

[0016] Assume that there are n agents in the multi-agent system. The first compound of the individual Q values of all agents' individual behavior values is:

[0017]

[0018] where Q i (τ i ,u i ) represents the individual Q value corresponding to the i-th agent, and the first weight matrix W1 is an n×m weight matrix. is the element in the i-th row and j-th column of the first weight matrix W1, i = 1, 2, 3, ..., n, j = 1, 2, 3, ..., m; h j Represents the jth element in the first compound result;

[0019] Based on the first compound result, a second compound is performed to obtain the global joint Q value Q tot :

[0020]

[0021] Where W1 is the m×n weight matrix W2 is a 1×m weight matrix, is the jth element in the second weight matrix W2.

[0022] In the present invention, the loss function of the individual evaluation network parameters and the hybrid network parameters corresponding to each agent is constructed based on the global joint Q value. Optimize the loss function and update the individual evaluation network parameters and hybrid network parameters corresponding to each agent at the same time. The loss function is as follows:

[0023]

[0024]

[0025] in Indicates the mathematical expectation of the expression in []. represents the set of individual evaluation network parameters of n agents, θ i represents the individual evaluation network parameters of the i-th agent, s t Represents the global state information of the multi-agent system at the current moment, represents the joint continuous action set consisting of the continuous action variables of all agents at the current moment, represents the joint discrete action set composed of the discrete action variables of all agents at the current moment, r t Represents the sum of the reward values of all agents calculated according to the preset reward function at the current moment, τ t represents the set of trajectory information of all agents at the current moment, τ t+1 Represents the set of trajectory information of all agents at the next moment, represents the experience pool, Q tot represents the global joint Q value, ω represents the hybrid network parameter, y tot represents the target global joint Q value, γ represents the discount factor, s t+1 Represents the global state information of the multi-agent system at the next moment, represents the joint continuous action set consisting of the continuous action variables of all agents at the next moment, represents the joint discrete action set composed of the discrete action variables of all agents at the next moment, ω - Represents the target hybrid network parameters.

[0026] In this invention, the policy network parameters of each agent are updated using a centralized policy gradient update mechanism as follows:

[0027]

[0028] in Indicates the mathematical expectation of the expression in []. represents the centralized policy gradient of the set of policy networks of all agents, represents the set of policy networks of n agents, μ i represents the policy network of the i-th agent, represents the set of policy network parameters of n agents, φ i represents the policy network parameters of the i-th agent, Q tot represents the global joint Q value, s t represents the global state information of the multi-agent system at the current moment, τ i represents the trajectory information of the i-th agent, r t Represents the sum of the reward values of all agents calculated according to the preset reward function at the current moment, τ t Represents the set of trajectory information of all agents at the current moment, represents the joint continuous action set consisting of the continuous action variables of all agents at the current moment, represents the joint discrete action set composed of the discrete action variables of all agents at the current moment, s t+1 Represents the global state information of the multi-agent system at the next moment, τ t+1 represents the set of trajectory information of all agents at the next moment, and D represents the experience pool.

[0029] In another aspect, the present invention provides a multi-agent decision model training device in a parameterized action space, comprising:

[0030] The first module is used to obtain the global state information of the multi-agent system at the current moment and the trajectory information of each agent in the multi-agent system, wherein the trajectory information includes the observation information of the agent at the current moment and the action information at the previous moment;

[0031] The second module is used to input the trajectory information of each agent into the policy network corresponding to each agent, and the policy network corresponding to each agent outputs discrete action variables and continuous action variables;

[0032] The third module is used to input the trajectory information of each agent at the current moment and the discrete action variables and continuous action variables corresponding to each agent into the individual evaluation network corresponding to each agent, and the individual evaluation network corresponding to each agent evaluates the individual Q value of the current individual behavior value of each agent;

[0033] The fourth module is used to take the global state information of the multi-agent system at the current moment and the individual Q values of the current individual behavior values of all agents as the input of the hybrid network. The hybrid network fits the joint Q value through a multi-layer nonlinear composite method;

[0034] The fifth module updates the strategy network parameters, individual evaluation network parameters and hybrid network parameters corresponding to each intelligent agent based on the joint Q value.

[0035] On the other hand, the present invention provides a multi-agent decision-making method in a parameterized action space, which determines the behavioral actions of each agent based on the multi-agent decision model in a parameterized action space obtained by the multi-agent decision model training method in the parameterized action space mentioned above.

[0036] On the other hand, the present invention provides a multi-agent decision-making device in a parameterized action space, comprising an input module and a multi-agent decision-making model, wherein the multi-agent decision-making model is obtained by using the multi-agent decision-making model training method in the parameterized action space.

[0037] The input module obtains the global state information of the multi-agent system at the current moment and the trajectory information of each agent in the multi-agent system and inputs it into the multi-agent decision model training method of the parameterized action space. The multi-agent decision model outputs the behavioral action decision of each agent.

[0038] In another aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0039] A multi-agent decision model training method in a parameterized action space is characterized by comprising:

[0040] Obtain the global state information of the multi-agent system at the current moment and the trajectory information of each agent in the multi-agent system, wherein the trajectory information includes the observation information of the agent at the current moment and the action information at the previous moment;

[0041] The trajectory information of each agent is input into the strategy network corresponding to each agent, and the strategy network corresponding to each agent outputs discrete action variables and continuous action variables;

[0042] The trajectory information of each agent at the current moment and the discrete action variables and continuous action variables corresponding to each agent are input into the individual evaluation network corresponding to each agent, and the individual evaluation network corresponding to each agent evaluates the individual Q value of the current individual behavior value of each agent;

[0043] The global state information of the multi-agent system at the current moment and the individual Q values of the individual behavior values of all agents at the current moment are used as the input of the hybrid network. The hybrid network fits the global joint Q value through a multi-layer nonlinear composite method.

[0044] Based on the global joint Q value, the strategy network parameters, individual evaluation network parameters and hybrid network parameters corresponding to each agent are updated until the training of the multi-agent decision model is completed.

[0045] In another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0046] A multi-agent decision model training method in a parameterized action space is characterized by comprising:

[0047] Obtain the global state information of the multi-agent system at the current moment and the trajectory information of each agent in the multi-agent system, wherein the trajectory information includes the observation information of the agent at the current moment and the action information at the previous moment;

[0048] The trajectory information of each agent is input into the strategy network corresponding to each agent, and the strategy network corresponding to each agent outputs discrete action variables and continuous action variables;

[0049] The trajectory information of each agent at the current moment and the discrete action variables and continuous action variables corresponding to each agent are input into the individual evaluation network corresponding to each agent, and the individual evaluation network corresponding to each agent evaluates the individual Q value of the current individual behavior value of each agent;

[0050] The global state information of the multi-agent system at the current moment and the individual Q values of the individual behavior values of all agents at the current moment are used as the input of the hybrid network. The hybrid network fits the global joint Q value through a multi-layer nonlinear composite method.

[0051] Based on the global joint Q value, the strategy network parameters, individual evaluation network parameters and hybrid network parameters corresponding to each agent are updated until the training of the multi-agent decision model is completed.

[0052] Compared with the prior art, the technical effects of the present invention are:

[0053] This invention uses a multi-agent decision-making algorithm based on a value decomposition network as a centralized evaluation framework. The structure of the value decomposition network ensures effective coordination of the multi-agent system, effectively avoiding the emergence of the "credit allocation" problem. On this basis, drawing on the policy network design for processing parameterized action spaces in the parameterized deep deterministic policy gradient algorithm, the present invention designs a policy network structure similar to PDDPG. This invention can effectively make decisions and train multi-agent collaborative tasks with parameterized action spaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0055] Figure 1 This is a flow chart of an embodiment of the present invention;

[0056] Figure 2 This is a structural diagram of a multi-agent decision-making model in one embodiment of the present invention;

[0057] Figure 3 A network structure diagram of a strategy network and an individual evaluation network in one embodiment of the present invention;

[0058] Figure 4 A network structure diagram of a hybrid network in one embodiment of the present invention;

[0059] Figure 5 This is an architecture diagram of a strategy network in a multi-aircraft coordinated penetration scenario in one embodiment;

[0060] Figure 6 This is an overall structural diagram of a simulation system for multi-aircraft coordinated penetration scenarios in one embodiment;

[0061] Figure 7 This is a graph showing the changing trend of round cumulative rewards in a multi-aircraft coordinated penetration scenario in one embodiment. DETAILED DESCRIPTION

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0063] One embodiment provides a method for training a multi-agent decision model in a parameterized action space, comprising:

[0064] Obtain the global state information of the multi-agent system at the current moment and the trajectory information of each agent in the multi-agent system, wherein the trajectory information includes the observation information of the agent at the current moment and the action information at the previous moment;

[0065] The trajectory information of each agent is input into the strategy network corresponding to each agent, and the strategy network corresponding to each agent outputs discrete action variables and continuous action variables;

[0066] The trajectory information of each agent at the current moment and the discrete action variables and continuous action variables corresponding to each agent are input into the individual evaluation network corresponding to each agent, and the individual evaluation network corresponding to each agent evaluates the individual Q value of the current individual behavior value of each agent;

[0067] The global state information of the multi-agent system at the current moment and the individual Q values of the individual behavior values of all agents at the current moment are used as the input of the hybrid network. The hybrid network fits the global joint Q value through a multi-layer nonlinear composite method.

[0068] Based on the global joint Q value, the strategy network parameters, individual evaluation network parameters and hybrid network parameters corresponding to each agent are updated until the training of the multi-agent decision model is completed.

[0069] In the multi-agent system described herein, observation information refers to the information obtained within each agent's observation range, including the agent's own position, velocity, and equipment parameters. Global state information refers to the aggregate observation information of all agents from the perspective of the White player. From a game perspective, both the friendly and enemy players have their own observation ranges and are unable to access information outside of their observation ranges. The White player's perspective provides access to all situational information within the entire game scenario.

[0070] In the multi-agent decision-making model of the parameterized action space described in the present invention, three network models are included: one is a strategy network for each agent to output discrete action variables and continuous action variables; one is an individual evaluation network for evaluating the value of individual behavior of each agent; and one is a hybrid network that combines all individual Q values that evaluate the value of individual behavior of each agent into a global joint Q value, wherein the global joint Q value is used to evaluate the value of the joint strategy of all agents.

[0071] In one embodiment of the present invention, a multi-agent decision model is proposed, whose structure is as follows: Figure 2As shown, there are n agents in the multi-agent system, that is, there are n policy networks and n individual evaluation networks. The i-th agent corresponds to the i-th policy network, and the i-th policy network corresponds to the i-th individual evaluation network. Get the trajectory information of the i-th agent at the current moment The trajectory information includes the observation information of the agent at the current moment and the action information at the previous moment. The strategy network is a double-headed output strategy network. According to the trajectory information of the i-th agent at the current moment Through the dual-head output structure of the i-th strategy network and the action parameters of the pair output (including continuous action parameters and discrete action parameters Perform action sampling to obtain continuous action variables and discrete action variables

[0072] The trajectory information of the i-th agent at the current moment And the discrete action variable corresponding to the i-th agent Continuous action variables As the input of the i-th individual evaluation network corresponding to the i-th agent The individual Q value of the current individual behavior value of the i-th agent is evaluated by the i-th individual evaluation network The individual Q values of all agents' individual behavior values are compounded through the hybrid network to obtain the global joint Q value By calculating the global joint Q value The centralized policy gradient of each agent's strategy guides the update of each agent's strategy network parameters. By optimizing the loss of the global joint Q value, the individual evaluation network and the hybrid network of all agents can be updated simultaneously.

[0073] The dual-head output of the policy network of the present invention is used to couple the decision-making requirements of the parameterized action space, and the centralized evaluation based on value decomposition is used for the effective collaboration of the multi-agent system.

[0074] The strategy network described in the present invention is a dual-head output MLP multi-layer neural network, which makes decisions based on the observation information of the intelligent agent at the current moment and the action information of the previous moment and outputs the discrete action variables and continuous action variables at the current moment.

[0075] Reference Figure 3In one embodiment, the policy network and individual evaluation network provided by the agent in this embodiment are both composed of multiple linear layers plus activation functions. The policy network is a dual-headed MLP multi-layer neural network. The policy network is based on the trajectory information of the agent at the current moment (including the observation information at the current moment and the action information decision at the previous moment). One head of the policy network outputs the discrete action variable at the current moment, and the other head outputs the continuous action variable at the current moment. The individual evaluation network takes the discrete action variables and continuous action variables output by the policy network as input, as well as the trajectory information at the current moment, and outputs an individual Q value for evaluating the value of the individual behavior of the current agent.

[0076] Reference Figure 4 , the hybrid network provided by one embodiment, the individual Q values of all agents Composite to joint Q value The hybrid network converts the global state information S of the multi-agent system at the current moment into t As input, the first weight matrix W1, the second weight matrix W2, and the first bias are generated through different fully connected networks. And the second bias b2, based on the weight and bias, the individual Q values of all agents are compounded twice to obtain the global joint Q value, where Represent the m elements in the first bias b1 respectively.

[0077] Assume that there are n agents in the multi-agent system. The first compound of the individual Q values of all agents' individual behavior values is:

[0078]

[0079] where Q i (τ i ,u i ) represents the individual Q value corresponding to the i-th agent, and the first weight matrix W1 is an n×m weight matrix. is the element in the i-th row and j-th column of the first weight matrix W1, i = 1, 2, 3, ..., n, j = 1, 2, 3, ..., m; h j Represents the jth element in the first compound result;

[0080] Based on the first compound result, a second compound is performed to obtain the global joint Q value Q tot :

[0081]

[0082] Where W1 is the m×n weight matrix W2 is a 1×m weight matrix, is the jth element in the second weight matrix W2.

[0083] The hybrid network can effectively fit the global joint Q value through a multi-layer nonlinear composite method. The hybrid network can effectively solve the "credit allocation" problem in the multi-agent system, and can also enhance the representation ability of the joint Q value.

[0084] In the entire model structure of the present invention, the global joint Q value used to evaluate the effectiveness of the joint action can be obtained as follows:

[0085] Q tot =g(s,Q1,...,Q i ;ω)

[0086] Where n is the number of agents, ω is the parameter of the hybrid network, s is the current state information, Q i is the individual Q value of the i-th agent, Q tot is the global joint Q-value used to evaluate the effectiveness of joint actions. As can be seen from the above relationship, the composite joint Q-value involves the individual Q-values and the hybrid network. Therefore, by optimizing the loss of the joint Q-value, the individual evaluation networks and the hybrid network of all agents can be updated simultaneously.

[0087] In one embodiment, it is proposed to construct the loss function of the individual evaluation network parameters and the hybrid network parameters corresponding to each agent based on the global joint Q value. Optimize the loss function and update the individual evaluation network parameters and hybrid network parameters corresponding to each agent at the same time. The loss function is as follows:

[0088]

[0089]

[0090] in Indicates the mathematical expectation of the expression in []. represents the set of individual evaluation network parameters of n agents, θ i represents the individual evaluation network parameters of the i-th agent, s t Represents the global state information of the multi-agent system at the current moment, represents the joint continuous action set consisting of the continuous action variables of all agents at the current moment, represents the joint discrete action set composed of the discrete action variables of all agents at the current moment, r t Represents the sum of the reward values of all agents calculated according to the preset reward function at the current moment, τ t represents the set of trajectory information of all agents at the current moment, τ t+1represents the set of trajectory information of all agents at the next moment, D represents the experience pool, Q tot represents the global joint Q value, ω represents the hybrid network parameter, y tot represents the target global joint Q value, γ represents the discount factor, s t+1 Represents the global state information of the multi-agent system at the next moment, represents the joint continuous action set consisting of the continuous action variables of all agents at the next moment, represents the joint discrete action set composed of the discrete action variables of all agents at the next moment, ω - Represents the target hybrid network parameters. It is the operation of using all individual Q values to obtain the global joint Q value to evaluate the effectiveness of the joint action. tot It is obtained by compounding the weights and biases generated by the hybrid network through the individual Q value, so ω,θ1,...,θ n They represent the parameters of the hybrid network and the parameters of the individual Q networks of n agents. tot The essence of is to evaluate the value of joint actions in the global state, so s t , As Q tot Input parameters.

[0091] In one embodiment, it is proposed that the policy network parameters of each agent are updated using a centralized policy gradient update mechanism as follows:

[0092]

[0093] in Indicates the mathematical expectation of the expression in []. represents the centralized policy gradient of the set of policy networks of all agents, represents the set of policy networks of n agents, μ i represents the policy network of the i-th agent, represents the set of policy network parameters of n agents, φ i represents the policy network parameters of the i-th agent, Q tot represents the global joint Q value, s t represents the global state information of the multi-agent system at the current moment, τ i represents the trajectory information of the i-th agent, r t Represents the sum of the reward values of all agents calculated according to the preset reward function at the current moment, τ t Represents the set of trajectory information of all agents at the current moment, represents the joint continuous action set consisting of the continuous action variables of all agents at the current moment, represents the joint discrete action set composed of the discrete action variables of all agents at the current moment, s t+1 Represents the global state information of the multi-agent system at the next moment, τ t+1 Represents the set of trajectory information of all agents at the next moment, Represents the experience pool. represents the set of policy networks of n agents, μ i (τ i ) means that the trajectory information of the i-th agent is taken as input τ i , the strategy is obtained through the strategy network, that is, [u P ,u T ] i =μ i (τ i ); Indicates the gradient of all agent strategy network parameters, Indicates the strategy gradient of all agents. tot is a composite function, so through Q tot The chain rule is used to find the gradient of all agent strategy network parameters, that is, Because [u P ,u T ] i =μ i (τ i ), so Q tot (s t ,μ1(τ1),......μ n (τ n )) is equivalent to

[0094] A specific application example of the present invention is provided below to illustrate the effectiveness of the present invention:

[0095] This embodiment is a multi-aircraft coordinated penetration mission scenario. Based on the above-mentioned parameterized action space multi-agent decision model training method, a multi-aircraft coordinated penetration intelligent decision model training method is proposed. The steps include:

[0096] Obtaining global state information of the multi-aircraft coordinated penetration system at the current moment and trajectory information of each aircraft in the multi-aircraft coordinated penetration system, wherein the trajectory information includes observation information of the aircraft at the current moment and action information at the previous moment;

[0097] The trajectory information of each aircraft is input into the policy network corresponding to each aircraft, and the policy network corresponding to each aircraft outputs discrete action variables and continuous action variables;

[0098] The trajectory information of each aircraft at the current moment and the discrete action variables and continuous action variables corresponding to each aircraft are input into the individual evaluation network corresponding to each aircraft, and the individual evaluation network corresponding to each aircraft evaluates the individual Q value of the current individual behavior value of each aircraft;

[0099] The global state information of the multi-aircraft coordinated penetration system at the current moment and the individual Q values of the individual behavior values of all aircraft at the current moment are used as inputs of the hybrid network. The hybrid network fits the global joint Q value through a multi-layer nonlinear composite method.

[0100] Based on the global joint Q value, the strategy network parameters, individual evaluation network parameters and hybrid network parameters corresponding to each aircraft are updated until the training of the intelligent decision-making model for multi-aircraft collaborative penetration is completed.

[0101] Reference Figure 5 The structural design of the strategy network in the intelligent decision-making model for multi-aircraft coordinated penetration. For the i-th aircraft, the i-th policy network first outputs continuous and discrete action variables based on its observation information. At the same time, the policy shielding module will make corrections based on the output action variables to avoid invalid or unreasonable actions in this decision. Secondly, the corrected output action variables are combined with action encoding, encoded into policy instructions executable by the aircraft, and then output. The policy shielding module is a policy correction module based on expert prior design.

[0102] Figure 6 The figure represents the overall structure of the simulation system for a multi-aircraft coordinated penetration scenario. The aircraft's intelligent decision-making system and the simulation system exchange data via the UDP data communication protocol. The logic of this data exchange is as follows: the simulation system first sends global situational information to the intelligent decision-making system. The intelligent decision-making system then obtains its own observation information from the situational information based on its own observation space design. This observation information serves as the input to its policy network to obtain the aircraft's action variable output, and then sends the encoded action instructions to the simulation system. After the simulation system responds to the aircraft's action instructions, it updates the global situational information and sends the updated situational information to the intelligent decision-making system. Data from each interaction between the simulation system and the intelligent decision-making system is stored as training samples in an experience pool. The intelligent decision-making model samples and trains and updates model parameters at regular intervals until the model converges, at which point the interactive training ceases.

[0103] In one simulation example, the mission scenario involves a simulation environment with three launch points and three target points deployed within a certain area. Five aircraft are deployed at each of the three launch points, with a target point assigned to each aircraft. Two threat zones A and three threat zones B are located between the launch and target points. The state space includes information related to the aircraft and the threat zones. The action space includes four penetration strategies: maneuvering, acceleration and deceleration, stealth, and jamming, along with their execution constraints. Three reward functions are designed: a reward for the relative position relative to the target, a reward for the time it takes to pass through the threat zone, and a reward for successfully striking the target. Observational information includes: aircraft ID, aircraft state, the strategy being executed, whether the aircraft has entered a threat zone, the ID of the most threatening threat zone, the ratio of the threat zone radius to the distance between the aircraft and the threat zone, angle information (including aircraft velocity angle and the angle between the aircraft and the target), the relative position between the aircraft and the target, the number of threat zones within a certain threshold range, the constraints on the execution of the penetration strategy, and the ID of the platform in the threat zone that has been jammed.

[0104] The adopted strategy network and individual evaluation network are as follows Figure 3 As shown, the hybrid network structure used is as follows Figure 4 As shown, the policy network determines the corresponding behavioral actions based on the current observation information of the aircraft, and the value assessment network generates individual Q values used to evaluate the value of each aircraft's behavioral actions under the current observation information. The policy network is designed as a dual-head output, capable of simultaneously outputting discrete action variables and continuous action variables for the aircraft. The hybrid network nonlinearly combines the individual Q values of all aircraft to obtain a global joint Q value used to evaluate the effectiveness of the joint action. The method for updating each network parameter is the same as that disclosed in the above embodiment and will not be repeated here.

[0105] In this embodiment, the relevant parameters involved in the method of the present invention are shown in the following table:

[0106]

[0107]

[0108] During the training process, the algorithm records the cumulative rewards of each round and draws the trend chart of the round rewards. Figure 7 The training results of this embodiment in a multi-aircraft scenario are presented. In an experiment involving coordinated penetration of five aircraft, the algorithm gradually converged after approximately 350,000 rounds of sampling training. To verify the effectiveness of the trained model in making penetration decisions for a multi-aircraft system, 500 penetration experiments were first conducted in the training scenario. Separately, 500 simulation experiments were conducted in two new scenarios with adjusted launch and target point relationships. The number of successful penetrations and the penetration success rate are shown in the following table.

[0109]

[0110] This embodiment provides an intelligent decision-making method for multi-aircraft coordinated penetration. Based on the intelligent decision-making model obtained by the training method for the intelligent decision-making model for multi-aircraft coordinated penetration described in the above embodiment, the method determines the behavior of each aircraft. This invention effectively enables decision-making and training for multi-aircraft coordinated missions with a parameterized action space.

[0111] In another embodiment, a multi-agent decision model training apparatus in a parameterized action space is provided, comprising:

[0112] The first module is used to obtain the global state information of the multi-agent system at the current moment and the trajectory information of each agent in the multi-agent system, wherein the trajectory information includes the observation information of the agent at the current moment and the action information at the previous moment;

[0113] The second module is used to input the trajectory information of each agent into the policy network corresponding to each agent, and the policy network corresponding to each agent outputs discrete action variables and continuous action variables;

[0114] The third module is used to input the trajectory information of each agent at the current moment and the discrete action variables and continuous action variables corresponding to each agent into the individual evaluation network corresponding to each agent, and the individual evaluation network corresponding to each agent evaluates the individual Q value of the current individual behavior value of each agent;

[0115] The fourth module is used to take the global state information of the multi-agent system at the current moment and the individual Q values of the current individual behavior values of all agents as the input of the hybrid network. The hybrid network fits the joint Q value through a multi-layer nonlinear composite method;

[0116] The fifth module updates the strategy network parameters, individual evaluation network parameters and hybrid network parameters corresponding to each intelligent agent based on the joint Q value.

[0117] In another embodiment, a multi-agent decision-making method for a parameterized action space is provided, and the behavioral actions of each agent are decided based on the multi-agent decision model for the parameterized action space obtained by the multi-agent decision model training method for the parameterized action space mentioned above.

[0118] In another embodiment, a multi-agent decision-making device for a parameterized action space is provided, comprising an input module and a multi-agent decision-making model. The multi-agent decision-making model is obtained using the multi-agent decision-making model training method for the parameterized action space described above:

[0119] The input module obtains the global state information of the multi-agent system at the current moment and the trajectory information of each agent in the multi-agent system and inputs it into the multi-agent decision model training method of the parameterized action space. The multi-agent decision model outputs the behavioral action decision of each agent.

[0120] In addition, this embodiment further provides a multi-agent decision-making system for a parameterized action space, comprising a computer device programmed to execute the steps of the multi-agent decision-making method for a parameterized action space described above in this embodiment, or a storage medium of the computer device storing a computer program programmed to execute the multi-agent decision-making method for a parameterized action space described above in this embodiment. In addition, this embodiment further provides a computer-readable medium storing a computer program programmed to execute the multi-agent decision-making method for a parameterized action space described above in this embodiment.

[0121] In summary, the present invention utilizes a decision-making algorithm to ensure the effective coordination of multiple agents, and combines the dual-headed policy output in the parameterized deep deterministic policy gradient algorithm to achieve effective coupling of the parameterized action space; through the structure of the value decomposition network, it ensures the effective coordination of the multi-agent system, effectively avoids the emergence of the "credit allocation" problem, and designs a policy network and evaluation network structure for processing the parameterized action space, which can effectively make decisions and train multi-aircraft collaborative tasks with parameterized action space.

[0122] On the other hand, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the multi-agent decision model training method for parameterized action space provided in any of the above embodiments. The computer device may be a server. The computer device comprises a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sample data. The network interface of the computer device is used to communicate with an external terminal via a network connection.

[0123] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multi-agent decision model training method for parameterized action space provided in any of the above embodiments.

[0124] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0125] Matters not covered by the present invention are known technologies.

[0126] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

[0128] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A multi-agent decision model training method with parameterized action space, characterized by: include: Obtaining global state information of the multi-agent system at the current moment and trajectory information of each agent in the multi-agent system, wherein the trajectory information includes observation information of the agent at the current moment and action information of the agent at the previous moment, wherein the multi-agent system is a multi-aircraft coordinated penetration system and the agents are aircraft; The trajectory information of each agent is input into the strategy network corresponding to each agent, and the strategy network corresponding to each agent outputs discrete action variables and continuous action variables; The trajectory information of each agent at the current moment and the discrete action variables and continuous action variables corresponding to each agent are input into the individual evaluation network corresponding to each agent, and the individual evaluation network corresponding to each agent evaluates the individual Q value of the current individual behavior value of each agent; The global state information of the multi-agent system at the current moment and the individual Q values of the individual behavior values of all agents at the current moment are used as the input of the hybrid network. The hybrid network fits the global joint Q value through a multi-layer nonlinear composite method. Based on the global joint Q value, the strategy network parameters, individual evaluation network parameters and hybrid network parameters corresponding to each agent are updated until the training of the multi-agent decision model is completed, wherein the loss function of the individual evaluation network parameters and hybrid network parameters corresponding to each agent is constructed based on the global joint Q value. , optimize the loss function and update the individual evaluation network parameters and hybrid network parameters corresponding to each agent at the same time, where the loss function is as follows: ； in Indicates the mathematical expectation of the expression in [ ], express The set of individual evaluation network parameters of each agent, Indicates the i The individual evaluation network parameters of each agent, Represents the global state information of the multi-agent system at the current moment, represents the joint continuous action set consisting of the continuous action variables of all agents at the current moment, represents the joint discrete action set composed of the discrete action variables of all agents at the current moment, It represents the sum of the reward values of all agents calculated according to the preset reward function at the current moment. Represents the set of trajectory information of all agents at the current moment, Represents the set of trajectory information of all agents at the next moment, represents the experience pool, represents the global joint Q value, represents the hybrid network parameters, represents the target global joint Q value, represents the discount factor, Represents the global state information of the multi-agent system at the next moment, represents the joint continuous action set consisting of the continuous action variables of all agents at the next moment, represents the joint discrete action set consisting of the discrete action variables of all agents at the next moment, Represents the target hybrid network parameters.

2. The multi-agent decision model training method for parameterized action space according to claim 1 is characterized in that: The policy network is a dual-head output MLP multi-layer neural network, which makes decisions based on the observation information of the agent at the current moment and the action information of the previous moment and outputs the discrete action variables and continuous action variables at the current moment.

3. The multi-agent decision model training method of parameterized action space according to claim 1 or 2, characterized in that: The hybrid network compounds the individual Q values of all agents' individual behavior values twice to obtain a global joint Q value, including: The global state information of the multi-agent system at the current moment is input into four different fully connected networks, and the four fully connected networks respectively output the first weight matrix , the second weight matrix , first bias and the second bias , Represents the first bias in m elements; Suppose there are n The first compounding of the individual Q values of all agents’ individual behavior values: ； in Indicates the i The individual Q value corresponding to the agent, the first weight matrix for The weight matrix, is the first weight matrix Middle i Rank j Elements of the column, , ; Represents the first compound result j elements; Based on the first composite result, a second composite is performed to obtain a global union Q value : ； in for The weight matrix , for The weight matrix, is the second weight matrix Middle j elements.

4. The multi-agent decision model training method for parameterized action space according to claim 3 is characterized in that: The policy network parameters of each agent are updated using a centralized policy gradient update mechanism as follows: ； in Indicates the mathematical expectation of the expression in [ ], represents the centralized policy gradient of the set of policy networks of all agents, express The collection of policy networks of agents, Indicates the i The policy network of each agent, express The set of policy network parameters of each agent, Indicates the i The policy network parameters of each agent, represents the global joint Q value, Represents the global state information of the multi-agent system at the current moment, Indicates the i The trajectory information of each agent, It represents the sum of the reward values of all agents calculated according to the preset reward function at the current moment. Represents the set of trajectory information of all agents at the current moment, represents the joint continuous action set consisting of the continuous action variables of all agents at the current moment, represents the joint discrete action set composed of the discrete action variables of all agents at the current moment, Represents the global state information of the multi-agent system at the next moment, Represents the set of trajectory information of all agents at the next moment, Represents the experience pool.

5. A multi-agent decision model training device with parameterized action space, characterized in that: include: The first module is configured to obtain the global state information of the multi-agent system at the current moment and the trajectory information of each agent in the multi-agent system, wherein the trajectory information includes the observation information of the agent at the current moment and the action information of the previous moment. The multi-agent system is a multi-aircraft coordinated penetration system, and the agents are aircraft. The second module is used to input the trajectory information of each agent into the policy network corresponding to each agent, and the policy network corresponding to each agent outputs discrete action variables and continuous action variables; The third module is used to input the trajectory information of each agent at the current moment and the discrete action variables and continuous action variables corresponding to each agent into the individual evaluation network corresponding to each agent, and the individual evaluation network corresponding to each agent evaluates the individual Q value of the current individual behavior value of each agent; The fourth module is used to take the global state information of the multi-agent system at the current moment and the individual Q values of the current individual behavior values of all agents as the input of the hybrid network. The hybrid network fits the global joint Q value through a multi-layer nonlinear composite method; The fifth module updates the strategy network parameters, individual evaluation network parameters and hybrid network parameters corresponding to each agent based on the global joint Q value until the training of the multi-agent decision model is completed, wherein the loss function of the individual evaluation network parameters and hybrid network parameters corresponding to each agent is constructed based on the global joint Q value. , optimize the loss function and update the individual evaluation network parameters and hybrid network parameters corresponding to each agent at the same time, where the loss function is as follows: ； in Indicates the mathematical expectation of the expression in [ ], express The set of individual evaluation network parameters of each agent, Indicates the i The individual evaluation network parameters of each agent, Represents the global state information of the multi-agent system at the current moment, represents the joint continuous action set consisting of the continuous action variables of all agents at the current moment, represents the joint discrete action set composed of the discrete action variables of all agents at the current moment, It represents the sum of the reward values of all agents calculated according to the preset reward function at the current moment. Represents the set of trajectory information of all agents at the current moment, Represents the set of trajectory information of all agents at the next moment, represents the experience pool, represents the global joint Q value, represents the hybrid network parameters, represents the target global joint Q value, represents the discount factor, Represents the global state information of the multi-agent system at the next moment, represents the joint continuous action set consisting of the continuous action variables of all agents at the next moment, represents the joint discrete action set consisting of the discrete action variables of all agents at the next moment, Represents the target hybrid network parameters.

6. A multi-agent decision-making method with parameterized action space, characterized in that: include: Based on the multi-agent decision model in parameterized action space obtained by the multi-agent decision model training method in parameterized action space as claimed in claim 1, the behavioral actions of each agent are decided.

7. A multi-agent decision-making device with parameterized action space, characterized in that: The method comprises an input module and a multi-agent decision model in a parameterized action space obtained by the multi-agent decision model training method in a parameterized action space according to claim 1; The input module obtains the global state information of the multi-agent system at the current moment and the trajectory information of each agent in the multi-agent system and inputs it into the multi-agent decision model of the parameterized action space obtained by the multi-agent decision model training method of the parameterized action space as claimed in claim 1, and the multi-agent decision model outputs the behavioral action decision of each agent.

8. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor executes the program to implement the steps of the multi-agent decision model training method in the parameterized action space described in claim 1.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program to implement the steps of the multi-agent decision model training method in the parameterized action space described in claim 1.

Images