A multi-agent collaborative interactive decision-making and control method in an interactive scenario

Abstract

The present invention proposes a multi-agent collaborative interactive decision-making and control method in an interactive scenario. First, each agent obtains the temporal state information of the confrontation scenario. In the decision-making system of each agent, the acquired temporal state information of the confrontation scenario is input into the LSTM network for prediction and feature extraction to obtain the hidden state. The hidden state is input into the SAC reinforcement learning model. The LSTM network and the SAC reinforcement learning model are trained using a designed reward function, and finally the maneuver decision action vector of each agent is obtained. Each agent uses the maneuver decision action vector as the control target value and controls the agent according to a discrete incremental PID controller. Based on the multi-agent SAC reinforcement learning algorithm, this method combines LSTM temporal processing and a dual experience replay mechanism. By designing a reasonable reward function mechanism, it can quickly predict the motion trajectory of dynamic objects in an environment with complex dynamic objects and make accurate decisions on the amount of its own continuous actions.

Description

Technical Field

[0001] The present invention relates to the field of intelligent agent decision-making and control, and specifically to a multi-agent collaborative interactive decision-making and control method in an interactive scenario. Background Art

[0002] Agents are AI entities capable of perceiving their environment and making autonomous decisions. Multi-agent systems (MAS) overcome individual limitations through collaboration, improving system performance through information sharing and task sharing. These systems are widely used in various fields. Due to issues such as partially observable environments and conflicts of interest between individuals and groups, MAS collaboration requires dynamic coordination rather than simply superimposing behaviors. Multi-agent reinforcement learning (MARL) incorporates the trial-and-error optimization mechanism of reinforcement learning into MAS. Using the Markov decision process (MDP) as a framework, MARL trains group collaboration strategies through environmental feedback. Since Littman proposed the basic framework, it has become a core technology in fields such as distributed decision-making and intelligent robotics.

[0003] As multi-agent systems are widely used in complex interactive scenarios (such as autonomous flight against aerial targets and autonomous driving against complex road conditions), agents face dynamic and uncertain environments. In such scenarios, key information such as the position and speed of obstacles or opponents changes rapidly over time, placing extremely high demands on the agents' decision-making and control capabilities.

[0004] Traditional multi-agent reinforcement learning approaches face significant technical bottlenecks when addressing agent decision-making and control tasks in complex interactive scenarios. Traditional approaches typically make decisions based solely on the current state, ignoring the continuity and temporal correlation of environmental state changes. This limitation is particularly pronounced in environments with unknown dynamic obstacles. Due to a lack of effective utilization of past state information, agents are unable to accurately capture the motion trajectories of dynamic objects and struggle to predict their future positions, resulting in a lack of foresight and accuracy in decision-making. For example, in autonomous driving scenarios, vehicles may be unable to make informed decisions based on changing road conditions, increasing the probability of traffic accidents. Furthermore, during training, traditional approaches often employ a single sampling strategy for experience replay, such as uniform sampling or prioritized experience replay. Uniform sampling undervalues ​​key experiences, resulting in slow learning and requiring agents to acquire effective decision-making strategies over time. Prioritized experience replay, while accelerating convergence, overly focuses on a few high-value experiences at the expense of comprehensive exploration of the environment. This results in poor adaptability and robustness in novel environments or unexpected situations, hindering their ability to flexibly respond to complex situations. Summary of the Invention

[0005] To ensure that intelligent agents can accurately predict dynamic environmental information, better adapt to the environment, and make effective decisions, this paper proposes a multi-agent collaborative interactive decision-making and control method for interactive scenarios. This method, based on a multi-agent SAC reinforcement learning algorithm, combines LSTM time series processing with a dual-experience replay mechanism. Through a well-designed reward function mechanism, it can rapidly predict the motion trajectories of dynamic objects (such as opposing intelligent agents and kinetic bodies) in environments with complex dynamic objects and accurately determine the amount of continuous action to be taken. Compared to traditional methods, this method not only accelerates training but also significantly improves the success rate of interactive confrontations.

[0006] The technical solution of the present invention is:

[0007] A multi-agent collaborative interactive decision-making and control method in an interactive scenario includes the following steps:

[0008] Step 1: Each agent obtains the temporal state information of the adversarial scenario;

[0009] Step 2: In the decision-making system of each agent, the acquired adversarial scenario temporal state information is input into the LSTM network for prediction and feature extraction to obtain the hidden state vector , used as input for subsequent reinforcement learning;

[0010] Step 3: In the decision-making system of each agent, the hidden state vector obtained in step 2 is input into the SAC reinforcement learning model. The LSTM network and the SAC reinforcement learning model are trained using the designed reward function to finally obtain the maneuver decision action vector of each agent.

[0011] Step 4: Each agent obtains the maneuver decision action vector through step 3 Then, the maneuver decision action vector As the control target value, the intelligent agent is controlled according to the discrete incremental PID controller.

[0012] Furthermore, in step 1, the temporal state information of the adversarial scenario obtained by the agent includes:

[0013] (1) The three-dimensional coordinates of the agent itself and the other agents on its side relative to the agent itself at the current time step and the historical time step;

[0014] (2) The three-dimensional coordinates of the other agent relative to the agent itself at the current time step and the historical time step;

[0015] (3) The three-axis velocity components of the agent itself and the other agents on its side relative to the agent itself at the current time step and the historical time step;

[0016] (4) The three-axis velocity components of the other agent relative to the agent itself at the current time step and the historical time step;

[0017] (5) The attitude angle of the agent at the current time step and the historical time step;

[0018] (6) The three-dimensional coordinates of the kinetic energy body carried by the other agent relative to the agent itself at the current time step and the historical time step;

[0019] (7) The three-axis velocity components of the kinetic energy body carried by the other agent relative to the agent itself at the current time step and the historical time step;

[0020] (8) The absolute value of the current velocity of the kinetic energy body carried by the other agent relative to the agent itself.

[0021] Furthermore, in step 2, the hidden state vector The dimension of the state information of the confrontation scene obtained by the agent is lower than the dimension of the temporal state information, and the hidden state vector , explicitly includes the predicted three-dimensional coordinates of the opponent agent relative to the own agent in the next time step.

[0022] Furthermore, in step 3, the SAC reinforcement learning model includes an Actor network and a Critic network;

[0023] The prediction and dimensionality-reduced hidden state vector obtained in step 2 Input Actor network to generate maneuver decision action vector a = [ pitch, roll, rudder , engine ] , each element in the vector represents the agent's pitch decision target value, roll decision target value, steering decision target value, and thrust decision target value respectively;

[0024] The hidden state vector and the generated maneuver decision action vector Input Critic network, output action value .

[0025] Furthermore, in step 3, during the training process, the designed reward function includes angle advantage reward, height advantage reward, speed advantage reward and win-loss reward, and also adds a trajectory prediction reward. The final reward function is the sum of the reward functions with random weights added.

[0026] Furthermore, the reward functions are:

[0027] Angle Advantage Bonus :

[0028]

[0029] is the sight angle;

[0030] Height Advantage Reward :

[0031]

[0032] in, The ideal height for setting; is the height of the agent itself, It is a high safety zone;

[0033] Speed ​​Advantage Bonus :

[0034]

[0035] in, is the agent’s own speed, is the effective speed range;

[0036] Win or Lose Rewards :

[0037] If you defeat the opponent, you will receive a reward Take a positive value. If defeated by the opponent, the victory or defeat reward Take negative values;

[0038] Trajectory prediction reward :

[0039]

[0040] in, is the position of the other agent n at the next time step predicted by the LSTM network, is the actual position of the other agent n at the next time step;

[0041] The final total reward function is:

[0042]

[0043] in , Indicates the weight of each part.

[0044] Furthermore, a dual experience replay mechanism is used in the training process of step 3. In the early stage of training, a random experience pool is used for uniform sampling to ensure that the agent can fully explore the environment. In the middle and late stages of training, a priority experience pool is used for priority sampling based on the reward value to accelerate the learning of high-value experience.

[0045] Furthermore, a basic action library is also stored in the decision-making system of the intelligent agent, which sets up a variety of typical maneuvering actions of the intelligent agent. Under certain set conditions, the intelligent agent is not controlled by the reinforcement learning model, but performs typical maneuvering actions according to the settings of the basic action library. When other set conditions are met, the reinforcement learning model is used again for control.

[0046] In addition, the present invention also provides an electronic device and a readable storage medium:

[0047] An electronic device comprises a processor and a memory, wherein the memory is used to store one or more programs;

[0048] When the one or more programs are executed by the processor, the above method is implemented.

[0049] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0050] Beneficial effects:

[0051] The multi-agent collaborative interactive decision-making and control method in an interactive scenario proposed by the present invention has the following advantages:

[0052] 1. Introducing time series information into the state space allows the agent to obtain state information that is no longer limited to the current moment, but includes rich data from the current and historical time steps, effectively capturing long-term dependencies in the data.

[0053] 2. Compared with the traditional reinforcement learning process method that directly inputs the state vector into the Actor network, the present invention first predicts and reduces the dimensionality of the high-dimensional state vector containing time series information through the LSTM unit, and converts the time series information into a hidden state vector Reinforcement learning not only reduces dimensionality, avoiding the dimensionality explosion that occurs when directly inputting high-dimensional state vectors into the Actor network, but also includes the prediction of the next time step of the other party, so that the predicted value gradually approaches the true relative position. This enables more accurate prediction of the future motion trajectory of dynamic objects, pre-planning response strategies, significantly improving the accuracy and timeliness of decision-making, and providing a more comprehensive and accurate basis for intelligent agent decision-making.

[0054] 3. The present invention adopts a dual experience replay mechanism to assist reinforcement learning. In the early stage of training, the mechanism adopts a random sampling strategy to enable the intelligent agent to fully explore the environment, obtain diverse state and action combinations, discover more potential effective strategies, and avoid falling into local optimal solutions; as the training progresses, it switches to a reward value-based priority sampling strategy in the middle and late stages, giving priority to learning those experiences that are more critical to improving system performance, accelerating the learning process, and improving training efficiency; by adopting different sampling strategies in different training stages, it effectively balances exploration and utilization in reinforcement learning, ensuring that the intelligent agent can quickly learn efficient decision-making strategies while fully understanding the environment.

[0055] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments with reference to the following drawings, in which:

[0057] Picture 1 This is a logic flow chart of the interaction between an intelligent agent using the method of the present invention and built-in AI of simulation platforms at various levels in an embodiment of the present invention;

[0058] Picture 2 This is a curve showing the change in the sum of the accumulated reward values ​​of the intelligent agents using the method of the present invention in an embodiment of the present invention.

[0059] Picture 3 This is the changing curve of the sum of the agent's cumulative reward values ​​in the traditional SAC algorithm. DETAILED DESCRIPTION

[0060] The following describes in detail embodiments of the present invention. The embodiments are exemplary and intended to explain the present invention, but are not to be construed as limiting the present invention.

[0061] In this embodiment, taking 4V4 intelligent agent confrontation interaction as an example, the multi-agent collaborative interaction decision-making and control method in the interactive scenario proposed by the present invention is used to train the intelligent agent's built-in decision-making model, and interact with the built-in AI intelligent agent of the existing interactive simulation platform to verify the decision-making effectiveness of the method of the present invention.

[0062] The specific steps include:

[0063] Step 1: The agent obtains the temporal state information of the adversarial scenario;

[0064] This embodiment is a 4V4 intelligent agent interactive confrontation scenario, including four A-party intelligent agents and four B-party intelligent agents. Each intelligent agent also has two kinetic bodies, which can be sent out by the intelligent agent and move according to a pre-set control rate.

[0065] Taking Agent 1 of Party A as an example, the acquired temporal state information of the confrontation scenario includes:

[0066] (1) In the world coordinate system, the three-dimensional coordinates of Party A's agent No. 1 and the other three agents of Party A relative to Party A's agent No. 1 at the current time step and the historical time step (such as the previous 10 steps), where the current The coordinate information of each time step is expressed as:

[0067] [ x 1(i) , y 1(i) , z 1(i) , x 2(i) , y 2(i) , z 2(i) , x 3(i) , y 3(i) , z 3(i) , x 4(i) , y 4(i) , z 4(i) ]

[0068] in is the three-dimensional coordinate time series matrix of agent 1 of party A in the world coordinate system, which is composed of the three-dimensional coordinates of agent 1 of party A at the current time step and the historical time step. 、 、 They are respectively the three-dimensional coordinate time series matrices of the other three agents of Party A relative to Agent 1 of Party A in the world coordinate system, which are composed of the relative three-dimensional coordinates of the other three agents of Party A relative to Agent 1 of Party A in the current time step and the historical time step.

[0069] (2) In the world coordinate system, the three-dimensional coordinates of the four agents of party B relative to agent 1 of party A at the current time step and the historical time step are:

[0070] [ x 5(i) , y 5(i) , z 5(i) , x 6(i) , y 6(i) , z 6(i) , x 7(i) , y 7(i) , z 7(i) , x 8(i) , y 8(i) , z 8(i) ]

[0071] in 、 、 、 They are respectively the three-dimensional coordinate time series matrices of the four agents of Party B relative to Agent 1 of Party A in the world coordinate system, which are composed of the relative three-dimensional coordinates of the four agents of Party B relative to Agent 1 of Party A at the current time step and the historical time step.

[0072] (3) In the world coordinate system, the three-axis velocity components of Agent 1 of Party A and the other three agents of Party A relative to Agent 1 of Party A at the current time step and the historical time step:

[0073] [ v 1x i , v 1y i , v 1z i , v 2x i , v 2y i , v 2z i , v 3x i , v 3y i , v 3z i , v 4x i , v 4y i , v 4z i ]

[0074] in is the three-dimensional velocity time series matrix of Agent 1 of Party A in the world coordinate system, which is composed of the three-axis velocity components of Agent 1 of Party A at the current time step and the historical time step. 、 、 They are respectively the three-dimensional velocity time series matrices of the other three agents of Party A relative to Agent 1 of Party A in the world coordinate system, which are composed of the relative three-axis velocity components of the other three agents of Party A relative to Agent 1 of Party A in the current time step and the historical time step.

[0075] (4) In the world coordinate system, the three-axis velocity components of the four agents of party B relative to agent 1 of party A at the current time step and the historical time step:

[0076] [ v 5x(i) , v 5y(i) , v 5z(i) , v 6x(i) , v 6y(i) , v 6z(i) , v 7x(i) , v 7y(i) , v 7z(i) , v 8x(i) , v 8y(i) , v 8z(i) ]

[0077] in 、 、 、 They are respectively the three-dimensional velocity time series matrices of the four agents of Party B relative to Agent 1 of Party A in the world coordinate system, which are composed of the relative three-axis velocity components of the four agents of Party B relative to Agent 1 of Party A in the current time step and the historical time step.

[0078] (5) The posture angles of the four agents of party A at the current time step and the historical time step include:

[0079] Pitch angle:

[0080]

[0081] Roll angle:

[0082]

[0083] Yaw angle:

[0084] ;

[0085] in 、 、 、 These are the posture angle matrices of the four agents on party A, which are composed of the posture angles of the four agents on party A at the current time step and the historical time step.

[0086] (6) Each of the four agents on party B carries two releasable kinetic bodies. Therefore, the temporal state information of the confrontation scenario also includes the three-dimensional coordinates of the eight kinetic bodies on party B relative to agent No. 1 on party A at the current time step and the historical time step in the world coordinate system:

[0087] , , , , , , ,

[0088] They are respectively the three-dimensional coordinate time series matrices of the eight kinetic bodies of Party B relative to the intelligent agent No. 1 of Party A in the world coordinate system, which are composed of the relative three-dimensional coordinates of the eight kinetic bodies of Party B relative to the intelligent agent No. 1 of Party A in the current time step and the historical time step.

[0089] (7) In the world coordinate system, the three-axis velocity components of the eight kinetic bodies of party B relative to the intelligent body No. 1 of party A at the current time step and the historical time step are:

[0090] , , , , , , ,

[0091] They are respectively the three-dimensional velocity time series matrices of the eight kinetic bodies of Party B relative to the intelligent agent No. 1 of Party A in the world coordinate system, which are composed of the relative three-dimensional velocity components of the eight kinetic bodies of Party B relative to the intelligent agent No. 1 of Party A in the current time step and the historical time step.

[0092] (8) In the world coordinate system, the absolute values ​​of the current velocities of the eight kinetic bodies relative to the A-party agent No. 1 , , , , , , , .

[0093] It can be seen that the confrontation scene time series state information obtained by a single agent is a high-dimensional state vector of 1196 dimensions. The decision-making system of each agent obtains the confrontation scene time series state information at a set time interval. The time interval in this embodiment is set to .

[0094] Step 2: In the decision-making system of each intelligent agent, the acquired temporal state information of the adversarial scenario is input into the LSTM network for prediction and feature extraction to obtain the hidden state vector, which is used as the input for subsequent reinforcement learning.

[0095] Taking Agent 1 of Party A as an example, the decision-making system of Agent 1 of Party A inputs the 1196-dimensional high-dimensional time series state information corresponding to the agent into the LSTM network of its own decision-making system for prediction and feature extraction, and obtains a 256-dimensional hidden state vector . It contains the key dependencies in the time series information, providing more accurate and comprehensive information for Party A's agent No. 1. Moreover, the last 12 dimensions of the 256-dimensional data are explicitly configured as the predicted three-dimensional coordinates of Party B's four agents relative to Party A's agent No. 1 in the next time step. This information is also predicted by the LSTM network. This is a key feature that is different from the existing technology that uses the LSTM network for only encoding. By predicting in advance the predicted three-dimensional coordinates of the four agents of Party B relative to Party A's agent No. 1 in the next time step, it can more accurately guide the decision-making of Party A's agent. The remaining 244 dimensions are implicit decision features, and the time series dependencies are automatically extracted through the LSTM network gating mechanism, mainly including the following information: (1) the coordinate trend, speed stability and posture coordination of the four agents of Party A; (2) the threat characteristics such as the speed change information of the agent of Party B and the collision risk of the kinetic body; (3) the effectiveness of historical actions and the memory of threat events (such as successful obstacle avoidance actions).

[0096] Here is a theoretical description of the LSTM network:

[0097] The LSTM network processes input sequence data through a series of gating mechanisms, effectively capturing long-term dependencies in the data. Its core formula is as follows:

[0098] Input Gate: , the input gate determines the current input How much information is added to the cell state? It is the sigmoid activation function, which maps the input value to the (0,1) interval, so that the input gate outputs a value between 0 and 1 to control the current input How much information is added to the cell state? It is the weight matrix of the input gate, which determines the weight distribution of the input information and the hidden state vector at the previous moment in the input gate. It is continuously adjusted during the training process to optimize the network performance. is a bias vector that provides additional learnable parameters for the calculation of the input gate, helping the model to better fit the data. Indicates that the hidden state vector of the previous moment and the current input This splicing operation enables the input gate to comprehensively consider current information and historical information, thereby more accurately deciding which information to incorporate into the cell state.

[0099] Forget Gate: , the forget gate determines the cell state at the previous moment How much information is about to be forgotten. It is the weight matrix of the forget gate, which controls the influence of the previous hidden state vector and the current input on the forget gate decision. The function of the forget gate is optimized by adjusting the weight during training. Is the bias vector, which adds flexibility to the calculation of the forget gate. Through the sigmoid activation function The forget gate outputs a value between 0 and 1. If the value is close to 1, it means that the system tends to retain most of the past information; if it is close to 0, it means that the system tends to forget most of the past information. In this way, the forget gate can dynamically adjust the degree of retention of historical information based on the current input and the previous state, allowing the cell state to better adapt to changes in the environment.

[0100] Cell status update: , It is a candidate cell state, which is obtained by processing the hyperbolic tangent activation function tanh. Tanh can highlight important information and suppress unimportant information, so that the candidate cell state can effectively capture the key features of the current input and historical state. is the weight matrix, are bias vectors, which are continuously optimized during the training process to adjust the generation of candidate cell states. The latter formula shows the update process of the cell state. Indicates that the cell state at the previous moment is retained according to the output of the forget gate The required part in ⊙ represents element-by-element multiplication, that is, only when When the corresponding element in is close to 1, The information at the corresponding position will be retained. It means that according to the output of the input gate, the candidate cell state In this way, the cell state can not only retain important historical information but also update the current information in a timely manner, thus better reflecting the dynamic changes of the environment.

[0101] Output gate: , and are the weight matrix and bias vector, which are optimized during training to determine how much information in the current cell state is output to the hidden state vector middle. The function maps the output value to the (0,1) interval to control the output ratio. Indicates that according to the control of the output gate, appropriate information is selected from the processed cell state as the hidden state vector The final hidden state vector is It includes the short-term features of the current time step and the long-term dependence of historical information. In the four-agent interactive confrontation scenario of this embodiment, it provides key information for the agent's decision-making, helping the agent predict the opponent's actions and plan its own actions.

[0102] Step 3: In the decision-making system of each agent, the hidden state vector obtained in step 2 is input into the SAC reinforcement learning model, and the LSTM network and the SAC reinforcement learning model are trained with the designed reward function to finally obtain the maneuver decision action vector of each agent.

[0103] The SAC reinforcement learning model includes an Actor network and a Critic network.

[0104] The prediction and dimensionality-reduced hidden state vector obtained in step 2 Input Actor network to generate maneuver decision action vector a = [ pitch, roll, rudder , engine ] , each element in the vector represents the agent's pitch decision target value, roll decision target value, steering decision target value, and thrust decision target value respectively.

[0105] The hidden state vector and the generated maneuver decision action vector Input Critic network, output action value .

[0106] Compared with the traditional reinforcement learning process method that directly inputs the state vector into the Actor network, the present invention first predicts and reduces the dimensionality of the high-dimensional state vector containing time series information through the LSTM unit, and converts the time series information into a hidden state vector. Reinforcement learning not only reduces the dimension and avoids the dimensionality explosion phenomenon caused by directly inputting high-dimensional state vectors into the Actor network, but also includes the prediction of the next time step of the other party, so that the predicted value gradually approaches the true relative position, thereby being able to more accurately predict the future motion trajectory of dynamic objects, plan response strategies in advance, and significantly improve the accuracy and timeliness of decision-making.

[0107] During the training process, the designed reward function includes angle advantage reward, height advantage reward, speed advantage reward and win-loss reward, and also adds trajectory prediction reward to help the intelligent agent better adapt to the dynamic environment. The final reward function is the sum of each reward function with random weights added.

[0108] (1) Angle advantage bonus

[0109] Physical meaning: sight angle The smaller the angle between our velocity vector and the line connecting us and the enemy, the stronger the target locking ability and the higher the reward.

[0110] formula:

[0111]

[0112] (2) Highly Advantageous Rewards

[0113] Physical meaning: the height of our agent Need to maintain in a safe range , to avoid falling to the ground or losing the range advantage.

[0114] formula:

[0115]

[0116] in, The ideal height for setting;

[0117] (3) Speed ​​advantage reward

[0118] Physical meaning: our agent's speed Need to be maintained within the effective range , to avoid stall or overload .

[0119] formula:

[0120]

[0121] The reward within the safety zone is linearly positively correlated with speed, encouraging drivers to maintain medium to high speeds to improve maneuverability.

[0122] (4) Winning and losing rewards

[0123] Physical meaning: represents whether our agent defeats or is defeated by the other party in the interactive confrontation. In this embodiment, if we defeat the other party, the winning reward is Take 100, if defeated by the opponent, the victory reward Take -100; in this embodiment, the basis for judging whether it is defeated is that when the attacked party is in the inescapable zone of the attacker's kinetic energy body, the attacked party is considered to be defeated.

[0124] (5) Trajectory prediction reward

[0125]

[0126] in, is the position of party B’s agent n at the next time step predicted by the LSTM network, is the actual position of party B’s agent n at the next time step.

[0127] The final total reward function is:

[0128]

[0129] in , Indicates the weight of each part.

[0130] In the reward function, the trajectory prediction reward guides the LSTM network to the hidden state vector The four agents of Party B are trained to predict the three-dimensional coordinates of agent No. 1 of Party A in the next time step, so that the LSTM network prioritizes retaining the enemy's recent coordinate data by adjusting the weights of the forget gate and input gate, so that the predicted value gradually approaches the true relative position; and it is jointly optimized with the angle reward and speed reward to dynamically adjust the pitch, roll and other action decisions of our agent to achieve coordinated action and path planning.

[0131] In order to improve training efficiency, this embodiment also introduces a dual experience replay mechanism, which adopts different sampling strategies at different stages of training. In the early stage of training, a random experience pool is used for uniform sampling to ensure that the agent can fully explore the environment. In the middle and late stages of training (for example, after 15,000,000 steps), a priority experience pool is used for priority sampling based on the reward value to accelerate the learning of high-value experience.

[0132] Step 4: Each agent obtains the maneuver decision action vector through step 3 Then, the maneuver decision action vector As the control target value, the agent is controlled by a discrete incremental PID controller, which is a common method in the control field.

[0133] Furthermore, in this embodiment, the agent's decision-making system also stores a basic action library, which contains a variety of typical maneuvers for the agent, such as target tracking and rapid climbing. Under certain conditions, the agent is not controlled by the reinforcement learning model, but instead executes typical maneuvers according to the basic action library. For example, when a target enters the recognition range of the agent, the agent executes the target tracking action in the basic action library. However, when the target leaves the recognition range or the agent is threatened by an opposing kinetic body, the agent reverts to the reinforcement learning model for control. This avoids problems such as overly long intelligent decision sequences, unequal decision sequences, and overly sparse rewards.

[0134] In this embodiment, the four intelligent agents of Party A are intelligent agents that use the method proposed in the present invention to make decisions and control, and the four intelligent agents of Party B are built-in AI intelligent agents of the interactive simulation platform. The hyperparameter settings of the mobile decision network training are shown in Table 1.

[0135] Table 1 Hyperparameter settings for maneuver decision network training

[0136]

[0137] Finally, the training model of 20,000 Epochs is used as the test network to interact with the built-in AI of the simulation platform at each level. The interaction logic is as follows: Picture 1 As shown in Table 2, the test results show that the winning rate of the present invention is significantly higher than that of the traditional SAC, and the traditional SAC refers to the SAC reinforcement learning model that does not adopt the LSTM network.

[0138] Table 2 Comparison of test results of four-agent interactive decision-making network

[0139]

[0140] Depend on Picture 2 、 Picture 3 As shown in the changes in the cumulative rewards of the agents in each round, the sum of the cumulative reward values ​​of all agents in the present invention, which is based on the SAC reinforcement learning algorithm and combines LSTM time series processing and dual experience replay mechanism, basically stabilizes above 400 after the 25th round and basically maintains at around 500. Compared with traditional SAC, this method can make the cumulative rewards reach the ideal value faster and more steadily.

[0141] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention.

Claims

1. A multi-agent collaborative interactive decision-making and control method in an interactive scenario, characterized by: The following steps are involved: Step 1: Each agent obtains the temporal state information of the adversarial scenario; Step 2: In the decision-making system of each agent, the acquired adversarial scenario temporal state information is input into the LSTM network for prediction and feature extraction to obtain the hidden state vector , used as input for subsequent reinforcement learning; Step 3: In the decision-making system of each agent, the hidden state vector obtained in step 2 is input into the SAC reinforcement learning model. The LSTM network and the SAC reinforcement learning model are trained using the designed reward function to finally obtain the maneuver decision action vector of each agent. The designed reward function includes angle advantage reward, height advantage reward, speed advantage reward and win-loss reward, and also adds trajectory prediction reward. The final reward function is the sum of the reward functions with random weights added; The specific reward functions are: Angle Advantage Bonus : is the sight angle; Height Advantage Reward : in, The ideal height for setting; is the height of the agent itself, It is a high safety zone; Speed ​​Advantage Bonus : in, is the agent’s own speed, is the effective speed range; Win or Lose Rewards : If you defeat the opponent, you will receive a reward Take a positive value. If defeated by the opponent, the victory or defeat reward Take negative values; Trajectory prediction reward : in, is the position of the other agent n at the next time step predicted by the LSTM network, is the actual position of the other agent n at the next time step; The final total reward function is: in , Indicates the weight of each part; Step 4: Each agent obtains the maneuver decision action vector through step 3 Then, the maneuver decision action vector As the control target value, the intelligent agent is controlled according to the discrete incremental PID controller.

2. The multi-agent collaborative interactive decision-making and control method in an interactive scenario according to claim 1, characterized in that: In step 1, the agent obtains the following temporal state information of the adversarial scenario: (1) The three-dimensional coordinates of the agent itself and the other agents on its side relative to the agent itself at the current time step and the historical time step; (2) The three-dimensional coordinates of the other agent relative to the agent itself at the current time step and the historical time step; (3) The three-axis velocity components of the agent itself and the other agents on its side relative to the agent itself at the current time step and the historical time step; (4) The three-axis velocity components of the other agent relative to the agent itself at the current time step and the historical time step; (5) The attitude angle of the agent at the current time step and the historical time step; (6) The three-dimensional coordinates of the kinetic energy body carried by the other agent relative to the agent itself at the current time step and the historical time step; (7) The three-axis velocity components of the kinetic energy body carried by the other agent relative to the agent itself at the current time step and the historical time step; (8) The absolute value of the current velocity of the kinetic energy body carried by the other agent relative to the agent itself.

3. The multi-agent collaborative interactive decision-making and control method in an interactive scenario according to claim 1, characterized in that: In step 2, the hidden state vector The dimension of the state information of the confrontation scene obtained by the agent is lower than the dimension of the temporal state information, and the hidden state vector , explicitly includes the predicted three-dimensional coordinates of the opponent agent relative to the own agent in the next time step.

4. The multi-agent collaborative interactive decision-making and control method in an interactive scenario according to claim 1 or 3, characterized in that: In step 3, the SAC reinforcement learning model includes an Actor network and a Critic network; The prediction and dimensionality-reduced hidden state vector obtained in step 2 Input Actor network to generate maneuver decision action vector , each element in the vector represents the agent's pitch decision target value, roll decision target value, steering decision target value, and thrust decision target value respectively; The hidden state vector and the generated maneuver decision action vector Input Critic network, output action value .

5. The multi-agent collaborative interactive decision-making and control method in an interactive scenario according to claim 1, characterized in that: During the training process of step 3, a dual experience replay mechanism is used. In the early stage of training, a random experience pool is used for uniform sampling to ensure that the agent can fully explore the environment. In the middle and late stages of training, a priority experience pool is used for priority sampling based on the reward value to accelerate the learning of high-value experience.

6. The multi-agent collaborative interactive decision-making and control method in an interactive scenario according to claim 1, characterized in that: The decision-making system of the intelligent agent also stores a basic action library, which contains a variety of typical maneuvering actions of the intelligent agent. Under certain set conditions, the intelligent agent is not controlled by the reinforcement learning model, but performs typical maneuvering actions according to the settings of the basic action library. When other set conditions are met, the reinforcement learning model is used again for control.

7. An electronic device comprising a processor and a memory, wherein the memory is used to store one or more programs; characterized in that: When the one or more programs are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

8. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images