Multi-agent target matching collaborative analysis method based on RMAPPO algorithm

Abstract

The invention belongs to the field of computer application, and discloses a multi-agent target matching collaborative analysis method based on an RMAPPO algorithm. The method comprises the following steps: (10) constructing a task scene scenario, and obtaining basic environment state information in a scene; and (20) constructing an opposite-side index system according to expert knowledge rules. (30) constructing a model data set and performing preprocessing; and (40) introducing a recurrent neural network to solve the problem of partial state observability. And (50) centrally training the value network and decentrally training the strategy network according to the RMAPPO algorithm. The method has the beneficial effects that the data set is constructed according to task scenarios, and the memory ability of the model is enhanced by introducing the recurrent neural network. According to the method, different intelligent agents can be accurately operated to perform different tasks, especially on the premise of ensuring that people are in a loop, an auxiliary decision-making reference scheme can be made for a decision-making person, and the timeliness and accuracy of modern command control are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer application, and designs a multi-agent target matching cooperative analysis method based on an RMAPPO algorithm. TECHNICAL BACKGROUND

[0002] In modern coping, unmanned coping systems have become an important part of multi-domain coping. Unmanned aerial vehicle clusters show great potential in tasks such as environment collection, electronic interference and accurate target matching. However, in complex electromagnetic environments and multi-target dynamic scenarios, how to realize the autonomous cooperation of unmanned aerial vehicle clusters, especially in target allocation and path planning, is still a key technical problem that needs to be solved. Traditional target allocation and path planning methods rely on centralized optimization or rule-driven mechanisms, which are difficult to cope with real-time in high-dynamic and strong-coping environments.

[0003] In recent years, reinforcement learning has been continuously developed in the application of autonomous decision-making of agents, and multi-agent reinforcement learning (MARL) provides a new solution for unmanned aerial vehicle cluster tasks. In particular, the multi-agent proximal policy optimization algorithm (MAPPO) is an extension of the PPO algorithm in the multi-agent scenario, which can handle the cooperation and coping problems between multiple unmanned aerial vehicles while ensuring the stability of policy updates. Through the centralized training, decentralized execution (CTDE) paradigm, MAPPO can utilize global information to improve learning efficiency in the training phase, and rely only on local observations for autonomous decision-making in the execution phase, thereby adapting to complex and variable task environments.

[0004] Applying the RMAPPO algorithm to the target allocation and pathfinding tasks of unmanned aerial vehicles can achieve the following advantages: first, cooperative allocation, the unmanned aerial vehicle group can autonomously learn a reasonable target allocation strategy according to target value, device efficiency and task constraints; second, intelligent pathfinding, unmanned aerial vehicles can dynamically plan paths to improve penetration and survival capabilities in the face of space interference, electronic interference and terrain obstacles; third, coping adaptation, in the presence of opponent intelligent defense, RMAPPO can improve the robustness of the strategy through coping training. Thus, this technology provides a feasible intelligent solution for future unmanned coping systems to autonomously cope in high-threat environments. SUMMARY

[0005] The present application aims to solve the problem that the traditional target assignment and path planning method relies on centralized optimization or rule-driven mechanism in a multi-target dynamic scene environment, and it is difficult to respond in real time in a high dynamic and strong response environment. Therefore, we propose a multi-agent target matching collaborative analysis method based on RMAPPO algorithm. Specifically, the following steps are included:

[0006] (10) Construct the task scenario assumption and obtain the basic environmental state information in the scene.

[0007] (11) According to the task scenario assumption, obtain the deployment information of our side and the opponent in the task scene, including longitude, latitude, altitude, and highland name, etc.

[0008] (12) Obtain the information of our MALD unmanned aerial vehicle, storm shadow unmanned aerial vehicle, MALD, storm shadow, and the oil of each unmanned aerial vehicle in the task scene, as well as the information of the opponent's 1st group, 2nd group, and airport.

[0009] (20) According to the expert knowledge rule, build the opponent's index system.

[0010] (21) In the Mozi simulation environment, the expert rule undertakes the definition function of the basic logic and behavior constraint of the task scene. When our agent enters the attack circle of the opponent, the opponent will automatically launch an attack according to the rule script; if our unit is out of the detection range of the opponent's sensor, the opponent will enter the search and alert state, etc.

[0011] (22) The expert rule defines the priority of device use, formation action mode, and electronic interference conditions, so that the simulation environment has strong rule-driven and controllability. In this way, complex response scenarios can be quickly constructed while ensuring the repeatability and stability of the simulation.

[0012] (30) Construct the model data set and do preprocessing.

[0013] (31) In the running process of Mozi simulation engine, high-dimensional situation data will be generated according to expert rules and environmental evolution. It includes the spatial position parameters (current latitude, longitude, altitude, speed, heading) of each agent, covers the state information (attitude angle, remaining fuel, equipment storage, damage assessment) of each agent itself, and the sensor and electronic related data (sensor GUID set, detection range, signal strength, target recognition probability). A single agent can collect more than 50-dimensional state characteristics, fully describing its dynamic properties and interaction relationships in the task scene environment. This high-dimensional data not only truly reproduces the complexity of the task process, but also provides a rich input space for subsequent data-driven modeling and agent training.

[0014] (32) Build drone dataset according to mission scenario, dataset includes attack_position, attack_target, attack_distance, orientation_angle, flight_speed, etc.

[0015] (40) Introduce recurrent neural network to solve partial state observability.

[0016] (41) Introduce recurrent neural network (RNN) to handle partial observability, let multiple agents have memory. In the Actor network, for agent i, the RNN input at time t is

[0017]

[0018] Where: is the current observation, is the previous action, f enc is an MLP layer that concatenates and projects the input to a fixed dimension.

[0019] Get the input vector:

[0020]

[0021] (42) The update rule of RNN is as follows:

[0022]

[0023] Where: is the hidden state of the previous time step, is the hidden state of the current time step.

[0024] The output hidden state of RNN is used to generate the action distribution:

[0025]

[0026] Where: is the actual sampled action of agent i at time step t.

[0027] (44) In the Critic network, use centralized training value network, input is the concatenation of all observations

[0028]

[0029] Where: f encFor a MLP layer, the inputs are concatenated and projected to a fixed dimension.

[0030] (45) The RNN update rule is as follows:

[0031]

[0032] where: is the hidden state of the previous time step, is the hidden state of the current time step.

[0033] (46) The value network input is as follows:

[0034]

[0035] where:

[0036] (50) According to the RMAPPO algorithm, the value network is centrally trained, and the strategy network is decentralized.

[0037] (51) The optimization goal of the strategy network is as follows:

[0038]

[0039] where: Advantage function is calculated using the GAE method, S is the strategy entropy, and sigma is the entropy coefficient hyperparameter.

[0040] The optimization goal of the value network is as follows:

[0041]

[0042] where: is the discounted reward, B is the batch size, and n represents the number of agents.

[0043] Compared with the traditional artificial analysis method, the beneficial effects of the present application are: through reinforcement learning, continuous interaction with the environment, autonomous adjustment of decision-making in dynamic and non-deterministic environment, guaranteeing the real-time and flexibility of target allocation and path planning. The centralized training and decentralized execution (CTDE) mechanism is adopted, each unmanned aerial vehicle is regarded as an independent agent to make local decision, and the whole can still maintain global cooperation, greatly improving the autonomous cooperative response ability of the unmanned aerial vehicle cluster.

[0044] The present application uses recurrent neural network RNN to approximate the strategy through neural network, the inference stage has high calculation efficiency, can process large-scale unmanned aerial vehicle and target allocation problem, and supports real-time online inference. The strategy can be continuously trained and fine-tuned in simulation or actual combat data, gradually improving the performance of unmanned aerial vehicles in task allocation and path finding, and realizing the real "getting stronger and stronger". Attached Figure Description

[0045] Figure 1 This is a flowchart of the method of the present invention.

[0046] Figure 2 This is the training graph for maximizing the strategy network in this method.

[0047] Figure 3 This method minimizes the loss training graph in the value network.

[0048] Figure 4 This is a diagram showing the modular composition of the method of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] This invention is a multi-agent target matching collaborative analysis method based on the RMAPP0 algorithm, and the specific steps are as follows:

[0051] (10) Construct a task scenario and obtain basic environmental state information within the scenario.

[0052] (11) Based on the task scenario, obtain the deployment information of our side and the other side in the task scenario, including longitude, latitude, elevation and highland name, etc.

[0053] (12) Obtain information on our MALD drone, Storm Shadow drone, MALD, Storm Shadow, fuel levels of each drone, and information on the other party's Group 1, Group 2, and airport within the mission scenario.

[0054] (20) Construct the other party's indicator system based on expert knowledge rules.

[0055] (21) In the Mozi simulation environment, expert rules are responsible for defining the basic logic and behavioral constraints of the task scenario. When our agent enters the opponent's attack range, the opponent will automatically launch an attack according to the rule script; if our agent leaves the detection range of the opponent's sensor, the opponent will switch to search and alert states, etc.

[0056] (22) Expert rules define equipment usage priorities, formation action patterns, and electronic interference conditions, thereby giving the simulation environment strong rule-driven nature and controllability. In this way, complex response scenarios can be quickly constructed while ensuring simulation repeatability and stability. (30) Construct model datasets and perform preprocessing.

[0057] (30) Construct the model dataset and perform preprocessing.

[0058] (31) During the running of Mozi simulation engine, high-dimensional situation data will be generated according to expert rules and environmental evolution. It contains the spatial position parameters (current latitude, longitude, height, speed, heading) of each agent, covers the state information of each agent itself (attitude angle, remaining fuel, equipment reserves, damage assessment), and the sensor and electronic related data (sensor GUID set, detection range, signal strength, target recognition probability). A single agent can collect more than 50-dimensional state characteristics, fully describing its dynamic attributes and interaction relationships in the task scene environment. This high-dimensional data not only truly reproduces the complexity in the response process, but also provides a rich input space for subsequent data-driven modeling and agent training.

[0059] (32) According to the task assumption, the unmanned aerial vehicle data set is constructed, including attack_position (attack position), attack_target (attack target), attack_distance (attack distance), orientation_angle (orientation angle), flight_speed (flight speed), etc.

[0060] (40) Introduce recurrent neural network to solve the problem of partial state observability.

[0061] (41) Introduce recurrent neural network (RNN) to handle partial observability and allow multiple agents to have memory. In the Actor network, for agent i, the RNN input at time t is

[0062]

[0063] Where: is the current observation, is the previous action, f enc is an MLP layer that concatenates and projects the input to a fixed dimension.

[0064] Get the input vector:

[0065]

[0066] (42) The update rule of RNN is as follows:

[0067]

[0068] Where: is the hidden state of the previous time step, is the hidden state of the current time step.

[0069] The output hidden state of RNN is used to generate the action distribution:

[0070]

[0071] where: is the actual sampled action of agent i at time step t.

[0072] (44) In the Critic network, a centralized training value network is adopted, and the input is the concatenation of all observations

[0073]

[0074] where: f enc is an MLP layer that concatenates and projects the input to a fixed dimension.

[0075] (45) The RNN update rule is as follows:

[0076]

[0077] where: is the hidden state of the previous time step, is the hidden state of the current time step.

[0078] (46) The input of the value network is as follows:

[0079]

[0080] where:

[0081] (50) According to the RMAPPO algorithm, the value network is centrally trained, and the strategy network is decentralized.

[0082] (51) The optimization goal of the strategy network is as follows:

[0083]

[0084] where: Advantage function is calculated using the GAE method, S is the strategy entropy, and σ is the entropy coefficient hyperparameter.

[0085] The optimization goal of the value network is as follows:

[0086]

[0087] where: is the discounted reward, B is the batch size, and n represents the number of agents.

[0088] Compared with the traditional manual analysis method, the application has the beneficial effects that: by continuously interacting with the environment through reinforcement learning, the decision-making can be autonomously adjusted in a dynamic and non-deterministic environment, ensuring the real-time and flexibility of target allocation and path planning. The centralized training and decentralized execution (CTDE) mechanism is adopted, each unmanned aerial vehicle is regarded as an independent agent to make local decision, and the whole can still maintain global cooperation, greatly improving the autonomous cooperative response ability of the unmanned aerial vehicle cluster.

[0089] Example analysis:

[0090] The model is trained using 250 groups of data, an RNN network is introduced, the batch_size of each group is 64, the Epoch is set to 200, the experience revisit buffer buffer_size is 500000, the environment is sampled from the Mozi simulation engine, the hidden_size is 64, the maximum step max_steps of a game is 50, the Gaussian noise is introduced, the sampling frequency sample_interval is 100, the network parameter updating frequency step_interval is 25, the Relu is used as the activation function, the policy_activate_masks is used as the available action mask, the optimizer.zero_grad() is used for gradient zero clearing, the MLP network is used for feature extraction of the observation value, the loss.backward() is used for backward propagation to calculate the gradient, and the optimizer.step() is used for updating the network parameters.

[0091] The application is not limited to the above embodiments, and anyone should know that any technical solution made under the inspiration of the application and having the same or similar technical solutions as the application falls within the protection scope of the application.

Claims

1. A multi-agent target matching and collaborative analysis method based on the RMAPPO algorithm, characterized in that, The steps of the method are as follows: (10) Construct a task scenario and obtain basic environmental state information within the scenario. (20) Construct the other party's indicator system based on expert knowledge rules. (30) Construct the model dataset and perform preprocessing. (40) Introduce recurrent neural networks to solve the problem of partial state observability. (50) The value network is trained in a centralized manner according to the RMAPPO algorithm, and the policy network is trained in a decentralized manner.

2. The multi-agent target matching and collaborative analysis method based on the RMAPPO algorithm according to claim 1, characterized in that, The specific contents of (10) are as follows: (11) Based on the task scenario, obtain the deployment information of our side and the other side in the task scenario, including longitude, latitude, elevation and highland name, etc. (12) Obtain information on our MALD drone, Storm Shadow drone, MALD, Storm Shadow, fuel levels of each drone, and information on the other party's Group 1, Group 2, and airport within the mission scenario.

3. The multi-agent target matching and collaborative analysis method based on the RMAPPO algorithm according to claim 1, characterized in that, The specific steps of (20) are as follows: (21) In the Mozi simulation environment, expert rules are responsible for defining the basic logic and behavioral constraints of the task scenario. When our agent enters the opponent's attack range, the opponent will automatically launch an attack according to the rule script; if our agent leaves the detection range of the opponent's sensor, the opponent will switch to search and alert states, etc. (22) Expert rules define equipment usage priorities, formation action patterns, and electronic interference conditions, thereby giving the simulation environment strong rule-driven nature and controllability. In this way, complex task scenarios can be quickly constructed while ensuring simulation repeatability and stability.

4. The multi-agent target matching and collaborative analysis method based on the RMAPP0 algorithm according to claim 1, characterized in that, The specific steps of (30) are as follows: (31) During operation, the Mozi simulation engine generates high-dimensional situational data based on expert rules and environmental evolution. This data includes spatial position parameters of each agent (current latitude, longitude, altitude, speed, heading), state information of each agent (attitude angle, remaining fuel, equipment reserves, damage assessment), and sensor and electronic data (sensor GUID set, detection range, signal strength, target recognition probability). A single agent can collect more than 50 dimensions of state features, comprehensively depicting its dynamic attributes and interaction relationships in the mission environment. This high-dimensional data not only realistically reproduces the complexity of the mission process but also provides a rich input space for subsequent data-driven modeling and agent training. (32) Construct a UAV dataset based on the mission scenario. The dataset includes attack_position, attack_target, attack_distance, orientation_angle, flight_speed, etc.

5. The multi-agent target matching and collaborative analysis method based on the RMAPP0 algorithm according to claim 1, characterized in that, The specific steps of (40) are as follows: (41) Introduce a recurrent neural network (RNN) to handle partial observability, allowing multiple agents to possess memory. The input is the local observations of each agent. Previous action Hidden layer states of RNN The output is the current action. and the updated hidden layer state 6. The multi-agent target matching and collaborative analysis method based on the RMAPPO algorithm according to claim 1, characterized in that, The specific steps of (50) include the following: (51) The optimization objective of the strategy network is as follows: in: Advantage function It is calculated using the GAE method, where S is the policy entropy and σ is the entropy coefficient hyperparameter. (52) The optimization objectives of the value network are as follows: in It represents the discount reward, B is the batch size, and n represents the number of agents.

Images