A method and device for air-ground heterogeneous platform collaborative decision-making based on multi-agent reinforcement learning

Abstract

The application discloses a ground-air heterogeneous platform cooperative decision-making method and device based on multi-agent reinforcement learning, relates to the technical field of multi-agent cooperative decision-making, and comprises the following steps: obtaining a current global state by splicing local observations of each agent; combining the local observations and platform type coding, and outputting a joint action through a strategy network; obtaining next-time local observations and immediate rewards after executing the action, storing experience elements containing the global state, the joint action, platform coding, next-time observations, a termination flag and the immediate rewards into an experience pool; judging whether the experience pool is full or reaches a specified number of steps: if not, continuing to cycle with new observations; if yes, inputting the global state and the platform coding into a critic network, calculating state values and advantage function estimates; alternately updating strategy and critic network parameters based on the advantage function estimates until convergence, and realizing ground-air heterogeneous platform cooperative decision-making by using the converged strategy network. The application can realize real-time joint decision-making among different heterogeneous platforms.

Description

Technical Field

[0001] This application relates to the field of multi-agent collaborative decision-making technology, and in particular to a method and apparatus for collaborative decision-making on ground-air heterogeneous platforms based on multi-agent reinforcement learning. Background Technology

[0002] Currently, in multi-platform collaborative applications such as wargaming and unmanned swarm warfare, existing technologies mainly fall into two categories: one is the method of independent learning for each platform, where each platform trains independently based on its own observation information, ignoring the mutual influence between teammates and opponents; the other is centralized or rule-based collaborative methods, such as assigning tasks according to priority and using fixed rules to control behavior and other constraints. In addition, conventional multi-agent joint training methods have also been used in wargaming and unmanned swarm warfare.

[0003] The main drawbacks of existing technologies include: constraint rules and scripts are set manually, which cannot adapt to dynamic changes on the battlefield and are difficult to deal with complex terrain and the coordination of multiple platform types; each platform fights independently according to fixed rules, making it difficult to achieve joint decision-making between different heterogeneous platforms; traditional optimization search has too much computational load, cannot make decisions in real time, and has very low decision-making efficiency in fast-paced attack and defense games.

[0004] Therefore, there is an urgent need for a technical solution that does not require manual pre-setting of rules, enables collaborative decision-making across heterogeneous platforms, has low computational load, and provides rapid response. Summary of the Invention

[0005] The purpose of this application is to provide a collaborative decision-making method and device for ground-air heterogeneous platforms based on multi-agent reinforcement learning, which can realize real-time joint decision-making between different heterogeneous platforms.

[0006] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning, including: Obtain local observations of several agents at the current time step and their corresponding platform type codes.

[0007] The global state at the current time step is obtained by stitching together the local observations of the several agents.

[0008] Based on the local observations of the aforementioned agents and the corresponding platform type encodings, a policy network is used to obtain joint actions.

[0009] The joint action is executed to obtain the local observation and immediate reward of the next time step, and the experience element of the current time step is stored in the experience pool; at the same time, it is determined whether the experience pool is full or has reached a specified number of steps, and a first judgment result is obtained; the experience element includes: the global state of the current time step, the platform type code, the joint action, the local observation of the next time step, the termination flag, and the immediate reward.

[0010] If the first judgment result is negative, then the local observation of the next time step is taken as the local observation of the current time step, and the process returns to the "obtain local observations of several agents at the current time step" step.

[0011] If the first judgment result is yes, then the current global state and the platform type code are input into the commentator network to obtain the current global state value corresponding to each platform.

[0012] Based on the current global state value corresponding to each platform and the instantaneous reward, the advantage function estimate of each platform is calculated.

[0013] Based on the advantage function estimation of each platform, the optimization objective of the policy network is maximized by gradient ascent and the value loss of the critic network is minimized by gradient descent. The parameters of the policy network and the critic network are updated alternately until convergence, and the converged policy network parameters are output.

[0014] Based on the converged policy network parameters, collaborative decision-making between ground-to-air heterogeneous platforms is performed.

[0015] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning.

[0016] Thirdly, this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning.

[0017] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning.

[0018] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a collaborative decision-making method and apparatus for ground-to-air heterogeneous platforms based on multi-agent reinforcement learning. By acquiring the local observations of several agents at the current time step and their corresponding platform type codes, it solves the problems of scattered perception data and inability to quantify platform attributes of ground-to-air heterogeneous platforms, achieving complete acquisition of real-time perception information and platform characteristics of each agent. By stitching together the local observations of the several agents to obtain the global state at the current time step, it solves the problems of limited observation perspective of a single platform and fragmented cross-platform environmental information, achieving multi-source local perception data fusion and constructing a complete global state representation. Based on the local observations of the several agents and their corresponding platform codes, the method achieves comprehensive decision-making. Type encoding, employing a policy network, yields joint actions, addressing the issues of independent decision-making and lack of action coordination across heterogeneous platforms. This enables multi-agent collaborative actions to be output by combining platform-specific attributes. By executing joint actions, local observations and immediate rewards for the next time step are obtained, and the experience elements of the current time step are stored in an experience pool. Simultaneously, a first judgment result is obtained by determining whether the experience pool is full or has reached a specified number of steps. This solves the problems of incomplete data retention in agent-environment interactions and the inability to batch collect training data, achieving ordered storage of all elements of the interaction process and completing the conditional judgment during the training data collection phase. If the first judgment result is negative, the local observations for the next time step are... As a local observation at the current time step, the process returns to obtain local observation steps for several agents at the current time step, solving the problem of unsustainable single-round interaction and interrupted data collection in dynamic environments, and realizing uninterrupted continuous interaction and continuous collection of training samples; if the first judgment result is yes, the current global state and the platform type encoding are input into the commentator network to obtain the current global state value corresponding to each platform. Based on the current global state value corresponding to each platform and the instantaneous reward, the advantage function estimate of each platform is calculated; based on the advantage function estimate of each platform, the optimization objective of the gradient ascent maximization strategy network and the gradient descent minimization commentator network are used. The system uses the value loss to alternately update the parameters of the policy network and the critic network until convergence, and outputs the converged policy network parameters. This solves the problems of traditional reinforcement learning in multi-platform heterogeneous environments, where the critic network cannot accurately estimate the global state value, the variance of the advantage function estimation is too high, and the policy network update direction is inconsistent and difficult to converge due to the large differences in reward scale and state transition rules of different platforms. The system achieves this by introducing platform type encoding so that the critic network can perceive platform differences and output more accurate state values, thereby obtaining low variance and stable advantage function estimation. Finally, it drives the policy network and the critic network to converge to the optimal policy parameters that are universal across platforms in the alternating adversarial update process.By executing collaborative decision-making between ground-to-air heterogeneous platforms based on the converged policy network parameters, the problems of insufficient cross-heterogeneous platform collaboration capability and poor task adaptation in traditional solutions are solved, realizing real-time and efficient joint collaborative decision-making between ground-to-air heterogeneous platforms in complex scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is an application environment diagram of a collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning in one embodiment of this application; Figure 2 A flowchart illustrating a collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning, provided as an embodiment of this application; Figure 3 This is a flowchart illustrating the construction of a heterogeneous multi-agent MAPPO attack and defense decision model according to an embodiment of this application. Figure 4 A rasterized map provided in one embodiment of this application; Figure 5 A flowchart illustrating the design of a heterogeneous multi-agent cooperative decision-making algorithm based on MAPPO, provided in one embodiment of this application; Figure 6 This is a schematic diagram illustrating the closed-loop interaction of system modeling and decision-making algorithms provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] This application belongs to the field of multi-agent cooperative decision-making, specifically involving an offensive and defensive game between three heterogeneous platforms on each side (red and blue) within a certain area. The red side consists of manned vehicles, unmanned vehicles, and drones, and acts as the primary decision-maker in both offense and defense. The blue side serves as the adversary environment with scripted strategies. In this scenario, the red side primarily addresses the cooperative search, maneuver, and attack decision-making problems under strongly coupled conditions.

[0023] Non-learning-based attack and defense decision-making methods, such as rule-based scripts, fixed task allocation, and traditional optimization search, suffer from shortcomings such as poor adaptability, slow real-time response, difficulty in collaborative decision-making, and coarse manual modeling. This solution aims to introduce a heterogeneous multi-agent reinforcement learning method (MAPPO) based on centralized training and distributed execution. By comparing it with existing non-learning methods, it addresses the difficulties in collaborative decision-making and dynamic adaptation among heterogeneous platforms within a limited area, enabling collaborative search, maneuvering, and attack decisions for manned / unmanned platforms in complex terrain.

[0024] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] The collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send the local observations of several agents at the current time step and their corresponding platform type codes to server 104. Server 104 receives the local observations of several agents at the current time step and their corresponding platform type codes to be processed. For the local observations of several agents at the current time step and their corresponding platform type codes, server 104 will concatenate and combine the local observations of the several agents to obtain the global state of the current time step. Based on the local observations of several agents and their corresponding platform type codes, a policy network is used to obtain a joint action. The joint action is executed to obtain the local observations and immediate reward of the next time step, and the experience elements of the current time step are stored in the experience pool. At the same time, it is determined whether the experience pool is full or has reached a specified number of steps to obtain a first judgment result. The experience elements include: the global state of the current time step, the platform type code, the joint action, the local observations of the next time step, the termination flag, and the immediate reward. If the first judgment result is negative, the local observations of the next time step are used as the local observations of the current time step, and the process returns to the "obtain local observations of several agents at the current time step" step. If the first judgment result is yes, then the current global state and the platform type code are input into the critic network to obtain the current global state value corresponding to each platform; based on the current global state value corresponding to each platform and the instant reward, the advantage function estimate of each platform is calculated; based on the advantage function estimate of each platform, the policy network parameters and critic network parameters are alternately updated by maximizing the optimization objective of the policy network through gradient ascent and minimizing the value loss of the critic network through gradient descent until convergence, and the converged policy network parameters are output; based on the converged policy network parameters, ground-air heterogeneous platform collaborative decision-making is performed; the server 104 can feed back the converged policy network parameters to the terminal 102. In addition, in some embodiments, the ground-air heterogeneous platform collaborative decision-making method based on multi-agent reinforcement learning can also be implemented by the server 104 or the terminal 102 alone. For example, the terminal 102 can directly process the local observations of several agents at the current time step and the corresponding platform type codes, or the server 104 can obtain the local observations of several agents at the current time step and the corresponding platform type codes from the data storage system and process them.

[0026] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, etc. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0027] In one exemplary embodiment, such as Figure 2 As shown, a collaborative decision-making method for ground-to-air heterogeneous platforms based on multi-agent reinforcement learning is provided. This method considers joint actions and is executed by computer devices, specifically by a terminal or server alone, or by both. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 209.

[0028] Step 201: Obtain the local observations of several agents at the current time step and the corresponding platform type codes.

[0029] Step 202: Combine the local observations of the several agents to obtain the global state at the current time step.

[0030] Step 203: Based on the local observations of the aforementioned agents and the corresponding platform type encodings, a policy network is used to obtain joint actions.

[0031] Step 204: Execute the joint action to obtain the local observation and immediate reward of the next time step, and store the experience element of the current time step into the experience pool; at the same time, determine whether the experience pool is full or has reached the specified number of steps, and obtain the first judgment result; the experience element includes: the global state of the current time step, the platform type code, the joint action, the local observation of the next time step, the termination flag, and the immediate reward.

[0032] Step 205: If the first judgment result is negative, then take the local observation of the next time step as the local observation of the current time step and return to the "Get local observations of several agents in the current time step" step.

[0033] Step 206: If the first judgment result is yes, then input the current global state and the platform type code into the commentator network to obtain the current global state value corresponding to each platform.

[0034] Step 207: Based on the current global state value and the instantaneous reward corresponding to each platform, calculate the advantage function estimate of each platform.

[0035] Step 208: Based on the advantage function estimation of each platform, maximize the optimization objective of the policy network through gradient ascent and minimize the value loss of the critic network through gradient descent, alternately update the parameters of the policy network and the critic network until convergence, and output the converged policy network parameters.

[0036] Step 209: Based on the converged policy network parameters, perform collaborative decision-making for the ground-air heterogeneous platform.

[0037] By implementing steps 201 to 209 above, this application obtains the local observations and platform type codes of each agent and concatenates them into a global state. It then generates joint actions and executes them in conjunction with the policy network. When the experience pool is full or a specified number of steps are reached, the global state value is calculated using the critic network, and the parameters of the policy network and the critic network are alternately updated through the advantage function estimation until convergence. This achieves efficient collaborative decision-making for heterogeneous ground-air platforms, solves the non-stationarity and partial observability problems in heterogeneous platform environments, and improves the stability and generalization ability of multi-agent collaborative strategies.

[0038] In another exemplary embodiment of this application, in order to accurately obtain the local observations of several agents at the current time step, the above step 201 is replaced by the following steps 2011 to 2013: Step 2011: Construct the original state space; the original state space includes: platform's own state information, friendly platform state information, enemy target state information, and local terrain and visibility state information.

[0039] Step 2012: Perform state preprocessing on the original state space to obtain the processed state space; the state preprocessing is to encode the original state space based on encoding rules.

[0040] Step 2013: Based on the processed state space, obtain the local observations of several agents at the current time step.

[0041] As an optional implementation, the encoding rules include: One-hot encoding is used for categorical variables in the original state space.

[0042] The Boolean variables in the original state space are encoded using 0 / 1.

[0043] Normalized encoding is used for continuous variables in the original state space.

[0044] In another exemplary embodiment of this application, in order to accurately obtain joint actions based on the local observations of the plurality of agents and the corresponding platform type encodings, a policy network is used, and the above step 203 is replaced by the following steps 2031 to 2033: Step 2031: Input the local observations and corresponding platform type encodings of the several agents into the policy network, filter out several optional actions corresponding to each agent from the global action set, and obtain the original logits corresponding to the optional actions; the global action set includes: discrete reconnaissance action instruction set and discrete attack action instruction set; the original logits corresponding to the optional actions include the original logits corresponding to illegal actions and the original logits corresponding to legal actions.

[0045] Step 2032: Based on the action masking mechanism, perform illegal action removal on the original logits corresponding to the optional actions to obtain the processed logits corresponding to the optional actions.

[0046] Step 2033: Perform softmax normalization and sampling on the logits corresponding to the processed optional actions to obtain the joint action.

[0047] As an optional implementation, the construction of the global action set specifically includes the following steps: Step 301: Construct the action space.

[0048] Step 302: Divide the action space into a reconnaissance action space and a strike action space; the reconnaissance action space mainly corresponds to the platform's maneuvering and search behavior in reconnaissance missions; the strike action space mainly corresponds to the platform's maneuvering, target selection, and firepower allocation behavior in strike missions.

[0049] Step 303: Decompose the reconnaissance action space into mutually orthogonal maneuvering and search dimensions; the maneuvering dimension includes several basic displacement commands, including several search directions along the two-dimensional coordinate axes and stopping actions; the search dimension includes electro-optical search and radar search; by combining the displacement commands with the search modes through Cartesian product, several discrete reconnaissance action commands are generated, forming a discrete reconnaissance action command set.

[0050] Step 304: Divide the strike mission action space into maneuver actions, target selection actions, and strike ammunition selection actions; combine the maneuver actions, target selection actions, and ammunition actions by Cartesian product to obtain several strike action combinations, forming a discrete strike action instruction set.

[0051] Step 305: Merge the discrete reconnaissance action instruction set with the discrete strike action instruction set to obtain the global action set.

[0052] As an optional implementation, the expression for the action masking mechanism is: ; in, Indicates the first An agent at time step For candidate actions Action mask value; Indicates the first An agent at time step The set of legal actions.

[0053] As an optional implementation, the formula for calculating the instant reward is: ; in, Indicates the first An agent at time step Total reward; Individual reward Indicates the first An agent at time step Individual rewards; Indicates the first An agent at time step Team-shared rewards; Indicates the first An agent at time step Assistance and guidance rewards.

[0054] In another exemplary embodiment of this application, step 208 is replaced by steps 2081 to 2085: Step 2081: Divide the experience elements in the experience pool into several small batches of samples according to a preset batch size.

[0055] Step 2082: In each round of updates, the parameters of the old policy network corresponding to the fixed sampling phase are fixed, and the ratio of action probability is calculated based on the action probability obtained by the current policy network and the action probability obtained by the old policy network; the action probability is the probability obtained by performing softmax normalization on the corresponding logits.

[0056] Step 2083: Construct the pruning objective function of the policy network based on the action probability ratio and the advantage function estimate, and update the policy network parameters through gradient ascent.

[0057] Step 2084: Construct a reward objective based on the current global state value and the advantage function estimate; construct a value loss function based on the error between the reward objective and the state value output by the critic network; and update the critic network parameters through gradient descent.

[0058] Step 2085: Repeat the policy network parameter update and the commentator network parameter update until the preset number of update rounds is reached or the preset convergence condition is met, and output the converged policy network parameters.

[0059] As an optional implementation, the optimization objective expression of the policy network is: ; in, This represents the optimization objective of the policy network; This indicates the total number of our platforms; This represents the time step for each trajectory; This represents the summation over all platforms and time steps; This means taking the smaller of the two values ​​to achieve the clipping constraint; Indicates platform At any moment The advantage function estimation; This means cutting the probability ratio to... interval; Indicates the cutting factor; This represents the entropy regularization coefficient; Represents policy entropy, which measures the randomness of a policy. This represents the complete action distribution of the current policy given the input.

[0060] The value loss expression for the critic network is: ; in, This indicates a loss of value for the critics' network; Represents the critic network (parameters) The output is a state value estimate, and the input is the global state. Platform type encoding ; Indicates time The global state; Indicates platform At any moment The return target, .

[0061] In an exemplary embodiment, this application mainly includes two parts: a heterogeneous multi-agent attack and defense game modeling module and a MAPPO-based collaborative decision-making module. It constructs the corresponding state space, action space and reward function, and then constructs a heterogeneous MAPPO collaborative decision-making model based on the centralized training and distributed execution (CTDE) method.

[0062] This application first constructs a heterogeneous multi-agent attack-defense game model (Markov decision model) for a 3V3 heterogeneous platform attack-defense scenario characterized by strong constraints, small sample size, and high coupling. It completes the state space design, action space design, reward mechanism, and system modeling, providing standardized input and constraint rules for collaborative decision-making. The overall modeling process is as follows: Figure 3As shown, first, an environment and map model is established, and then the following steps are executed in sequence: S1 State space design and state preprocessing, S2 Action space design and legal action set generation, and S3 Reward function design, finally constructing a Markov decision model.

[0063] Specifically, the S1 state space design: Assuming each agent The status that may be involved in operations such as maneuvering, reconnaissance, and strike is: platform's own status information. Status information of neighboring platforms Enemy target status information and local terrain and visibility information Therefore, the state space is .

[0064] Platform's own status information It consists of platform type, payload type, ammunition type, current position and speed, remaining fuel, and remaining ammunition; friendly platform status information. Composed of information such as the type of friendly platform, relative distance and speed, task mode, and availability; enemy target status information. It consists of enemy type, relative distance from this platform, location, speed, threat level, and survival status; local terrain and visibility information. It consists of the location of obstacles, accessibility, etc.

[0065] The state preprocessing in this application is achieved by state encoding. The encoding rules are uniformly as follows: 1) To avoid manually assigning values ​​based on size, categorical variables such as platform type, enemy type, and ammunition type adopt one-hot encoding; 2) Boolean variables such as availability and target survival status adopt 0 / 1 encoding; 3) Continuous variables such as position, speed, distance, remaining ammunition, remaining fuel, threat level, obstacle position, and accessibility adopt normalized encoding.

[0066] S2 motion space design: Action space design determines how an agent responds to the battlefield situation and is also a crucial link connecting state awareness and strategy execution. Manned and unmanned vehicles are primarily responsible for mobility, reconnaissance, and strike missions, while unmanned aerial vehicles (UAVs) are primarily responsible for reconnaissance and situational awareness missions. A unified action space is essential. Includes reconnaissance action space and striking action space The expression is ,in This mainly corresponds to the platform's maneuvering and search behavior during reconnaissance missions. This mainly corresponds to the platform's mobility, target selection, and firepower allocation behavior during strike missions.

[0067] Reconnaissance actions divide their action space into maneuvering. and search Two orthogonal dimensions, their expression is: This achieves a balance between lightweight decision-making and efficient execution. The reconnaissance mission proceeds in two steps: Nine basic displacement commands are selected from eight search directions along the two-dimensional coordinate axes and a library of stopping actions. These are then combined with search action modes (electro-optical search and radar search) and formed into a set of actions through Cartesian product. Ultimately forming 9 Two sets of discrete reconnaissance action instructions.

[0068] The action space for strike missions is divided into: maneuvering actions Target selection action , Action to select ammunition Its expression is: By combining the maneuvers (9), targets (n), and ammunition (p types) using Cartesian products, the attack action combination is obtained: And formed 9 n p types of discrete strike action instruction sets.

[0069] Depending on the battlefield situation, the platform's action space can be divided into a set of legal actions and a set of illegal actions. The set of legal actions for a platform is not static, but is determined by the platform type, its survival status, terrain constraints, weapon status, and resource status. For example, drones are inherently prohibited from performing strike actions; when the terrain is impassable, movement actions in the corresponding direction are disabled; when the fire system is not ready or ammunition is depleted, strike-related actions are disabled; when the platform is damaged in combat, only invalid occupancy actions or termination marking actions are retained. For any action... Define action mask Then it can be written as: .

[0070] In the policy network, the logits corresponding to illegal actions are assigned a minimum value, thus ensuring that softmax sampling is performed only on the set of legal actions. This approach not only guarantees consistency between training and execution but also significantly reduces the waste of samples caused by exploring illegal actions.

[0071] S3 reward function design: Considering the characteristics of the heterogeneous combat scenario in this embodiment, the reward function design guides the platform to gradually learn effective maneuvering, search, and attack strategies through a reasonable reward and punishment mechanism. The reward function consists of individual rewards. Team-shared rewards and assistance / guidance rewards It consists of three parts, and introduces and It is composed of weighted coefficients for team rewards and guidance rewards, and its expression is: The specific composition of individual rewards differs depending on the mission mode: search missions mainly consist of target discovery rewards, area exploration rewards, search pattern matching rewards, and repeated search penalties; strike missions mainly consist of target engagement rewards, engagement accuracy rewards, target-ammunition matching rewards, and resource consumption penalties.

[0072] This application creates a 1km zone between the red and blue teams. The local offensive and defensive game environment within a 1km grid area is divided into three independent regions: near-ground, ultra-low-altitude, and low-altitude (e.g., Figure 4 (As shown).

[0073] Since the primary focus is on establishing our collaborative decision-making model, the training phase concentrates on joint optimization among our three heterogeneous agents. The attack and defense decision is now transformed into a heterogeneous multi-agent Markov game, with the following expression: ; in For our intelligent agents set, For the global state space, For the platform The set of legal actions For the environment state transition function, For the platform Instant rewards, among which It was a joint operation. It is a time step Global state, It is a time step The global state; the state transition of multiple agents is affected by themselves, friendly neighbors, enemies, and terrain, and joint actions will also have a joint effect on the reward.

[0074] The system displays the heterogeneous multi-agent attributes of manned vehicles, unmanned vehicles, and drones in the state space, and adds legal action judgment and action space division in the action space. Then, the reward function simultaneously takes into account the four-dimensional strategies of maneuvering, searching, cooperating, and attacking.

[0075] The collaborative decision-making module based on Multi-Agent Proximal Policy Optimization (MAPPO) employs a centralized training and distributed execution framework to address the instability issue in small-scale, strongly coupled attack and defense scenarios. The algorithm execution flow is as follows: Figure 5First, the network parameters, experience pool, and environment of the decision network (Actor) and the critic network (Critic) are initialized. After the agent acquires local observations, the Actor network generates an action distribution and applies an action mask to sample actions. Then, joint actions are executed, the next state and reward are obtained, and the data is stored in the experience pool. When the experience pool is full or a specified number of steps T is reached, the centralized Critic calculates the global state value estimate and advantage function estimate (GAE). Then, proximal policy optimization (PPO) is performed on the Actor network to update the policy, and value loss is updated on the Critic network. If the algorithm does not converge or the maximum number of rounds is not reached, it returns to continue collecting data. If the convergence condition is met or the maximum number of rounds is reached, the converged multi-agent decision policy is output, and the process ends. The specific steps are as follows: Step 1: Initialization: Initialize the policy network parameters With commentator network parameters Clear the experience cache Reset the simulated battlefield environment.

[0076] Step 2: Local Observation and Global State Construction: Before inputting the state into the policy network, it undergoes unified preprocessing. Local observations from all platforms are concatenated and combined in a unified order to obtain the global state. .in For intelligent agents The local state, , This provides supplementary statistical information at the team level.

[0077] Step 3: Action Mask Generation and Action Sampling: A combination of policy network data sharing and platform type encoding is used to achieve heterogeneous platform decision-making. The Actor network uses the local states of the agent... Platform type encoding As input, output raw logits: .in For policy network parameters, This represents the mapping function of the Actor network. In the actual decision-making process, only legal actions are performed, and the logits corresponding to illegal actions are directly converted into minimum value penalties.

[0078] Step 4: Joint Action Execution and Experience Storage: Execute joint actions. Used to drive the testing environment and earn rewards Next state and termination flag Store the experience tuple into the experience pool. Then repeat steps 2 through 4 until the iteration is complete.

[0079] Step 5: Calculation of Advantage Function and Return Objective: A critic network design with centralized input and platform-dependent output is adopted. The critic network uses a global state... Platform type encoding As input, and by incorporating critic network parameters. The output platform is in the current global state value. Then, advantage function estimation is performed on each platform based on GAE, thereby providing stable gradient information for the policy network; a reward objective is constructed based on the advantage function estimation and the current global state value, wherein the reward objective is the target value for training the Critic.

[0080] Step 6: Policy and Critics Network Update: The policy network parameters and critic network parameters are updated alternately by maximizing the optimization objective of the policy network through gradient ascent and minimizing the value loss of the critic network through gradient descent.

[0081] Specifically, the experience elements in the experience pool are divided into several mini-batches according to a preset batch size; in each round of updates, the parameters of the old policy network corresponding to the sampling phase are fixed, and the action probability ratio is calculated based on the action probability obtained by the current policy network and the action probability obtained by the old policy network; the action probability is the probability obtained by softmax normalization of the corresponding logits; the pruning objective function of the policy network is constructed based on the action probability ratio and the advantage function estimation, and the parameters of the policy network are updated by gradient ascent; the value loss function is constructed based on the error between the reward objective and the state value output by the critic network, and the parameters of the critic network are updated by gradient descent.

[0082] Step 7: Convergence judgment and output; Repeat steps 2 to 6 until convergence is achieved and the policy network parameters are output.

[0083] In summary, as Figure 6 As shown, the system is first modeled, defining the state space, action space, and reward function to provide a foundation for subsequent decision-making. Then, the MAPPO decision algorithm selects a legal action based on the current state. After the agent executes the action, the environment updates the state and returns a reward. The new state and reward information are then fed back to the MAPPO algorithm to drive iterative optimization of the policy, forming a closed-loop interaction process.

[0084] This application has the following advantages compared with the prior art: Create a heterogeneous state and action space: including four-dimensional information of self, friendly neighbors, enemies and terrain visibility, using normalized, discrete binary variables and maintaining their consistent length, unifying the action space and dynamically filtering illegal actions, thereby improving system robustness and action feasibility.

[0085] Multi-dimensional action comprehensive rewards: Combining individual and team shared rewards with auxiliary guidance rewards, including tracking, attack, coordination and terrain guidance, to solve the problem of ambiguous action reward allocation and enhance reconnaissance-attack coordination.

[0086] MAPPO centralized training and distributed execution: The value network utilizes the global state, and the policy network is based on local information observation and action masking, combined with GAE and PPO updates, to achieve the use of global information to mitigate instability and distributed real-time decision-making and multi-agent information sharing.

[0087] It should be noted that the MAPPO algorithm is essentially an extension of PPO in multi-agent environments. By introducing the policy optimization mechanism of PPO into multi-agent systems, it achieves more efficient policy learning in both cooperative and competitive environments. This algorithm is a policy gradient-based multi-agent reinforcement learning algorithm, particularly suitable for mixed cooperative and competitive multi-agent scenarios.

[0088] This application establishes a model framework for heterogeneous multi-agent MAPPO attack and defense decision-making, including state space, action space, cooperative reward, and type encoding. Based on this framework, when facing platform type changes or algorithm updates, there is no need to redesign the heterogeneous state space and action space, nor to construct new preprocessing and reward structures for incomplete information or battle damage scenarios. It can be reused simply by changing the platform's own fields and masking rules, or by connecting to other CTDE algorithms through a unified interface.

[0089] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 7 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores data used in the process of implementing a collaborative decision-making method for ground-to-air heterogeneous platforms based on multi-agent reinforcement learning. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a collaborative decision-making method for ground-to-air heterogeneous platforms based on multi-agent reinforcement learning.

[0090] Figure 7The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0091] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0092] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0093] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations and be authorized by the owner of the corresponding device.

[0094] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0095] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0096] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning, characterized in that, The method includes: Obtain local observations of several agents at the current time step and their corresponding platform type codes; The global state at the current time step is obtained by splicing together the local observations of the aforementioned agents. Based on the local observations of the aforementioned agents and the corresponding platform type encodings, a policy network is used to obtain joint actions. The joint action is executed to obtain the local observation and immediate reward of the next time step, and the experience element of the current time step is stored in the experience pool; at the same time, it is determined whether the experience pool is full or has reached a specified number of steps, and a first judgment result is obtained; the experience element includes: the global state of the current time step, the platform type code, the joint action, the local observation of the next time step, the termination flag, and the immediate reward. If the first judgment result is negative, then the local observation of the next time step is taken as the local observation of the current time step, and the process returns to the "Get local observations of several agents at the current time step" step. If the first judgment result is yes, then the current global state and the platform type code are input into the commentator network to obtain the current global state value corresponding to each platform; Based on the current global state value and the instantaneous reward corresponding to each platform, the advantage function estimate of each platform is calculated; Based on the advantage function estimation of each platform, the optimization objective of the policy network is maximized by gradient ascent and the value loss of the critic network is minimized by gradient descent. The parameters of the policy network and the critic network are updated alternately until convergence, and the converged policy network parameters are output. Based on the converged policy network parameters, collaborative decision-making between ground-to-air heterogeneous platforms is performed.

2. The collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning according to claim 1, characterized in that, Obtain local observations from several agents at the current time step, specifically including: Construct an original state space; the original state space includes: platform's own state information, friendly platform state information, enemy target state information, and local terrain and visibility state information; The original state space is preprocessed to obtain a processed state space; the state preprocessing is to encode the original state space based on encoding rules. Based on the processed state space, local observations of several agents at the current time step are obtained.

3. The collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning according to claim 2, characterized in that, The encoding rules specifically include: One-hot encoding is used for categorical variables in the original state space; Boolean variables in the original state space are encoded using 0 / 1. Normalized encoding is used for continuous variables in the original state space.

4. The collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning according to claim 1, characterized in that, Based on the local observations of the aforementioned agents and the corresponding platform type encodings, a policy network is used to obtain joint actions, specifically including: The local observations of the aforementioned agents and their corresponding platform type encodings are input into the policy network. Several optional actions corresponding to each agent are selected from the global action set to obtain the original logits corresponding to the optional actions. The global action set includes: a discrete reconnaissance action instruction set and a discrete attack action instruction set. The original logits corresponding to the optional actions include: the original logits corresponding to illegal actions and the original logits corresponding to legal actions. Based on the action masking mechanism, illegal actions are removed from the original logits corresponding to the optional actions to obtain the processed logits corresponding to the optional actions. The logits corresponding to the processed optional actions are subjected to softmax normalization and sampling to obtain the joint action.

5. The collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning according to claim 4, characterized in that, The expression for the action masking mechanism is: ； in, Indicates the first An agent at time step For candidate actions Action mask value; Indicates the first An agent at time step The set of legal actions.

6. The collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning according to claim 1, characterized in that, The formula for calculating the instant reward is as follows: ； in, Indicates the first An agent at time step Total reward; Individual reward Indicates the first An agent at time step Individual rewards; Indicates the first An agent at time step Team-shared rewards; Indicates the first An agent at time step Assistance and guidance rewards.

7. The collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning according to claim 1, characterized in that, Based on the aforementioned advantage function estimation, the optimization objective of the policy network is maximized by gradient ascent, and the value loss of the critic network is minimized by gradient descent. The parameters of the policy network and the critic network are alternately updated until convergence, and the converged policy network parameters are output, specifically including: The experience elements in the experience pool are divided into several small batches of samples according to a preset batch size; In each round of updates, the parameters of the old policy network corresponding to the fixed sampling phase are used to calculate the ratio of action probabilities based on the action probabilities obtained by the current policy network and the action probabilities obtained by the old policy network; the action probabilities are the probabilities obtained by performing softmax normalization on the corresponding logits. The pruning objective function of the policy network is estimated based on the action probability ratio and the advantage function, and the parameters of the policy network are updated by gradient ascent. A reward objective is constructed based on the current global state value and the advantage function estimate. A value loss function is constructed based on the error between the reward objective and the state value output by the critic network. The critic network parameters are then updated using gradient descent. Repeat the policy network parameter update and the critic network parameter update until the preset number of update rounds is reached or the preset convergence condition is met, and output the converged policy network parameters. The optimization objective expression for the policy network is: ； in, This represents the optimization objective of the policy network; This indicates the total number of our platforms; This represents the time step for each trajectory; This represents the summation over all platforms and time steps; This means taking the smaller of the two values ​​to achieve the clipping constraint; Indicates platform At any moment The advantage function estimation; This means cutting the probability ratio to... interval; Indicates the cutting factor; This represents the entropy regularization coefficient; Represents policy entropy, which measures the randomness of a policy. This represents the complete action distribution of the current policy given the input; The value loss expression for the critic network is: ； in, This indicates a loss of value for the critics' network; Represents the critic network (parameters) The output is a state value estimate, and the input is the global state. Platform type encoding ; Indicates time The global state; Indicates platform At any moment The return target, .

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the collaborative decision-making method for ground-air heterogeneous platforms based on multi-agent reinforcement learning as described in any one of claims 1-7.

Images